Weekly Update 2: Completion of Version Control Integration

A little late this week, as I was moving into my summer apartment.

• The biggest (and really only) news is that I finished the Subversion hooks, and (slightly) improved the backend in general. It is now possible to deny commit access to users based on whether or not they have a Drupal account with that particular Subversion repository. Branches and tags, however, are not supported, as the Subversion backend does not support them. There is the framework for doing so, as the repository creation/edit form accepts path patterns for trunk, branches and tags, so adding support should not be too difficult.

• The post-commit hooks allows Drupal to gather log information as each commit is made, rather than on a cron run. There isn't much else to it, but it can provide for a bit more immediacy of actions, and in the future, could be an event for the Rules module (something I will work on later in the summer).

• Both of the hooks come with a set of SimpleTests for ensuring their correct functioning. The tests take a little while to run, as creating, checking out, and committing to a subversion repository take a while (and don't play that nice with the OS-level filecache, since new objects are being created each time), but are helpful for verifying that everything is in working order. They test both positive and negative cases, making sure the hooks fail gracefully when passed invalid information. They currently all pass (as one should hope !), and are able to catch a decently wide range of errors in the operation of the hooks.

• A minor aside, but it bears mentioning, since I spent several hours testing it, but I discovered that Subversion allows any UTF-8 character for any string (comments, user names, and files). However, certain tools restrict especially the username to a subset of those characters, depending on the configuration options for that program. For example, the svnserve 'passwd' file format uses brackets and equals signs as part of its syntax, so those characters are prohibited in entries in that file, thus preventing Subversion users from having those names. However, if authentication is done in a different way, such as with ssh keys, this restriction no longer applies.

Well, that's all for now, stay tuned next week as I complete the Git repository hooks.

Weekly Update 1: Completion of Version Control Integration

This week was not a terribly busy one, with most of my time spent getting acquainted with the APIs of Drupal, the versioncontrol module, and Subversion. I was able to complete the pre-commit hook for Subversion, as well as the basic configuration for the other Subversion hooks.

I also identified a number of areas in which the SVN backend needs more work, such as recognizing branches and tags. The framework is in place, as the repository creation form includes fields for the branch and tag patterns, but currently no code makes use of those fields. Subversion is a tricky case, as it has no native concept of branches or tags, so the branch of a commit depends on the path of the files it modifies in the repository.

With the prep work out of the way, I hope to push forward to complete the Subversion hooks and begin on the next step of the project, the hooks for Git.

The future is now! (or in a few weeks)

Go take a look at the video on Google Wave and tell me that it isn't the future of everything. Go on, I dare you.

Tabs of Genius

Since I am going to be coding full-time (or nearly) for my summer job, I took this as an opportunity to undertake some long-overdue changes to my Emacs installation. Among the changes are:

• Adding Drupal file extensions (.module, .incto the recognized PHP patterns


• Using notify.el to pop up the cool new notification system in the latest version of Ubuntu whenever my nickname is mentioned on an IRC channel which I have joined.


• Replacing my own ido-enhanced M-x with the pimptacularSmex program.


• Updated nXhtml to the latest version. For a good example of how NOT to write commit messages, check out the (otherwise excellent) repository on Launchpad.


• Move some more packages from my elisp directory to ELPA. This lets me cut down on the amount of path configuration and autoloading in my configuration files, as well as managing its own updates.


• Replace pabbrev with Smart Tab, (hence the name of this post) which uses the massively powerful hippie-expand to return better results. I ended up developing it from a simple function rebinding tab to a full-fledged global-minor-mode (which does nothing but rebind tab :) and a git repository on github. My next move is to clean up all the little issues and try to get it included in ELPA. It is my first time maintaining any sort of package for public use, and a lot more goes into it than I thought. It's kind of neat.


• Shuffle some configuration files about and update the copyright dates on my config files. Not like anyone is going to infringe them, right?


• Spent FAR more time than should ever be necessary hunting down a (relatively benign) bug in anything.el which clobbered the key binding for occur. Long story short: there is actually a difference between copy-keymap and set-keymap-parent. I'm going to see about pushing that change back upstream, though I'm not exactly sure what the "official" repository for anything.el is, aside from some files on EmacsWiki.


• Spent a great deal of time figuring out how to get Smart Tab and auto-complete to play nice with each other. It eventually turned out that the bulk of my problems was that I was setting up my autoloads after loading Custom, so libraries which were expecting Custom to initialize them were left hanging.


• Started, but got sidetracked from, setting up some form of tags file for at least PHP, since that is what I'm going to be spending most of my time working on. Unfortunately, given the nature of the interaction between the program which generates the tags and Emacs, it is non-trivial to have an automatically continually updated base of tags. Additionally, the format Emacs uses is apparently fairly inefficient (requires linear scans for all/most operations), but there is a rich ecosystem of libraries and helpers around it. I'll have to look at it some more later.


• Looked into being able to use ido-completing-read everywhere the vanilla completing-read is used, making variable and function lookups much more pleasant. It is not as easy as simply defaliasing ido-completing-read to completing-read, since ido-completing-read itself calls completing-read during its execution. I asked StackOverflow if anyone had a solution, and nothing has come back so far. It would be really convenient if there was a completing-read-function variable which you could simply set to code>completing-read or ido-completing-read and be done with it. What's worse, ido-completing-read is a drop-in replacement for completing-read, so most libraries likely would not notice or care.

Whew! That was a lot! I haven't had a good Emacs indulgence in a while, and I forgot how fun it is!

Many of the changes I made were a bit more under-the-hood, but I've already fallen in love with Smex, and from what I hear, Anything.el is pure bliss (I haven't had a chance to set it up and get used to it fully).

The big thing I'd like to spend some more time on is improving the situation around EmacsWiki. It is a great site, with a healthy community of interested people, but a Wiki is a pretty bad form of version control. Once need only look at something like line numbering, where there is one for Xemacs only, line-num.el, linum.el, setnu.el, setnu+.el, lineo.el, and a number of patches for each of them, mingling around in a long page which also serves as the bug tracker. I would like to see much more organization (though obviously not at the expense of productivity) and ease of deployment (like with ELPA). Even for packaging, there are at least six different and incomplete (in that they don't index all or even most packages) implementations of packaging programs. Competition and diversity are good, but there comes a point when it means that nothing actually ends up usable.

Well, I've certainly got a lot cut out for myself, but it all looks interesting. And maybe, when I'm done fiddling around with Emacs, I might actually get to some Drupal programming. You know, the thing that's supposed to be my job.

Holy Ubuntu!

Wow, I love Ubuntu.

I just found out that if you hover the mouse over a audio file in Ubuntu (I'm running 9.04, I don't know about the older versions), it will automatically start playing it. No clicking, no nothing!

Not a terribly big deal, but a very cool little thing. Hooray for finding out cool stuff!

I got GSoC 2009!

My proposal for the Google Summer of Code 2009 was accepted! I will be working on Drupal, specifically completing work on the Version Control Integration API with the goal of allowing Drupal to switch from CVS to a DVCS (most likely Git) for its main version control.

It will be a difficult task, but much of the baseline work has already been done, so I will be mostly filling in the gaps in functionality and finishing up the few areas where the Git backend lags behind the CVS one. You can look at my proposal here, and I will endeavor to make relatively frequent status updates on this blog. I am chrono325 on Drupal.org, and will try to spend some time on the #drupal irc channel on freenode, so if you have questions, that's how to get in touch.

Yay for Google!

SSD Anthology

My dad asked me a few questions about this article. In short, SSD drives suffer from performance degradation due to the fact that they are read from and written to in 4KB pages but can only be erased in 512KB blocks. This means that if you have a full block and want to change a single bit within it, the operating system sends the 4KB page which has been modified, but the SSD needs to erase and rewrite the 512KB block.

This "solution" sounds a lot like reinstalling Windows and all of your programs. ie. way more work and trouble than anyone other than a hard core geek would put up with.

Yup, you're right.

I'm waiting until they come up with an automatic solution. Something like "click 'yes' to speed up your drive"

In that case, you would want the drive to just do it automatically, because why wouldn't you want it to just automatically be as fast as possible? It would be like having a car which had an ignition and a separate button labeled "press this to actually start the car." If it were as simple and transparent as pressing a button, it should be done without bothering you.

As I said, the real problem is that the filesystem doesn't know enough about the structure of the flash drive to cooperate optimally. This is mostly a problem with existing filesystems which are organized for HDDs, but is also a problem of the flash drives which do not expose sufficient information to the filesystems for them to be able to make the best choices.

This is not just a case of one company or the other simply being stupid or lazy (though there is some of that). When flash drives were first introduced, there were no filesystems to take advantage of them since there was no demand for creating such a filesystem, and writing filesystems is REALLY HARD. Really, really hard. This means that you aren't going to create a new filesystem unless you have a really good reason for doing so, since, as I said, writing filesystems is REALLY HARD. The correct thing for the SSD drive manufacturers to do was to hide the underlying complexity from the filesystems of the day and use a strategy which was good enough when a filesystem treated the SSD as a hard drive. This gave rise to the fancy block reordering schemes described in the article.

The problem is that these reordering schemes run into the problems described by the articles (all of the blocks are used up and must be erased before new data can be written) which could be mostly solved (or at least largely mitigated) by filesystems which knew more about the structure of the SSD. Unfortunately, one of the factors over which SSD makers compete is their block reordering scheme, so they have an incentive to keep that a secret and prevent people from circumventing it (thereby making their fancy reordering scheme irrelevant). Taken to the extreme, the only field on which to compete would be the makeup of the memory chips themselves, which would push the different SSD makers to a more commodity status (and therefore lower margins). From a user's view, this would be an ideal scenario as long as your operating system and filesystem could take advantage of the additional information provided by the memory chips.

We are in a sort of transitional period during which there is quite a bit of flux and uncertainty. It is very, very useful to have a small number of filesystems which everyone can read and write, since it makes data portability that much easier and possible. This is the main reason (along with its minimal storage and computational overhead) why the otherwise horribly outdated FAT family of filesystems are still in widespread usage by removable storage. Superior alternatives exist (mostly on Linux, due to the relative ease of writing new filesystems for it), but aren't compatible with other operating systems, and so are useless for consumer flash drives. For a flash-aware filesystem to gain widespread support, it would need to have compatibility with the major operating systems, a catch-22 for any new filesystems.

The other hurdle, which is less visible to end-users, is the way the drives expose information about themselves. For a flash-aware filesystem to be truly effective, it would need some way of gathering information about the characteristics of the underlying flash drive and have direct access to it, bypassing any (or the most dramatic) block reordering schemes. The technical ideal would be to have a single open, royalty-free, well-written and extensible standard for performing this kind of communication. The problem is that allowing such a standard would bypass the competitive differentiator of a manufacturer's reordering scheme, which individual manufacturers would likely resist. If such a standard could be agreed upon, it would go a long way towards enabling better cooperation between the hardware and software.

Whether any of this comes to pass remains to be seen. What is fairly certain, however, is that there will be a lot of volatility in the SSD space as these issues (and more) are figured out.