My proposal for the Google Summer of Code 2009 was accepted! I will be working on Drupal, specifically completing work on the Version Control Integration API with the goal of allowing Drupal to switch from CVS to a DVCS (most likely Git) for its main version control.
It will be a difficult task, but much of the baseline work has already been done, so I will be mostly filling in the gaps in functionality and finishing up the few areas where the Git backend lags behind the CVS one. You can look at my proposal here, and I will endeavor to make relatively frequent status updates on this blog. I am chrono325 on Drupal.org, and will try to spend some time on the #drupal irc channel on freenode, so if you have questions, that's how to get in touch.
Yay for Google!
Friday, April 24, 2009
Sunday, April 19, 2009
SSD Anthology
My dad asked me a few questions about this article. In short, SSD drives suffer from performance degradation due to the fact that they are read from and written to in 4KB pages but can only be erased in 512KB blocks. This means that if you have a full block and want to change a single bit within it, the operating system sends the 4KB page which has been modified, but the SSD needs to erase and rewrite the 512KB block.
Yup, you're right.
In that case, you would want the drive to just do it automatically, because why wouldn't you want it to just automatically be as fast as possible? It would be like having a car which had an ignition and a separate button labeled "press this to actually start the car." If it were as simple and transparent as pressing a button, it should be done without bothering you.
As I said, the real problem is that the filesystem doesn't know enough about the structure of the flash drive to cooperate optimally. This is mostly a problem with existing filesystems which are organized for HDDs, but is also a problem of the flash drives which do not expose sufficient information to the filesystems for them to be able to make the best choices.
This is not just a case of one company or the other simply being stupid or lazy (though there is some of that). When flash drives were first introduced, there were no filesystems to take advantage of them since there was no demand for creating such a filesystem, and writing filesystems is REALLY HARD. Really, really hard. This means that you aren't going to create a new filesystem unless you have a really good reason for doing so, since, as I said, writing filesystems is REALLY HARD. The correct thing for the SSD drive manufacturers to do was to hide the underlying complexity from the filesystems of the day and use a strategy which was good enough when a filesystem treated the SSD as a hard drive. This gave rise to the fancy block reordering schemes described in the article.
The problem is that these reordering schemes run into the problems described by the articles (all of the blocks are used up and must be erased before new data can be written) which could be mostly solved (or at least largely mitigated) by filesystems which knew more about the structure of the SSD. Unfortunately, one of the factors over which SSD makers compete is their block reordering scheme, so they have an incentive to keep that a secret and prevent people from circumventing it (thereby making their fancy reordering scheme irrelevant). Taken to the extreme, the only field on which to compete would be the makeup of the memory chips themselves, which would push the different SSD makers to a more commodity status (and therefore lower margins). From a user's view, this would be an ideal scenario as long as your operating system and filesystem could take advantage of the additional information provided by the memory chips.
We are in a sort of transitional period during which there is quite a bit of flux and uncertainty. It is very, very useful to have a small number of filesystems which everyone can read and write, since it makes data portability that much easier and possible. This is the main reason (along with its minimal storage and computational overhead) why the otherwise horribly outdated FAT family of filesystems are still in widespread usage by removable storage. Superior alternatives exist (mostly on Linux, due to the relative ease of writing new filesystems for it), but aren't compatible with other operating systems, and so are useless for consumer flash drives. For a flash-aware filesystem to gain widespread support, it would need to have compatibility with the major operating systems, a catch-22 for any new filesystems.
The other hurdle, which is less visible to end-users, is the way the drives expose information about themselves. For a flash-aware filesystem to be truly effective, it would need some way of gathering information about the characteristics of the underlying flash drive and have direct access to it, bypassing any (or the most dramatic) block reordering schemes. The technical ideal would be to have a single open, royalty-free, well-written and extensible standard for performing this kind of communication. The problem is that allowing such a standard would bypass the competitive differentiator of a manufacturer's reordering scheme, which individual manufacturers would likely resist. If such a standard could be agreed upon, it would go a long way towards enabling better cooperation between the hardware and software.
Whether any of this comes to pass remains to be seen. What is fairly certain, however, is that there will be a lot of volatility in the SSD space as these issues (and more) are figured out.
This "solution" sounds a lot like reinstalling Windows and all of your programs. ie. way more work and trouble than anyone other than a hard core geek would put up with.
Yup, you're right.
I'm waiting until they come up with an automatic solution. Something like "click 'yes' to speed up your drive"
In that case, you would want the drive to just do it automatically, because why wouldn't you want it to just automatically be as fast as possible? It would be like having a car which had an ignition and a separate button labeled "press this to actually start the car." If it were as simple and transparent as pressing a button, it should be done without bothering you.
As I said, the real problem is that the filesystem doesn't know enough about the structure of the flash drive to cooperate optimally. This is mostly a problem with existing filesystems which are organized for HDDs, but is also a problem of the flash drives which do not expose sufficient information to the filesystems for them to be able to make the best choices.
This is not just a case of one company or the other simply being stupid or lazy (though there is some of that). When flash drives were first introduced, there were no filesystems to take advantage of them since there was no demand for creating such a filesystem, and writing filesystems is REALLY HARD. Really, really hard. This means that you aren't going to create a new filesystem unless you have a really good reason for doing so, since, as I said, writing filesystems is REALLY HARD. The correct thing for the SSD drive manufacturers to do was to hide the underlying complexity from the filesystems of the day and use a strategy which was good enough when a filesystem treated the SSD as a hard drive. This gave rise to the fancy block reordering schemes described in the article.
The problem is that these reordering schemes run into the problems described by the articles (all of the blocks are used up and must be erased before new data can be written) which could be mostly solved (or at least largely mitigated) by filesystems which knew more about the structure of the SSD. Unfortunately, one of the factors over which SSD makers compete is their block reordering scheme, so they have an incentive to keep that a secret and prevent people from circumventing it (thereby making their fancy reordering scheme irrelevant). Taken to the extreme, the only field on which to compete would be the makeup of the memory chips themselves, which would push the different SSD makers to a more commodity status (and therefore lower margins). From a user's view, this would be an ideal scenario as long as your operating system and filesystem could take advantage of the additional information provided by the memory chips.
We are in a sort of transitional period during which there is quite a bit of flux and uncertainty. It is very, very useful to have a small number of filesystems which everyone can read and write, since it makes data portability that much easier and possible. This is the main reason (along with its minimal storage and computational overhead) why the otherwise horribly outdated FAT family of filesystems are still in widespread usage by removable storage. Superior alternatives exist (mostly on Linux, due to the relative ease of writing new filesystems for it), but aren't compatible with other operating systems, and so are useless for consumer flash drives. For a flash-aware filesystem to gain widespread support, it would need to have compatibility with the major operating systems, a catch-22 for any new filesystems.
The other hurdle, which is less visible to end-users, is the way the drives expose information about themselves. For a flash-aware filesystem to be truly effective, it would need some way of gathering information about the characteristics of the underlying flash drive and have direct access to it, bypassing any (or the most dramatic) block reordering schemes. The technical ideal would be to have a single open, royalty-free, well-written and extensible standard for performing this kind of communication. The problem is that allowing such a standard would bypass the competitive differentiator of a manufacturer's reordering scheme, which individual manufacturers would likely resist. If such a standard could be agreed upon, it would go a long way towards enabling better cooperation between the hardware and software.
Whether any of this comes to pass remains to be seen. What is fairly certain, however, is that there will be a lot of volatility in the SSD space as these issues (and more) are figured out.
Tuesday, February 24, 2009
Programming Project Resume
This is a list of my personal programming projects. Most of them were one-time projects to scratch an itch or play around with something and are not maintained.
- A listing of most of my personal projects is available here.
- I am the maintainer for smart tab, a small Emacs package to allow the tab key to automagically indent or complete, depending on the context.
- My .emacs directory.
- I am the webmaster for the Brown Ballroom Dance Team website. Notable features of the site include faceted video search of nearly 2000 videos (see an example), similar management of costumes owned by the team (viewable only by team members), and user profile management.
- I worked on the Drupal version control integration libraries for the 2009 Google Summer of Code. The packages I worked on are here, here, and here.
- I added support for many CCK fields to the Drupal Apache Solr integration project and refined the existing framework for supporting CCK fields.
- I am working on Ezbl, a web browser based on the Uzbl browser project. It provides a glue between Emacs and Uzbl, forwarding commands and managing history and cookies.
- I have contributed to package.el, the package manager for Emacs. Some of my work has been integrated into the development branch for Emacs 24, and I am working on an overhaul of the codebase.
Friday, January 30, 2009
My Intelligence is Better Than Your Intelligence!
I happened to stumble across the book Emotional Intelligence sitting on Nathaniel's bookshelf today and commented on it. I have not read it, so take any analysis I have of the book with a giant, massive grain of salt. The premise seems to be that there is this concept, Emotional Intelligence, or EI, that measures, in the words of Wikipedia, "ability, to identify, assess, and manage the emotions of one's self, of others, and of groups." The claim seems to be that it is a better/good/different predictor of "success" than IQ.
I had a conversation with my dad a little while ago in which he made the point that the EI theory is not very sound, and that its only predictors for success are ones which measure factors already known to correlate with "success," such as g.
What is g, you ask? Here you go.
The basic idea seems to be in the second paragraph, in which it talks about how for each specific task, there seemed to be a task-specific component (spatial reasoning, etc.) and some general component that correlates with all of the tests. So a person's performance in math is a combination of his g and his math-specific abilities. Someone with a high g may still do poorly at math, since his math-specific skills may drag him down, but he will do better than someone with an equivalent math score but lower g.
My dad's point, if I remember correctly, was that EQ tests were predictive only insofar as they measured g, which is already known to be predictive of certain kinds of success. Apparently, the tests were equally predictive if the sections of an EQ test which did not measure g were removed.
This is an article (for pay, unfortunately) that says that EQ and IQ together are "a more powerful predictor of 'success' than either measure alone."
I would have to learn more about EQ, IQ, andgbefore making any kind of firm assertion, but as far as I can tell, the concept of g is generally accepted within psychology.
It is an interesting concept, one which I would like to learn more about. Whether EI is really a better and different measurement of success than IQ, what g is, more specifically, and what factors really contribute to intelligence are all interesting areas to pursue. I'll report back with any findings I have.
P.B. (Post Blog) I stumbled across this while looking at stuff and thought it might be interesting.
I had a conversation with my dad a little while ago in which he made the point that the EI theory is not very sound, and that its only predictors for success are ones which measure factors already known to correlate with "success," such as g.
What is g, you ask? Here you go.
The basic idea seems to be in the second paragraph, in which it talks about how for each specific task, there seemed to be a task-specific component (spatial reasoning, etc.) and some general component that correlates with all of the tests. So a person's performance in math is a combination of his g and his math-specific abilities. Someone with a high g may still do poorly at math, since his math-specific skills may drag him down, but he will do better than someone with an equivalent math score but lower g.
My dad's point, if I remember correctly, was that EQ tests were predictive only insofar as they measured g, which is already known to be predictive of certain kinds of success. Apparently, the tests were equally predictive if the sections of an EQ test which did not measure g were removed.
This is an article (for pay, unfortunately) that says that EQ and IQ together are "a more powerful predictor of 'success' than either measure alone."
I would have to learn more about EQ, IQ, andgbefore making any kind of firm assertion, but as far as I can tell, the concept of g is generally accepted within psychology.
It is an interesting concept, one which I would like to learn more about. Whether EI is really a better and different measurement of success than IQ, what g is, more specifically, and what factors really contribute to intelligence are all interesting areas to pursue. I'll report back with any findings I have.
P.B. (Post Blog) I stumbled across this while looking at stuff and thought it might be interesting.
Saturday, January 10, 2009
Doing Science in the Sink
I just performed an interesting and unintentional scientific experiment with an empty milk carton. After finishing off a gallon of (skim!) milk, I went to rinse it out so that the milk didn't sit and smell. When I first turned the faucet on, it was set to spew out hot water, which I realized was unnecessary, so I changed it to cold water after a few seconds. I put the top on, and shook.
When I went to open the carton, I noticed that the sides were puffed out and when I took the top off, a bit of air rushed out. Interesting, I thought, why does this happen?
After a few tests, I found that I was reliably able to reproduce this effect by filling the carton with a few seconds worth (around 3ish) of the hottest water the sink could put out, switching to the coldest water it could put out, and then putting the lid on and shaking. Every time, it would puff out the bottle. Also, filling it with just hot water or just cold water did not have an effect.
After spending a few minutes testing, I had isolated the effect, but did not know the cause. Luckily, my dad happened to be around, and suggested that the answer had something to do with the fact that cold water has a higher capacity for dissolved gas than warm water.
It is just a guess, but I think it could be that the cold water brings extra oxygen (or whatever is dissolved in it) into the carton, and when it is mixed with the hot water, its ability to retain the dissolved gas decreases. The warm and cold water stay separate long enough that there is time to put the cap on, and shaking the bottle mixes them up and causes the cold water to release its gas.
If this is correct (and there is an excellent chance that it is not; I haven't taken chem since 10th grade), it would also mean that the cold water gives up more gas by heating up than the hot water absorbs by cooling down. This seems to suggest that the gas absorption to temperature function is non-linear (or my measuring methods of "about 3 seconds" are not very precise).
Try it at home! All you need is an empty milk carton (or any largeish container which you can close quickly), a faucet which can switch between hot and cold water fairly quickly, and your imagination!
When I went to open the carton, I noticed that the sides were puffed out and when I took the top off, a bit of air rushed out. Interesting, I thought, why does this happen?
After a few tests, I found that I was reliably able to reproduce this effect by filling the carton with a few seconds worth (around 3ish) of the hottest water the sink could put out, switching to the coldest water it could put out, and then putting the lid on and shaking. Every time, it would puff out the bottle. Also, filling it with just hot water or just cold water did not have an effect.
After spending a few minutes testing, I had isolated the effect, but did not know the cause. Luckily, my dad happened to be around, and suggested that the answer had something to do with the fact that cold water has a higher capacity for dissolved gas than warm water.
It is just a guess, but I think it could be that the cold water brings extra oxygen (or whatever is dissolved in it) into the carton, and when it is mixed with the hot water, its ability to retain the dissolved gas decreases. The warm and cold water stay separate long enough that there is time to put the cap on, and shaking the bottle mixes them up and causes the cold water to release its gas.
If this is correct (and there is an excellent chance that it is not; I haven't taken chem since 10th grade), it would also mean that the cold water gives up more gas by heating up than the hot water absorbs by cooling down. This seems to suggest that the gas absorption to temperature function is non-linear (or my measuring methods of "about 3 seconds" are not very precise).
Try it at home! All you need is an empty milk carton (or any largeish container which you can close quickly), a faucet which can switch between hot and cold water fairly quickly, and your imagination!
Tuesday, January 6, 2009
Interesting Look at the Financial Meltdown
I saw this article sitting on the kitchen table the other day and then set about to find it on the internets.
It is fascinating to see how badly the whole financial system was set up that a guy, Harry Markopolos, could spend 9 years specifically detailing how Madoff's company could not be anything other than a Ponzi Scheme, and be basically ignored by the regulatory bodies whose job it was to investigate exactly those issues.
His argument went something along the lines of:
The real meat of the column, however, is how the Madoff scandal (and the financial crash in general) was caused not so much by greed, but by a breathtakingly stupid system. Evie would be proud.
Take a look at the article for a better explanation, but the basic problem was that the people running the SEC were using it as a stepping stone to get a job at the same places they were supposed to be regulating. So if Bob the SEC Director of Enforcement was really hard on, you know, criminals who conned people, when he went to them asking for a job, they would probably roll their eyes at "Bob the Buzzkill." On the other hand, if he turned a blind eye, or, even better, actively helped them steal more money, come hiring day, he would be "Bob, the cool guy who got us all those strippers and booze," a much more qualified candidate.
It seems like the solution will come in the form of a system which does not reward and encourage the exact opposite of what it is intended to achieve. How to do that, especially in the face of an established system which, until recently, was doing quite well for itself is an exercise left to the reader.
It is fascinating to see how badly the whole financial system was set up that a guy, Harry Markopolos, could spend 9 years specifically detailing how Madoff's company could not be anything other than a Ponzi Scheme, and be basically ignored by the regulatory bodies whose job it was to investigate exactly those issues.
His argument went something along the lines of:
- Madoff claims to be handling roughly $50 billion in securities.
- The markets in which he claims to be involved are smaller than that.
- If he actually did put $50 billion into the market, we would see those trades somewhere.
- We don't see them, so he can't have invested the amount of money he claims.
The real meat of the column, however, is how the Madoff scandal (and the financial crash in general) was caused not so much by greed, but by a breathtakingly stupid system. Evie would be proud.
Take a look at the article for a better explanation, but the basic problem was that the people running the SEC were using it as a stepping stone to get a job at the same places they were supposed to be regulating. So if Bob the SEC Director of Enforcement was really hard on, you know, criminals who conned people, when he went to them asking for a job, they would probably roll their eyes at "Bob the Buzzkill." On the other hand, if he turned a blind eye, or, even better, actively helped them steal more money, come hiring day, he would be "Bob, the cool guy who got us all those strippers and booze," a much more qualified candidate.
It seems like the solution will come in the form of a system which does not reward and encourage the exact opposite of what it is intended to achieve. How to do that, especially in the face of an established system which, until recently, was doing quite well for itself is an exercise left to the reader.
On the Baddity of Cities
I told you all! Cities are bad for you!
I have heard some criticism of the theory that the article presents, that the hypothesis is not really supported by the data, or that the fact that simply looking at pictures of nature improved performance on certain tests means that something else could be at work. I think these are valid, and that the topic needs some more research, but it is an interesting and intuitively correct hypothesis.
If I spend more than a few days in a city, I do tend to get physically tired and overwhelmed. Each time I go to New York, I am reminded of this. I find that everywhere I go, I am constantly scanning the surrounding area for threats and information. After a few minutes waiting at a subway stop, I have looked at the face of everyone in the terminal, and if anyone new comes in, I turn, look at them, and go back to waiting. It is subconscious, but it causes a lot of stress, since my environment is constantly changing and I am struggling to keep up. If I focus on ignoring my surroundings, I can avoid looking around, but that takes concentration and I still feel uneasy when I hear a turnstile turn but don't look at who came through.
One of my favorite parts of Andover was being able to take a short walk from my dorm and head to the ~100 acre sanctuary, which was a big forested area with a looping path going through it and two ponds. It was my favorite place to go jogging, and was a great way to relax from the pressure of the place. There was rarely anyone else there, or at least it was big enough that I didn't run into people very often, so I could just concentrate on running. In contrast, I haven't found anywhere at Brown where I enjoy running, since I hate running in the city. I tried it a few times, and it was just unbearable. I have to be looking out for cars and people, there are things and places to avoid and I have to remember my route back. If I'm running, I don't want to have to think about anything, and I can't do that in a city (even Providence).
So this article completely made sense, and I would like to see more studies to better establish the validity of the claim for a broader range of people. For me at least, I have known this for a long time.
On the bright side, it could be a lot worse.
I have heard some criticism of the theory that the article presents, that the hypothesis is not really supported by the data, or that the fact that simply looking at pictures of nature improved performance on certain tests means that something else could be at work. I think these are valid, and that the topic needs some more research, but it is an interesting and intuitively correct hypothesis.
If I spend more than a few days in a city, I do tend to get physically tired and overwhelmed. Each time I go to New York, I am reminded of this. I find that everywhere I go, I am constantly scanning the surrounding area for threats and information. After a few minutes waiting at a subway stop, I have looked at the face of everyone in the terminal, and if anyone new comes in, I turn, look at them, and go back to waiting. It is subconscious, but it causes a lot of stress, since my environment is constantly changing and I am struggling to keep up. If I focus on ignoring my surroundings, I can avoid looking around, but that takes concentration and I still feel uneasy when I hear a turnstile turn but don't look at who came through.
One of my favorite parts of Andover was being able to take a short walk from my dorm and head to the ~100 acre sanctuary, which was a big forested area with a looping path going through it and two ponds. It was my favorite place to go jogging, and was a great way to relax from the pressure of the place. There was rarely anyone else there, or at least it was big enough that I didn't run into people very often, so I could just concentrate on running. In contrast, I haven't found anywhere at Brown where I enjoy running, since I hate running in the city. I tried it a few times, and it was just unbearable. I have to be looking out for cars and people, there are things and places to avoid and I have to remember my route back. If I'm running, I don't want to have to think about anything, and I can't do that in a city (even Providence).
So this article completely made sense, and I would like to see more studies to better establish the validity of the claim for a broader range of people. For me at least, I have known this for a long time.
On the bright side, it could be a lot worse.
Subscribe to:
Posts (Atom)