Scott Kveton

For the past two years I’ve been hearing about grid computing and how utility computing is going to revolutionize the industry. I never really saw the application for it. When I saw that Sun’s grid play had zero customers by October of last year, I was feeling pretty smug about it. Does grid computing even matter?

Flash back over the past couple of months and watching projects like Mozilla, Meebo, Flickr, Zooomr and others try to keep up with demand under increasing load and the “slashdot” and “digg” effects. The barrier to entry for some new projects can be painful if a) the project is resource intensive and b) they experience exponential growth in a short time period. How can companies ramp up their infrastructure in a timely fashion while still doing work at-the-speed-of-business (not to use a horrible cliche but it seems apt here)?

The barrier to entry is still pretty low for new Web 2.0 companies. There was so much sunk cost left over from the bubble that things like bandwidth, office space and others are much cheaper to acquire. Coupling that with maturing open source tools and you have a formula for getting your service out to users much more quickly. The one thing that hasn’t really changed though is the time to purchase, configure and deploy the “servers” for driving these services. I’ll give a few examples.

Sometime in September 2005, Meebo launched to great acclaim. If you’ve never seen Meebo, think of it as an AJAX-ified web-based instant messenger client for AIM, Google Talk, Yahoo! Instant Messenger, etc. It actually is a web front-end to Gaim running on Linux from what I can gather. Over the course of the last few months Meebo has been working hard to ramp up their infrastructure to meet the growing demand. It appears they have upwards of 35+ application servers (or instances) serving up their instant messaging goodness. It has taken months to get this infrastructure up and running not to mention the sheer cost of the gear. Their setup is fairly specialized with the Gaim usage but most of it appears to be LAMP stack based.

In October 2005, another Web 2.0 application launched; Zooomr. Zooomr has been best described (not by me) as Flickr on steroids. Just last week they were dugg which rendered the service almost completely unusable. Looking at their infrastructure, it is definitely based on a LAMP stack (without the A and P == Python). The bummer here is there is this great new web service that simply could not scale when the demand hit. The lost opportunity cost could be pretty significant. How many people will remember to go back other than the really core users?

Looking at the above two examples, I started to think there has to be a better way. What if you could just leverage some sort of grid of LAMP machines to make this happen and then pay per cycle while you bring up your new infrastructure? Folks like Meebo and Zooomr would be able to launch very quickly, meet the Slashdot and Digg effects with ease while continually increasing usage.

What would it take to implement this? Well, you could have this as a value-added service of a co-location facility. They have a cluster of machines ready-to-go plus some staging infrastructure for people to test what they want to deploy. That could get pretty cost intensive for one co-location facility pretty quickly. However, what if you did this across multiple companies and co-location facilities? Now you’d have your grid component (there would have to be some standardization here) and the ability to fail across the infrastructure very easily. Hook in some caching and redundant database services and now you’ve got a pretty compelling tool.

Taking the above examples, they could have deployed across this LAMP grid and been able to scale that much further, that much faster right when they released. This would remove a serious constraint for these companies (and many others) if it could be done economically.

I’ve pitched this idea to a few folks and they think its pretty neat. Of course you can make anything look good on a white board. The big question is, can this be done and more importantly would it be economical and effective for these small companies to use and for the co-location facilities to deploy?