Scott Kveton

For the past two years I’ve been hearing about grid computing and how utility computing is going to revolutionize the industry. I never really saw the application for it. When I saw that Sun’s grid play had zero customers by October of last year, I was feeling pretty smug about it. Does grid computing even matter?

Flash back over the past couple of months and watching projects like Mozilla, Meebo, Flickr, Zooomr and others try to keep up with demand under increasing load and the “slashdot” and “digg” effects. The barrier to entry for some new projects can be painful if a) the project is resource intensive and b) they experience exponential growth in a short time period. How can companies ramp up their infrastructure in a timely fashion while still doing work at-the-speed-of-business (not to use a horrible cliche but it seems apt here)?

The barrier to entry is still pretty low for new Web 2.0 companies. There was so much sunk cost left over from the bubble that things like bandwidth, office space and others are much cheaper to acquire. Coupling that with maturing open source tools and you have a formula for getting your service out to users much more quickly. The one thing that hasn’t really changed though is the time to purchase, configure and deploy the “servers” for driving these services. I’ll give a few examples.

Sometime in September 2005, Meebo launched to great acclaim. If you’ve never seen Meebo, think of it as an AJAX-ified web-based instant messenger client for AIM, Google Talk, Yahoo! Instant Messenger, etc. It actually is a web front-end to Gaim running on Linux from what I can gather. Over the course of the last few months Meebo has been working hard to ramp up their infrastructure to meet the growing demand. It appears they have upwards of 35+ application servers (or instances) serving up their instant messaging goodness. It has taken months to get this infrastructure up and running not to mention the sheer cost of the gear. Their setup is fairly specialized with the Gaim usage but most of it appears to be LAMP stack based.

In October 2005, another Web 2.0 application launched; Zooomr. Zooomr has been best described (not by me) as Flickr on steroids. Just last week they were dugg which rendered the service almost completely unusable. Looking at their infrastructure, it is definitely based on a LAMP stack (without the A and P == Python). The bummer here is there is this great new web service that simply could not scale when the demand hit. The lost opportunity cost could be pretty significant. How many people will remember to go back other than the really core users?

Looking at the above two examples, I started to think there has to be a better way. What if you could just leverage some sort of grid of LAMP machines to make this happen and then pay per cycle while you bring up your new infrastructure? Folks like Meebo and Zooomr would be able to launch very quickly, meet the Slashdot and Digg effects with ease while continually increasing usage.

What would it take to implement this? Well, you could have this as a value-added service of a co-location facility. They have a cluster of machines ready-to-go plus some staging infrastructure for people to test what they want to deploy. That could get pretty cost intensive for one co-location facility pretty quickly. However, what if you did this across multiple companies and co-location facilities? Now you’d have your grid component (there would have to be some standardization here) and the ability to fail across the infrastructure very easily. Hook in some caching and redundant database services and now you’ve got a pretty compelling tool.

Taking the above examples, they could have deployed across this LAMP grid and been able to scale that much further, that much faster right when they released. This would remove a serious constraint for these companies (and many others) if it could be done economically.

I’ve pitched this idea to a few folks and they think its pretty neat. Of course you can make anything look good on a white board. The big question is, can this be done and more importantly would it be economical and effective for these small companies to use and for the co-location facilities to deploy?

We’ve been having a rousing discussion on the POSSE list over the last couple of days spurred on by David Myers, the founder of openforest.org. He posted a great set of questions that I took the time to follow-up on. I’m posting the questions and my responses to them here as a chance to vet some of the ideas to a wider audience.

1. In his Open for Business column, Jon Perr writes “Much remains to be done, of course, to bring about Oregon’s success as an open source mecca.” What are some of the top items on that to-do list?

We have so many components in place right now; OSDL, OSL, OTBC, PSU, OSU, etc … More acronyms than you can shake a stick at. However, what we’re lacking are the new companies to take advantage of these assets.

During RecentChangesCamp, there was a group of students visiting from France that came to see about this “open technology cluster” here in Oregon. They had done quite a bit of research and were traveling around Oregon visiting all of the people/groups in the cluster. Their conclusion? “Where’s the beef?” There isn’t a cluster here because there are almost no startups. I actually had to concur with them; they were dead on.

I almost feel like we don’t have anybody thinking big enough (myself included). I’m not saying we need an uber huge company here or that we need to sacrifice what makes this Oregon. Just the opposite. I believe that our way of life here really melds with this always-connected, merit-based economy that is emerging on the Internet. If you could encapsulate the Oregon way of life in a web service, I think you’d have a gold mine. It would just suck if somebody did it in the bay area first …

So what’s on the todo list? Great question … But to me its finding where the rubber meets the road in Portland; connect the movers/shakers, stir it all together and get them to think big. We’ve got all of the pieces; now we just need to put it all together.

(I think I just answered your question with another question — sorry ’bout that)

2. Does the democratic-yet-meritocratic nature of the open source community make it an effective organizing principle for extending the development model beyond software? Like developing a grassroots-powered media network, for instance. Or perhaps writing legislation? Couldn’t the US Constitution be thought of as an early open source development project?

I saw a great presentation from Mark Shuttleworth (of Ubuntu-fame) who talked about this specific issue. He said that the reason we had open source instead of open law or open medicine first was because of the tools. As soon as email, diff and patch revealed themselves on the Internet, we had the most rudimentary tools of collaboration. As these tools have matured over time, they are starting to have applicability to folks outside of the technology space. Its no longer just geeks writing code to help them write more code. Its becoming technology for the sake of X where X is democracy, music sharing and anything else you can imagine. Take 10 lawyers, put them in a room with a 200 page word document and say “collaborate”! No wonder lawyers are so miserable; the tools for collaboration for them stink. Imagine using a wiki instead of a word doc? Holy crap … Now we’re talkin’.

In Thomas Friedman’s book “The World is Flat” (if you haven’t read this you really should — IMHO its one of the #1 drivers for open source adoption at the C-level right now) there is a quote (and I can’t remember it exactly) about how the industrial revolution was all about the innovations of single individuals. Take Henry Ford as an example. The knowledge revolution will be about the innovations of communities; people coming together to do great things. I think open source is just one great example of this. However, I believe the best is yet to come.

3. If Oregon attempts to pitch itself as an “open source” clone of the technology hubs in Silicon Valley or Redmond would we risk endangering the sort of collaborative, cooperative and sustainably-minded ethos seemingly indigenous to our corner of the country? Isn’t this the kind of sensibility that plays to the strengths of open source development?

I don’t think we want to pitch ourselves as a clone of anything. We’re too small of a State to just try and copy something or even to compare ourselves to Silicon Valley or Redmond. I think we have to play on our strengths you list above; its what got us here.