July 2005

Another beautiful weekend here in Portland brings with it throngs of open source developers, activists and groupies to the lovely Rose City. O’Reilly is slowly taking over the Oregon Convention Center (OCC) for what promises to be the biggest OSCON ever.

I’m here at the OCC right now for the Saturday and Sunday FLOSS Summit that has been organized by the likes of Dave Neary (GNOME Board) and Allison Randal (The Perl Foundation). The first day brought with it 13 people from Eclipse, Python, Perl, Gentoo, LUA, Plone, Apache, OSL, Drupal and Creative Commons and we had some great discussions about how each organization is setup the way that it is, etc.

Themes coming from the first day were varied. We all agreed that each of these organizations is slightly different but that there are significant advantages if we could work together on things like legal and trademarks. The key is keeping the pragmatists in the discussion and not the zealots (”No way can our Perl accountant work on the Python accounting!! Sacrilege!”). Instead of forming yet-another-foundation around the interest of these groups, we decided that it would make the most sense to approach existing organizations that are tasked with this sort of role and see if they would be willing to work with us. Barring that, we’d continue to use the list to see what good we could do.

The second day will bring some legal expertise from the likes of Larry Rosen and a few others. This day should be much more well attended.

The coolest thing about Saturday happened right after dinner. We were all heading back to our cars and we happened upon a family walking past us on the street. The mom and dad were hippies with some serious tattoos and their son points at my t-shirt and says “Whoa! Look, its a Firefox!”

On Netcraft’s main page they mention that “the Mozilla Foundation site has been experiencing intermittent performance problems, which began early Sunday.”

Over this past weekend, while doing software upgrades, our static content web servers in one of our LVS clusters was mistakenly updated. We run Apache 2.0 using the worker mpm. This is the threaded mpm and thus does not work well with the PHP module (technically PHP is not thread-safe). The static content web servers (3 of them behind our OSL cluster) mistakenly had PHP installed and then removed. However, during the system update process it got re-installed. This caused our Apache instance to not want to start properly. This, plus the intersection of some vacation time for the systems folks meant this slipped through the cracks.

It should be noted that as soon as it was discovered the fix was trivial and service was quickly restored:

This also revealed gaps in our monitoring via our Nagios instance which also have been resolved.

I thought I would follow-up on this as it was in fact a mis-configuration on our part and not so much a capacity issue.

As many people have seen and heard, Drupal was able to raise close to $11,000 for purchase of infrastructure to help grow their project in a reliable fashion. The OSL has offered up to host this gear here.

When we originally started discussing upgrading the Drupal infrastructure, the discussions we had with the Drupal leaders were to just try to raise enough money to buy a server. Originally we said let’s shoot for $1000 and we can buy a cheapy machine and then we’ll be in better shape. However, as discussions progressed, people decided that if we’re going to do it, we should do it right with “enterprise”-class gear. Dual power supplies, hardware RAID, etc. To do this, I off-handedly quoted out a Dell 1850 with 2 x 2.8Ghz Xeon’s, 2 x 73G disks, hardware RAID, dual Gbit NIC’s and dual power supplies. The cost was $3056 on Dell’s site so I told everyone; why don’t we shoot for $3000 then?

Of course we all know what happened. $11,000 later and a generous donation from Sun and the picture had changed a little bit. I talked with my Dell rep and asked what they could do with the quote if we bought three of them. He was able to knock the price down to $2522 each. It was decided then that we would buy 3 Dell’s of the specifications above and integrate that with the Sun V20z that was donated.

As time has gone on, some folks have expressed concern about going with Dell for the servers and using the donations in a way that is not directly helping the F/OSS eco-system.

I wanted to just try and describe the reasoning behind the choices we have made here.

I agree that HP, IBM and Sun have all done good things for the open source community. I would even say that Dell has done a few things for the open source community but let’s be honest, at the end of the day, Dell isn’t an R&D company.

I’d like to point out that here at the OSL we use a lot of different gear. We’ve got HP’s, IBM, Dell, Sun, Apple, Netwinder, Genesi and an assortment of other gear to help provide our services to the community. There are a couple of things that we do when we consider hardware purchases. We consider price, compatibility, reliability and supportability. I’d like to talk about all of these as they relate to the usage of the Drupal donations.

We had ruled out white boxes from the get go (even before the outpouring of support from the community) so that leaves the “enterprise” vendors. Looking over the quotes we got back from these vendors (and doing some math on their websites) we could get the equivalent gear from HP for $3500, from IBM for $4500 and the equipment from Sun that was closely matched in specifications for $5000 but without dual power supplies or turn-key remote management. Bearing that in mind, we could buy 3 Dell’s and 2 of anything else. For the sake of redundancy and bang-for-buck we said let’s do more Dell’s and hopefully help avoid another outage.

Compatibility is a toughy and some people may laugh at me on that. However, would it be wise to go out and buy the shiniest gear in the world? I’ll even give you a recent Dell example and one that made us continue to use Dell’s. We had bought a new Dell 1850 with the new Intel EM64T processors. We installed RHEL 4 and away we went. This was a heavily loaded machine (you wouldn’t believe me if I told you but it supported at one time 20 million unique clients a day). After a few days it would kernel panic. Uh-oh. Reboot. A few days later, same deal. Guess what though? It was on Red Hat’s compatible hardware list which meant they had to (and did) support it. We had a couple of patches in a few days and next thing you know we’re up and rolling. The box has been up ever since.

Reliability is a critical factor. We wanted something that would have a certain level of reliability built-in (dual power supplies, remote management) but something that could be easily serviced (see below). The fact is; gear breaks. There are moving parts and components that fail. So, you have to factor in how easy it is to get support for a machine.

That leads to supportability; we wanted to be able to support this equipment as easily as possible. By that I mean we wanted to be able to (ideally) fill out a web form when a part fails and get a new one shipped to us. When the RAID controller says a disk has failed I’m inclined to believe it. I don’t want to have to sit on hold waiting for a support person that is going to ask “please make sure that you’ve run Windows Update on the machine first”. Again, mileage may vary for you and your vendor and I haven’t had experience with every single vendor in the world. I just know that what we have had with Dell (and HP for that matter) has always been great service.

Doing a little technorati digging yielded this blog entry on ahawkins.org. I feel pretty confident that I could go out on the Internet and find bad things about every single hardware vendor in the world. Heck, I’ll guarantee it. So then, what do you have to go on? You have to go on your experiences with vendors yourself. We have had a great deal of luck with our Dell gear here as well as coordinating problems with failed drives, bum motherboards, etc. We have even had our Dell rep offer up gear to help us with transitions to data centers, etc. On the other hand, our relationship with HP has not been as good mainly because they have had some turn over in their sales force and we’ve seen that HP is going through some changes. That said, I know folks that have the complete opposite relationship with HP and Dell. It all comes down to the personal relationships you have forged with a vendor.

Finally, it should be noted that we actually have spare gear here on hand for supporting projects like Drupal. As a matter of fact, we ran out of temporary gear which is why we haven’t landed Drupal here yet and are waiting for the gear order to happen. With the addition of Drupal and now Participatory Culture we’re going to be landing some cold spares here specifically to support these projects in their time of need. No more downtime; rip out the procs (or disks or power supplies or memory) and get people back up ASAP.

Our goal was to help Drupal get the most bang for their buck while still keeping it maintainable by our people here at the OSL. We don’t have any special deal with Dell (or HP, Sun, or IBM for that matter). We’re just another customer to them. However, our personal relationships, the pricing, reliability and supportability of the Dell equipment were the main factors in going that route.

As I’ve said before and will probably say again and again; your mileage may vary.

Even though we got the garden in late this year, things appear to be doing really well. Normally by this time we’ve been able to harvest quite a few sweet peppers but only a few so far.

This Summer has been an odd weather year to say the least. Where normally we’ve had a few hot spells, so far things have been really mild. Why only this past week did we hit the 90’s for the first time. Even when we went to plant in early June, the weather was drab, chilly and quite wet. In fact, I had to delay planting by another week and didn’t really get things in the ground until 6/9/2005.

Looking at one month of progress, I’m going to cover how the peppers, tomatoes and eggplant are doing.

I tried some new things with the peppers this year. Namely I did starts from seed starting in mid-April. The starts did relatively well but took quite some time to “take off”. Looking back on it, I wish I had started these guys back in January possibly in my office. There’s always next year on that. In any case, most of those peppers have really started to thrive and are starting to catch up with the purchased starts from Bi-mart.

A little note on Bi-mart. Bi-mart was the original warehouse store … sort of the Oregonian version of Costco or Sam’s Club. I went and visited there on the weekend around 6/9/2005 and low-and-behold a truck pulls up right as I’m getting there with some of the most amazing pepper starts I’ve ever seen. I asked the guy driving the truck and he said they were out of a local farm in Woodburn (about 45 miles north of Corvallis). He had lots of sweet peppers (California Wonder and Bell) as well as habenero, thai and jalapeno. These were all immaculate and in 1 gallon containers. The cost for these beauties? Frickin’ $1.50 each … are you kidding me?! I snapped up 3 sweet, a jalapeno and a habenero. I went back a week later to find they had been cleaned out except for one fantastic looking thai hot pepper plant and a “cherry bomb” hot pepper plant. I snapped those up as well …

The lesson learned here is that it’s silly for me to do the sweet peppers from seed when Bi-mart is such a great resource. Next year I’ll be doing some more exotic stuff from seed and saving shelf space in our garden window for herbs, etc. All of my sweet (and jalapeno, thai and cherry bomb) peppers will be coming from Bi-mart now.

I’ve had a bit of a tough time with bugs this year. It appears the mild winter made it really easy for the Japanese Beetles to come back in force. I’ve only caught (and subsequently dispatched) a few of them but they are gaining momentum. I finally resorted to some Safer soap and that appears to be at least slowing them down. They have just moved on to the flowers (zinnias and sunflowers).

I had a rough go of it with a few peppers that I started from seed. These of course were the ones I was most excited about. My fish, black hungarian and senorita peppers I’m most excited about. The fish have really started to take off now. It looks like I’m going to actually get some fruit off of both the black hungarian and the senorita’s as well. They went from this, to this in less than 4 weeks.

I am also doing two long cayenne pepper plants as well. Last year my long cayenne plant did gang busters. I actually took seeds from that plant and did the starts myself. They aren’t off to an amazing start, but I’m hoping to compensate by having two of them going at once.

My tomatoes are going bonkers. We are doing three varieties this year (and even have some un-determined volunteers showing up); Cosmonaut Volkov (slicer), Silvery Fir Tree and a Sun Gold. These are going crazy. Still very few flowers right now so I’m getting a little nervous about my cages; they simply will not hold up these beasties.

We are also doing two varieties (’asian swallow’ on the left, ‘dusky’ on the right) of eggplants this year. Last year we tried to do some eggplant and it did not do so well. The year before that we had a great harvest so we felt like it was sort of hit-or-miss. This year, we tried doing the eggplants with black plastic over them (around their base to keep the soil temperature up at night). So far so good. These things are looking amazing.

It’s only July and this keeps getting better and better all the time.

I wanted to pass along the information that Rafael Ebron is going to be heading up the development of UMO (a.k.a. addons.mozilla.org) effective immediately. Rafael brings with him experience from the Netscape SmartUpdate project as well as being a full-time Mozilla Foundation staff member. He is going to be an excellent leader of this project.

I haven’t been able to dedicate the time needed to UMO in the last few months and its been affecting the progress of the project. To that end, I’m going to be stepping away from UMO completely except in regards to any capacity planning/management, etc that has been coming with the OSL’s work with the MoFo and their infrastructure.

I’ve enjoyed my time working on this project with you all; this is an amazing project and one that I think has some amazing potential. Sorry for my delinquency in the last few months.

The open source community never ceases to amaze me.

For those of you not following closely, I’ll recap. Drupal is a PHP-based content management system that has really started to gain some momentum. Drupal currently drives SpreadFirefox as well as a bunch of other sites and is the basis for secondary products like what the folks at CivicSpace are doing. In recent months, their shared server has really started to show some strain. They are often taking outages and increased server load due to their popularity.

In February, Chris Messina connected us with the Drupal community when their first started experiencing difficulties with their exisiting infrastructure. We had some discussions and offered up our services because the Drupal community is a good one.

We didn’t even seek out Drupal, they found us. Chris Messina connected them to us when they started having problems a few months ago. We offered up our services because Drupal is a great (and growing) open source community. Drupal really fits with the OSL’s mission of being “about community” because they are a tool that enables communities.

About two weeks ago, the Drupal folks got serious about sorting out their hosting and started a fundraiser email thread. We talked about what it would take to host the site, etc and came up with a modestly configured enterprise-class machine. The total cost was $3000 that we were going to shoot for. We finally sorted out the wording on the email and web page for the fundraiser and then let it fly around 1am PST 7/10/2005.

Within 13 hours they had met (and vastly exceeded) their goal of $3000. In addition to offers of cash, people have been offering hardware, hosted services and co-location to Drupal at a phenomenal rate. We’ll be able to meet Drupal’s needs and then some.

The amazing thing here is that a community has spoken and the OSL is helping by giving them a venue to realize their voice. It’s an amazing time to be a part of the open source world.