The Hour of Doom: Conquered?

Yesterday at 4pm PST (aka 00:00 GMT) the “Hour of Doom” hit the Mozilla infrastructure as it does at the start of every month. However, this time we were ready.

I worked with Dave this past week to get the application servers for our PHP services behind the LVS cluster here at the OSL. With OSCON and Linux World Expo coming up I knew we would not have many cycles to get it done until after. Getting this built out now would allow us to brace ourselves in part for the deluge of requests to addons.mozilla.org as well as to prepare for what has been deemed the “hour of doom”.

The best way to describe the “hour of doom” is like having to pay down a bill every month on something silly you did while drunk months and months ago (note: the author has never done this, but he had a friend once that did …). The silly thing we did in this case was a bug in Firefox that would cause older clients (it has been fixed now) to continually phone home for the first 7 days of the month starting right on the nose at 00:00 GMT. We discovered this bug in January (somebody correct me if I’m wrong there) and have been having to deal with it ever since.

In the past few months, Dave, Polvi and Corey have been bustin’ their humps getting the new LVS clusters (yes, its plural) up to handle these bursts in traffic as well as have a roadmap for growth over the next few years for the MoFo. Polvi made some great configuration scripts for managing the clusters and Dave has been a maniac with the Red Hat Network configurations (its pretty darn slick now — we can change the role of any server behind the cluster in a matter of minutes). Corey has been a trooper fixing boxes here that were broken in shipping as well as moving earth to help get the cluster into place in a timely fashion.

So yesterday at 00:00 GMT it hit and although there were a few hiccups, the monster held together through an amazing deluge of traffic (upwards of 100 Mbit of just AUS traffic). Dave and Polvi did some final tweaking that brought the load down considerably and should have us ready for next month … here’s to smooth sailin’ and butt kicker SA’s … Good job guys …