Firefox 1.0
After much anticipation, Firefox 1.0 is out now. And because there wasn’t enough said about it on the NET today, I’m going to anty in as well.
This morning around 12am PST Firefox 1.0 hit the streets. The download infrastructure held up pretty well for most of the day but finally crashed around 2pm PST. Fortunately we had a plan b). More on that in a bit.
Just about every site out on the NET had a link to www.mozilla.org and subsequently it was soon to be a goner. We threw in a couple of machines to the www.mozilla.org rotation and fortunately they don’t do anything funny with their website. All straight up HTML/CSS and no PHP or CGI’s … thank goodness. 
By about 7am PST they we were serving up close to 9000 requests/sec across all of the web servers in the rotation. One of the machines we threw in was a Dell 1850 with the new Intel Xeon’s with 64-bit extensions. It promptly fell over with some sort of memory errors. Switching the machine to nptl threads (from LinuxThreads) and using mpm worker with Apache 2.0 we were able to really rein in the box. By the afternoon it was happily chugging along at 500-800 requests/second with a load average of a little less than 0.20. All of this brought to you by Gentoo Linux. We’ll see how the web infrastructure holds up after the NY Times ad hits.
One of the biggest issues with any of the Mozilla products is getting the bits out to the end-users in a fashion that won’t require them to be clicking around some web tree trying to figure out what they need. In addition, we have about 40+ mirrors across the globe, each with varying available bandwidth, that we need to spread the load across. On top of all of that, we’d love to know how many downloads for each product, OS, etc from each mirror. This of course is a bit of a daunting process.
Mike Morgan and I wrote a mirror administration application that would handle the above issues. By 2:11pm PST we had it in production and it worked flawlessly. We were able to spread the load over the rest of the available mirrors and weight them based on available bandwidth. We would query each mirror every 15 minutes to make sure they were available and had the right files for each product. After the first hour we had processed a little over 50,000 downloads. All available through a pretty web interface all running on a Gentoo Linux machine that wasn’t even creeping above a 0.10 load average. We’ll see how this holds up during the big releases we have to handle in the near future.
