Scott Kveton

The dust has finally settled on the Firefox 1.0 release and we’re finally getting a picture of what exactly happened last week.

Tuesday was a pretty hectic day but I have to say that all of the Mozilla and Gentoo folks that pitched in to help were absolutely heroic. We figure there were about 100 million hits across the three www.mozilla.org servers for a total of about 7 million unique visitors. Quite amazing considering we had one old and heavily loaded box available as of Monday morning before the release.

As of Monday night, we had 10 mirrors in the main ftp.mozilla.org rotation. The majority of these were machines hosted at AOL that had extra IP’s so we could add them several times into the round-robin. These mirrors did brilliantly until they just about collapsed around 2pm PST on Tuesday.

Plan b) came in the form of a download redirector that would use all of the mirrors at once. At the time of this writing we’re looking at close to 2.7 million downloads of Firefox leveraging our new download redirector (that may be released in the near future). Now we just need to trim up the main archive so that we can then advertise for more mirrors. If we’re going to hit 15% market share, we’re going to need a lot more mirrors.

The other issue we were having was with update.mozilla.org service. This site sees connections from every Firefox client that is on the network. Fortunately the clients connect at random times so the load is pretty steady. The downer is that there are so many clients … David Miller put a Squid cache in front of the update.m.o service and we saw an immediate improvement but it was still a bit sluggish. We added one more squid cache on another machine and that made all of the difference in the world. We’re able to handle a lot more connections now and the application server doing the heavy lifting in terms of version checking, etc no longer has to burden itself with the likes of handing out images over-and-over … its all in the Squid caches. BTW – 8GB of memory on one box with your Squid cache means you can put everything in memory … wow is that fun and impressive to watch in action.

Watching this release take off … watching everybody respond so well … seeing it all in action was one of the most impressive things I have ever seen. Yes, we had some hiccups in service. But we got through it and we’re making plans for the future.

Thanks go out to the boys at CNN that helped tweak our squid configurations. Lance & Stuart from Gentoo for configuring up the Nocona box that just wouldn’t stay up how we’d configured it. Blizzard for putting up another www.mozilla.org mirror. All of the mirrors across the globe; cshields, Peter Losher, Neil Bright, etc. Everybody in #bmo on irc.mozilla.org that pitched in and countless others that I know I’m missing.