Scott Kveton

I always thought that if I were a CIO and had to make a decision about open source it would be a hard one. Afterall, you have to take so many things into consideration. What is your “main thing”? What strengths does your organization possess? What does your “product” run on? These are all very important questions for somebody in charge of the technical direction of an organization.

Most people know me as an advocate of open source; a guy that will always do the right thing at the end of the day. I’ll not try to hide it; I believe open source solutions are good ones in most cases. So its hard for to imagine being in the shoes of a CIO and having to make that sort of decision. Let’s be honest; in 20 years from now, you may be in a position of a CIO but your skill set will be entirely different. What would set you apart from others in seeing open source as a valid choice and then being able to execute on it?

Well, I believe I have a couple of opinions on that matter …

1. Evaluate your options: Look around you. What are you competitors doing? What are your peers doing? This sounds silly but a lot of people don’t do this. In the very rare case you will in fact be blazing some sort of trail with the decisions you make but it won’t happen on accident. If you’re not taking time out to evaluate your options and think about your overall strategy then you aren’t doing your job.
2. Choose people: Your people are your biggest asset. Again, sounds simple but most folks miss it. Where are your people’s strengths? Who are your shining stars? What path do they want to take? Where are your shining stars headed? This can sometimes be an indication of their decision making skills … if they are on the path to professional growth and not just in a holding pattern they may be on to something that you can tap. Look for the people in your organization that analyze the problems at hand. People who make snap judgements without looking at all possibilities are the same people that will end up getting you locked into some vendor/product that will cost you in the long run. The people who take a moment to examine the facts are a huge asset and totally infectious in an organization that promotes transparancy.
3. Know tactical loss vs. strategic win: I was talking with some folks on the Indiana University staff and we were discussing their choice to totally scrap their working course management software in favor of re-building it with a J2EE-based tool chain instead of sticking the its current development track which was Microsoft ASP/.NET. This is a huge decision to make. 95,000 students, faculty and staff that were quite happy with the solution they already had and they decided to scrap it in favor of building it again. This is called a short term tactical loss leading to a strategic win. That was over 2 years ago and the migration is complete. IU is now better leveraging their strengths in J2EE for the CMS development as well as being able to tie that software into their other J2EE applications that drive ERP and FIS. So many institutions think on the short-term when we will actually be here for the long-term and we might as well just do it right the first time.
4. Know your strengths: Where do the strengths of your company lie? I met with the CIO of a small medical software company a few months ago. By small I mean less than $1 billion a year in revenue. He had a hard path to take. As the new CIO at this company he had to make a decision; what development platform should his company choose for their systems. The expertise of his company was Windows, pure and simple. What was his decision? Go with .NET as a platform for the future.
5. Look at the facts, ignore the pundits: How Laura DiDio still has a job is completely beyond me. She is the Yankee Group’s “unbiased” technologist at large often pontificating on why TCO of Windows is way lower than any other solution. Funny that Microsoft are the ones that are paying the tab on these “reports” they put out. Anyhoo … how do you sift through fact and biased opinion? Even your people will do this … the fact is people don’t like the unknown and it forces them to push what they do know even if the technology or path is flawed. One thing you should always keep in your cadre of CIO tools are some technologists that you can depend on. People that you can put hard questions to and get an honest response from. If you don’t have something like this or you don’t have people in your organization that you can depend on for this then you are just flying blind.

These are just some of the things I have seen while looking up through the myriad of organizations that I’ve been involved with. Thanks everybody … I’ll be here all week …

The dust has finally settled on the Firefox 1.0 release and we’re finally getting a picture of what exactly happened last week.

Tuesday was a pretty hectic day but I have to say that all of the Mozilla and Gentoo folks that pitched in to help were absolutely heroic. We figure there were about 100 million hits across the three www.mozilla.org servers for a total of about 7 million unique visitors. Quite amazing considering we had one old and heavily loaded box available as of Monday morning before the release.

As of Monday night, we had 10 mirrors in the main ftp.mozilla.org rotation. The majority of these were machines hosted at AOL that had extra IP’s so we could add them several times into the round-robin. These mirrors did brilliantly until they just about collapsed around 2pm PST on Tuesday.

Plan b) came in the form of a download redirector that would use all of the mirrors at once. At the time of this writing we’re looking at close to 2.7 million downloads of Firefox leveraging our new download redirector (that may be released in the near future). Now we just need to trim up the main archive so that we can then advertise for more mirrors. If we’re going to hit 15% market share, we’re going to need a lot more mirrors.

The other issue we were having was with update.mozilla.org service. This site sees connections from every Firefox client that is on the network. Fortunately the clients connect at random times so the load is pretty steady. The downer is that there are so many clients … David Miller put a Squid cache in front of the update.m.o service and we saw an immediate improvement but it was still a bit sluggish. We added one more squid cache on another machine and that made all of the difference in the world. We’re able to handle a lot more connections now and the application server doing the heavy lifting in terms of version checking, etc no longer has to burden itself with the likes of handing out images over-and-over … its all in the Squid caches. BTW – 8GB of memory on one box with your Squid cache means you can put everything in memory … wow is that fun and impressive to watch in action.

Watching this release take off … watching everybody respond so well … seeing it all in action was one of the most impressive things I have ever seen. Yes, we had some hiccups in service. But we got through it and we’re making plans for the future.

Thanks go out to the boys at CNN that helped tweak our squid configurations. Lance & Stuart from Gentoo for configuring up the Nocona box that just wouldn’t stay up how we’d configured it. Blizzard for putting up another www.mozilla.org mirror. All of the mirrors across the globe; cshields, Peter Losher, Neil Bright, etc. Everybody in #bmo on irc.mozilla.org that pitched in and countless others that I know I’m missing.

After much anticipation, Firefox 1.0 is out now. And because there wasn’t enough said about it on the NET today, I’m going to anty in as well.

This morning around 12am PST Firefox 1.0 hit the streets. The download infrastructure held up pretty well for most of the day but finally crashed around 2pm PST. Fortunately we had a plan b). More on that in a bit.

Just about every site out on the NET had a link to www.mozilla.org and subsequently it was soon to be a goner. We threw in a couple of machines to the www.mozilla.org rotation and fortunately they don’t do anything funny with their website. All straight up HTML/CSS and no PHP or CGI’s … thank goodness. By about 7am PST they we were serving up close to 9000 requests/sec across all of the web servers in the rotation. One of the machines we threw in was a Dell 1850 with the new Intel Xeon’s with 64-bit extensions. It promptly fell over with some sort of memory errors. Switching the machine to nptl threads (from LinuxThreads) and using mpm worker with Apache 2.0 we were able to really rein in the box. By the afternoon it was happily chugging along at 500-800 requests/second with a load average of a little less than 0.20. All of this brought to you by Gentoo Linux. We’ll see how the web infrastructure holds up after the NY Times ad hits.

One of the biggest issues with any of the Mozilla products is getting the bits out to the end-users in a fashion that won’t require them to be clicking around some web tree trying to figure out what they need. In addition, we have about 40+ mirrors across the globe, each with varying available bandwidth, that we need to spread the load across. On top of all of that, we’d love to know how many downloads for each product, OS, etc from each mirror. This of course is a bit of a daunting process.

Mike Morgan and I wrote a mirror administration application that would handle the above issues. By 2:11pm PST we had it in production and it worked flawlessly. We were able to spread the load over the rest of the available mirrors and weight them based on available bandwidth. We would query each mirror every 15 minutes to make sure they were available and had the right files for each product. After the first hour we had processed a little over 50,000 downloads. All available through a pretty web interface all running on a Gentoo Linux machine that wasn’t even creeping above a 0.10 load average. We’ll see how this holds up during the big releases we have to handle in the near future.