Netcraft Reports on Mozilla.org

On Netcraft’s main page they mention that “the Mozilla Foundation site has been experiencing intermittent performance problems, which began early Sunday.”

Over this past weekend, while doing software upgrades, our static content web servers in one of our LVS clusters was mistakenly updated. We run Apache 2.0 using the worker mpm. This is the threaded mpm and thus does not work well with the PHP module (technically PHP is not thread-safe). The static content web servers (3 of them behind our OSL cluster) mistakenly had PHP installed and then removed. However, during the system update process it got re-installed. This caused our Apache instance to not want to start properly. This, plus the intersection of some vacation time for the systems folks meant this slipped through the cracks.

It should be noted that as soon as it was discovered the fix was trivial and service was quickly restored:

This also revealed gaps in our monitoring via our Nagios instance which also have been resolved.

I thought I would follow-up on this as it was in fact a mis-configuration on our part and not so much a capacity issue.