Service interruption, April 7, 2010
Some service outages occurred today preventing HTTP access to the server. We believe the problem has been resolved and everything is operating normally again.
The issue was the IIS server was failing to service requests and returning an ECONNRESET. The problem was noticed around 3 PM PST and again around 5 PM PST. The first occurrence was temporarily resolved by restarting the web services while the investigation began. The cause had not been discovered until the 5 PM PST outage forced a server reset. Around this time, the investigation revealed some configuration issues with a mandatory update of the server firewall/antivirus/security system. Some time ago it was discovered that an optional email scanner feature was creating an excessive number of TCP endpoints that were not being terminated. During the update of the security software, this feature was inadvertently re-enabled and was very likely (but not 100% verified to be) the cause of the failures today. This feature was disabled. Also around this time it was noticed that the update did not correctly bring in the previously stable firewall configuration from the earlier version and this needed to be reconfigured. This required additional maintenance outside the standard maintenance window of 7 PM - 9 PM PST due to the urgency of having proper firewall configuration. We apologize for the inconvenience and service interruption. We've done everything in our power tonight to ensure that everything is configured properly and the situation will be monitored diligently in case the cause of the initial ECONNRESET problem is not yet fixed so that swift action can be taken.
