Server issue, Dec 23, 2009
A server issue occurred this morning. The result of this issue was that some site functions and AyaNova systems were unavailable. The basic cause was essentially that the databases stopped responding to internal requests from the web server.
Due to the time of day and the nature of the problem as it could be best understood at the time, I made some adjustments to various configurations and then did a full shutdown / reboot in order to get everything up and running as quickly as possible. The system is back up and hopefully the changes made will ensure that the system will remain stable throughout the day until further investigation can be done and no further interruption will occur.
The problem is not yet fully understood due to the extensive log data that I will need to examine in order to properly investigate, which I will be doing throughout today. Any further adjustments to the server will be made during the standard maintenance window of 7-9 PM tonight if necessary.
For the technical among you and in the interest of full disclosure:
The problem spanned Microsoft SQL and extended to MySQL causing them both to fail to respond to requests simultaneously. Restarting various services revealed that the IIS FTP service refused to restart with an error regarding “insufficient storage” (of course it is not a disk related error), and while the databases restarted without incident, they failed to respond to requests. The Event Viewer reported nothing of interest, however the Microsoft SQL logs have some information that will need to be examined more to understand. My sense of the matter based on the FTP service failure message is that it lies somewhere in the arena of the Microsoft MSDTC system and how it internally maps ports in the networking layers for requests. Some significant Windows Updates were applied on Sunday which were tested ahead of time, and worked fine Monday and Tuesday, but as always unforeseen issues can appear later as may be the case here. The adjustments made to hopefully reduce the problem for today were reducing the port usage range for MySQL, reducing the total number of active connections in the AyaNova IIS Application Pool. There was a known MSDTC problem noted in earlier versions of IIS, but resolved in the version we are running making the workarounds for the previous versions invalid for this one, which is another reason I suspect the recent Windows Updates may be at work here.
