Happenings – April Fools Week

Over the past couple of days I’ve been fighting text messages from Postini that one of our Internet connections used to transfer e-mail was flapping for about 10-90 seconds every night. It would happen almost like clockwork: exactly 12:38AM every morning. Thinking that it was trouble with our ISP, I notified them of the issue. They didn’t seem concerned but scheduled a software update and reboot on our equipment anyway.
The Monday following April 1st, at 8:36AM the Exchange server lost connection with the rest of the network. Going in to VSphere, Ben found the system unresponsive. Just as he was about to hit the virtual power button, the screen unfroze. Perusing the event log was uneventful. The only thing that we could find of significance was a SCSI event about delayed writes to the disk. I was really worried about disk contention on the VM host but this system has been running trouble-free for about a year. The database has not grown either.
Regardless, I took the database offline and ran an ESEUTIL to defrag it. Then I defragged the virtual disk. Viewing the logs after the Information Store came back online, I found event 9580:

“Virus scanning is enabled but diagnostic logging for ‘Virus Scanning’ category is turned off. To see diagnostic events related to virus scanning, increase logging level for ‘Virus Scanning category using Exchange System Manager.”

So, into the ESM I went to turn on virus debug logging. Now I’m getting the events delivered to The Dude and the event viewer. Come to find out, the McAfee GroupShield product was not working properly. We’re now on the phone with McAfee tech support to figure out what went wrong.