Ever wondering what happened with an Exchange Server database corruption?

I presented this blog entry (ways to solve Exchange database corruption) at the upcoming NY Exchange User Group meeting on Tuesday, 1/8/08 at 6pm (see www.nyexug.com for more info).

Symptoms
Since I work with numerous clients and every client of mine uses an Exchange Server (in-house or hosted), I see a lot of things relating to Exchange most typical IT techs do not see. So, about a month ago, a client's Exchange 2003 Server starting showing signs of being non-responsive. The server would take an hour or so to boot-up if it did, and then when it did, logging in remotely was extremely difficult (it would hang after user/pwd login, or extremely slow refresh periods) via RDP (aka MS Terminal Services). My first thought was hard disk failure or other disk related problem. It was running on a RAID 1 setup with SAS 15k drives but those showed no signs of issues.

Found Issue
After reviewing the Event Logs, I saw reports of disk non-responsiveness. So, I checked the Dell OpenManage status for the disk subsystem, and it reported everything as fine. So, nothing there. I then performed a basic search with last modified date to last day, and reviewed all the recently modified files. Everything looked good until I saw that the 20 person's company's Public Folder Store had grown to 12GB from about 114MB. Yes, 114MB to 12GB! Information Store database files looked fine. So, I started reviewing the backup logs and noticed the Public Folder Store database was growing hundreds of MBs a day. Something was out of definitely wrong with it.

Approaches to Solve Corruption Problem
Before you do anything. Dismount the database in question. Then shutdown the Information Store service and copy the .edb & .stm to another hard drive or at a minimum to another folder. Then attempt to use the Microsoft's utility of Eseutil. Eseutil is a dangerous tool, so please make sure you thoroughly read the documentation & before a backup before proceeding. Sometimes, you can't make a simply copy/paste of the database. If so, you don't have much choice, but to proceed. As I warned above, you could lose data performing this. And only run if your databases are experiencing problems.

Commands to run from your bin directory and pointing to your databases.
"eseutil /g" - Integrity check in read only mode.
"eseutil /k" - Checksum check for header damage.
"eseutil /p" – Repairs table and page. Perform isinteg next.
"eseutil /d" - defrags db's and reduce the size based on empty space. You can safely run this if you dismount the databases.
"isinteg -s emailsrv -fix -test alltests" - emailsrv = name of server. Used to test and insure all corruption is solved.

After I ran these commands, the Public Folder store was reduced to 114MB and the server was responsive again.

Resources to Review Before Proceeding
MS Exchange Team Blog about database fixing with eseutil
http://msexchangeteam.com/archive/2004/06/18/159413.aspx

MS Support - Ramifications of running the eseutil /p or edbutil /d /r command in Exchange
http://support.microsoft.com/kb/259851

MS Exchange.org - isinteg & eseutil demo-ed
http://msexchange.org/tutorials/Exchange-ISINTEG-ESEUTIL.html

MS Support - eseutil checksum feature
http://support.microsoft.com/default.aspx?scid=kb;en-us;823167

MS Support - eseutil file header damage
http://support.microsoft.com/default.aspx?scid=kb;en-us;825088

MS Exchange Team Blog on 2007 SP1 Changes for ESE
http://msexchangeteam.com/archive/2007/11/30/447640.aspx