Ever wondering what happened with an Exchange Server database corruption?

I presented this blog entry (ways to solve Exchange database corruption) at the upcoming NY Exchange User Group meeting on Tuesday, 1/8/08 at 6pm (see www.nyexug.com for more info).

Symptoms
Since I work with numerous clients and every client of mine uses an Exchange Server (in-house or hosted), I see a lot of things relating to Exchange most typical IT techs do not see. So, about a month ago, a client's Exchange 2003 Server starting showing signs of being non-responsive. The server would take an hour or so to boot-up if it did, and then when it did, logging in remotely was extremely difficult (it would hang after user/pwd login, or extremely slow refresh periods) via RDP (aka MS Terminal Services). My first thought was hard disk failure or other disk related problem. It was running on a RAID 1 setup with SAS 15k drives but those showed no signs of issues.

Found Issue
After reviewing the Event Logs, I saw reports of disk non-responsiveness. So, I checked the Dell OpenManage status for the disk subsystem, and it reported everything as fine. So, nothing there. I then performed a basic search with last modified date to last day, and reviewed all the recently modified files. Everything looked good until I saw that the 20 person's company's Public Folder Store had grown to 12GB from about 114MB. Yes, 114MB to 12GB! Information Store database files looked fine. So, I started reviewing the backup logs and noticed the Public Folder Store database was growing hundreds of MBs a day. Something was out of definitely wrong with it.

Approaches to Solve Corruption Problem
Before you do anything. Dismount the database in question. Then shutdown the Information Store service and copy the .edb & .stm to another hard drive or at a minimum to another folder. Then attempt to use the Microsoft's utility of Eseutil. Eseutil is a dangerous tool, so please make sure you thoroughly read the documentation & before a backup before proceeding. Sometimes, you can't make a simply copy/paste of the database. If so, you don't have much choice, but to proceed. As I warned above, you could lose data performing this. And only run if your databases are experiencing problems.

Commands to run from your bin directory and pointing to your databases.
"eseutil /g" - Integrity check in read only mode.
"eseutil /k" - Checksum check for header damage.
"eseutil /p" – Repairs table and page. Perform isinteg next.
"eseutil /d" - defrags db's and reduce the size based on empty space. You can safely run this if you dismount the databases.
"isinteg -s emailsrv -fix -test alltests" - emailsrv = name of server. Used to test and insure all corruption is solved.

After I ran these commands, the Public Folder store was reduced to 114MB and the server was responsive again.

Resources to Review Before Proceeding
MS Exchange Team Blog about database fixing with eseutil
http://msexchangeteam.com/archive/2004/06/18/159413.aspx

MS Support - Ramifications of running the eseutil /p or edbutil /d /r command in Exchange
http://support.microsoft.com/kb/259851

MS Exchange.org - isinteg & eseutil demo-ed
http://msexchange.org/tutorials/Exchange-ISINTEG-ESEUTIL.html

MS Support - eseutil checksum feature
http://support.microsoft.com/default.aspx?scid=kb;en-us;823167

MS Support - eseutil file header damage
http://support.microsoft.com/default.aspx?scid=kb;en-us;825088

MS Exchange Team Blog on 2007 SP1 Changes for ESE
http://msexchangeteam.com/archive/2007/11/30/447640.aspx

A warm helo & ehlo to everyone...

On my first blog posting.... I realize, most people do not know what "helo" and "ehlo" mean. So picking a blog name with this might not be the smartest marketing idea, but I'm not so concerned with that (more important to educate folks at least a little bit). So, Without either term (helo & ehlo), email would not work as it does. Meaning, no email would be sent or received. So, maybe you should read the next paragraph to under a bit more.

All email servers communicate using either term (technically, this is how the initiating server starts communicating with another mail server). "helo" is the term for saying hello to another server when the initiating server is not very sophisticated and just wants to perform a basic email send communication. More sophisticated servers that support more features (aka extensions) and wish to use them (e.g. security, reliability, speed, etc), start off with "ehlo", also meaning hello, but also "tell me what commands" you support. This allows the other server to know, that this server can support x feature and will respond to handle it and perform the email communication in the enhanced manner.

So, of course my favorite email server, Microsoft Exchange Server (2003 currently) supports this feature & so does my favorite email gateway server (IceWarp's Merak Mail Server v.8.x). Unfortunately, some folks use firewalls to block (aka proxy) smtp commands, which can slow down email flow and prevent the use of ehlo. So, I recommend disabling such smtp firewall features (e.g. Cisco's fixup protocol) since this is a false sense of security and slows down communication. I've also had not heard of a security risk for smtp or other used commands on Exchange & Merak. So, this is not a real risk in my book (I tend to error on the side of security, so if I thought it was a danger, I would not suggest it). The benefits of "ehlo" outweigh any theoretical risk. So, enough of the soap box for this blog posting. I'll step down...

So, my plans for the future: the blog will tend to focus on a number of topics since I am interested in such, email in general in relation to server-side spam filtering techniques, email security, and performance, Microsoft Exchange Server, wireless (e.g. 802.11a/b/g/n), physical security, and other technology related things I get involved with.