"Cheating" on an Exchange 2003 Hardware Upgrade

[Project completed in June 2008. Figured this was in my drafts, I should post it.]

Hello Everyone,

I "cheated" on an Exchange 2003 hardware upgrade I did last weekend (Fri-Sat), or that's at least how I feel since this was hands down the fastest and easiest upgrade I've ever done (and it was about 80GB of db's on an older server with direct attached storage). At the end of the weekend, I started to think maybe I should carry around one of these "things" for my clients for upgrades. I'll share what this "thing" was later in the posting. I don't want folks to think I'm pushing products. My role in the project was to insure the replacement of the Exchange Server hardware went smoothly. The client was in production 24/7 and literally the office was staffed 6 days a week. So, I was concerned originally how to insure minimal downtime.

Background on existing hardware & performance
We were upgrading from an Exchange 2003 Server that was installed with 3 hard drives in a RAID 5 hard drive configuration (direct attached storage) for the OS, transaction logs, and Exchange databases. Company had about 60 users and 30 BlackBerrys or so. 1 BES user adds a load similar to 2 Outlook users. So, total company usage was about 120 users. Performance was an issue, so some users were configured for cached mode to "improve" performance. Cached mode should not be required for LANs, unless Outlook end users are receiving "retrieving data from network". Recommended another DAS server that used RAID 1 for the OS, RAID 1 for transaction logs, and RAID 10 for the Exchange databases.

The Migration
So, I checked the OS install (another admin handled that), Exchange install (using the /disasterrecovery switch for setup and service pack 2), Exchange config, and insuring the email & public folder migration completed successful. Only catch was during this server replacement, there was to be no downtime and no use of Exchange clustered services. Hmmm, that's a challenge. Or so one would think.

The Cheat
The client I was working for happened to have a 3rd party product (keeping read to find out) for Exchange that in essence allowed the "cheating". And I mean this in a very good way. It saved me a LOT of time. Meaning, we told the 3rd party product to take over all the existing Exchange services (MAPI, SMTP, OWA, IMAP, etc) and data for the Outlook, OWA, ActiveSync, & BlackBerry was available to all users. All their data was still available. This took a few minutes (3 or 4 minutes) on the switch-over. Now everyone was operating off the appliance and end users didn't know this besides restarting Outlook and re-authenticating to OWA (ActiveSync & BES users had a slight delay. BES users could be out of service for up to 15 minutes, but that's a limitation of BES). Once the appliance took over, we copied over the Exchange databases (.edb/.stm's) using robocopy to the new Exchange 2003 Server. We considered upgrading to 2007, but the appliance and all the associated Exchange applications would have had to been upgraded, and it wasn't cost effective (TCO reminder). So, after we started robocopy-ing, we went home.

Day 2 of the Migration & Failback
I'm not going to go into the details, but migrating took a few hours including getting the SSL certs for OWA and handling all that. Once the new hardware was setup with Exchange, it was time to bring back all the new email. As I previously said, the reason to copy over the databases to the new server, was the appliance then doesn't need to copy over all data, and just new email/data. Once we copied over the databases and transaction logs, we were able to get the Exchange Server fully operational and enable the failback from the appliance. We then failed back from the appliance to the new Exchange Server. This took a lot longer to check that all data was copied from the appliance to the new Exchange Server. This took 10 hours or so and then everyone had to relaunch Outlook and re-authenticate against the new server.

Appliance Details
The "cheat" was an Exchange high availability appliance from Teneros. For the record, this company has been bankrupt and while still in operation as of 7/10, I do NOT recommend purchasing this product. Even though this appliance runs 2 operating systems, Linux and Windows, the entire configuration is on 2 web pages. Meaning, the Teneros support team is really what runs this product, not the Exchange admin. As per the web interface, to say the amount of information and configuration is sparse, is putting it lightly. Overall the product worked well and we ran into 2 glitches due to permissions and resetting process of the AD name due to poor documentation. And the migration process took longer than expected since the status of synchronization is not very accurate. Not a big deal, since end users are working during the failover and failback. Overall solution is very impressive, but I have some doubts since I'm not a big fan of trusting secret functionality of a black box type solution. I like to know how applications work and I do have concerns over Exchange updates or patches breaking the Teneros functionality. If you are curious, pricing is around $10k, give or take a few thousands. If you wanted to see the demo, Teneros did present at the NY Exchange User Group meeting back in November of December of 2007 or check out their website.

-Ben

Tip - Needing to move 10GB+ files use ESEUTIL

Ehlo All,

ESEUTIL utility is not just for Exchange Servers. Consider it for your large (think 10GB+ files) copies/moves. I'm preparing for an Exchange Server storage expansion (removing some existing hard drives and replacing them with larger ones) which requires I move a client's Exchange databases to another storage location and then move them back. We're talking about over 270GB of databases. So, when you move large files you do not want to use copy/paste, robocopy, richcopy, xcopy, etc since those buffer read/writes (aka caching) and is significantly slower than unbuffered read/writes (non-caching). For big files, you want to use an application that uses unbuffered read/writes which is (you guessed it), eseutil.

Microsoft Windows Server Performance Team Blog Article about using ESEUTIL for faster copies.

Notes from article:
"There are x86 & x64 versions of ESEUTIL, so make sure you use the right version for your operating system. The syntax for ESEUTIL is very simple: eseutil /y /d . Of course, since we're using command line syntax - we can use ESEUTIL in batch files or scripts." Once I test the scripts, I'll edit and update these article to help everyone.

Enjoy,
-Ben

Which Version are you running (SPx w/Rollup x?) - Easily Tell Now!

Ehlo All,

Ever wonder if you have Rollup 6 or 7 installed on Exchange 2007 SP1. It can be very complicated since Exchange just lists a build version # 8.1.340.1 or 8.1.359.2. I work with Exchange a lot, and I can't remember them. So, thankfully Microsoft has a handy wiki page that does.

http://social.technet.microsoft.com/wiki/contents/articles/exchange-server-and-update-rollups-builds-numbers.aspx

Enjoy,
-Ben

Article - Discussing Leapfrogging from 2003 to 2010

Ehlo All,

This is a good non-technical article from SearchExchange.com that discusses IT admins that have gone from Exchange Server 2003 to 2010 and others that are considering it. Hence bypassing Exchange 2007. I would agree with this, since the biggest change involved in moving to 2007 or 2010 is the x64 operating system. Once you have planned for x64, make the jump if you can to 2010. Jumping from 2003 to 2007 improves I/O by 70%, while going from 2003 to 2010 improves I/O by 90%. So, this will allow you to support more users on a single server than before. One of the big cost savings in upgrading.

The only word of caution is make sure your Exchange eco-system software is supported (e.g. backups, spam filtering, archiving, replication, mobile messaging [aka BlackBerry], etc). Jumping Exchange versions typically requires software upgrade especially for a x64 operating system.

-Ben

Windows 2008 R2 and Exchange 2007? Microsoft says no.

Hello All,

Anyone who has not deployed Exchange Server 2007 and wants to do it on Windows 2008 R2 (it'll be launched this month), the answer is it is NOT supported. Not a big deal, since I would just wait a bit longer and have the benefits of I suspect to be the hands down best Exchange version yet, Exchange 2010. If you are wondering, 2003 is still the king. Some articles discussing this R2 rejection: Windows IT Pro and the Microsoft Exchange Team postings.

Server OS Support
Exchange Server 2003 supports Window Server 2000, Windows Server 2003, & Windows Server 2003 R2. (3 OSes supported)

Exchange Server 2007 supports Windows Server 2003, Windows Server 2003 R2, & Windows Server 2008. (3 OSes supported)

Exchange Server 2010 supports Windows Server 2008 & Windows Server 2008 R2. (2 OSes supported currently)

Microsoft decided against QA and supporting Exchange 2007 on Windows Server 2008, which is understandable since supporting 4 OSes is a time consuming process and 2010 is getting launch within the month. So, wait a little longer, and try and deploy Exchange Server 2010 once your eco-system is supported (e.g. BES, GFI, AppAssure, Backup Exec, etc).

-Ben

Article - Domain Controllers are critical to Exchange, Know Which DC is Used for Communication

Hello All,

A working Active Directory foundation is required for Exchange Server, so have you ever wondered which Domain Controller (DC) was being used by your Exchange Server. Now you can. Read on if you're interested.

I recently read an article about how Exchange Server knows what Active Directory servers are available. For Exchange 2000, 2003, and 2007, the answer lies with an Exchange process called "Directory Services Access", aka DSAccess. It polls all DC every 15 minutes and makes optimizations based on the results. You can read the summary article here (via SearchExchange.org) or a full detailed technical article here (via MSExchange.org) or another one from SearchExchage.org here. Start with the summary, and then the technical articles.

Tidbit - Under 2000 and 2003, DSAccess ran under the Exchange service called System Attendant. Exchange 2007 changed that, and now it runs under a separate Windows service running on your Exchange Servers called "Microsoft Exchange Active Directory Topology Service".

Until next time... may your email flow well.
-Ben

Virtualization and Exchange - It's Microsoft Supported and Works Well!

Ehlo All,

Sadly, there is a LOT of marketing non-truths (aka lies) being spread about Virtualization and Exchange Server (e.g. it doesn't work, Microsoft won't support it, bad performance, etc). I was so tired of hearing this, I finally decided to give a presentation at the NY Exchange Server User Group Meeting this past week that discussed an intro to virtualization and Exchange and exactly what is supported (Exchange 2003 & 2007 is VM supported, VMware ESX and Exchange 2007 SP1 is supported, and much more). My Exchange environment is fully Microsoft PSS (Professional Support Services) supported since it's Exchange 2007 SP2, Windows 2008, and VMware ESX 3.5 U2. Amazing, I know. I've included the topics presented in the evening, and feel free to spend sometime reading through PDF. I've included referenced footnotes as well.

Presentation Topics
- intro to virtualization
- pro's / con's of virtualization
- what servers work with virtualization
- Microsoft's Exchange Support Policy explained in detail
- different Exchange version support
- VM performance comparison to old school hardware deployments (and yes, it's old school)
- why virtualization is more important than performance
- what Exchange Server really needs

Review the September 8, 2009 NYExUG presentation by Ben Serebin, and post any questions you have on the blog and I'll happily answer them.

.
QUIT

Free Microsoft Internet Tool for Verifying Exchange is setup correctly

Ehlo All,

"Have you ever installed an Exchange server and wanted to verify your Internet facing services were setup and configured properly? Things like Exchange ActiveSync, AutoDiscover, Outlook Anywhere (RPC/HTTP), and inbound email." Well, the Microsoft Exchange Team Blog finally announced an Exchange testing website I used in a presentation months ago. It's a great time saver. It's called Exchange Remote Connectivity Analyzer (ExRCA) tool which can be accessed at https://www.TestExchangeConnectivity.com.

This is a great free resource. Available for:
- Exchange 2003
- Exchange 2007
- Windows Mobile 5
- Windows Mobile 6.1+

Enjoy,
-Ben

SearchExchange Top 10 Tips for 2008

Ehlo All,

SearchExchange (of TechTarget) posts their top 10 tips for 2008 for Microsoft Exchange Server (primarily 2007) article. I would call it 5 tips, but there are 10 listed. :-)

-Ben

Easy access to .edb/.stm files via Ontrack PowerControls

Ehlo All,

So, I recently upgraded to Exchange 2007, and realized one day I might need access to my old Exchange 2003 data. Even though I performed move mailboxes for most accounts, I didn't did it for all. I figured, I should save the databases, .edb & .stm files "just in case". Well, the "just in case" time came almost one month later after my upgrade.

So, while I run AppAssure's MailRetriever which is designed to mount offline Exchange 2003/2007 databases, it's at my datacenter and I had the .edb/.stm files at my home office. I considered transfering the files, but they are close to 40GBs, so I decided to test out under a virtual machine Ontrack's PowerControls. So, I fired up my handy VMware Workstation 6.5 & XP Pro w/Office 2003 and mounted over a 100Mb network via UNC/drive the folder share containing the files. Once mapping to a drive letter, the .edb/.stm files quickly and easily exported the data (some task items) to a PST. Opening the 23GB .edb w/o logs took about 15 seconds which was mindblowing which was OVER a network connection. I then tested the PST data in Outlook which looked good and compressed it and emailed the 1.3Mb off to the user that needed it. That's it. It was so easy, I was shocked. It literally took me maybe 10 minutes if that long.

Overall, Ontrack's PowerControls software package is very impressive.

-Ben

OWA Light Comparison from Exchange 2003 to 2007

Ehlo All,

So, I upgraded to Exchange 2007 this past week. The primary focus of this blog post is OWA Light difference between 2003 and 2007. Reason is, I frequently use OWA on non-IE browser to access Public Folders and was surprised to see existing features in OWA 2003 removed. So, I wondered what else was missing in action (or added) from OWA Light on 2007. Here is the list. [-] means feature is missing from new version. [+] means feature is added in 2007.

If you have a lot of OWA Light users, I would seriously consider staying on Exchange 2003 or leaving OWA users on Exchange 2003 while having other users on Exchange 2007.

What others talked about....
Exchange Product Team Discussing OWA Light 2007 - it seems OWA Light only gained feature according to the Exchange Team.

Figured I would share since OWA is discussed...
Exchange Product Team Discussing Exchange 14 (aka E14). Not much detail except talk about the 3.5 million beta testers and "empowering" end users with distribution list & GAL control. Not very sexy.

OWA 2007 OWA Light - to summarize, is the bare essential in webmail. So, that's email, calendar, & contacts. Nada mas.
[-] Calendar (Daily View Only)
[-] NO Public Folders Access
[-] NO Tasks
[+] Quickly jump to a subfolder (via drop-down)
[+] Easily rename/move/delete folders
[+] Ability to search in Address Book
[+] Ability to change AD password
[+] View a partnered Windows Mobile device

OWA 2003 OWA Light
[+] Calendar (Daily & Weekly)
[+] Public Folders
[+] Tasks

To say I'm disappointed would be an understatement. Come on Microsoft, add features, don't remove otherwise folks will jump to other web based email products like GMa... you know what I'm talking about. :-)

I'll blog more about differences in the future and the migration in general.

-Ben

Favorite URLs for my Exchange & Admin Work Posted...

Ehlo All,

I figured I would share the websites I use for my Exchange and other admin work so other admin's would benefit. So, enjoy. Any comments or suggestions for other sites, let me know in the comments.

-Ben

How To Article on Virtualization for Exchange 2003 & Office Communications Server Support for the BlackBerry

This is a how to article on the process to virtualize Exchange 2003 with Hyper-V writing by an Exchange MVP, Brien Posey. Good walk-through of the process. And, since this is Hyper-V vm'ed, this is supported by Microsoft Support.

http://searchexchange.techtarget.com/tip/0,289483,sid43_gci1334779,00.html?track=NL-359&ad=667516&asrc=EM_NLT_4749386&uid=1446997

And BlackBerry now supports Office Communication Server 2007.
http://na.blackberry.com/eng/services/server/exchange/ocs2007.jsp

Enjoy,
-Ben

New Formula: Exchange + Virtualization = Microsoft Support

Yes, the new formula is correct. And I did not expect to see this so soon. Microsoft has announced that you can run Exchange 2003/2007 in a Virtual Server 2005 R2/Hyper-V virtualized environment and get Microsoft Professional Support Services. Here is the official Microsoft release about this. http://technet.microsoft.com/en-us/library/cc794548.aspx There are a details for each configuration (e.g. all 2007 roles supported except Unified Messaging), but this is a step in the right direction.

Also, Windows ITPro's article on this mentions that Microsoft has created a Server Virtualization Validation Program (SVVP) and that VMware is in the process of attempting to achieve certification. This is not a show stopper in my eyes since VMware supports clients Windows & Exchange hosts currently.

-Ben

"Cheating" on an Exchange 2003 Hardware Upgrade

Hello Everyone,

I "cheated" on an Exchange 2003 hardware upgrade I did 2 weekends ago (Fri-Sat), or that's at least how I feel since this was hands down the fastest and easiest upgrade I've ever done (and it was about 80GB of db's on an older server with direct attached storage). At the end of the weekend, I started to think maybe I should carry around one of these "things" for my clients for upgrades. I'll share what this "thing" was later in the posting. I don't want folks to think I'm pushing products. My role in the project was to insure the replacement of the Exchange Server hardware went smoothly. The client was in production 24/7 and literally the office was staffed 6 days a week. So, I was concerned originally how to insure minimal downtime.

Background on existing hardware & performance
We were upgrading from an Exchange 2003 Server that was installed with 3 hard drives in a RAID 5 hard drive configuration (direct attached storage) for the OS, transaction logs, and Exchange databases. Company had about 60 users and 30 BlackBerrys or so. 1 BES user adds a load similar to 2 Outlook users. So, total company usage was about 120 users. Performance was an issue, so some users were configured for cached mode to "improve" performance. Cached mode should not be required for LANs, unless Outlook end users are receiving "retrieving data from server". Always a bad sign to see unless you have poor network connection. Recommended another DAS server that used RAID 1 for the OS, RAID 1 for transaction logs, and RAID 10 for the Exchange databases.

The Migration for a hardware refresh
So, I checked the OS install (another admin handled that), Exchange install (using the /disasterrecovery switch for setup and service pack 2), Exchange config, and insuring the email & public folder migration completed successful. Only catch was during this server replacement, there was to be no downtime and no use of Exchange clustered services. Hmmm, that's a challenge. Or so one would think.

The Cheat
The client I was working for happened to have a 3rd party product (keeping read to find out) for Exchange that in essence allowed the "cheating". And I mean this in a very good way. It saved us a LOT of time. Meaning, we told the 3rd party product to take over all the existing Exchange services (MAPI, SMTP, OWA, IMAP, etc) and all data for the Outlook, OWA, ActiveSync, & BlackBerry was available to all users. This took a few minutes (3 or 4 minutes) on the switch-over. Meaning the appliance takes some time to take control. Once that was done, everyone was operating off the appliance and end users didn't know this besides restarting Outlook and re-authenticating to OWA (ActiveSync & BES users had a slight delay. BES users could be out of service for up to 15 minutes, but that's a limitation of BES). Once the appliance took over, we copied over the Exchange databases (.edb/.stm's) using robocopy to the new Exchange 2003 Server. We considered upgrading to 2007, but the appliance and all the associated Exchange applications would have had to been upgraded, and it wasn't cost effective (TCO reminder). So, after we started robocopy-ing, we went home.

Day 2 of the Migration & Failback
I'm not going to go into the details, but migrating took a few hours including getting the SSL certs for OWA and handling all that. Once the new hardware was setup with Exchange, it was time to bring back all the new email. As I previously said, the reason to copy over the databases to the new server, was the appliance then doesn't need to copy over all data, and just new email/data. So, this is a huge time saver. Once we copied over the databases and transaction logs, we were able to get the Exchange Server fully operational and enable the failback from the appliance. We then failed back from the appliance to the new Exchange Server. This took a lot longer to check that all data was copied from the appliance to the new Exchange Server. This took 10 hours or so and then everyone had to relaunch Outlook and re-authenticate against the new server.

Appliance Details
The "cheat" was an Exchange high availability appliance from Teneros. Even though this appliance runs 2 operating systems, Linux and Windows, the entire configuration is on 2 web pages. Meaning, the Teneros support team is really what runs this product, not the Exchange admin. As per the web interface, to say the amount of information and configuration is sparse, is putting it lightly. Overall the product worked well and we ran into 2 glitches due to permissions and resetting process of the AD name due to poor documentation. And the migration process took longer than expected since the status of synchronization is not very accurate. Not a big deal, since end users are working during the failover and failback. Overall solution is very impressive, but I have some doubts since I'm not a big fan of trusting secret functionality of a black box type solution. I like to know how applications work and I do have concerns over Exchange updates or patches breaking the Teneros functionality. If you are curious, pricing is around $10k, give or take a few thousands. If you wanted to see the demo, Teneros did present at the NY Exchange User Group meeting back in November of December of 2007 or check out their website.

There are many other software solutions on the market that do this, and so I'll blog about when I work with them. My user group has had demo's and presentation on a # of them, but this was the first real world usage of one.

Let me know your thoughts on this.

-Ben

Ever wondering what happened with an Exchange Server database corruption?

I presented this blog entry (ways to solve Exchange database corruption) at the upcoming NY Exchange User Group meeting on Tuesday, 1/8/08 at 6pm (see www.nyexug.com for more info).

Symptoms
Since I work with numerous clients and every client of mine uses an Exchange Server (in-house or hosted), I see a lot of things relating to Exchange most typical IT techs do not see. So, about a month ago, a client's Exchange 2003 Server starting showing signs of being non-responsive. The server would take an hour or so to boot-up if it did, and then when it did, logging in remotely was extremely difficult (it would hang after user/pwd login, or extremely slow refresh periods) via RDP (aka MS Terminal Services). My first thought was hard disk failure or other disk related problem. It was running on a RAID 1 setup with SAS 15k drives but those showed no signs of issues.

Found Issue
After reviewing the Event Logs, I saw reports of disk non-responsiveness. So, I checked the Dell OpenManage status for the disk subsystem, and it reported everything as fine. So, nothing there. I then performed a basic search with last modified date to last day, and reviewed all the recently modified files. Everything looked good until I saw that the 20 person's company's Public Folder Store had grown to 12GB from about 114MB. Yes, 114MB to 12GB! Information Store database files looked fine. So, I started reviewing the backup logs and noticed the Public Folder Store database was growing hundreds of MBs a day. Something was out of definitely wrong with it.

Approaches to Solve Corruption Problem
Before you do anything. Dismount the database in question. Then shutdown the Information Store service and copy the .edb & .stm to another hard drive or at a minimum to another folder. Then attempt to use the Microsoft's utility of Eseutil. Eseutil is a dangerous tool, so please make sure you thoroughly read the documentation & before a backup before proceeding. Sometimes, you can't make a simply copy/paste of the database. If so, you don't have much choice, but to proceed. As I warned above, you could lose data performing this. And only run if your databases are experiencing problems.

Commands to run from your bin directory and pointing to your databases.
"eseutil /g" - Integrity check in read only mode.
"eseutil /k" - Checksum check for header damage.
"eseutil /p" – Repairs table and page. Perform isinteg next.
"eseutil /d" - defrags db's and reduce the size based on empty space. You can safely run this if you dismount the databases.
"isinteg -s emailsrv -fix -test alltests" - emailsrv = name of server. Used to test and insure all corruption is solved.

After I ran these commands, the Public Folder store was reduced to 114MB and the server was responsive again.

Resources to Review Before Proceeding
MS Exchange Team Blog about database fixing with eseutil
http://msexchangeteam.com/archive/2004/06/18/159413.aspx

MS Support - Ramifications of running the eseutil /p or edbutil /d /r command in Exchange
http://support.microsoft.com/kb/259851

MS Exchange.org - isinteg & eseutil demo-ed
http://msexchange.org/tutorials/Exchange-ISINTEG-ESEUTIL.html

MS Support - eseutil checksum feature
http://support.microsoft.com/default.aspx?scid=kb;en-us;823167

MS Support - eseutil file header damage
http://support.microsoft.com/default.aspx?scid=kb;en-us;825088

MS Exchange Team Blog on 2007 SP1 Changes for ESE
http://msexchangeteam.com/archive/2007/11/30/447640.aspx