Message Dehydration isn't a good thing for Exchange!

Ehlo All,


Quote of the week: "limit is 94% before message dehydration occurs."

A normally very stable client's Exchange Server 2007 stopped processing inbound emails and this was the issue above. Client reported all internal email was working though. I logged into their Exchange Server and reviewed the normal issues inside the Exchange Management Console and nothing jumped out (e.g. databases mounted, 3GB free space on C, 700GB+ free space on database partition (D), no quota limits, receive connectors present/enabled, etc). Strange. I decided to check the email filtering solution in front of the Exchange Server. Since I always recommend clients use my company's email filtering service (SpamCop - I'll discuss this is another post) since it allows me to quickly troubleshoot issues and provide very secure email service. Reviewing SpamCop outbound queue to the client's Exchange Server illuminated the error:


Deferred: : host 55.55.55.55 said: 452 4.3.1 Insufficient system resources (in reply to...

More details about this error can be found on this posting. Checking the client's Application Event Log for this error, I found it "The Microsoft Exchange Transport service is rejecting message submissions because the available disk space has dropped below the configured threshold.". Which was surprising since the C drive had a decent amount of space available (3GB+). What changed? Good old SBS's Windows Server Update Services had downloaded every update under the sun and the storage space threshold was passed and triggered back pressure. I uninstalled WSUS and that eliminated 10GB and back pressure was eased and email flow started. I plan to remove some other functionality as well and plan to do a scheduled reboot as well to free up more space. To sum up, SBS causes more problems than it's worth for Exchange deployments. I prefer clean Exchange installs over Exchange SBS installs.

-Ben

Fixing ActiveSync on an Exchange 2007 Server

Ehlo All,


So, a client's iPhone with ActiveSync stopped working with their Exchange Server 2007. If rebooting and deleting and setting it back up doesn't fix it, confirm your server's AS is working. An good way to do this is to test with Microsoft Exchange test website found here. So after some investigating, it turned up that the IIS Virtual Directory for "ActiveSync" wasn't responding correctly. One can test this by "https://myservername/Microsoft-Server-ActiveSync" and make sure it prompts for a username and pwd. My client's server didn't do that. It reported 501 service unavailable error. I therefore deleted the ActiveSync virtual directory and re-created it and it fixed the issue. A great blog posting to explain this can be found here. To clarify, the "XXXXX" in his example for normal Exchange installs is "Default Web Site".


Handy URLs:
Detailed instructions on deleting and recreating virtual directories for Exchange 2007
http://my.opera.com/RavenOverride/blog/2009/06/17/how-to-recreate-all-virtual-directories-for-exchange-2007

Microsoft's Test Website for Exchange/ActiveSync/Outlook/SMTP
https://www.testexchangeconnectivity.com

-Ben

Article - HA for Exchange 2010 is more costly than Exchange 2007

Ehlo All,

This is an excellent article especially if you're using or considering deploying Exchange 2007 in an HA configuration with SCR,. Also, the new HA mailbox solution in Exchange 2010 is called Database Availability Groups (aka DAGs). This requires the Enterprise Edition for Windows Server 2008, and we do not know if it requires Exchange 2010 Standard or Enterprise. Either way, the requirement of Server Enterprise Edition makes it costly.

Excellent website (Exchange-Genie.com) that explains with pictures the following HA solutions for Exchange 2007 & 2010, CCR, LCR, SCC, SCR, and DAG. In this case, a picture is worth a thousand words.

Information is power,
-Ben

Intro to Exchange 2007 & SSL Certificates

[Written in December 2008. Not sure why I didn't post it back then. Found in my drafts.]

Ehlo All,

SSL Certificates (aka SSL certs, certs) are complicated. Exchange is complicated. Quick story... during my research, I read about a Microsoft employee (Exchange admin) who thought a bug existed with a SSL cert vendor's special "Exchange 2007" cert. So he contacted an Exchange Team PM about the issue and it turned out he didn't properly configure the Exchange SSL cert generation. Moral of the story, add SSL certs & Exchange 2007 together, and it can be a real challenge. So, I recommend you read this intro if you're considering or even using Exchange 2007 & real SSL certs especially since many things have changed.

First, the good news with Exchange 2007. Microsoft now includes self-signed certs on install. So, you're secure out of the box. Meaning, everything is encrypted, but you'll get errors each time you access a SSL protected site with it. So, you'll want to fix this.

Well, everything you know about Exchange 2003 SSL certificates can be thrown out, or that's what I've learned so far. You can use wildcard SSL certs, but you'll run into issues in the future (Outlook auto-discovery has issues, Windows Mobile 5 has issues, etc), so I would recommend you embrace the new SSL cert that Microsoft wants you to use OR learn about a special DNS configuration that lets you avoid this (thanks to a MS June 07 update for Outlook 2007).

As per the new certs, they are best described by DigiCert (I liked their explanation & enhanced it some) is:

Exchange 2007 Certificates Definition
The new SSL certificates are called Unified Communications Certificates (aka UC Certificates, UCC Certificates, or SAN certificates) which give you full control of the Subject Alternative Name field so you can secure as few or as many host names as you like with just one SSL certificate. These are NOT wildcard certs since they secure specific hostnames you define within this one cert. Wildcard certs secure any subdomain (e.g. *.mydomain.com), while UCC certs secure (autodiscover.mydomain.com, webmail.mydomain.com, mail.mydomain.com, exchsrv.mydomain.local, etc).

Microsoft's Recommended List of UCC Cert Providers as of 12/08
Comodo - http://www.comodo.com/msexchange/index.html
DigiCert - http://www.digicert.com/unified-communications-ssl-tls.htm
Entrust - http://www.entrust.net/ssl-certificates/unified-communications.htm
Microsoft's Recommended List of UCC Cert Providers

Special DNS Configuration to Avoid UCC Certs
So, you decided you didn't want to spend about $250-300 on a UCC cert for one year. I can understand. There is another option I hinted to above that relates to a June 2007 Outlook 2007 update that added a special feature to avoid the need for a UCC cert for autodiscovery or complex admin configuration. It involves setting a special DNS record to get around this. The DNS record is a SRV record. Once you have this SRV record set, the Exchange 2007 server's externl adn internal URLs need to be this one server as identified in the SRV DNS record.

Learn more about this in Microsoft White Paper on Auto-Discovery in Exchange & Outlook 2007. http://technet.microsoft.com/en-us/library/bb332063.aspx

MS Article on how to setup DNS SRV record for auto-discovery functionality
MS KB article about auto-discovery issues and fixes

-Ben

Which Version are you running (SPx w/Rollup x?) - Easily Tell Now!

Ehlo All,

Ever wonder if you have Rollup 6 or 7 installed on Exchange 2007 SP1. It can be very complicated since Exchange just lists a build version # 8.1.340.1 or 8.1.359.2. I work with Exchange a lot, and I can't remember them. So, thankfully Microsoft has a handy wiki page that does.

http://social.technet.microsoft.com/wiki/contents/articles/exchange-server-and-update-rollups-builds-numbers.aspx

Enjoy,
-Ben

Article - Quick Tip to improve OWA caching on Exchange 2007

Ehlo All,

Quick tip to improve OWA caching on Exchange 2007. Most admins have plenty of CPU on their Exchange Server and not enough bandwidth. So, this enables more compression (gzip baby) and reduces bandwidth usage. A win win for the vast majority of us.

OWA 2007 configuration tricks to boost performance
http://searchexchange.techtarget.com/generic/0,295582,sid43_gci1371540,00.html

-Ben

Article - Discussing cleaning up AD for Exchange 2010

Ehlo All,

If you plan to upgrade to Exchange 2010 from 2007, read this good article by Exchange MVP Brien Posey to learn how to clean up your Active Directory during the upgrade process.

Article - SearchExchange AD Clean-up from Exchange 2007 to 2010

I'm planning an Exchange 2010 deployment for a client in January, but it's a clean install with no pre-existing Exchange, so I won't need this, but for my own upgrade from 2007 to 2010, this would come in handy.

-Ben

Exchange 2007 will support Windows 2008 R2

Hello All,

The Microsoft Exchange Team announced today (11/4/09) that Exchange 2007 will be supported under Windows Server 2008 R2 once an update is released for this. Originally, it was not going to be supported, but MS changes their mind. Full details can be found here.

http://msexchangeteam.com/archive/2009/11/04/453026.aspx

Enjoy,
-Ben

Test your Exchange Restore Capability - Tips for Restoring

Ehlo All,

So, I've been using AppAssure's MailRetriever (part of Replay product) on Exchange 2007 (& other types of servers) for about 9 months now, and have been impressed for it's ability for continuous data protection (aka snapshots) and ability to natively write VM files (e.g. VMware Workstation, VM Server, VM Player, ESX, ESXi, and Hyper-V). I've used it numerous times to restore individual Exchange items and traditional files. BUT, a major concern has arisen recently. I've been working with AppAssure support for over a week to resolve a significant issue with item level restores and no good news yet. My major concerns are that the Replay product reports a restore success, but it's not. I've replicated the issue 3 times and support has been notified and working on it. Restoring an entire user's mailbox folder (e.g. Sent Items) results in almost 500 missing items (from about 85k). Yet, OWA/Outlook/exmerge to PST report about 500 more items that the restore and matches the correct folder item count.

During this time, I needed a way to test restores. So, originally, I would create a test AD (Test Account1) and email account (taccount1@somefake.addomain.com) and then restore my data to this account. And then I would delete the AD and email account, and re-create on the next restore. So, I've gotten good at deleting AD accounts and purging mailboxes. But, I found a better approach. Just delete the email account and purge it from the database and then re-create it for your next test. Why delete the AD account, it's not necessary. Some helpful sites to learn the commands.

1) Delete Exchange attributes from AD User (disconnects mailbox from AD user)
http://technet.microsoft.com/en-us/library/aa997210.aspx
Here is what you need to type in bold.
[PS] C:\Windows\System32>disable-mailbox bserebin@somefake.addomain.com

Confirm
Are you sure you want to perform this action?
Disabling Mailbox "bserebin@somefake.addomain.com" will remove the Exchange
properties from the Windows user object and mark the mailbox in the database
for removal.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help
(default is "Y"):y
-------------------------
2) Deletes disconnect mailbox(es) in Exchange Server 2007
Type the following 3 commands found on this page (skip option "Removing a single entry")
http://msmvps.com/blogs/andersonpatricio/archive/2007/10/08/removing-disconnected-mailboxes-in-exchange-server-2007.aspx

Also, In my testing of this issue, I made a full export (copy) of my DC & Exchange 2007 servers. And then I run them (NIC configured) via VMware Server to host only mode. This makes the servers visible to only the VMware Server. I then have a full shadow production environment. So, now I can test against the backup software. I originally detected the issue when I did a full mailbox restore to another AD account on my Exchange Server and compared sizes & item count and noticed the large discrepancy. I'm also in the process of getting Backup Exec 12.5 and ARCserve (r12.5) to test restores via those products as well. Curious to see how well they work.

So, far my Backup Exec 12.5.2231A x64 testing has been painful since the default install on Windows 2008 from a demo downloaded via their website does not show the Exchange Server databases. Same problem using 12.5.2231A x86 on 2003 occurred. I followed the support solution (document ID 315712) but it didn't resolve it. I'll continue troubleshooting later this week. I'll report my feedback on ARCserve later this week too.

-Ben

Windows 2008 R2 and Exchange 2007? Microsoft says no.

Hello All,

Anyone who has not deployed Exchange Server 2007 and wants to do it on Windows 2008 R2 (it'll be launched this month), the answer is it is NOT supported. Not a big deal, since I would just wait a bit longer and have the benefits of I suspect to be the hands down best Exchange version yet, Exchange 2010. If you are wondering, 2003 is still the king. Some articles discussing this R2 rejection: Windows IT Pro and the Microsoft Exchange Team postings.

Server OS Support
Exchange Server 2003 supports Window Server 2000, Windows Server 2003, & Windows Server 2003 R2. (3 OSes supported)

Exchange Server 2007 supports Windows Server 2003, Windows Server 2003 R2, & Windows Server 2008. (3 OSes supported)

Exchange Server 2010 supports Windows Server 2008 & Windows Server 2008 R2. (2 OSes supported currently)

Microsoft decided against QA and supporting Exchange 2007 on Windows Server 2008, which is understandable since supporting 4 OSes is a time consuming process and 2010 is getting launch within the month. So, wait a little longer, and try and deploy Exchange Server 2010 once your eco-system is supported (e.g. BES, GFI, AppAssure, Backup Exec, etc).

-Ben

Learning about the services Exchange 2007 installs?

Ehlo All,

Quick question. The "Microsoft Exchange File Distribution" Service gets installed on which Exchange Server 2007 roles? The answer is Client Access and Unified Messaging server roles. This Microsoft TechNet article that explains all the services installed by Exchange setup for 2007 which can help in case of troubleshooting. If an issue occurs, you'll know where to look first which can be helpful since there are 21 services that Exchange installs depending on the server role. That's a lot services. This article can help you understand them.

-Ben

Article - Domain Controllers are critical to Exchange, Know Which DC is Used for Communication

Hello All,

A working Active Directory foundation is required for Exchange Server, so have you ever wondered which Domain Controller (DC) was being used by your Exchange Server. Now you can. Read on if you're interested.

I recently read an article about how Exchange Server knows what Active Directory servers are available. For Exchange 2000, 2003, and 2007, the answer lies with an Exchange process called "Directory Services Access", aka DSAccess. It polls all DC every 15 minutes and makes optimizations based on the results. You can read the summary article here (via SearchExchange.org) or a full detailed technical article here (via MSExchange.org) or another one from SearchExchage.org here. Start with the summary, and then the technical articles.

Tidbit - Under 2000 and 2003, DSAccess ran under the Exchange service called System Attendant. Exchange 2007 changed that, and now it runs under a separate Windows service running on your Exchange Servers called "Microsoft Exchange Active Directory Topology Service".

Until next time... may your email flow well.
-Ben

Virtualization and Exchange - It's Microsoft Supported and Works Well!

Ehlo All,

Sadly, there is a LOT of marketing non-truths (aka lies) being spread about Virtualization and Exchange Server (e.g. it doesn't work, Microsoft won't support it, bad performance, etc). I was so tired of hearing this, I finally decided to give a presentation at the NY Exchange Server User Group Meeting this past week that discussed an intro to virtualization and Exchange and exactly what is supported (Exchange 2003 & 2007 is VM supported, VMware ESX and Exchange 2007 SP1 is supported, and much more). My Exchange environment is fully Microsoft PSS (Professional Support Services) supported since it's Exchange 2007 SP2, Windows 2008, and VMware ESX 3.5 U2. Amazing, I know. I've included the topics presented in the evening, and feel free to spend sometime reading through PDF. I've included referenced footnotes as well.

Presentation Topics
- intro to virtualization
- pro's / con's of virtualization
- what servers work with virtualization
- Microsoft's Exchange Support Policy explained in detail
- different Exchange version support
- VM performance comparison to old school hardware deployments (and yes, it's old school)
- why virtualization is more important than performance
- what Exchange Server really needs

Review the September 8, 2009 NYExUG presentation by Ben Serebin, and post any questions you have on the blog and I'll happily answer them.

.
QUIT

An example of what not to do. Article published on setting up an Edge 2007 Server without EdgeSync.

Ehlo All,

Yes, you read the title correctly, an Edge 2007 Server without EdgeSync. Why would anyone ever deploy an Edge 2007 Server without EdgeSync, the synchronization functionality? You got me. But, Neil Hobson published on msexchange.org an article how to set it up. You can check it out here, but I do not recommend this configuration, and would be very curious to know why the quoted organization couldn't use it. He really should have elaborated why it was done, since this is NOT recommended.

For some background, check out this MSDN blog which has the pro's/con's for EdgeSync. Check it out here. To summarize you lose gateway based recipient filtering, safelist block/accept, config and admin is easier on setup, and traffic is encrypted by default. If you do without EdgeSync, do not spend the money on an Exchange license. Save it for another mail gateway product. Sometimes, msexchange.org publishes article of questionable value, this is one of them until they elaborate the purpose of runnings without EdgeSync.

.
QUIT

Great Time Saving Tip for Installing Exchange 2007 SP1 on Windows 2008

Ehlo All,

Want to safely speed up install times and make less mistakes (e.g. forgetting to install a prerequisite, role, or feature) on your installs of Exchange 2007 SP1 on Windows 2008. The smart creators of Exchange (aka Microsoft Exchange Server Team) via their blog announced and released a GREAT tool. Normally, you have to separately install numerous prerequisites, roles, and features which can take a while. So, they released an all-in-one way to do this via XML files for each role option. Here are the 6 options with 2 clarified. Or you can manually install the needed software via this Microsoft TechNet article.

Exchange-Base = installs Server Manager, PowerShell, and RSAT-ADDS
Exchange-CAS
Exchange-ClusMBX = clustered mailbox
Exchange-Edge
Exchange-MBX
Exchange-UM

.
QUIT

Exchange 2007 SP2 is coming soon (before Oct)

Ehlo All,

According to the official Microsoft Exchange Server Team Blog posting, Exchange 2007 SP2 is arriving before October 2009. Technically, they said Q3 2009, but easier to say before October, than July, August, or September. 3 biggest changes will be:

1) ability to backup with an OS provided tool within Windows 2008 (right now you can't w/o 3rd party)
2) auditing Exchange events (e.g. configuration changes, etc)
3) interoperability with Exchange 2010 (I plan to jump as quickly as I can to 2010 once my 2 critical Exchange add-on apps work)

.
QUIT

Exchange 2007 SP1 Update Rollup 9 Released

Ehlo All,

MS Exchange Server Team Blog announced on 7/17/09 that the Exchange 2007 SP1 update rollup 9 was released. You can read more about it here. I've downloaded it and loaded it on my own production server and no issues to report. I did notice when I installed it, I had skipped rollup 8, and even on reboot, Windows Updates wanted rollup 8 installed. Which is wrong, since rollups are cumulative. I hide rollup 8. Oh well, don't make a mistake and install a downgraded version.

.
QUIT

Field Notes - Exchange 2007 Edge Troubleshooting

Ehlo All,

So, I figured I would share more of my daily work from the field. That means more troubleshooting, upgrade issues, successes, and all around adventure. A client of REEF Solutions (using REEF Solutions' hosted clustered spam, virus, and DoS protection solution which has handled about 550,000+ messages a day over the past month) with their own in-house IT staff was working on an Exchange 2007 migration from 2003 that had email flow problems and almost 9,000 valid messages were stuck in the Edge queue. I was called in to assist after the client's IT was on the phone with Microsoft Professional Support Services for over 3 hrs and there was no solution and they were considering reinstalling Edge. Client was restless, since email downtime was suppose to end after 7 days or so, but it didn't.

Background
The client had migrated to 2 new servers, an Exchange 2007 Mailbox/CAS/Hub & Edge both on Windows 2008 Server 64 bit. During the upgrade they implemented an Exchange 2007 Edge Server. This was to replace an existing non-Exchange smtp gateway server. They previous had a single Exchange 2003 environment. After the Edge implementation, email would flow from the Mailbox Server to Edge to Internet, but not the reverse. Client IT had tested and telneting between the Edge and Mailbox worked, and vice versa, but email would not flow. Edge was in a DMZ. MS PSS had done a lot of things, but the email was still not flowing. During the the entire week long downtime, REEF Solutions had queued up email off-site (9k of valid non-spam messages) for the client.

Troubleshooting and Solution
1) Running the built-in Exchange troubleshooting analyzer reported errors on both servers. Running it on Mailbox reported not seeing Edge, and vice versa. This was because the DMZ didn't have those ports open for RPC and other ports. Not a big deal, but makes troubleshooting harder.
2) pinging the Mailbox and Edge servers NETBIOS name worked from both servers.
3) from Mailbox and Edge, telneting via port 25 to generate "homemade" email both ways was successful.
4) on Mailbox ran "Test-EdgeSynchronization" and it passed with flying colors.
5) on Mailbox ran "Test-EdgeSynchronization -VerifyRecipient bgates@yourdomain.com" and it was successful. Obviously, pick an email in your domain. This is testing the AD Application Mode (ADAM) replication [1 way from AD -> Edge] for storage of configuration and recipient information. This is because Edge is a non-domain computer and doesn't have access to AD like a normal domain based server.
6) checked the hosts files on both servers. And added due to a known IPv6 issue, the NETBIOS and FQDN of each server and the other server in their hosts file. So, if your mailbox server was called "mailboxsrv", in the hosts file would say "192.168.1.2 mailboxsrv" and then line 2 would be "192.168.1.2 mailboxsrv.corp.yourdomain.com" and comment out the ::1 localhost entry to "#::1 localhost".
7) on Mailbox server in EMC - Organization Configuration - Hub Transport - Send Connectors - EdgeSync - Inbound to Mailbox Server - Route mail through the following smart hosts: {your mailbox server IP})
8) on Edge, saw an Event log error for a non-valid SSL cert, so on the Mailbox and Edge server, if I recall, under EMC - Hub Transport - Send Connectors - Network - unchecked "Enable Domain Security (Mutual-Auth TLS)". This is an excellent article by MVP Elan Shudnow that discusses transport layer security between Edge and Transport.
9) on Mailbox, ran "Start-EdgeSynchronization" and the configuration changes I made replicated to the Edge server.
10) since all inbound port 25 is restricted from REEF's clustered email filtering solution, I generated email from their and tested inbound flow from cluster - edge - mailbox, and it was successful. And then I tested outbound email and it worked. Then the 9k message queue quickly reduced down to 0.

FYI: if you need to reinstall the Edge or Transport Server and have messages in your queue, you can backup it up, re-install Edge or Transport services, and then restore the database. Edge queue database is ESE based, like Exchange. An excellent article by explaining the backup and restore process by Joshua Raymond is here.

Problem solved.
.
QUIT

Exchange 2007's Outlook Anywhere Overview Article

Ehlo All,

Here is a good overview and refresher article about Exchange 2007's feature of Outlook Anywhere. I added my 2 cents via the comments about not needing expensive SAN/UCC SSL certificates for auto-discovery functionality. There's a lot of mis-information about this on the internet. Which reminds me, an upcoming blog posting will discuss Exchange 2007's auto-discovery feature and how it's not the answer to the holy grail which many make it out to be.

-Ben

In Q3 2009 - Exchange 2007 SP2 will bring backup to Windows 2008

Ehlo All,

For Windows 2008 Server admins running Exchange 2007, Microsoft never provided a backup utility similar to ntbackup.exe for Windows 2003 Exchange admins. FYI: Exchange 2007 on Windows 2003 can be backed up, just not on Windows 2008. BK Winstead of Windows IT Pro posted about Microsoft plans in Q3 2009 to release an Exchange 2007 SP2 update that will allow backups. Microsoft's Exchange Team blogged about this and additional features (e.g. Exchange change auditing, Public Folder quotas, etc) found in SP2.

-Ben