Message Dehydration isn't a good thing for Exchange!

Ehlo All,


Quote of the week: "limit is 94% before message dehydration occurs."

A normally very stable client's Exchange Server 2007 stopped processing inbound emails and this was the issue above. Client reported all internal email was working though. I logged into their Exchange Server and reviewed the normal issues inside the Exchange Management Console and nothing jumped out (e.g. databases mounted, 3GB free space on C, 700GB+ free space on database partition (D), no quota limits, receive connectors present/enabled, etc). Strange. I decided to check the email filtering solution in front of the Exchange Server. Since I always recommend clients use my company's email filtering service (SpamCop - I'll discuss this is another post) since it allows me to quickly troubleshoot issues and provide very secure email service. Reviewing SpamCop outbound queue to the client's Exchange Server illuminated the error:


Deferred: : host 55.55.55.55 said: 452 4.3.1 Insufficient system resources (in reply to...

More details about this error can be found on this posting. Checking the client's Application Event Log for this error, I found it "The Microsoft Exchange Transport service is rejecting message submissions because the available disk space has dropped below the configured threshold.". Which was surprising since the C drive had a decent amount of space available (3GB+). What changed? Good old SBS's Windows Server Update Services had downloaded every update under the sun and the storage space threshold was passed and triggered back pressure. I uninstalled WSUS and that eliminated 10GB and back pressure was eased and email flow started. I plan to remove some other functionality as well and plan to do a scheduled reboot as well to free up more space. To sum up, SBS causes more problems than it's worth for Exchange deployments. I prefer clean Exchange installs over Exchange SBS installs.

-Ben

Fixing ActiveSync on an Exchange 2007 Server

Ehlo All,


So, a client's iPhone with ActiveSync stopped working with their Exchange Server 2007. If rebooting and deleting and setting it back up doesn't fix it, confirm your server's AS is working. An good way to do this is to test with Microsoft Exchange test website found here. So after some investigating, it turned up that the IIS Virtual Directory for "ActiveSync" wasn't responding correctly. One can test this by "https://myservername/Microsoft-Server-ActiveSync" and make sure it prompts for a username and pwd. My client's server didn't do that. It reported 501 service unavailable error. I therefore deleted the ActiveSync virtual directory and re-created it and it fixed the issue. A great blog posting to explain this can be found here. To clarify, the "XXXXX" in his example for normal Exchange installs is "Default Web Site".


Handy URLs:
Detailed instructions on deleting and recreating virtual directories for Exchange 2007
http://my.opera.com/RavenOverride/blog/2009/06/17/how-to-recreate-all-virtual-directories-for-exchange-2007

Microsoft's Test Website for Exchange/ActiveSync/Outlook/SMTP
https://www.testexchangeconnectivity.com

-Ben

Techstravaganza Event with amazing Exchange Speakers (free)

Ehlo All,

           My user group (NYExUG) and 4 other UGs are organizing an amazing event that is a free Tech-Ed type event which has 5 tracks (Exchange, SharePoint, PowerShell, Server/Office, Ask the Experts) and 5 sessions in each.

           For those who attended this past week’s NYExUG meeting, I mentioned that at the upcoming Techstravaganza event we might have an Exchange Superstar presenting. Well, that has happened! Ross Smith (Microsoft) will be presenting (thanks to Bob Hunt). You do NOT want to miss this event! Just to give you an idea, this guy frequently presents at major conferences and even runs a session called “Stump the Experts”. With “Stump the Experts”, if you’re able to ask an Exchange question that Ross can’t answer, you win an Xbox. He’s never given away an Xbox. This guy knows Exchange. Period. Learn from a superstar. He’ll be presenting on Exchange 2010 High Availability / DAGs.

Techstravaganza

Held at Microsoft’s NY office

Friday, March 18 (8am-5pm).

This event will have 5 tracks of Exchange, PowerShell, SharePoint, Windows/Client Server, and Ask The Experts.

Register asap since spots are limited

http://techstravanganzanyc.eventbrite.com/

Main Website

http://www.techstravaganza.com

Hope to see you at the event...

-Ben

1st Known Spamming from the "Cloud"

Ehlo All,

This is my 1st confirmed spam (mid 2010) I have ever seen come from the "Cloud". The "winner" of this honor goes to Amazon. Congratulations (sarcasm). According to my latest research Amazon still does not allow PTR (rDNS) records which is typically required/strongly recommend to avoid outbound email being labeled as spam. How do the many mail servers running on Amazon's Cloud handle it? They relay their email from Amazon's environment onto another host (e.g. authsmtp, google "smtp relay service", etc) and then the other service forwards.

Background on me
I see a LOT of spam since my firm handles filtering for most of our clients via our geographically diverse clustered anti-spam/virus/DoS solution. Our clients on an average day get a total of about 300-400k connections a day (spam/real). This provides me a lot of experience/exposure with spam filtering. If you are wondering why we run our own systems it is because it offers more flexibility, significantly lower latency for email messages (aka delay), and faster response than the big guys.

View from my Spam Filtering Solution which Quarantines Suspect Email Like This.

Spam Header Details
Received: from mm-notify-out-209-61.amazon.com (mm-notify-out-209-61.amazon.com [72.21.209.61])


by mail.rbkgroup.com with ESMTP id 67cz6639988tcu.19.20100625083501;

Fri, 25 Jun 2010 11:35:01 +0200

Date: Fri, 25 Jun 2010 11:35:01 +0200

X-Barracuda-BBL-IP: 72.21.209.61

X-Barracuda-RBL-IP: 72.21.209.61

From: "Buy.com"

Reply-To: Nobody

To:

Message-ID: <02630844.67618272250016768122.JavaMail.em-build@na-mm-relay.amazon.com>

Subject: Thanks for your order!

X-AMAZON-CLIENT-HOST: digital-docs-dope-5002.iad5.amazon.com

X-ASG-Orig-Subj: Thanks for your order!

Bounces-to: 20100625083501q4b3332ggg949lm9p0629fm7g208en6r@bounces.amazon.com

X-AMAZON-CLIENT-SENDTIME: Fri, 25 Jun 2010 11:35:01 +0200

X-AMAZON-MAIL-RELAY-TYPE: notification

X-AMAZON-RTE-VERSION: 2.0

MIME-Version: 1.0

Content-Type: text/html; charset=UTF-8

Content-Transfer-Encoding: 7bit


--------------------------------------------

Any questions, let me know.
-Ben

DoS of DNS by an Exchange Focused Backup Software (AppAssure Replay)

 Ehlo All,

Imagine to my surprise that my favorite Exchange & Windows backup solution (AppAssure Replay 4.5.1.27532) was attempting to cause a denial of service (DoS). This version has a major problem with it's use of DNS lookups within the problem. Within 3 days, one Replay Server had performed over 450,000 queries of the hostname I used for Replay replication. This is almost 100 queries every minute 24 hours a day. That's what the product is doing. This is a serious issue. I've alerted the vendor, so I'm sure a fix will be included in a future release. In the mean-time, see below for the work-around until that happens.

The Issue
Inside AppAssure's Replay for replication, you specify a "Replication Target Host Name". This can be a hostname or IP address. See below for setting within Replay.

 This "Select Replication Target" configuration is per protected server (e.g. your Exchange Server, etc). I normally use a hostname for these types of settings since I'm a big fan of using DNS instead of IPs when possible (saves time when changing IPs & saves brain memory space for Exchange Server things). So, when you add your Replication Target hostname, the Replication target and source perform lookups more often than the snap-shot period (x min/hrs). In reality, Replay should only perform a DNS lookup when a replication needs to occur and NOT almost a 100 per minute.

AppAssure's Replay abusing DNS lookups. View from my Firewall hostname query  logs.

The Workaround Until a Permanent Fix is Released by AppAssure
If you use a hostname within the Replication option, make sure you add the corresponding information inside the hosts file (c:\windows\System32\drivers\etc\hosts - format is IP address space and hostname - use notepad to open the "hosts" file) on the source AND target Replay Replication Server. This avoids the use of an external DNS query and the query is handled by the operating system. So, this speeds up the process of performing a lookup and reduces your hostname's name server load. Otherwise prepare for your DNS to be attacked by your Replay environment.

Sadly, this isn't the first time I have seen a product mis-use DNS, but it's one of the worst in recent memory.

-Ben

Article - Constitution Finally is Defended over illegal email searches by US Government

 Hello All,

Glad to see someone cover this topic and that this "piece of paper" above means something. "This topic" is about defending our rights (4th Amendment which is unreasonable searches) from the US government illegal intrusions into email. US Dept of Justice illegally had an ISP copy someone's email and was slapped on the wrist, thank goodness for a Court of Appeals. See the Windows IT Pro article from Paul Robichaux discuss this. I hope this ISP get's sued as well now.

-Ben

Article - HA for Exchange 2010 is more costly than Exchange 2007

Ehlo All,

This is an excellent article especially if you're using or considering deploying Exchange 2007 in an HA configuration with SCR,. Also, the new HA mailbox solution in Exchange 2010 is called Database Availability Groups (aka DAGs). This requires the Enterprise Edition for Windows Server 2008, and we do not know if it requires Exchange 2010 Standard or Enterprise. Either way, the requirement of Server Enterprise Edition makes it costly.

Excellent website (Exchange-Genie.com) that explains with pictures the following HA solutions for Exchange 2007 & 2010, CCR, LCR, SCC, SCR, and DAG. In this case, a picture is worth a thousand words.

Information is power,
-Ben