Recommended Exchange Deployments are Multi-role Now

At the last free "Tech Ed style" event in NYC held at the Microsoft Offices, we had Ross Smith IV present on

Exchange 2010 High Availability/Database Availability Groups. If you don't know Ross, he's a VERY senior Microsoft employee who wrote the Exchange Storage and Server Role Calculator. He knows Exchange, period. End of story. So, when he said that everyone should deploy Exchange 2010 in a multi-role configuration to improve performance and not break apart the roles, you need to take his recommendation seriously. This was the 1st time I had heard this. I had a long conversation with him about this in NYC, and he explained that for performance and the ability for leveraging failover capacity it is better to keep all the roles together. In theory, you could deploy less. Since if you were going to deploy 2 CAS and 2 Mailbox, you could in theory just deploy 3 consolidated roles. Well, Microsoft TechNet finally released some guidance on this. That only took 6 months. Don't forget to use a hardware/VM load balancer when deploy your multi-role Exchange Servers.

TechNet article title: Understanding Multiple Server Role Configurations in Capacity Planning
http://technet.microsoft.com/en-us/library/dd298121.aspx

-Ben

Hackers & Malware - Dangers Everywhere - Not Just Scare Tactics

Hackers and malware were busy this week at clients of REEF Solutions.

Good news first, we identified a serious denial of service vulnerability during a network infrastructure review for a financial firm. So bad, a simple command would reboot a core network device. That was a highlight of the review. And this was not even a security audit, I am sure it'll be only worse.

Bad news now

• A client's system was infected with TDSS, one of the nasty [known] malware products  (think encryption, p2p command and control, http/https tunneling, malware competition removal, and MBR infection). Malware vendors even offer a Firefox plug-in to allow paying customers to surf via infected machines to provide anonymous cover. To summarize, TDSS is extremely dangerous. More technical details here. As of now, the only tool that can remove it or most of it is Kaspersky. Ideally, we should have wiped the system, but the client would not permit this.
• A hacker attacked via RDP and compromised a system. We detected the compromise and took immediate action to isolate and remediate the attack. If we had not caught it faster, this could have been a serious issue. The key is to have an Intrusion Detection System in place, even if it's just a firewall based solution. You need to be aware of what is happening on your network. I recommend additional policies such as resetting all administrator passwords, not permitting  "administrator" usernames, requiring 15+ characters passwords, email alerting w/3rd party logging tool on administrator level logins, and layered security products (firewall based scanning, servers based, proxy based, DNS scanning, etc).

Sadly, malware and attackers are not sitting idly by. There are some real threats out there. Stay safe...

-Ben

Sprint 4G appears to be hacked at DEFCON

Hello All,

It appears there could have been a successful man in the middle attack (MiTM) on Sprint 4G at DEFCON. Numerous Android devices were attacked during this period. I hope Android users didn’t "upgrade" or re-enter "their passwords" during the multiple day event. Dangerous, but there is a solution that the carriers and handheld manufacturers could implement to protect against this (see solution below).

News from:
http://www.extremetech.com/computing/92370-4g-and-cdma-reportedly-hacked-at-def-con
http://seclists.org/fulldisclosure/2011/Aug/76

A friend and I were discussing this and this what his response was:

 

I'm betting that they did it one of two ways on 802.16/ClearWire/Sprint4G.

They gained physical access to the local tower, and did MiTM from the tower.  WiMax is Mobile IP from the tower to the provider edge.  It would be a lot easier to do MiTM on at a Mobile IP tower, rather than a LTE network.

The other way is, someone in the group worked for / had access to enough of the parts to make a fake WiMax base station.  Based on the signal strength reports, and slow speeds, this is what I bet they did.  WiMax uses either EAP-TLS or EAP-TTLS.  I'm guessing either they had access to the certs to appear valid, or the end devices did not properly implement EAP-TLS and EAP-TTLS, and just accepted any certificate

A pretty cool hack if they did.  Hopefully it can be shown, and the 4G devices can implement proper security.

Update to above.....

After further research, it definitely looks like the latter method (fake WiMax Base station).  They talk about signal strength and upload speeds.  Those wouldn't be affected by the first method (getting into a valid tower).

Unfortunately the supplicate (client) is probably just configured to accept any client.  

For example in the Cradlepoint, you just specify the carrier / realm, but that's it.  No username, etc.   No certs.  No other options.

Another example, the Sprint SmartView client, it doesn't have the ability to specify anywhere anything related to authentication and certificates.

One would need a fake WiMax base station (that can do 2.6ghz) in order to test to see if the supplicate takes any certificate.

Side note: certain applications offer the ability to register against a specific TLS certificate serial number such as Apple Mail.  I hope other devices/applications allow this in the future.

The MiTM Attack Solution

If phones only accepted carrier based certs and had a proper implementation of EAP-TLS or EAP-TTLS this would protect against this sort of attack. 

Sadly, the solution is going to take a bit of work and time. So, don't automatically "trust" voice over data. Protect your data and it can be more secure than your data.

 

-Ben

New Microsoft ActiveSync Compatibilty Program fails on helpfulness

Ehlo All,

Curious about knowing...

• what ActiveSync functionality is available with which version of Exchange?
• which mobile devices have higher Active Functionality?

Well, this new Microsoft ActiveSync compatibility program for OEMs won't help, but read on about it.
Microsoft recently announced the Exchange ActiveSync (EAS) Logo Program for OEMs (think HTC, Google, Apple, Motorola, Microsoft, etc) which should have been used to identify and bring clarify to the level of EAS support a mobile device included. Sadly, it does not do this since there is 1 level for EAS Logo Program and it includes very basic functionality. So, if the device says "ActiveSync", this is pretty much equal to the EAS Logo Program. BK (author of post below) had it right that there should be multiple levels. For example, "basic", "enhanced", "ultimate". So, if an ActiveSync device said "Ultimate", you would know it supports every feature under the sun for EAS against Exchange 2010. Oh well, maybe version 2 of the program will get this improvement.

Windows IT Pro Post about new EAS Logo Program by BK Winstead

-Ben

Message Dehydration isn't a good thing for Exchange!

Ehlo All,


Quote of the week: "limit is 94% before message dehydration occurs."

A normally very stable client's Exchange Server 2007 stopped processing inbound emails and this was the issue above. Client reported all internal email was working though. I logged into their Exchange Server and reviewed the normal issues inside the Exchange Management Console and nothing jumped out (e.g. databases mounted, 3GB free space on C, 700GB+ free space on database partition (D), no quota limits, receive connectors present/enabled, etc). Strange. I decided to check the email filtering solution in front of the Exchange Server. Since I always recommend clients use my company's email filtering service (SpamCop - I'll discuss this is another post) since it allows me to quickly troubleshoot issues and provide very secure email service. Reviewing SpamCop outbound queue to the client's Exchange Server illuminated the error:


Deferred: : host 55.55.55.55 said: 452 4.3.1 Insufficient system resources (in reply to...

More details about this error can be found on this posting. Checking the client's Application Event Log for this error, I found it "The Microsoft Exchange Transport service is rejecting message submissions because the available disk space has dropped below the configured threshold.". Which was surprising since the C drive had a decent amount of space available (3GB+). What changed? Good old SBS's Windows Server Update Services had downloaded every update under the sun and the storage space threshold was passed and triggered back pressure. I uninstalled WSUS and that eliminated 10GB and back pressure was eased and email flow started. I plan to remove some other functionality as well and plan to do a scheduled reboot as well to free up more space. To sum up, SBS causes more problems than it's worth for Exchange deployments. I prefer clean Exchange installs over Exchange SBS installs.

-Ben

Fixing ActiveSync on an Exchange 2007 Server

Ehlo All,


So, a client's iPhone with ActiveSync stopped working with their Exchange Server 2007. If rebooting and deleting and setting it back up doesn't fix it, confirm your server's AS is working. An good way to do this is to test with Microsoft Exchange test website found here. So after some investigating, it turned up that the IIS Virtual Directory for "ActiveSync" wasn't responding correctly. One can test this by "https://myservername/Microsoft-Server-ActiveSync" and make sure it prompts for a username and pwd. My client's server didn't do that. It reported 501 service unavailable error. I therefore deleted the ActiveSync virtual directory and re-created it and it fixed the issue. A great blog posting to explain this can be found here. To clarify, the "XXXXX" in his example for normal Exchange installs is "Default Web Site".


Handy URLs:
Detailed instructions on deleting and recreating virtual directories for Exchange 2007
http://my.opera.com/RavenOverride/blog/2009/06/17/how-to-recreate-all-virtual-directories-for-exchange-2007

Microsoft's Test Website for Exchange/ActiveSync/Outlook/SMTP
https://www.testexchangeconnectivity.com

-Ben

Techstravaganza Event with amazing Exchange Speakers (free)

Ehlo All,

           My user group (NYExUG) and 4 other UGs are organizing an amazing event that is a free Tech-Ed type event which has 5 tracks (Exchange, SharePoint, PowerShell, Server/Office, Ask the Experts) and 5 sessions in each.

           For those who attended this past week’s NYExUG meeting, I mentioned that at the upcoming Techstravaganza event we might have an Exchange Superstar presenting. Well, that has happened! Ross Smith (Microsoft) will be presenting (thanks to Bob Hunt). You do NOT want to miss this event! Just to give you an idea, this guy frequently presents at major conferences and even runs a session called “Stump the Experts”. With “Stump the Experts”, if you’re able to ask an Exchange question that Ross can’t answer, you win an Xbox. He’s never given away an Xbox. This guy knows Exchange. Period. Learn from a superstar. He’ll be presenting on Exchange 2010 High Availability / DAGs.

Techstravaganza

Held at Microsoft’s NY office

Friday, March 18 (8am-5pm).

This event will have 5 tracks of Exchange, PowerShell, SharePoint, Windows/Client Server, and Ask The Experts.

Register asap since spots are limited

http://techstravanganzanyc.eventbrite.com/

Main Website

http://www.techstravaganza.com

Hope to see you at the event...

-Ben

1st Known Spamming from the "Cloud"

Ehlo All,

This is my 1st confirmed spam (mid 2010) I have ever seen come from the "Cloud". The "winner" of this honor goes to Amazon. Congratulations (sarcasm). According to my latest research Amazon still does not allow PTR (rDNS) records which is typically required/strongly recommend to avoid outbound email being labeled as spam. How do the many mail servers running on Amazon's Cloud handle it? They relay their email from Amazon's environment onto another host (e.g. authsmtp, google "smtp relay service", etc) and then the other service forwards.

Background on me
I see a LOT of spam since my firm handles filtering for most of our clients via our geographically diverse clustered anti-spam/virus/DoS solution. Our clients on an average day get a total of about 300-400k connections a day (spam/real). This provides me a lot of experience/exposure with spam filtering. If you are wondering why we run our own systems it is because it offers more flexibility, significantly lower latency for email messages (aka delay), and faster response than the big guys.

View from my Spam Filtering Solution which Quarantines Suspect Email Like This.

Spam Header Details
Received: from mm-notify-out-209-61.amazon.com (mm-notify-out-209-61.amazon.com [72.21.209.61])


by mail.rbkgroup.com with ESMTP id 67cz6639988tcu.19.20100625083501;

Fri, 25 Jun 2010 11:35:01 +0200

Date: Fri, 25 Jun 2010 11:35:01 +0200

X-Barracuda-BBL-IP: 72.21.209.61

X-Barracuda-RBL-IP: 72.21.209.61

From: "Buy.com"

Reply-To: Nobody

To:

Message-ID: <02630844.67618272250016768122.JavaMail.em-build@na-mm-relay.amazon.com>

Subject: Thanks for your order!

X-AMAZON-CLIENT-HOST: digital-docs-dope-5002.iad5.amazon.com

X-ASG-Orig-Subj: Thanks for your order!

Bounces-to: 20100625083501q4b3332ggg949lm9p0629fm7g208en6r@bounces.amazon.com

X-AMAZON-CLIENT-SENDTIME: Fri, 25 Jun 2010 11:35:01 +0200

X-AMAZON-MAIL-RELAY-TYPE: notification

X-AMAZON-RTE-VERSION: 2.0

MIME-Version: 1.0

Content-Type: text/html; charset=UTF-8

Content-Transfer-Encoding: 7bit


--------------------------------------------

Any questions, let me know.
-Ben

DoS of DNS by an Exchange Focused Backup Software (AppAssure Replay)

 Ehlo All,

Imagine to my surprise that my favorite Exchange & Windows backup solution (AppAssure Replay 4.5.1.27532) was attempting to cause a denial of service (DoS). This version has a major problem with it's use of DNS lookups within the problem. Within 3 days, one Replay Server had performed over 450,000 queries of the hostname I used for Replay replication. This is almost 100 queries every minute 24 hours a day. That's what the product is doing. This is a serious issue. I've alerted the vendor, so I'm sure a fix will be included in a future release. In the mean-time, see below for the work-around until that happens.

The Issue
Inside AppAssure's Replay for replication, you specify a "Replication Target Host Name". This can be a hostname or IP address. See below for setting within Replay.

 This "Select Replication Target" configuration is per protected server (e.g. your Exchange Server, etc). I normally use a hostname for these types of settings since I'm a big fan of using DNS instead of IPs when possible (saves time when changing IPs & saves brain memory space for Exchange Server things). So, when you add your Replication Target hostname, the Replication target and source perform lookups more often than the snap-shot period (x min/hrs). In reality, Replay should only perform a DNS lookup when a replication needs to occur and NOT almost a 100 per minute.

AppAssure's Replay abusing DNS lookups. View from my Firewall hostname query  logs.

The Workaround Until a Permanent Fix is Released by AppAssure
If you use a hostname within the Replication option, make sure you add the corresponding information inside the hosts file (c:\windows\System32\drivers\etc\hosts - format is IP address space and hostname - use notepad to open the "hosts" file) on the source AND target Replay Replication Server. This avoids the use of an external DNS query and the query is handled by the operating system. So, this speeds up the process of performing a lookup and reduces your hostname's name server load. Otherwise prepare for your DNS to be attacked by your Replay environment.

Sadly, this isn't the first time I have seen a product mis-use DNS, but it's one of the worst in recent memory.

-Ben