NYExUG 3/12 Meeting Follow-up - Troubleshooting Tips

We had an excellent Exchange User Group meeting this past Tuesday about Troubleshooting Tips. Even though we ran later than we had in a while, we could not cover everything. I've highlighted comments and feedback received during the meeting and after. Thank you to everyone for your feedback. This is what makes us a community.

NYExUG Exchange Troubleshooting and Tips Presentation

Correction: the Exchange Server User Monitor (ExMon) tool does not list ActiveSync versions, but other AS performance stats.

Dirt cheap $60 UCC/SAN (5 names) certificate I recommend is https://certificatesforexchange.com/ which is backended via GoDaddy.

RDP Manager I use is called RoyalTS ($35) which has a lot of flexibility, functionality, stability, and works on XP and above. http://www.code4ward.net/main/

Website tool highlights from my presentation

• Microsoft owned Remotely Testing Outlook (RPC), ActiveSync, Web Services, Email (smtp) and even Office 365 https://www.testexchangeconnectivity.com
• Review DNS Records for Auto Discovery - http://centralops.net
• Review DNS Records for basic MX/A Records – http://tools.appriver.com
• Checking for RBLs – http://www.mxtoolbox.com
• [application] Stress testing application for generating email (Server Traffic Test) - http://www.icewarp.com/downloads/tools/ No reboot and stable application.

Attendees Feedback (thank you)

• The website designed to check your TLS configuration. http://www.checktls.com
• "Out of control transaction logs. On a number of occasions we have had the transaction logs grow significantly (one every second or so). This can be caused by a rogue application sending emails via your HT or a bad out of  office configuration. We have experienced both.  The last one was a user's out of office settings. They had used the rules section in the out of office. Viewing the transaction log showed the user forwarding the same email every second or so. Turning of the out of office resolved the issue."
• Exchange environment summary report based (# of Exchange Servers & mailboxes, DB sizes, DAG status, etc.  http://www.stevieg.org/2011/06/exchange-environment-report
• Post on troubleshooting ActiveSync issues from the Exchange Team Blog.  http://blogs.technet.com/b/exchange/archive/2012/01/31/a-script-to-troubleshoot-issues-with-exchange-activesync.aspx
• Tony Redmond wrote an excellent article about ActiveSync not working as a result of 2010 user being a member of the priv’ group on his blog site. If he’s truly 2003 user, then the only thing I can think of is setting up similar profile on a different iphone. If that stil doesn’t work then it’s the account & he may need to look into deleting the EAS association via adsiedit & redo the EAS profile on the device. An Exchange MVP (Michael B Smith) has commented several times in the past on the MSExchange forum re: the ills people have been experiencing with iphones – in our own environment we’ve seen disappearing emails/corrupted calendars/and all sorts of wackiness. I can forward forum posts if people are interested but my desktop team has been beaten into submission about what to do/not to do when it comes to syncing EAS devices with Exchange.  http://thoughtsofanidlemind.wordpress.com/2010/10/08/ex2010-insufficient-access/
• Free Microsoft RDP manager mentioned was Remote Desktop Connection Manager (I didn't like the last version, so I know nothing about this one -Ben)  http://www.microsoft.com/download/en/details.aspx?id=21101

Any comments, post them or email me. Thanks.
-Ben

HP SAN vs "Dividing by Zero". 0-1. SAN hardware crashes.

Hello All,

Never, ever, ever, ever divide by zero otherwise very bad things can happen. A client's less than 1 year old HP SAN environment crashed last week thanks to a SAN firmware bug (dividing by zero) which caused a kernel panic. And of course this only happened when uptime hit 208.5 days. Hence, HP calls it the "208 Day" bug. I call it poor software development.

Absolutely ridicuous. I have never really liked the HP SAN hardware which was brought over from Lefthand Networks. For mission critical environments, I'm a believer in proper SAN hardware such as the Dell EqualLogic line. REEF has deployed HP and Dell SAN hardware, and without a doubt, the Dell SAN hardware is better. Even the HP software has problems with Hyper-V and running under Windows Core. Disappointing. The Dell SAN hardware is better built and cheaper, and this is why REEF Solutions' is a Dell Premier Partner.

HP bug which causes reboots after 208.5 days

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=4118659&prodTypeId=12169&objectID=c03090774

-Ben

P.S. The IT Director of the client who experienced the problem at least has a good sense of humor. This "dividing by zero" programming mistake is clearly a common issue. Enjoy the image below.

Preparing for D Day for Me...

Hello All,

I've been doing a lot of "house keeping" lately before "D" Day. "D" day being delivery day. My wife is due with our 3rd child. While my wife is nesting, I'm doing the equivalent for an IT person. We had a false alarm when we thought it was happening, so now I feel like I'm living on borrowed time and have all this "extra" time. In the last week I've done the following:

• getting our REEF NY & TX SonicWall firewalls updated to the latest code (VPN tunnel speed to my TX off-site environment doubled in speed)
• rolling out a SonicWall based network bandwidth and auditing solution (we currently monitor it using another solution) for REEF's networks.
• NY based on-site servers replication operating system re-installed (for REEF environment, the on-site server is 2008 R2 based. The replication data was not touched, since it is iSCSI based.)
• NY based on-site servers replication software upgraded (to improve performance, noticeable positive difference between AppAssure Replay 4.6.1,31257 and 4.7.2.40512 [found a bug in the replication UI and alerted AppAssure about it and received a support response in 5 minutes. Impressive. I wish all AppAssure support techs responded so quickly]). For REEF environment. Enjoy the image below.
• TX based off-site servers replication upgraded (same AppAssure Replay versions upgraded)
• rolled out my digital photo album solution based on a BlackBerry PlayBook. Considered an iPad, but security, performance, and low cost of the 64GB PlayBook ($300) made it the better solution.
• NY on-site server operating system re-installed (for clients environment, the server environment is Windows 2003 x86 based. Currently using a stable release of Ahsay. Planning to upgrade to latest stable version shortly.
• working on deploying a new wireless SonicWall based solution so guests at home will be on a separate VLAN based network. In preparation for all those home visitors.

Notice the replication speed showing “10.22MBit/sec”. It should be “Mb”, not MB. A capital “B” is BYTES, while a lower case “b” is bits. This is on the latest version 4.7.2.40512. Dev has been alerted per support's response.

Back to spending time with the existing kids and wife,
-Ben

Recommended Exchange Deployments are Multi-role Now

At the last free "Tech Ed style" event in NYC held at the Microsoft Offices, we had Ross Smith IV present on

Exchange 2010 High Availability/Database Availability Groups. If you don't know Ross, he's a VERY senior Microsoft employee who wrote the Exchange Storage and Server Role Calculator. He knows Exchange, period. End of story. So, when he said that everyone should deploy Exchange 2010 in a multi-role configuration to improve performance and not break apart the roles, you need to take his recommendation seriously. This was the 1st time I had heard this. I had a long conversation with him about this in NYC, and he explained that for performance and the ability for leveraging failover capacity it is better to keep all the roles together. In theory, you could deploy less. Since if you were going to deploy 2 CAS and 2 Mailbox, you could in theory just deploy 3 consolidated roles. Well, Microsoft TechNet finally released some guidance on this. That only took 6 months. Don't forget to use a hardware/VM load balancer when deploy your multi-role Exchange Servers.

TechNet article title: Understanding Multiple Server Role Configurations in Capacity Planning
http://technet.microsoft.com/en-us/library/dd298121.aspx

-Ben

Hackers & Malware - Dangers Everywhere - Not Just Scare Tactics

Hackers and malware were busy this week at clients of REEF Solutions.

Good news first, we identified a serious denial of service vulnerability during a network infrastructure review for a financial firm. So bad, a simple command would reboot a core network device. That was a highlight of the review. And this was not even a security audit, I am sure it'll be only worse.

Bad news now

• A client's system was infected with TDSS, one of the nasty [known] malware products  (think encryption, p2p command and control, http/https tunneling, malware competition removal, and MBR infection). Malware vendors even offer a Firefox plug-in to allow paying customers to surf via infected machines to provide anonymous cover. To summarize, TDSS is extremely dangerous. More technical details here. As of now, the only tool that can remove it or most of it is Kaspersky. Ideally, we should have wiped the system, but the client would not permit this.
• A hacker attacked via RDP and compromised a system. We detected the compromise and took immediate action to isolate and remediate the attack. If we had not caught it faster, this could have been a serious issue. The key is to have an Intrusion Detection System in place, even if it's just a firewall based solution. You need to be aware of what is happening on your network. I recommend additional policies such as resetting all administrator passwords, not permitting  "administrator" usernames, requiring 15+ characters passwords, email alerting w/3rd party logging tool on administrator level logins, and layered security products (firewall based scanning, servers based, proxy based, DNS scanning, etc).

Sadly, malware and attackers are not sitting idly by. There are some real threats out there. Stay safe...

-Ben

Sprint 4G appears to be hacked at DEFCON

Hello All,

It appears there could have been a successful man in the middle attack (MiTM) on Sprint 4G at DEFCON. Numerous Android devices were attacked during this period. I hope Android users didn’t "upgrade" or re-enter "their passwords" during the multiple day event. Dangerous, but there is a solution that the carriers and handheld manufacturers could implement to protect against this (see solution below).

News from:
http://www.extremetech.com/computing/92370-4g-and-cdma-reportedly-hacked-at-def-con
http://seclists.org/fulldisclosure/2011/Aug/76

A friend and I were discussing this and this what his response was:

 

I'm betting that they did it one of two ways on 802.16/ClearWire/Sprint4G.

They gained physical access to the local tower, and did MiTM from the tower.  WiMax is Mobile IP from the tower to the provider edge.  It would be a lot easier to do MiTM on at a Mobile IP tower, rather than a LTE network.

The other way is, someone in the group worked for / had access to enough of the parts to make a fake WiMax base station.  Based on the signal strength reports, and slow speeds, this is what I bet they did.  WiMax uses either EAP-TLS or EAP-TTLS.  I'm guessing either they had access to the certs to appear valid, or the end devices did not properly implement EAP-TLS and EAP-TTLS, and just accepted any certificate

A pretty cool hack if they did.  Hopefully it can be shown, and the 4G devices can implement proper security.

Update to above.....

After further research, it definitely looks like the latter method (fake WiMax Base station).  They talk about signal strength and upload speeds.  Those wouldn't be affected by the first method (getting into a valid tower).

Unfortunately the supplicate (client) is probably just configured to accept any client.  

For example in the Cradlepoint, you just specify the carrier / realm, but that's it.  No username, etc.   No certs.  No other options.

Another example, the Sprint SmartView client, it doesn't have the ability to specify anywhere anything related to authentication and certificates.

One would need a fake WiMax base station (that can do 2.6ghz) in order to test to see if the supplicate takes any certificate.

Side note: certain applications offer the ability to register against a specific TLS certificate serial number such as Apple Mail.  I hope other devices/applications allow this in the future.

The MiTM Attack Solution

If phones only accepted carrier based certs and had a proper implementation of EAP-TLS or EAP-TTLS this would protect against this sort of attack. 

Sadly, the solution is going to take a bit of work and time. So, don't automatically "trust" voice over data. Protect your data and it can be more secure than your data.

 

-Ben

New Microsoft ActiveSync Compatibilty Program fails on helpfulness

Ehlo All,

Curious about knowing...

• what ActiveSync functionality is available with which version of Exchange?
• which mobile devices have higher Active Functionality?

Well, this new Microsoft ActiveSync compatibility program for OEMs won't help, but read on about it.
Microsoft recently announced the Exchange ActiveSync (EAS) Logo Program for OEMs (think HTC, Google, Apple, Motorola, Microsoft, etc) which should have been used to identify and bring clarify to the level of EAS support a mobile device included. Sadly, it does not do this since there is 1 level for EAS Logo Program and it includes very basic functionality. So, if the device says "ActiveSync", this is pretty much equal to the EAS Logo Program. BK (author of post below) had it right that there should be multiple levels. For example, "basic", "enhanced", "ultimate". So, if an ActiveSync device said "Ultimate", you would know it supports every feature under the sun for EAS against Exchange 2010. Oh well, maybe version 2 of the program will get this improvement.

Windows IT Pro Post about new EAS Logo Program by BK Winstead

-Ben