HP SAN vs "Dividing by Zero". 0-1. SAN hardware crashes.

Hello All,

Never, ever, ever, ever divide by zero otherwise very bad things can happen. A client's less than 1 year old HP SAN environment crashed last week thanks to a SAN firmware bug (dividing by zero) which caused a kernel panic. And of course this only happened when uptime hit 208.5 days. Hence, HP calls it the "208 Day" bug. I call it poor software development.

Absolutely ridicuous. I have never really liked the HP SAN hardware which was brought over from Lefthand Networks. For mission critical environments, I'm a believer in proper SAN hardware such as the Dell EqualLogic line. REEF has deployed HP and Dell SAN hardware, and without a doubt, the Dell SAN hardware is better. Even the HP software has problems with Hyper-V and running under Windows Core. Disappointing. The Dell SAN hardware is better built and cheaper, and this is why REEF Solutions' is a Dell Premier Partner.

HP bug which causes reboots after 208.5 days

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=4118659&prodTypeId=12169&objectID=c03090774

-Ben

P.S. The IT Director of the client who experienced the problem at least has a good sense of humor. This "dividing by zero" programming mistake is clearly a common issue. Enjoy the image below.

Preparing for D Day for Me...

Hello All,

I've been doing a lot of "house keeping" lately before "D" Day. "D" day being delivery day. My wife is due with our 3rd child. While my wife is nesting, I'm doing the equivalent for an IT person. We had a false alarm when we thought it was happening, so now I feel like I'm living on borrowed time and have all this "extra" time. In the last week I've done the following:

• getting our REEF NY & TX SonicWall firewalls updated to the latest code (VPN tunnel speed to my TX off-site environment doubled in speed)
• rolling out a SonicWall based network bandwidth and auditing solution (we currently monitor it using another solution) for REEF's networks.
• NY based on-site servers replication operating system re-installed (for REEF environment, the on-site server is 2008 R2 based. The replication data was not touched, since it is iSCSI based.)
• NY based on-site servers replication software upgraded (to improve performance, noticeable positive difference between AppAssure Replay 4.6.1,31257 and 4.7.2.40512 [found a bug in the replication UI and alerted AppAssure about it and received a support response in 5 minutes. Impressive. I wish all AppAssure support techs responded so quickly]). For REEF environment. Enjoy the image below.
• TX based off-site servers replication upgraded (same AppAssure Replay versions upgraded)
• rolled out my digital photo album solution based on a BlackBerry PlayBook. Considered an iPad, but security, performance, and low cost of the 64GB PlayBook ($300) made it the better solution.
• NY on-site server operating system re-installed (for clients environment, the server environment is Windows 2003 x86 based. Currently using a stable release of Ahsay. Planning to upgrade to latest stable version shortly.
• working on deploying a new wireless SonicWall based solution so guests at home will be on a separate VLAN based network. In preparation for all those home visitors.

Notice the replication speed showing “10.22MBit/sec”. It should be “Mb”, not MB. A capital “B” is BYTES, while a lower case “b” is bits. This is on the latest version 4.7.2.40512. Dev has been alerted per support's response.

Back to spending time with the existing kids and wife,
-Ben

Recommended Exchange Deployments are Multi-role Now

At the last free "Tech Ed style" event in NYC held at the Microsoft Offices, we had Ross Smith IV present on

Exchange 2010 High Availability/Database Availability Groups. If you don't know Ross, he's a VERY senior Microsoft employee who wrote the Exchange Storage and Server Role Calculator. He knows Exchange, period. End of story. So, when he said that everyone should deploy Exchange 2010 in a multi-role configuration to improve performance and not break apart the roles, you need to take his recommendation seriously. This was the 1st time I had heard this. I had a long conversation with him about this in NYC, and he explained that for performance and the ability for leveraging failover capacity it is better to keep all the roles together. In theory, you could deploy less. Since if you were going to deploy 2 CAS and 2 Mailbox, you could in theory just deploy 3 consolidated roles. Well, Microsoft TechNet finally released some guidance on this. That only took 6 months. Don't forget to use a hardware/VM load balancer when deploy your multi-role Exchange Servers.

TechNet article title: Understanding Multiple Server Role Configurations in Capacity Planning
http://technet.microsoft.com/en-us/library/dd298121.aspx

-Ben

Hackers & Malware - Dangers Everywhere - Not Just Scare Tactics

Hackers and malware were busy this week at clients of REEF Solutions.

Good news first, we identified a serious denial of service vulnerability during a network infrastructure review for a financial firm. So bad, a simple command would reboot a core network device. That was a highlight of the review. And this was not even a security audit, I am sure it'll be only worse.

Bad news now

• A client's system was infected with TDSS, one of the nasty [known] malware products  (think encryption, p2p command and control, http/https tunneling, malware competition removal, and MBR infection). Malware vendors even offer a Firefox plug-in to allow paying customers to surf via infected machines to provide anonymous cover. To summarize, TDSS is extremely dangerous. More technical details here. As of now, the only tool that can remove it or most of it is Kaspersky. Ideally, we should have wiped the system, but the client would not permit this.
• A hacker attacked via RDP and compromised a system. We detected the compromise and took immediate action to isolate and remediate the attack. If we had not caught it faster, this could have been a serious issue. The key is to have an Intrusion Detection System in place, even if it's just a firewall based solution. You need to be aware of what is happening on your network. I recommend additional policies such as resetting all administrator passwords, not permitting  "administrator" usernames, requiring 15+ characters passwords, email alerting w/3rd party logging tool on administrator level logins, and layered security products (firewall based scanning, servers based, proxy based, DNS scanning, etc).

Sadly, malware and attackers are not sitting idly by. There are some real threats out there. Stay safe...

-Ben

Sprint 4G appears to be hacked at DEFCON

Hello All,

It appears there could have been a successful man in the middle attack (MiTM) on Sprint 4G at DEFCON. Numerous Android devices were attacked during this period. I hope Android users didn’t "upgrade" or re-enter "their passwords" during the multiple day event. Dangerous, but there is a solution that the carriers and handheld manufacturers could implement to protect against this (see solution below).

News from:
http://www.extremetech.com/computing/92370-4g-and-cdma-reportedly-hacked-at-def-con
http://seclists.org/fulldisclosure/2011/Aug/76

A friend and I were discussing this and this what his response was:

 

I'm betting that they did it one of two ways on 802.16/ClearWire/Sprint4G.

They gained physical access to the local tower, and did MiTM from the tower.  WiMax is Mobile IP from the tower to the provider edge.  It would be a lot easier to do MiTM on at a Mobile IP tower, rather than a LTE network.

The other way is, someone in the group worked for / had access to enough of the parts to make a fake WiMax base station.  Based on the signal strength reports, and slow speeds, this is what I bet they did.  WiMax uses either EAP-TLS or EAP-TTLS.  I'm guessing either they had access to the certs to appear valid, or the end devices did not properly implement EAP-TLS and EAP-TTLS, and just accepted any certificate

A pretty cool hack if they did.  Hopefully it can be shown, and the 4G devices can implement proper security.

Update to above.....

After further research, it definitely looks like the latter method (fake WiMax Base station).  They talk about signal strength and upload speeds.  Those wouldn't be affected by the first method (getting into a valid tower).

Unfortunately the supplicate (client) is probably just configured to accept any client.  

For example in the Cradlepoint, you just specify the carrier / realm, but that's it.  No username, etc.   No certs.  No other options.

Another example, the Sprint SmartView client, it doesn't have the ability to specify anywhere anything related to authentication and certificates.

One would need a fake WiMax base station (that can do 2.6ghz) in order to test to see if the supplicate takes any certificate.

Side note: certain applications offer the ability to register against a specific TLS certificate serial number such as Apple Mail.  I hope other devices/applications allow this in the future.

The MiTM Attack Solution

If phones only accepted carrier based certs and had a proper implementation of EAP-TLS or EAP-TTLS this would protect against this sort of attack. 

Sadly, the solution is going to take a bit of work and time. So, don't automatically "trust" voice over data. Protect your data and it can be more secure than your data.

 

-Ben

New Microsoft ActiveSync Compatibilty Program fails on helpfulness

Ehlo All,

Curious about knowing...

• what ActiveSync functionality is available with which version of Exchange?
• which mobile devices have higher Active Functionality?

Well, this new Microsoft ActiveSync compatibility program for OEMs won't help, but read on about it.
Microsoft recently announced the Exchange ActiveSync (EAS) Logo Program for OEMs (think HTC, Google, Apple, Motorola, Microsoft, etc) which should have been used to identify and bring clarify to the level of EAS support a mobile device included. Sadly, it does not do this since there is 1 level for EAS Logo Program and it includes very basic functionality. So, if the device says "ActiveSync", this is pretty much equal to the EAS Logo Program. BK (author of post below) had it right that there should be multiple levels. For example, "basic", "enhanced", "ultimate". So, if an ActiveSync device said "Ultimate", you would know it supports every feature under the sun for EAS against Exchange 2010. Oh well, maybe version 2 of the program will get this improvement.

Windows IT Pro Post about new EAS Logo Program by BK Winstead

-Ben

Message Dehydration isn't a good thing for Exchange!

Ehlo All,


Quote of the week: "limit is 94% before message dehydration occurs."

A normally very stable client's Exchange Server 2007 stopped processing inbound emails and this was the issue above. Client reported all internal email was working though. I logged into their Exchange Server and reviewed the normal issues inside the Exchange Management Console and nothing jumped out (e.g. databases mounted, 3GB free space on C, 700GB+ free space on database partition (D), no quota limits, receive connectors present/enabled, etc). Strange. I decided to check the email filtering solution in front of the Exchange Server. Since I always recommend clients use my company's email filtering service (SpamCop - I'll discuss this is another post) since it allows me to quickly troubleshoot issues and provide very secure email service. Reviewing SpamCop outbound queue to the client's Exchange Server illuminated the error:


Deferred: : host 55.55.55.55 said: 452 4.3.1 Insufficient system resources (in reply to...

More details about this error can be found on this posting. Checking the client's Application Event Log for this error, I found it "The Microsoft Exchange Transport service is rejecting message submissions because the available disk space has dropped below the configured threshold.". Which was surprising since the C drive had a decent amount of space available (3GB+). What changed? Good old SBS's Windows Server Update Services had downloaded every update under the sun and the storage space threshold was passed and triggered back pressure. I uninstalled WSUS and that eliminated 10GB and back pressure was eased and email flow started. I plan to remove some other functionality as well and plan to do a scheduled reboot as well to free up more space. To sum up, SBS causes more problems than it's worth for Exchange deployments. I prefer clean Exchange installs over Exchange SBS installs.

-Ben