DoS of DNS by an Exchange Focused Backup Software (AppAssure Replay)

 Ehlo All,

Imagine to my surprise that my favorite Exchange & Windows backup solution (AppAssure Replay 4.5.1.27532) was attempting to cause a denial of service (DoS). This version has a major problem with it's use of DNS lookups within the problem. Within 3 days, one Replay Server had performed over 450,000 queries of the hostname I used for Replay replication. This is almost 100 queries every minute 24 hours a day. That's what the product is doing. This is a serious issue. I've alerted the vendor, so I'm sure a fix will be included in a future release. In the mean-time, see below for the work-around until that happens.

The Issue
Inside AppAssure's Replay for replication, you specify a "Replication Target Host Name". This can be a hostname or IP address. See below for setting within Replay.

 This "Select Replication Target" configuration is per protected server (e.g. your Exchange Server, etc). I normally use a hostname for these types of settings since I'm a big fan of using DNS instead of IPs when possible (saves time when changing IPs & saves brain memory space for Exchange Server things). So, when you add your Replication Target hostname, the Replication target and source perform lookups more often than the snap-shot period (x min/hrs). In reality, Replay should only perform a DNS lookup when a replication needs to occur and NOT almost a 100 per minute.

AppAssure's Replay abusing DNS lookups. View from my Firewall hostname query  logs.

The Workaround Until a Permanent Fix is Released by AppAssure
If you use a hostname within the Replication option, make sure you add the corresponding information inside the hosts file (c:\windows\System32\drivers\etc\hosts - format is IP address space and hostname - use notepad to open the "hosts" file) on the source AND target Replay Replication Server. This avoids the use of an external DNS query and the query is handled by the operating system. So, this speeds up the process of performing a lookup and reduces your hostname's name server load. Otherwise prepare for your DNS to be attacked by your Replay environment.

Sadly, this isn't the first time I have seen a product mis-use DNS, but it's one of the worst in recent memory.

-Ben

Article - Constitution Finally is Defended over illegal email searches by US Government

 Hello All,

Glad to see someone cover this topic and that this "piece of paper" above means something. "This topic" is about defending our rights (4th Amendment which is unreasonable searches) from the US government illegal intrusions into email. US Dept of Justice illegally had an ISP copy someone's email and was slapped on the wrist, thank goodness for a Court of Appeals. See the Windows IT Pro article from Paul Robichaux discuss this. I hope this ISP get's sued as well now.

-Ben

Article - HA for Exchange 2010 is more costly than Exchange 2007

Ehlo All,

This is an excellent article especially if you're using or considering deploying Exchange 2007 in an HA configuration with SCR,. Also, the new HA mailbox solution in Exchange 2010 is called Database Availability Groups (aka DAGs). This requires the Enterprise Edition for Windows Server 2008, and we do not know if it requires Exchange 2010 Standard or Enterprise. Either way, the requirement of Server Enterprise Edition makes it costly.

Excellent website (Exchange-Genie.com) that explains with pictures the following HA solutions for Exchange 2007 & 2010, CCR, LCR, SCC, SCR, and DAG. In this case, a picture is worth a thousand words.

Information is power,
-Ben

Intro to Exchange 2007 & SSL Certificates

[Written in December 2008. Not sure why I didn't post it back then. Found in my drafts.]

Ehlo All,

SSL Certificates (aka SSL certs, certs) are complicated. Exchange is complicated. Quick story... during my research, I read about a Microsoft employee (Exchange admin) who thought a bug existed with a SSL cert vendor's special "Exchange 2007" cert. So he contacted an Exchange Team PM about the issue and it turned out he didn't properly configure the Exchange SSL cert generation. Moral of the story, add SSL certs & Exchange 2007 together, and it can be a real challenge. So, I recommend you read this intro if you're considering or even using Exchange 2007 & real SSL certs especially since many things have changed.

First, the good news with Exchange 2007. Microsoft now includes self-signed certs on install. So, you're secure out of the box. Meaning, everything is encrypted, but you'll get errors each time you access a SSL protected site with it. So, you'll want to fix this.

Well, everything you know about Exchange 2003 SSL certificates can be thrown out, or that's what I've learned so far. You can use wildcard SSL certs, but you'll run into issues in the future (Outlook auto-discovery has issues, Windows Mobile 5 has issues, etc), so I would recommend you embrace the new SSL cert that Microsoft wants you to use OR learn about a special DNS configuration that lets you avoid this (thanks to a MS June 07 update for Outlook 2007).

As per the new certs, they are best described by DigiCert (I liked their explanation & enhanced it some) is:

Exchange 2007 Certificates Definition
The new SSL certificates are called Unified Communications Certificates (aka UC Certificates, UCC Certificates, or SAN certificates) which give you full control of the Subject Alternative Name field so you can secure as few or as many host names as you like with just one SSL certificate. These are NOT wildcard certs since they secure specific hostnames you define within this one cert. Wildcard certs secure any subdomain (e.g. *.mydomain.com), while UCC certs secure (autodiscover.mydomain.com, webmail.mydomain.com, mail.mydomain.com, exchsrv.mydomain.local, etc).

Microsoft's Recommended List of UCC Cert Providers as of 12/08
Comodo - http://www.comodo.com/msexchange/index.html
DigiCert - http://www.digicert.com/unified-communications-ssl-tls.htm
Entrust - http://www.entrust.net/ssl-certificates/unified-communications.htm
Microsoft's Recommended List of UCC Cert Providers

Special DNS Configuration to Avoid UCC Certs
So, you decided you didn't want to spend about $250-300 on a UCC cert for one year. I can understand. There is another option I hinted to above that relates to a June 2007 Outlook 2007 update that added a special feature to avoid the need for a UCC cert for autodiscovery or complex admin configuration. It involves setting a special DNS record to get around this. The DNS record is a SRV record. Once you have this SRV record set, the Exchange 2007 server's externl adn internal URLs need to be this one server as identified in the SRV DNS record.

Learn more about this in Microsoft White Paper on Auto-Discovery in Exchange & Outlook 2007. http://technet.microsoft.com/en-us/library/bb332063.aspx

MS Article on how to setup DNS SRV record for auto-discovery functionality
MS KB article about auto-discovery issues and fixes

-Ben

"Cheating" on an Exchange 2003 Hardware Upgrade

[Project completed in June 2008. Figured this was in my drafts, I should post it.]

Hello Everyone,

I "cheated" on an Exchange 2003 hardware upgrade I did last weekend (Fri-Sat), or that's at least how I feel since this was hands down the fastest and easiest upgrade I've ever done (and it was about 80GB of db's on an older server with direct attached storage). At the end of the weekend, I started to think maybe I should carry around one of these "things" for my clients for upgrades. I'll share what this "thing" was later in the posting. I don't want folks to think I'm pushing products. My role in the project was to insure the replacement of the Exchange Server hardware went smoothly. The client was in production 24/7 and literally the office was staffed 6 days a week. So, I was concerned originally how to insure minimal downtime.

Background on existing hardware & performance
We were upgrading from an Exchange 2003 Server that was installed with 3 hard drives in a RAID 5 hard drive configuration (direct attached storage) for the OS, transaction logs, and Exchange databases. Company had about 60 users and 30 BlackBerrys or so. 1 BES user adds a load similar to 2 Outlook users. So, total company usage was about 120 users. Performance was an issue, so some users were configured for cached mode to "improve" performance. Cached mode should not be required for LANs, unless Outlook end users are receiving "retrieving data from network". Recommended another DAS server that used RAID 1 for the OS, RAID 1 for transaction logs, and RAID 10 for the Exchange databases.

The Migration
So, I checked the OS install (another admin handled that), Exchange install (using the /disasterrecovery switch for setup and service pack 2), Exchange config, and insuring the email & public folder migration completed successful. Only catch was during this server replacement, there was to be no downtime and no use of Exchange clustered services. Hmmm, that's a challenge. Or so one would think.

The Cheat
The client I was working for happened to have a 3rd party product (keeping read to find out) for Exchange that in essence allowed the "cheating". And I mean this in a very good way. It saved me a LOT of time. Meaning, we told the 3rd party product to take over all the existing Exchange services (MAPI, SMTP, OWA, IMAP, etc) and data for the Outlook, OWA, ActiveSync, & BlackBerry was available to all users. All their data was still available. This took a few minutes (3 or 4 minutes) on the switch-over. Now everyone was operating off the appliance and end users didn't know this besides restarting Outlook and re-authenticating to OWA (ActiveSync & BES users had a slight delay. BES users could be out of service for up to 15 minutes, but that's a limitation of BES). Once the appliance took over, we copied over the Exchange databases (.edb/.stm's) using robocopy to the new Exchange 2003 Server. We considered upgrading to 2007, but the appliance and all the associated Exchange applications would have had to been upgraded, and it wasn't cost effective (TCO reminder). So, after we started robocopy-ing, we went home.

Day 2 of the Migration & Failback
I'm not going to go into the details, but migrating took a few hours including getting the SSL certs for OWA and handling all that. Once the new hardware was setup with Exchange, it was time to bring back all the new email. As I previously said, the reason to copy over the databases to the new server, was the appliance then doesn't need to copy over all data, and just new email/data. Once we copied over the databases and transaction logs, we were able to get the Exchange Server fully operational and enable the failback from the appliance. We then failed back from the appliance to the new Exchange Server. This took a lot longer to check that all data was copied from the appliance to the new Exchange Server. This took 10 hours or so and then everyone had to relaunch Outlook and re-authenticate against the new server.

Appliance Details
The "cheat" was an Exchange high availability appliance from Teneros. For the record, this company has been bankrupt and while still in operation as of 7/10, I do NOT recommend purchasing this product. Even though this appliance runs 2 operating systems, Linux and Windows, the entire configuration is on 2 web pages. Meaning, the Teneros support team is really what runs this product, not the Exchange admin. As per the web interface, to say the amount of information and configuration is sparse, is putting it lightly. Overall the product worked well and we ran into 2 glitches due to permissions and resetting process of the AD name due to poor documentation. And the migration process took longer than expected since the status of synchronization is not very accurate. Not a big deal, since end users are working during the failover and failback. Overall solution is very impressive, but I have some doubts since I'm not a big fan of trusting secret functionality of a black box type solution. I like to know how applications work and I do have concerns over Exchange updates or patches breaking the Teneros functionality. If you are curious, pricing is around $10k, give or take a few thousands. If you wanted to see the demo, Teneros did present at the NY Exchange User Group meeting back in November of December of 2007 or check out their website.

-Ben

Microsoft (online/print) and My Photos

Hello All,

Think you've seen me on the Microsoft website, you have! A couple of people have asked me about this. So, yes, it's true. Microsoft hired me about 9 months ago for a photo shoot. So, you'll see me online and in print media. Here's an example someone sent me recently.

I was hired for the look of an "IT manager". One of the coolest scenes was inside the train control center in Newark Penn Station. Anyone sees that, let me know and take a screen-shot.

Thanks,
-Ben

The Truth Behind ActiveSync & Enterprise Licensing

Hello All,

Sadly I see this happen more often than it should. A potential client was lied to by another IT vendor's sales team. They claimed that for "iPhone ActiveSync remote wipe functionality, the Enterprise Client Access Licenses were required for Exchange Server 2010". Turns out this is a lie! Oops... caught by me.

The Enterprise Client Access License (aka eCAL) does not relate/control remote wipe for ActiveSync. The eCAL adds the following control for ActiveSync clients that support these features. Both server and client side need to support these ActiveSync policies for them to be in effect. Make sure you read that last sentence twice. I've put that in bold for you. This is critical in your understanding of ActiveSync device features. Pretty much each device that includes ActiveSync offers varying levels of ActiveSync capabilities.

iPhone support remote wipe (they also offer a limited capabilities for ActiveSync). The available ActiveSync functionality on the iPhone is documented here.

List of Exchange Server 2010 Enterprise Client Access License (eCAL) Functionality Abilities (screen-shots below)
Allow removable storage
Allow camera
Allow Wi-FI
Allow infrared
Allow Internet sharing from the device
Allow remote desktop from the device
Allow synchronization from a desktop
Allow Bluetooth
Allow browser
Allow consumer mail
Allow unsigned applications
Allow unsigned installation packages

-Ben