Imagine to my surprise that my favorite Exchange & Windows backup solution (AppAssure Replay 4.5.1.27532) was attempting to cause a denial of service (DoS). This version has a major problem with it's use of DNS lookups within the problem. Within 3 days, one Replay Server had performed over 450,000 queries of the hostname I used for Replay replication. This is almost 100 queries every minute 24 hours a day. That's what the product is doing. This is a serious issue. I've alerted the vendor, so I'm sure a fix will be included in a future release. In the mean-time, see below for the work-around until that happens.
The Issue
Inside AppAssure's Replay for replication, you specify a "Replication Target Host Name". This can be a hostname or IP address. See below for setting within Replay.
This "Select Replication Target" configuration is per protected server (e.g. your Exchange Server, etc). I normally use a hostname for these types of settings since I'm a big fan of using DNS instead of IPs when possible (saves time when changing IPs & saves brain memory space for Exchange Server things). So, when you add your Replication Target hostname, the Replication target and source perform lookups more often than the snap-shot period (x min/hrs). In reality, Replay should only perform a DNS lookup when a replication needs to occur and NOT almost a 100 per minute.
 |
| AppAssure's Replay abusing DNS lookups. View from my Firewall hostname query logs. |
The Workaround Until a Permanent Fix is Released by AppAssure
If you use a hostname within the Replication option, make sure you add the corresponding information inside the hosts file (c:\windows\System32\drivers\etc\hosts - format is IP address space and hostname - use notepad to open the "hosts" file) on the source AND target Replay Replication Server. This avoids the use of an external DNS query and the query is handled by the operating system. So, this speeds up the process of performing a lookup and reduces your hostname's name server load. Otherwise prepare for your DNS to be attacked by your Replay environment.
Sadly, this isn't the first time I have seen a product mis-use DNS, but it's one of the worst in recent memory.
-Ben
3 comments:
Hi Ben:
I'm one of the developers of Replay at AppAssure Software. Your blog post came up on our Google Alerts feed, and I wanted to take the opportunity to respond.
First of all, thanks for bringing this to our attention. The development team values customer feedback, especially about things we can do better.
When you configure replication in Replay, the two Replay Cores communicate periodically so we can display status information about the remote core in the GUI for the local core. Nearly twice per second is clearly much too frequent, which is obviously a bug in the code that determines the refresh interval. However even with this fixed, we will query the remote core more often than just when replication needs to happen.
Second, I'm curious why these queries are hitting your DNS server every single time. What is the TTL for the DNS records for your core's hostname? I would think the DNS cache on the Replay Core would be able to satisfy most of these hostname lookups. That's not to say this is an environmental problem, but for purposes of our internal QA I am legitimately curious how you have your DNS configured.
Thanks again for your feedback.
Adam
Hello Adam,
Thank you for your posting.
TTL of the A record for the replication hostname is 1800 seconds (30 minutes). The Replay Core (Target) of replay3.reefsolutions.com is pointing to a pair of local subnet based Windows 2003 DNS Servers. Why aren't they caching lookups, not sure. BUT, Replay on 2008 R2 appears to be ignoring DNS entries and is doing it's own direct DNS lookups. It's bypassing the server's DNS entries. I ran a debug on the Windows 2003 DNS servers, and NOT a single lookup query was detected (which is correct since I'm using the HOSTS file now) yet the firewall log is showing additional "replay3.reefsolutions.com" lookups.
I also found another problem with DNS lookups for Replay versions (4.4.1.26806 & 4.5.1.27532). Replay is attempting to performing DNS lookups for replicated server names even though servers aren't local. For example, a replicated server name of TX-MAIL is appearing in my DR site's DNS servers debug logs for "tx-mail.mylocaldomain.local".
Another bug: modifying the "Retention Policy" is now reporting "An error occurred during save operation: Error resolving hostname '{insert server name}'. - No such host is known. This was previously resolved in 4.4.1.26806. Inability to use UI to change retention is not possible again. This bug was re-introduced. I'll open another AppAssure support ticket on this. Come on guys!!!
-Ben
That is quite curious. I can assure you Replay does not bypass the Windows IP stack for DNS lookups; it uses the same name resolution APIs as any other Windows application, so I'm mystified as to where those DNS requests are coming from.
The second issue you describe is actually by design; even on a replicated core, it is possible to perform restores directly to the source agent, if the replicated core is on the same subnet. In order to determine whether this is possible, we attempt to connect to the agent on some interval, just as we would with a local agent.
Let's continue this conversation through our ticketing system.
Thanks,
Adam
Post a Comment