On 12-04-2012 05:28, Poppy, Ben wrote:
To be honest, I'm not sure what the cause is 100%.
Me neither.
The setup we have, should not have any dependencies on our DR site. Our xymon servers at our primary site use DNS servers in our primary site. They monitor a bunch of servers at our DR site, but the dependency ends there (and all that should mean is the servers show RED when DR site is down).
Aha - but you DO have tests in each setup that checks systems on the other site ? Would that happen to include any DNS or NTP checks ?
I suspect that you have each of your Xymon's setup to test availability of the DNS servers on both the primary and the DR site. That could be a problem.
Another bit of information, during this 5 hour outage, both of our xymon servers went from showing properly (where DR servers were showing RED conn as they weren't reachable, but the servers we monitor in our primary site were up), to everything going purple in conn (and other tests).. It would alternate back and forth over the course of the outage (I didn't detect a regular timeframe of when it switched from RED to PURPLE)..
The interesting thing is that they switch to purple, indicating that something is stalled.
I have seen something like this happen when we had a number of DNS checks in the Xymon servers, and network access to these failed (broken switch to a customer network). This caused xymon to stall on these DNS checks, and all of the network tests went purple.
I know that this is difficult to test, because obviously you cannot just cut the connection between the two sites to try it out. But you could try applying this patch which changes the DNS lookup code to use the same kind of timeout settings as the development version - the 4.3.x versions suffer from a common misunderstanding about how the C-ARES library handles timeout that make DNS timeouts take much too long.
One possible way of testing it would be if you can firewall access from e.g. your DR site Xymon server to the primary site's DNS server. If you are running Xymon on a Linux server, then "iptables" can do that for you. If your primary site DNS server is 10.1.2.3, then
iptables -I OUTPUT 1 -d 10.1.2.3 -j DROP iptables -I INPUT 1 -s 10.1.2.3 -j DROP
will cause all traffic to/from this server to be dropped.
Regards, Henrik