So, can you do DNS queries from the xymon server when DC3 & 4 are down?
"Poppy, Ben" 03/20/12 11:50 AM >>> So they are pointing to 2 DC's that stay up this entire time, we'll call them DC1 and DC2. Then we shutdown DR-DC3 and DR-DC4. When those servers are down, we begin to have issues.
-----Original Message----- From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Monday, March 19, 2012 7:46 PM To: Poppy, Ben Cc: xymon at xymon.com Subject: Re: [Xymon] Purple storm
On Tue, Mar 20, 2012 at 5:15 AM, Poppy, Ben wrote:
I have an interesting problem that happened last night. We are working
on a DR test. Part of that test includes shutting down some DC's in our DR datacenter. When that happened, most tests that are initiated from the xymon servers (http, dns, ssh, ftp, etc) to the monitored server went purple.
For network tests, Xymon resolves the IP address from the servername (typically using DNS), and then uses that IP address to perform the test. The IP address in the hosts.cfg file is not normally used for network tests. So if your DNS fails, Xymon's network tests fail also.
You can prevent this, and use the IP address supplied in hosts.cfg, by adding "testip" to each hosts.cfg entry that requires it. You can add it to a ".default." entry so that it applies to all hosts.
J
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Yes, that's the strange part, we can still manually do digs and nslookups from the xymon server to other DNS servers.
-----Original Message----- From: Phil Crooker [mailto:Phil.Crooker at orix.com.au] Sent: Tuesday, March 20, 2012 12:41 AM To: Poppy, Ben Cc: xymon at xymon.com Subject: Re: [Xymon] Purple storm
So, can you do DNS queries from the xymon server when DC3 & 4 are down?
"Poppy, Ben" 03/20/12 11:50 AM >>> So they are pointing to 2 DC's that stay up this entire time, we'll call them DC1 and DC2. Then we shutdown DR-DC3 and DR-DC4. When those servers are down, we begin to have issues.
-----Original Message----- From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Monday, March 19, 2012 7:46 PM To: Poppy, Ben Cc: xymon at xymon.com Subject: Re: [Xymon] Purple storm
On Tue, Mar 20, 2012 at 5:15 AM, Poppy, Ben wrote:
I have an interesting problem that happened last night. We are working
on a DR test. Part of that test includes shutting down some DC's in our DR datacenter. When that happened, most tests that are initiated from the xymon servers (http, dns, ssh, ftp, etc) to the monitored server went purple.
For network tests, Xymon resolves the IP address from the servername (typically using DNS), and then uses that IP address to perform the test. The IP address in the hosts.cfg file is not normally used for network tests. So if your DNS fails, Xymon's network tests fail also.
You can prevent this, and use the IP address supplied in hosts.cfg, by adding "testip" to each hosts.cfg entry that requires it. You can add it to a ".default." entry so that it applies to all hosts.
J
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
--
This message from ORIX Australia might contain confidential and/or privileged information. If you are not the intended recipient, any use, disclosure or copying of this message (or of any attachments to it) is not authorised.
If you have received this message in error, please notify the sender immediately and delete the message and any attachments from your system. Please inform the sender if you do not wish to receive future communications by email.
ORIX handles personal information according to a Privacy Policy that is consistent with the National Privacy Principles. Please let us know if you would like a copy. It is also available at http://www.orix.com.au .
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
Would you be able to run a tcpdump or use a network sniffer to see what the server is doing when you're getting the long response times?
Maybe that will help you see what it is trying to reach when that is happening.
On 3/20/12 1:51 PM, "Poppy, Ben" <poppy.ben at marshfieldclinic.org> wrote:
Yes, that's the strange part, we can still manually do digs and nslookups from the xymon server to other DNS servers.
-----Original Message----- From: Phil Crooker [mailto:Phil.Crooker at orix.com.au] Sent: Tuesday, March 20, 2012 12:41 AM To: Poppy, Ben Cc: xymon at xymon.com Subject: Re: [Xymon] Purple storm
So, can you do DNS queries from the xymon server when DC3 & 4 are down?
"Poppy, Ben" 03/20/12 11:50 AM >>> So they are pointing to 2 DC's that stay up this entire time, we'll call them DC1 and DC2. Then we shutdown DR-DC3 and DR-DC4. When those servers are down, we begin to have issues.
-----Original Message----- From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Monday, March 19, 2012 7:46 PM To: Poppy, Ben Cc: xymon at xymon.com Subject: Re: [Xymon] Purple storm
On Tue, Mar 20, 2012 at 5:15 AM, Poppy, Ben wrote:
I have an interesting problem that happened last night. We are working
on a DR test. Part of that test includes shutting down some DC's in our DR datacenter. When that happened, most tests that are initiated from the xymon servers (http, dns, ssh, ftp, etc) to the monitored server went purple.
For network tests, Xymon resolves the IP address from the servername (typically using DNS), and then uses that IP address to perform the test. The IP address in the hosts.cfg file is not normally used for network tests. So if your DNS fails, Xymon's network tests fail also.
You can prevent this, and use the IP address supplied in hosts.cfg, by adding "testip" to each hosts.cfg entry that requires it. You can add it to a ".default." entry so that it applies to all hosts.
J
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
--
This message from ORIX Australia might contain confidential and/or privileged information. If you are not the intended recipient, any use, disclosure or copying of this message (or of any attachments to it) is not authorised.
If you have received this message in error, please notify the sender immediately and delete the message and any attachments from your system. Please inform the sender if you do not wish to receive future communications by email.
ORIX handles personal information according to a Privacy Policy that is consistent with the National Privacy Principles. Please let us know if you would like a copy. It is also available at http://www.orix.com.au .
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
I think the interesting sniffer would be on the DC's that remain up. Just to make sure I got this straight, you've got two DC's on the LAN with Xymon and two DC's in your DR site. You shutdown the DC's in the DR site and now queries are timing out (or something) to the DC's on the LAN.
If that's the case, I would first look at DNS on the DC's. If you do a packet capture on the DC's filtered by UDP 53 and the only packets from or to your Xymon server, then this would show whether the queries are making to your remaining DC's and if there is any delay in the response. It wouldn't surprise me is Windows was so worried about the missing domain controllers that it forgot to respond to DNS queries. If there's not, then refer to the tcpump you have going on on your Xymon server to make sure the packets are making it back with acceptable latency.
I'm making the assumption that your DNS configuration has the zone in question as a primary, Active Directory integrated zone, also, are you running a caching DNS server on your Xymon system? I've seen some odd results with DNS caching. What order are the name servers in /etc/resolv.conf? I was playing with Debian this weekend and for some reason, no matter what I did it would not use any other name server than the first one in the list except for dig and nslookups, but not for regular queries.
Of course you could always just install your favorite DNS server on your Xymon system and transfer the zones as secondary zones from your Windows boxes, that'll definitely solve the problem and remove the prerequisite of your DC's staying up so Xymon works. ...matter of fact, I think I'll do that myself....
Jamison Maxwell
-----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Don Kuhlman Sent: Tuesday, March 20, 2012 3:05 PM To: xymon at xymon.com Subject: Re: [Xymon] Purple storm
Would you be able to run a tcpdump or use a network sniffer to see what the server is doing when you're getting the long response times?
Maybe that will help you see what it is trying to reach when that is happening.
On 3/20/12 1:51 PM, "Poppy, Ben" <poppy.ben at marshfieldclinic.org> wrote:
Yes, that's the strange part, we can still manually do digs and nslookups from the xymon server to other DNS servers.
-----Original Message----- From: Phil Crooker [mailto:Phil.Crooker at orix.com.au] Sent: Tuesday, March 20, 2012 12:41 AM To: Poppy, Ben Cc: xymon at xymon.com Subject: Re: [Xymon] Purple storm
So, can you do DNS queries from the xymon server when DC3 & 4 are down?
"Poppy, Ben" 03/20/12 11:50 AM >>> So they are pointing to 2 DC's that stay up this entire time, we'll call them DC1 and DC2. Then we shutdown DR-DC3 and DR-DC4. When those servers are down, we begin to have issues.
-----Original Message----- From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Monday, March 19, 2012 7:46 PM To: Poppy, Ben Cc: xymon at xymon.com Subject: Re: [Xymon] Purple storm
On Tue, Mar 20, 2012 at 5:15 AM, Poppy, Ben wrote:
I have an interesting problem that happened last night. We are working
on a DR test. Part of that test includes shutting down some DC's in our DR datacenter. When that happened, most tests that are initiated from the xymon servers (http, dns, ssh, ftp, etc) to the monitored server went purple.
For network tests, Xymon resolves the IP address from the servername (typically using DNS), and then uses that IP address to perform the test. The IP address in the hosts.cfg file is not normally used for network tests. So if your DNS fails, Xymon's network tests fail also.
You can prevent this, and use the IP address supplied in hosts.cfg, by adding "testip" to each hosts.cfg entry that requires it. You can add it to a ".default." entry so that it applies to all hosts.
J
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
--
This message from ORIX Australia might contain confidential and/or privileged information. If you are not the intended recipient, any use, disclosure or copying of this message (or of any attachments to it) is not authorised.
If you have received this message in error, please notify the sender immediately and delete the message and any attachments from your system. Please inform the sender if you do not wish to receive future communications by email.
ORIX handles personal information according to a Privacy Policy that is consistent with the National Privacy Principles. Please let us know if you would like a copy. It is also available at http://www.orix.com.au .
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On Wed, Mar 21, 2012 at 9:11 AM, Jamison Maxwell <jamison at newasterisk.com> wrote:
What order are the name servers in /etc/resolv.conf? I was playing with Debian this weekend and for some reason, no matter what I did it would not use any other name server than the first one in the list except for dig and nslookups, but not for regular queries.
That's all normal behaviour. The standard resolver library on Linux will only use second and third entries in resolv.conf when the first one is unavailable for 5 seconds (see "man resolv").
The dig, nslookup and host programs don't use the resolver library, and instead use their own in-built resolver that behaves differently to the standard resolver - but still uses the "nameserver" entries from resolv.conf. For this reason, when diagnosing DNS problems with applications, using dig/nslookup can give you results different to the application you're testing; it's better to use things like "ping" or "telnet" to do a lookup, as they use the same resolver library as most other applications, including Xymon.
participants (5)
-
Don.Kuhlman＠schawk.com
-
jamison＠newasterisk.com
-
jlaidman＠rebel-it.com.au
-
Phil.Crooker＠orix.com.au
-
poppy.ben＠marshfieldclinic.org