Hi
I'm running Xymon v4.3.10 on Linux, and I'm quite sure it's compiled with c-ares support.
I have 12 new DNS servers that were added to Xymon about one month ago. All of my server entries in hosts.cfg have "testip". The tasks.cfg runs xymonet with "--dns-timeout=3". The hosts entries look like so:
10.10.10.1 dnshost1.example.com # testip dns=NS:example.com,SOA: example.com
About a week ago, connectivity to all of these servers failed, and at the same time, the xymonnet run time jumped from less than 15 seconds to about 330 seconds, so about 315 seconds extra. The xymonnet page says 295 seconds is taken up by DNS tests.
If the increase in time taken is about 315 and is entirely due to the 12 servers failing, then each failed server is adding about 26 seconds to the total run time.
I don't think this should be happening like this. With two DNS checks per server, the DNS checks should be taking 6 seconds each to time-out, not 26. If I run xymonnet with "--timing --no-update" and specify only one hostname, I can view the results and the timing. This shows that the ping check gets reported after about 3 seconds, and then the DNS tests are executed and take 26 seconds total.
My naiive assumption was that when a server failed a ping (and didn't have "noclear" defined in hosts.cfg) then the network checks would be skipped. On re-reading the man page for hosts.cfg, it dawned on me that a failed ping simply suppresses failed test /results/, but doesn't stop the tests from being run.
So the real problem is that the "--dns-timeout=3" is not being taken into consideration by xymonnet. If I run xymonnet with "--debug" it tells me:
1900 2015-06-03 12:02:20 ares_search: tlookup='example.com', class=1, type=2 1900 2015-06-03 12:02:20 ares_search: tlookup='example.com', class=1, type=6 1900 2015-06-03 12:02:20 Processing 0 DNS lookups with ARES 1900 2015-06-03 12:02:46 Finished ARES queue after loop 423
This is peculiar. Why would it say "processing 0 DNS lookups" when there are two lookups to test? Could this be because xymonnet hasn't actually been built with ARES support and I didn't know it? Is there a good way to tell? If I add "--no-ares" I get the same results perhaps suggesting a lack of ARES support. On the other hand, if I add "timeout:3" and "attempts:1" into resolv.conf, I also get the same results. If I run "nm /path/to/xymonnet | grep gethostby" it returns "ares_gethostbyname".
Just for fun, I compiled Xymon v4.3.21 and ran the xymonnet binary from there, with no change in behaviour. I also tried removing the "--dns-timeout" option so that it defaults to 30 seconds, but still no change - 26 seconds for two DNS tests.
So, I'm not really sure what the problem is, but xymonnet certainly isn't behaving as I would expect.
Cheers Jeremy