Hey Henrik/all, I need more hand-holding.

I modified the command line for the bbtest-net command to include the dnslog option, the entry now looks like this:

[bbnet] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout=10 --dnslog=/var/log/hobbit/dns.log LOGFILE $BBSERVERLOGS/bb-network.log INTERVAL 10m

After restarting the whole Xymon application I looked at the open connections that the bbtest-net process had and saw this:

root at hbt0.lon:/home/mward>pfiles /proc/2944 2944: bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout= Current rlimit: 256 file descriptors 0: S_IFCHR mode:0666 dev:285,3 ino:41103 uid:0 gid:3 rdev:13,2 O_RDONLY /dev/null 1: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913 O_WRONLY|O_APPEND|O_CREAT /var/log/hobbit/bb-network.log 2: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913 O_WRONLY|O_APPEND|O_CREAT /var/log/hobbit/bb-network.log 3: S_IFREG mode:0644 dev:61,261 ino:42256 uid:1985 gid:1985 size:0 O_WRONLY|O_CREAT|O_TRUNC /var/log/hobbit/dns.log 4: S_IFSOCK mode:0666 dev:280,0 ino:25389 uid:0 gid:0 size:0 O_RDWR|O_NONBLOCK SOCK_STREAM SO_SNDBUF(49152),SO_RCVBUF(49640),IP_NEXTHOP(232.193.0.0) sockname: AF_INET6 ::ffff:10.44.107.48 port: 43062 peername: AF_INET6 ::ffff:10.44.107.51 port: 389 5: S_IFSOCK mode:0666 dev:280,0 ino:59068 uid:0 gid:0 size:0 O_RDWR|O_NONBLOCK FD_CLOEXEC SOCK_DGRAM SO_DGRAM_ERRIND,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.224.0.0) sockname: AF_INET 212.74.77.181 port: 50902 peername: AF_INET 212.74.77.108 port: 53

So you can see that the bbtest-net process has the log file opened OK on file ID 3. The problem is that the log file is empty :-(

root at hbt0.lon:/home/mward>ls -al /var/log/hobbit/dns.log -rw-r--r-- 1 hobbit hobbit 0 Oct 12 12:05 /var/log/hobbit/dns.log

I did reduce the number of tests being made as there were a load of duplicates but I have plenty of DNS tests still configured, here you can see one configured entry:

root at hbt0.lon:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts /opt/hobbit/server/etc/bb-hosts-* | egrep -v ":#" | wc -l 79 root at hbt0.lon:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts | head -1 /opt/hobbit/server/etc/bb-hosts:212.121.128.1 ns0.de.colt.net # ssh dns=ns:de.colt.net COMMENT:212.121.128.1

and all these take 1,351 seconds to complete testing.

So it seems I need to add another step in my process to resolve my DNS issues, how can I persuade bbtest-net to actually log something?

|\/|artin

-----Original Message----- From: Ward, Martin [mailto:Martin.Ward at colt.net] Sent: 09 October 2009 14:26 To: hobbit at hswn.dk Subject: RE: [hobbit] Which DNS query is causing delays

Thanks Henrik, I am not using the --no-ares option so have put the dnslog entry in as you recommend.

Thanks and I'll let you know how I get on.

|\/|artin

...

-----Original Message----- From: Henrik "Størner [mailto:henrik at hswn.dk] Sent: 09 October 2009 12:26 To: hobbit at hswn.dk Subject: Re: [hobbit] Which DNS query is causing delays

In <1F7B01020EC4D04DA17703634B9E888E0A01CCD1 at ULPGCTMVMAI003.EU.COLT> "Ward, Martin" <Martin.Ward at colt.net> writes:

...

I have noticed recently that my bb-test runs are taking a long time to complete, 6500 seconds instead of the maximum of 600. The info on the bb-test results page shows that the DNS tests are taking most of the time, but I have around 200 of these, does anyone have any suggestions on how I can find out which DNS tests are causing the delays without having to click on every DNS test in all of my Xymon website?

Are you using a local DNS cache ? I would strongly recommend that you do - the resolver library in Xymon is pretty aggressive, and has been known to knock out DNS servers if you have many hosts (although 200 doesn't sound like an awful lot).

It shouldn't take that long unless you're using the standard resolver library (i.e. you are running bbtest-net with the "--no-ares" option), in which case Xymon cannot control the timeout for the DNS lookup.

Try running with the option "--dnslog=FILENAME", this will log those DNS requests that fail. It is probably one of those.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

---

---

The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.

The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security at colt.net and delete the message and any attachments without retaining any copies.

Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.

The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security at colt.net and delete the message and any attachments without retaining any copies.

Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

-----Original Message----- From: Henrik "Størner [mailto:henrik at hswn.dk] Sent: 12 October 2009 15:49 To: hobbit at hswn.dk Subject: Re: [hobbit] Which DNS query is causing delays

In <1F7B01020EC4D04DA17703634B9E888E0E96BE8B at ULPGCTMVMAI003.EU.COLT> "Ward, Martin" <Martin.Ward at colt.net> writes:

...

I modified the command line for the bbtest-net command to include the dnslo= g option, the entry now looks like this: The problem is that the log file is empty :-(

It will only log those DNS lookups that fail, not those that succeed.

OK, so it now seems likely that all the DNS tests are working, they are just taking a very long time to do it, so the question is: Is there an easy way to get the DNS resolving times for all my Xymon DNS queries without having to view every web page?

|\/|artin

The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.

The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security at colt.net and delete the message and any attachments without retaining any copies.

Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

On Fri, Oct 09, 2009 at 11:26:23AM +0000, Henrik Størner wrote:

In <1F7B01020EC4D04DA17703634B9E888E0A01CCD1 at ULPGCTMVMAI003.EU.COLT> "Ward, Martin" <Martin.Ward at colt.net> writes:

...

I have noticed recently that my bb-test runs are taking a long time to complete, 6500 seconds instead of the maximum of 600. The info on the bb-test results page shows that the DNS tests are taking most of the time, but I have around 200 of these, does anyone have any suggestions on how I can find out which DNS tests are causing the delays without having to click on every DNS test in all of my Xymon website?

Are you using a local DNS cache ? I would strongly recommend that you do - the resolver library in Xymon is pretty aggressive, and has been known to knock out DNS servers if you have many hosts (although 200 doesn't sound like an awful lot).

It shouldn't take that long unless you're using the standard resolver library (i.e. you are running bbtest-net with the "--no-ares" option), in which case Xymon cannot control the timeout for the DNS lookup.

Try running with the option "--dnslog=FILENAME", this will log those DNS requests that fail. It is probably one of those.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

Hello,

when we are talking about, I have one question in this area (DNS). I don't understand why http tests resolve hostnames when you explicitly define ip address in test. Is this intention to make dns test together ? (I have small patch to disable it :-).

Thank for answer,

Regards,

-- Milan Kocian