[hobbit] Runtime longer than time limit (300)

Chris Wopat wrote:

Unsure if this is a bug or not. I noticed that for a few days I've been getting "Runtime 483 longer than time limit (300)" and the bbtest column shows:

TIME SPENT <snip> DNS tests executed 450.074868 <snip> TIME TOTAL 482.741716

I was able to successfully debug this by adding "--debug" to the [bbnet] CMD in hobbitlaunch.cfg. This revealed:

2009-09-01 08:26:01 ares_search: tlookup='dns.server.com', class=1, type=1 2009-09-01 08:26:01 Processing 0 DNS lookups with ARES 2009-09-01 08:33:31 Finished ARES queue after loop 20

Note that it took 7 minutes to complete this. Where 'dns.server.com' is a server that we took offline on the day the issue began. We didn't remove this from bb-hosts and instead marked it blue.

The fix was to comment it out from bb-hosts. It says "loop 20" above so I was searching for config options that listed 20 as a value but didn't come up with anything.

The issue is resolved on my end but I'd like to find out if this is a bug or expected behavior. It seems that a single DNS check on a host that's down shouldn't delay things by 7 minutes.

I'm running Xymon v4.2.3 on FreeBSD 7.2-RELEASE.

I've also tuned my DNS tests. The [bbnet] section in hobbitlaunch.cfg is the place to do it. See 'man bbtest-net' for options. I've upped the general timeout because I have a slow web server. I might shift known ones to urlplus because I can do custom timeouts on each server. My settings are: [bbnet] ENVFILE /usr/lib/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD bbtest-net --report --ping --checkresponse --dns-timeout=15 --timeout=20 --concurrency=512 LOGFILE $BBSERVERLOGS/bb-network.log INTERVAL 5m

Check the 'bbtest' column for your hobbit server for stats on bbtest-net runtimes - there's an RRD graph in there too.

David.

-- David Baldwin - IT Unit Australian Sports Commission www.ausport.gov.au Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616 david.baldwin at ausport.gov.au Leverrier Street Bruce ACT 2617

Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au

This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.