On Wed, Apr 26, 2006 at 03:24:15PM -0400, Schwimmer, Eric E *HS wrote:

We're monitoring 1420 IPs in hobbit, and it takes fping ~40 seconds to go through them all:

Is that a number you get from the "bbtest" status or from running fping by hand?

Are you doing other network tests in Hobbit than just ping? Hobbit does the ping tests in parallel with the other tests.

<snip> [root at hobbit fping]# fping -i5 -b12 -f ips -r1 -t250 -B2 -q -s

Are you using those parameters also on the FPING command in hobbitserver.cfg? Or is it just for your testing ?

Now, this seems a bit lengthy to me. I mean, if the avg round trip time is 5.83 ms, and there are 1430 hosts, should the total time in transit for all hosts should be 8336ms, or 8 seconds... right?

No, it should be less - because fping pings several hosts in parallel.

You have "-i5" which causes a 5 ms delay between each ping. So that's (5/1000)*1430 = 7.15 seconds where it does nothing. The default setting is "-i25" - i.e. 5 times higher - which would actually match your ~40 seconds nicely.

Don't forget that there is probably also some time spent doing ARP lookups for all of these IP's. Unless you have "testip" on all of the entries in bb-hosts (or run bbtest-net with "--dns=ip"), you'll also spend some time on DNS lookups (hint: use a local caching DNS server on the Hobbit server).

Even when I remove the hosts that aren't responding, the results on are par with those above.

Our polling interval is once every 60 seconds (which we want to maintain, because we like to know ASAP when something drops even one ping), so it's not a problem yet. We add hosts on a daily basis, however, so it will be a problem some time in the future and I'd like to fix it before it becomes a problem.

Well, the good news is that it probably won't become a problem. Because fping pings multiple hosts in parallel, the runtime doesn't change very much when you add more hosts.

If it does become an issue, spread the load. Setup an extra server to do half the network tests, and configure your bb-hosts file with "NET:net-a" and "NET:net-b" tags on the hosts. Then you set BBLOCATION="net-a" on one box, and "BBLOCATION=net-b" on the other. Then they'll only test those hosts where the NET:... setting matches. Unless it's an OS limitation, you could probably do that on a single box and just have two instances of the [bbnet] task in hobbitlaunch.cfg - instead of running bbtest-net directly, they would run a shell-script which sets the BBLOCATION environment just before running bbtest-net.

Regards, Henrik