On Thu, Feb 05, 2009 at 12:25:48PM -0500, David Paper wrote:

I've got a bb-hosts file with 8 server process instances getting tested.
Each instance gets tested with 3 HTTP requests (2 GET, 1 POST). All 8 server processes live on the same physical OS instance. This results in 24 HTTP requests getting sent from hobbit within 1/100th of a second.
This causes the load on the host to spike, and generates contention w/in each server to satisfy the requests. This same setup is repeated for hundreds of hosts and hundreds of processes.

Yeah, Xymon can be pretty agressive about testing network services. It's unfortunate that you hit the same server with multiple requests at once, although it is a bit unusual that a web/application server has a problem with just 24 requests simultaneously.

But of course, it depends on what your web application does.

Is there a way to tell hobbit to take all of the entries in bb-hosts and test them in a random order w/in the 1 minute testing interval? This would end up staggering the arrival of each HTTP test somewhat and lessen contention within each HTTP server and on each host.

I'm afraid not, at least not directly.

What you *could do was to setup more than one task to run the network tests, using the "NET:foo" definition in bb-hosts to create groups of tests than can be run simultaneously. From you description it sounds as if you have 8 "pseudo-hosts" defined in bb-hosts, each of them with 3 HTTP tests. I.e.

0.0.0.0 web1 # http://web1/test1 http://web1/test2
post;http://web1/test3;blababla 0.0.0.0 web2 # http://web2/test1 http://web2/test2
post;http://web2/test3;blababla

etc. for your 8 server processes. Is that correct ? Or would it at least be possible to configure it that way ?

If you can do that, then you can assign each of web1, web2, web3 and so on a different NET: setting, and then you run the checks for each of these NET's one at a time.

E.g. if you want to run the tests for "web1" and "web2" separately from "web3" and "web4", then your bb-hosts file would be like this:

0.0.0.0 web1 # http://web1/test1 http://web1/test2
post;http://web1/test3;blababla
NET:testgroup1 0.0.0.0 web2 # http://web2/test1 http://web2/test2
post;http://web2/test3;blababla NET:testgroup1 0.0.0.0 web3 # http://web3/test1 http://web3/test2
post;http://web3/test3;blababla NET:testgroup2 0.0.0.0 web4 # http://web4/test1 http://web4/test2
post;http://web4/test3;blababla NET:testgroup2

So you have two "NET:" groups of tests - "testgroup1" and "testgroup2". Then you change to [bbnet] task to run this script "runnetworktests.sh" instead of just launching bbtest-net:

#!/bin/sh

Run each NET group of tests separately

This script passes all commandline options to bbtest-net

BBLOCATION=testgroup1 bbtest-net $* BBLOCATION=testgroup2 bbtest-net $*

Finally, run the tests that have no NET definition

bbtest-net --test-untagged $*

exit 0

The [bbnet] task (in hobbitlaunch.cfg) would then have

CMD runnetworktests.sh --report --ping --checkresponse

instead of the default "CMD bbtest-net --report --ping --checkresponse"

The only "problem" with this is that you get to do the configuration of what tests can run simultaneously by hand. And each invocation of bbtest-net has to parse all of the bb-hosts file, there is a small overhead in doing that.

Regards, Henrik