On Sat, Jan 27, 2007 at 06:41:36PM -0700, Alan Sparks wrote:
Have used Hobbit with BB clients at a previous gig, but now looking to set one up with Hobbit clients at a new gig. This will be an installation of close to 1000 nodes. Having started testing using the clients, have some concerns and questions:
- I am concerned that the server performs all the proc and disk tests... I'm concerned that several hundred servers will overwhelm the Hobbit server. I see the client can be configured to run the checks locally, provided the configuration is copied to the clients. Has anyone any practical experience with this many clients, and whether local checks are better in this case?
The client handling on the Hobbit server is almost exclusively CPU bound, so how many clients can be handled depends on the speed of your processor.
I'm gradually converting all of our Unix servers to use Hobbit clients. At the moment I have 295 hosts with the Hobbit client on them, and the hobbitd_client program (which takes care of processing the client data) is using ~0.3% CPU time. This is on a pretty old Hobbit server (a Sun E220 with a single, 450 MHz CPU - an equivalent x86 processor would be something like a 1 GHz PIII).
In other words, I doubt if it will be a problem. If I'm mistaken, then the next Hobbit version will let you split the load over multiple servers; this code is already in production use and is available in the current snapshots.
- Most of the servers are Web and database servers... meaning that port, proc, and netstat reports are very large. I've seen so far that the Hobbit client data sent to the server is almost 500k or larger, causing truncation problems and problems with the server-side checks (proc tests did not work until I used either client-local tests, or disabled the ports client data). Can client data be disabled without affecting the ability to run checks? Is there a way to streamline the data so that effective monitoring is possible, without sending 1/2 MB data for each client every 5 minutes?
The first thing to do would be to increase the max. size allowed for the "client" messages. See the hobbitserver.cfg(5) man-page, look for the MAXMSG_* settings.
You can remove the "netstat -an" command from the client-side scripts, or limit it to eg only listing the listener ports. Of course, you will not be able to use the PORT checks on the Hobbit server, but that's the trade-off you'll have to make.
Regards, Henrik