Hi Matthew,

On 22-12-2011 00:27, Matthew Neumark wrote:

Currently I'm managing a XyMon server which consists of around 5,000 devices. We are looking to keep continually adding more and more devices to it as time goes on. The issue is our system is currently always using max system resources we keep allocating to the server. BTW devmon seems to be the highest system resource hog. Stats: About 5,000 devices XyMon 4.3.0-0-beta2 DevMon 0.2 4 CPU(s) 16 GB RAM Suse Linux Enterprise 10 (32-Bit) 300GB Enterprise SAN Storage - Fiber Channel - (3 Years of archived data stored) Do it do me any good to give the system more resources? CPU(s) or RAM? What is the experience that other users have with monitoring this many devices? What system configurations are you using to support this many monitored hosts?

Your installation is about the same size as the one I have at work. I recently upgraded it because it could no longer keep up with the load, and based on that I would say that your hardware specs should be more than adequate to handle the number of devices.

The only real difference between your system and mine is that I changed to SSD disks for storing the RRD-files (graphs) - I don't know how your FC disks compare with SSD's, but I could certainly see a significant effect of that change; when stopping Xymon it used to take 15-20 minutes for xymond_rrd to flush all of the cached RRD updates to disk, but after changing to the SSD disks it only takes a few seconds. The interesting thing of course is how long they will last, since the number of write operations is limited on these devices; I plan to replace them once a year to be on the safe side.

Have a look at your vmstat1 graph for the Xymon server (on the "trends" status page), and see how much time is being spent in I/O wait state - if that is in the 20-25% range, then you probably have a problem with I/O bandwidth, and adding an SSD disk could help. (I say 20-25% because as far as I know, Linux sends all I/O operations through one CPU, so if you have 4 CPU's and one of them is fully busy doing I/O, then it will show up in vmstat as I/O wait taking up 1/4 if the time).

Is there any swap being used ? (Check the "free" output). I wouldn't expect that there is much swap going on with 16 GB of RAM. So more RAM probably will not help.

I've never used Devmon, so I don't know how much of a "hog" it is. If it really is the one using all of the ressources, a solution (or workaround, really) might be to split the Devmon load between more servers - you can still have them report their data to the same Xymon server, you will only move the running of Devmon to a different node.

Just for the record, my current system is an HP DL380 G7, 2 dual-core 2.4 GHz CPU's, 24 GB RAM, 6x300 GB SAS 10K diske in a RAID-1 configuration, and 2 64 GB SSD disks in RAID-1. It is currently handling about 4200 servers with clients installed, and an additional 3000 entries in hosts.cfg for network devices, websites etc. All in all 50000 statuses being tracked. On average, the CPU load is 6% busy. To be fair, I must add that I have most of the network tests running on another node (for ease of firewall setup, mostly) and that node is 15% busy.

Regards, Henrik