On Wed, Apr 10, 2013 at 5:51 PM, White, Bruce <bewhite at fellowes.com> wrote:
Over 1000 devices monitored here and only real issue is rrd keeping up. I have been told an ssd for the rrd files will solve this issue.
~2000 hosts and that will double or triple in the next few weeks. I really don't see any IO issues in the slightest. 6 x 15k RPM SCSI drives in Raid 5 on a Dell PowerEdge 2950 with 8 gigs of ram and the thing is snoring (LA: 0.25)
Regards, Cami
We're currently processing ~2K incoming messages a second on a single xymond instance. This is a pretty beefy box, but it's also handling lots of other concurrent monitoring tasks that we're slowly moving over to xymon... including a non-fping-enabled Icinga install >.<
]# xymon localhost "xymondboard test=info fields=hostname" | wc -l 42459
(Not all of those are full hosts; some are application nodes with statuses being generated server-side out of client-side jvm stats or the like.)
At these levels it's important to ensure you're using whatever NUMA capabilities your system has properly, since message passing is basically just shoveling incoming TCP data around within memory. Also, you might want to tweak net.ipv4.ip_local_port_range and enable net.ipv4.tcp_tw_reuse and/or net.ipv4.tcp_tw_recycle on Linux to eke more simultaneous testing out of xymonnet.
One of the beauties of Xymon's architecture is the ability to cleanly disconnect the components... Xymongen can run on some other box, xymond_locator can be used to send rrd data off somewhere if IO becomes an issue, xymonnet pollers can be distributed, and xymonproxy can be used as needed to aggregate and smooth out incoming status reports, etc.
There are lots of different mechanisms for "scaling" efficiently depending on your particular needs, but I'd bet that on decently modern server hardware you'll probably want to scale for HA purposes long before you actually /need/ the additional power.
HTH,
-jc