Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running.
My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are.
-- Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374
Hi Michael,
At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4
That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7
Primary and standby external test VMs (WebLogic, custom tests) (same as above)
-----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Tuesday, May 22, 2012 1:01 PM To: xymon at xymon.com Subject: [Xymon] Most number of hosts
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running.
My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are.
-- Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Hi Michael
Here we are monitoring > 10 000 hosts over 18 hobbit clusters (1 cluster per datacenter) but we have one cluster with > 3600 hosts:
Hardware is
HP DL380 G7 2 x Xeon X5650 RAID 1 for OS RAID 5 for data with 4 SSD hard drives (165Gb) for xymon data. 8 Gb RAM.
xymongen:
Statistics: Hosts : 3692 Pages : 497 Status messages : 46298
- Red : 318 ( 0.69 %)
- Red (non-propagating) : 0 ( 0.00 %)
- Yellow : 347 ( 0.75 %)
- Yellow (non-propagating) : 0 ( 0.00 %)
- Clear : 902 ( 1.95 %)
- Green : 43537 (94.04 %)
- Purple : 82 ( 0.18 %)
- Blue : 1112 ( 2.40 %)
xymonnet:
Statistics: Hosts total : 3660 Hosts with no tests : 5 Total test count : 9645 Status messages : 10034 Alert status msgs : 0 Transmissions : 332
DNS statistics:
hostnames resolved : 1763
succesful : 628
failed : 6
calls to dnsresolve : 1727
TCP test statistics:
TCP tests total : 5993
HTTP tests : 2222
Simple TCP tests : 3771
Connection attempts : 5993
bytes written : 376789
bytes read : 25004690
[root at xxxxxxxx:/opt/xymon/server/etc/hosts.d/customer ] 21 grep -c proc /proc/cpuinfo 24
load average: 2.03, 1.97, 2.00
CPU %idle average: 90% !!!
you can have many hosts; but i think the most important things is the number of check per hosts and the external scripts running on your xymon server. (oracle check; backup check, reporting, etc) which increases the load.
Having SSD is a good option because in the past, the main load was I/O wait.
Cheers Nico
Le 22 mai 2012 à 20:11, <greg.shea at emc.com> a écrit :
Hi Michael,
At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4
That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7
Primary and standby external test VMs (WebLogic, custom tests) (same as above)
-----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Tuesday, May 22, 2012 1:01 PM To: xymon at xymon.com Subject: [Xymon] Most number of hosts
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running.
My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are.
-- Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
hi nico
how did you setup your hobbit cluster ? hearbeat/pacemaker + drdb , corosync/pacemaker + drdb, keapalived/opensvc + lvm ?
Regards, Wilfrid :)
Le 2012-05-22 20:46, Nico a écrit :
Hi Michael
Here we are monitoring > 10 000 hosts over 18 hobbit clusters (1 cluster per datacenter) but we have one cluster with > 3600 hosts:
Hardware is
HP DL380 G7 2 x Xeon X5650 RAID 1 for OS RAID 5 for data with 4 SSD hard drives (165Gb) for xymon data. 8 Gb RAM.
xymongen:
Statistics: Hosts : 3692 Pages : 497 Status messages : 46298
- Red : 318 ( 0.69 %)
- Red (non-propagating) : 0 ( 0.00 %)
- Yellow : 347 ( 0.75 %)
- Yellow (non-propagating) : 0 ( 0.00 %)
- Clear : 902 ( 1.95 %)
- Green : 43537 (94.04 %)
- Purple : 82 ( 0.18 %)
- Blue : 1112 ( 2.40 %)
xymonnet:
Statistics: Hosts total : 3660 Hosts with no tests : 5 Total test count : 9645 Status messages : 10034 Alert status msgs : 0 Transmissions : 332
DNS statistics:
hostnames resolved : 1763
succesful : 628
failed : 6
calls to dnsresolve : 1727
TCP test statistics:
TCP tests total : 5993
HTTP tests : 2222
Simple TCP tests : 3771
Connection attempts : 5993
bytes written : 376789
bytes read : 25004690
[root at xxxxxxxx:/opt/xymon/server/etc/hosts.d/customer ] 21 grep -c proc /proc/cpuinfo 24
load average: 2.03, 1.97, 2.00
CPU %idle average: 90% !!!
you can have many hosts; but i think the most important things is the number of check per hosts and the external scripts running on your xymon server. (oracle check; backup check, reporting, etc) which increases the load.
Having SSD is a good option because in the past, the main load was I/O wait.
Cheers Nico
Le 22 mai 2012 à 20:11, <greg.shea at emc.com [5]> a écrit :
Hi Michael,
At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4
That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7
Primary and standby external test VMs (WebLogic, custom tests) (same as above)
-----Original Message----- From: xymon-bounces at xymon.com [1] [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Tuesday, May 22, 2012 1:01 PM To: xymon at xymon.com [2] Subject: [Xymon] Most number of hosts
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running.
My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are.
-- Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com [3] 216-515-7374
Xymon mailing list Xymon at xymon.com [4] http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Links:
[1] mailto:xymon-bounces at xymon.com [2] mailto:xymon at xymon.com [3] mailto:michael.beatty at sherwin.com [4] mailto:Xymon at xymon.com [5] mailto:greg.shea at emc.com
Heartbeat/LVM/DRBD :)
Envoyé de mon iPhone
Le 25 mai 2012 à 22:05, wbeauchaine <wbeauchaine at katei.net> a écrit :
hi nico
how did you setup your hobbit cluster ? hearbeat/pacemaker + drdb , corosync/pacemaker + drdb, keapalived/opensvc + lvm ?
Regards, Wilfrid :)
Le 2012-05-22 20:46, Nico a écrit :
Hi Michael
Here we are monitoring > 10 000 hosts over 18 hobbit clusters (1 cluster per datacenter) but we have one cluster with > 3600 hosts:
Hardware is
HP DL380 G7 2 x Xeon X5650 RAID 1 for OS RAID 5 for data with 4 SSD hard drives (165Gb) for xymon data. 8 Gb RAM.
xymongen:
Statistics: Hosts : 3692 Pages : 497 Status messages : 46298
- Red : 318 ( 0.69 %)
- Red (non-propagating) : 0 ( 0.00 %)
- Yellow : 347 ( 0.75 %)
- Yellow (non-propagating) : 0 ( 0.00 %)
- Clear : 902 ( 1.95 %)
- Green : 43537 (94.04 %)
- Purple : 82 ( 0.18 %)
- Blue : 1112 ( 2.40 %)
xymonnet:
Statistics: Hosts total : 3660 Hosts with no tests : 5 Total test count : 9645 Status messages : 10034 Alert status msgs : 0 Transmissions : 332
DNS statistics:
hostnames resolved : 1763
succesful : 628
failed : 6
calls to dnsresolve : 1727
TCP test statistics:
TCP tests total : 5993
HTTP tests : 2222
Simple TCP tests : 3771
Connection attempts : 5993
bytes written : 376789
bytes read : 25004690
[root at xxxxxxxx:/opt/xymon/server/etc/hosts.d/customer ] 21 grep -c proc /proc/cpuinfo 24
load average: 2.03, 1.97, 2.00
CPU %idle average: 90% !!!
you can have many hosts; but i think the most important things is the number of check per hosts and the external scripts running on your xymon server. (oracle check; backup check, reporting, etc) which increases the load.
Having SSD is a good option because in the past, the main load was I/O wait.
Cheers Nico
Le 22 mai 2012 à 20:11, <greg.shea at emc.com [5]> a écrit :
Hi Michael,
At one point, I had 6115 hosts running from 1 physical Xymon server. Some tests, specifically WebLogic, I run from a separate VM so if the java process hangs, it doesn't affect the main Xymon server. Physical HW Dell PE 2950 2 Xeon 5110 CPUs 8GB RAM 2 146GB, 10K RPM SAS (RAID1 root disk) 2 146GB. 15K RPM SAS (apps & data) Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Xymon 4.3.4
That's all changing now because of the push to virtualize everything, so I used the opportunity to build in some redundancy. Currently we have 2750 hosts running in this environment and will be migrating the rest of the old server over before EOY. Primary and standby VMs 2 Virtual Xeon E7549 CPUs 8GB RAM 120GB root disk and 200GB apps disk Red Hat Enterprise Linux Server release 5.5 (Tikanga) Xymon 4.3.7
Primary and standby external test VMs (WebLogic, custom tests) (same as above)
-----Original Message----- From: xymon-bounces at xymon.com [1] [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Tuesday, May 22, 2012 1:01 PM To: xymon at xymon.com [2] Subject: [Xymon] Most number of hosts
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running.
My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I'm hoping to get some confidence that going into this project, I'm not going to be hit with a realization that Xymon simply cannot handle this type of environment. Its a lot of research and development that I would be flushing down the drain. I do have experience with my previous organization where we monitored about 1500 clients under the Hobbit project.... so I know the capabilities... I just don't know what the boundaries are.
-- Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com [3] 216-515-7374
Xymon mailing list Xymon at xymon.com [4] http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Links:
[1] mailto:xymon-bounces at xymon.com [2] mailto:xymon at xymon.com [3] mailto:michael.beatty at sherwin.com [4] mailto:Xymon at xymon.com [5] mailto:greg.shea at emc.com
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On 22-05-2012 19:00, Michael Beatty wrote:
Is there any close proximity number available as to how many clients Xymon can handle? I've seen in the wiki pages that one user is advertising over 10,000 however, that seems to be in a "clustered" setting. I see a few more users suggesting over 2,000. My concern lies in that my plans stand at some where in the middle of that, and also, there really is no gauge as to what hardware and network topology those systems are running.
My environment will be in the ball park of about 5,000 systems and many of them are on satellite bands with some latency concerns, pings tests come back at around 3 seconds.
I don't have much experience with high-latency connections - you'll obviously have to tweak some network timeout settings.
The main performance issue in Xymon these days are the disk I/O from updating all of the RRD files. Here are some numbers fresh off my main server:
$ iostat Linux 2.6.32-5-amd64 (myserver) 05/22/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle 1.95 0.06 0.79 0.65 0.00 96.56
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn cciss/c0d0 5.81 40.83 142.54 23238617 81132930 cciss/c0d1 79.20 31.19 3220.84 17755352 1833330720
It's been up for close to a week. Both of these disks are RAID-1 devices; c0d0 are normal 10K RPM disks, c0d1 are SSD disks and holds ONLY the xymon/rrd and xymon/tmp directories. As you can see, the write activity on the SSD disks is huge - they are about a year old now, and I guess I'll have to replace them soon.
CPU- and memory-wise, this box is doing nothing at all. It has 2 quad-core Xeon's and 24 GB RAM, but it only uses about 6 GB - the rest is used for disk cache (the way Linux does automatically).
Since the box has so much RAM and the RRD files only take up about 8 GB right now, I have been thinking about moving them to a RAM disk, and just rsync'ing them to the SSD disks once an hour or so.
It handles about 4000 full clients, and another 2500 for which we only do network tests.
Regards, Henrik
participants (5)
-
greg.shea＠emc.com
-
henrik＠hswn.dk
-
Michael.Beatty＠sherwin.com
-
nicolas＠lienard.name
-
wbeauchaine＠katei.net