New Hobbit stuff: Scalability and H/A work
A couple of weeks ago, I was asked if our Hobbit system at work could handle monitoring of one more customer. Of course, I said - no problem. Well, there was one gotcha: This customer has 1100+ servers that need to be monitored. Which means my Hobbit installation is about to double in the number of hosts monitored. Hmm ...
This will be interesting to watch. I am fairly confident that Hobbit can handle it, with one exception: The disks on my Hobbit server will be overloaded. It already spends about 50% of it's time in I/O wait, so doubling the number of hosts with cpu/memory/disk etc. graphs will probably crash it.
So something needs to be done - fast: These hosts should go into our Hobbit before Christmas. That is why I currently may seem a bit absent from the mailing list.
The way I plan to handle it will be by distributing the load of RRD updates onto several servers, each handling a subset of the total set of hosts; Hobbit will automatically detect which of the 3 RRD-servers handles a specific host and direct rrd-updates to that server. New hosts are distributed across all of the RRD servers in a weighted round-robin fashion.
The way this is going to be done means that it can be used not only for distributing the load of the RRD file updates, but also for distributing the other hobbitd_* modules (alerting, history logs, client data processing etc). In other words, this will be a major win for Hobbit in large installations.
It also has one more benefit: I think this can be evolved to handle automatic failover, so you can run multiple Hobbit servers that process the same data - meaning all of the on-disk data will be identical across all of the Hobbit servers. This should make it possible to setup a group of Hobbit servers for very high availability of the monitoring system. I haven't worked out all of the implementation details yet, but I think it is possible.
Regards, Henrik
Henrik Stoerner wrote:
A couple of weeks ago, I was asked if our Hobbit system at work could handle monitoring of one more customer. Of course, I said - no problem. Well, there was one gotcha: This customer has 1100+ servers that need to be monitored. Which means my Hobbit installation is about to double in the number of hosts monitored. Hmm ...
This will be interesting to watch. I am fairly confident that Hobbit can handle it, with one exception: The disks on my Hobbit server will be overloaded. It already spends about 50% of it's time in I/O wait, so doubling the number of hosts with cpu/memory/disk etc. graphs will probably crash it.
Strange that I/O seem to be an issue for you. What kind of system do you run the hobbit server on?
I have ~3300 rrd files updated here on a blade with an old 40 Gig 5400 2,5" hard drive and it is almost idle.
Cheers, Gildas
On Tue, Oct 31, 2006 at 04:48:34PM +0000, Gildas Le Nadan wrote:
Henrik Stoerner wrote:
This will be interesting to watch. I am fairly confident that Hobbit can handle it, with one exception: The disks on my Hobbit server will be overloaded. It already spends about 50% of it's time in I/O wait, so doubling the number of hosts with cpu/memory/disk etc. graphs will probably crash it.
Strange that I/O seem to be an issue for you. What kind of system do you run the hobbit server on?
The server is an oldish Sun E220 with two new 72 GB SCSI disks, 10K rpms.
I have ~3300 rrd files updated here on a blade with an old 40 Gig 5400 2,5" hard drive and it is almost idle.
I have about 25000 RRD files :-) That's about 80 updates every second.
Regards, Henrik
From: henrik at hswn.dk (Henrik Stoerner) Reply-To: hobbit at hswn.dk To: hobbit at hswn.dk Subject: [hobbit] New Hobbit stuff: Scalability and H/A work Date: Tue, 31 Oct 2006 17:03:34 +0100
A couple of weeks ago, I was asked if our Hobbit system at work could handle monitoring of one more customer. Of course, I said - no problem. Well, there was one gotcha: This customer has 1100+ servers that need to be monitored. Which means my Hobbit installation is about to double in the number of hosts monitored. Hmm ...
This will be interesting to watch. I am fairly confident that Hobbit can handle it, with one exception: The disks on my Hobbit server will be overloaded. It already spends about 50% of it's time in I/O wait, so doubling the number of hosts with cpu/memory/disk etc. graphs will probably crash it.
So something needs to be done - fast: These hosts should go into our Hobbit before Christmas. That is why I currently may seem a bit absent from the mailing list.
The way I plan to handle it will be by distributing the load of RRD updates onto several servers, each handling a subset of the total set of hosts; Hobbit will automatically detect which of the 3 RRD-servers handles a specific host and direct rrd-updates to that server. New hosts are distributed across all of the RRD servers in a weighted round-robin fashion.
So this is load balancing, each hobbit server keep a portion all-hosts's rrd files.
The way this is going to be done means that it can be used not only for distributing the load of the RRD file updates, but also for distributing the other hobbitd_* modules (alerting, history logs, client data processing etc). In other words, this will be a major win for Hobbit in large installations.
It also has one more benefit: I think this can be evolved to handle automatic failover, so you can run multiple Hobbit servers that process the same data - meaning all of the on-disk data will be identical across all of the Hobbit servers. This should make it possible to setup a group of Hobbit servers for very high availability of the monitoring system. I haven't worked out all of the implementation details yet, but I think it is possible.
Please check out FTSha(R1), the opensource cluster/ha for Solaris. So the only part you need to work on is the "Distributed BB messages distribution".
R1: http://www.fstha.com/compare.html
Regards
tj
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Stay in touch with old friends and meet new ones with Windows Live Spaces http://clk.atdmt.com/MSN/go/msnnkwsp0070000001msn/direct/01/?href=http://spa...
I have created an 'ext' script that is set in the clientlaunch.cfg to run on an interval of 7 days "INTERVAL 7d"
After it has run, going GREEN as it should, in about 15 minutes it goes purple, and will go Red shortly thereafter. But it is not scheduled to run for 7 days.
I even tried putting an entry on the server-side: hobbit-clients.cfg: sysdb 8d
This to no avail.
What should I do to get it to only have this go red or purple if it fails on the next time it is run? It appears that every time the hobbit client reports, that it views this weekly scheduled event as a failure.
--
Shawn Saunders MIST Team member Jet Propulsion Laboratory 4800 Oak Grove Drive Pasadena, CA 91109 MS 230-310 Office: (818) 354-4667 / 230-310A Mobile: (818) 458-0789 FAX: (818) 393-9022
Any fool can make things bigger, more complex, and more violent. It takes a touch of genius-and a lot of courage-to move in the opposite direction. Albert Einstein
When you send your status message, use 'status+7d'.
The exact syntax is on the man page for the bb command.
Shawn Saunders wrote:
I have created an 'ext' script that is set in the clientlaunch.cfg to run on an interval of 7 days "INTERVAL 7d"
After it has run, going GREEN as it should, in about 15 minutes it goes purple, and will go Red shortly thereafter. But it is not scheduled to run for 7 days.
I even tried putting an entry on the server-side: hobbit-clients.cfg: sysdb 8d
This to no avail. What should I do to get it to only have this go red or purple if it fails on the next time it is run? It appears that every time the hobbit client reports, that it views this weekly scheduled event as a failure.
-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com
Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007
oh oh oh !!! loook verrry interesting :)))
i have 2 questions: -if the heavy work (rrd updates) is distributed on 2 or more dedicated "rrd servers", the data wont be redondant on each of these "rrd server", right ? -what would happen if an "rrd server" crashes ??
recently i changed my 2 hobbit servers, which now have 2G of ram each would it make sense to have hobbit evolve and take full advantage of that memory ?
olivier
-----Message d'origine----- De : Henrik Stoerner [mailto:henrik at hswn.dk] Envoyé : mardi 31 octobre 2006 17:04 À : hobbit at hswn.dk Objet : [hobbit] New Hobbit stuff: Scalability and H/A work
A couple of weeks ago, I was asked if our Hobbit system at work could handle monitoring of one more customer. Of course, I said - no problem. Well, there was one gotcha: This customer has 1100+ servers that need to be monitored. Which means my Hobbit installation is about to double in the number of hosts monitored. Hmm ...
This will be interesting to watch. I am fairly confident that Hobbit can handle it, with one exception: The disks on my Hobbit server will be overloaded. It already spends about 50% of it's time in I/O wait, so doubling the number of hosts with cpu/memory/disk etc. graphs will probably crash it.
So something needs to be done - fast: These hosts should go into our Hobbit before Christmas. That is why I currently may seem a bit absent from the mailing list.
The way I plan to handle it will be by distributing the load of RRD updates onto several servers, each handling a subset of the total set of hosts; Hobbit will automatically detect which of the 3 RRD-servers handles a specific host and direct rrd-updates to that server. New hosts are distributed across all of the RRD servers in a weighted round-robin fashion.
The way this is going to be done means that it can be used not only for distributing the load of the RRD file updates, but also for distributing the other hobbitd_* modules (alerting, history logs, client data processing etc). In other words, this will be a major win for Hobbit in large installations.
It also has one more benefit: I think this can be evolved to handle automatic failover, so you can run multiple Hobbit servers that process the same data - meaning all of the on-disk data will be identical across all of the Hobbit servers. This should make it possible to setup a group of Hobbit servers for very high availability of the monitoring system. I haven't worked out all of the implementation details yet, but I think it is possible.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
participants (6)
-
gn1＠sanger.ac.uk
-
henrik＠hswn.dk
-
olivier.beau＠telecomitalia.fr
-
rsmrcina＠wi.rr.com
-
shawn.saunders＠jpl.nasa.gov
-
tj_yang＠hotmail.com