It would be nice to have an elegant solution, but I don't worry about it that much because 1) linux servers go down so infrequently, and 2) it would be pretty trivial to set up your own redundancy between hobbit servers. I can think of half a dozen ways to do it off the top of my head. For example:
Main Hobbit Server (MHS) does its thing normally. Backup Hobbit Server (BHS) syncs/mirrors the drive of the MHS server via rsync or whatever, and runs a copy of hobbit which monitors only one other system: the MHS. If the BHS detects that the MHS is down, the alert triggers a script that brings up its mirror copy of the server.
Doug
-----Original Message----- From: Josh Luthman [mailto:josh at imaginenetworksllc.com] Sent: Monday, June 16, 2008 1:58 PM To: hobbit at hswn.dk Subject: Re: [hobbit] grouping methods
This is quite obviously a well found problem and sought after feature
- getting redundant Hobbit servers.
Please help us, code monkeys =)
Josh
On Mon, Jun 16, 2008 at 1:45 PM, Sloan <joe at tmsusa.com> wrote:
Josh Luthman wrote:
Not sure what the real reasoning is behind this but if you
have 1000
servers monitored behind 3 hobbit servers each, figure one Hobbit server goes down you lost 1000/3000 being monitored. If you have 3000 servers being monitored behind 1 hobbit server, that one point of failure leaves you blind of all 3000 servers.
We do it with redundancy. Each server in our various data centers is monitored by two bb servers, with one of the two set up to send notifications, but in all other aspects the monitoring is active/active, and we get only one notification for alerts, rather than a pair of redundant notifications.
We've not had a bb server go down in all the years we've been using it, but sometimes wan connectivity goes away due to circumstances beyond our control, and a bb server in Arizona can't talk to the corresponding bb server in California, so the normally passive monitoring server goes into failover mode, and begins sending notification for alerts, since it can't verify that the other bb server is alive.
Thus, we always receive notifications for all alerts, and in the worst case we may get redundant notifications in the case of a split brain situation, which is the lesser of the evils.
Once this notification failover capability makes it into hobbit, we can finally switch from bb to hobbit.
Joe
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk