So you are using a custom script to monitor instead of hobbitclient-*.sh ? This really isn't an option for me, since I have literally dozens of servers that all have the same hobbit homedir, although I have written some custom scripts that check the hostname and only run if they are launched on the host that they need to be on.

I think I will just have the oncall persons manually edit hobbit-clients.cfg in the case of a failover (oncall gets woken up anyhow). They can just uncomment/comment definitions for whichever host is the master.

It would be nice if you could set dependencies for PROC tests, then I could just make all of the PROC tests dependant upon something, like one of the failover daemons, or a flag on the filesystem, etc.

-Charles

Haertig, David F (Dave) wrote:

I do this with a custom monitoring script (I don't use the standard Hobbit 'procs' test).

There should be something you can check via script that tells you if a server is primary or not. In my case, a database filesystem is mounting on the primary but not on the secondary. So my script uses 'df' to look for that filesystem. You could use 'mount' as well. If that database filesystem is mounted the script does the normal test for processes and reports red/green. But if it's not mounted, the script reports a clear condition.

-----Original Message----- From: Charles Jones [mailto:jonescr at cisco.com] Sent: Friday, September 21, 2007 1:12 PM To: hobbit at hswn.dk Subject: [hobbit] Advice on how to handle HA monitoring

We have 2 hosts, HostA and HostB. They are part of an HA cluster via HP ServiceGuard. There is a virtual IP and DNS name of "virtual" that automatically goes to whichever of HostA and HostB is the primary at the time.

I am currently monitoring both HostA and HostB via Hobbit. Currently HostA is the primary, and I am doing various PROC checks. Currently on HostB, I am not doing process checks.

My problem is, how do I smoothly handle a failover scenario (HostB becoming the primary)? When a failover occurs, all of the procs on HostA are stopped (either by the server crashing, or manualy by ServiceGuard), and the same procs are started up on HostB.

I'm trying to think of ways to monitor both hosts, but only monitor procs on the one that is primary. So far the best I can come up with is to run the hobbit clients in local mode, and maybe have the ServiceGuard scripts swap out the config files and restart the Hobbit clients when there is a failover. That would probably work, BUT in this case the Hobbit homdir is also the same (SAN mount) on both machines, so moving or editing a file on one does the same on the other :(

Simply shutting down the hobbit client on the non-primary is not an option, as then it would no longer be monitored at all.

Any ideas? :)

-Charles

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

On 9/21/07, Charles Jones <jonescr at cisco.com> wrote:

So you are using a custom script to monitor instead of hobbitclient-*.sh ? This really isn't an option for me, since I have literally dozens of servers that all have the same hobbit homedir, although I have written

It does not matter. You can still write an external script like this

[external] ENVFILE /path/to/local/etc/hobbitclient.cfg CMD /path/to/local/external/script LOGFILE /path/to/local/logs/external.log INTERVAL 5m

That will work just fine even though you are sharing same hobbit client

some custom scripts that check the hostname and only run if they are launched on the host that they need to be on.

I think I will just have the oncall persons manually edit hobbit-clients.cfg in the case of a failover (oncall gets woken up anyhow). They can just uncomment/comment definitions for whichever host is the master.

Ouch!

It would be nice if you could set dependencies for PROC tests, then I could just make all of the PROC tests dependant upon something, like one of the failover daemons, or a flag on the filesystem, etc.

-Charles

Haertig, David F (Dave) wrote:

...

I do this with a custom monitoring script (I don't use the standard Hobbit 'procs' test).

There should be something you can check via script that tells you if a server is primary or not. In my case, a database filesystem is mounting on the primary but not on the secondary. So my script uses 'df' to look for that filesystem. You could use 'mount' as well. If that database filesystem is mounted the script does the normal test for processes and reports red/green. But if it's not mounted, the script reports a clear condition.

-----Original Message----- From: Charles Jones [mailto:jonescr at cisco.com] Sent: Friday, September 21, 2007 1:12 PM To: hobbit at hswn.dk Subject: [hobbit] Advice on how to handle HA monitoring

We have 2 hosts, HostA and HostB. They are part of an HA cluster via HP ServiceGuard. There is a virtual IP and DNS name of "virtual" that automatically goes to whichever of HostA and HostB is the primary at the time.

I am currently monitoring both HostA and HostB via Hobbit. Currently HostA is the primary, and I am doing various PROC checks. Currently on HostB, I am not doing process checks.

My problem is, how do I smoothly handle a failover scenario (HostB becoming the primary)? When a failover occurs, all of the procs on HostA are stopped (either by the server crashing, or manualy by ServiceGuard), and the same procs are started up on HostB.

I'm trying to think of ways to monitor both hosts, but only monitor procs on the one that is primary. So far the best I can come up with is to run the hobbit clients in local mode, and maybe have the ServiceGuard scripts swap out the config files and restart the Hobbit clients when there is a failover. That would probably work, BUT in this case the Hobbit homdir is also the same (SAN mount) on both machines, so moving or editing a file on one does the same on the other :(

Simply shutting down the hobbit client on the non-primary is not an option, as then it would no longer be monitored at all.

Any ideas? :)

-Charles

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu

On Friday 21 September 2007, Haertig, David F (Dave) wrote:

I do this with a custom monitoring script (I don't use the standard Hobbit 'procs' test).

There should be something you can check via script that tells you if a server is primary or not. In my case, a database filesystem is mounting on the primary but not on the secondary. So my script uses 'df' to look for that filesystem. You could use 'mount' as well. If that database filesystem is mounted the script does the normal test for processes and reports red/green. But if it's not mounted, the script reports a clear condition. I do almost the same for AIX clustering. I have a config file so I know the ip adres of each resource group (this config file is generated based on the cluster configuration). I run a script on each host that checks the ip adres and if an ip adres is found in the config file, the MACHINE environment is changed to the name of the resource group and the hobbitclient.sh script is called. That way I have an extra hobbitclient.sh execute in the cluster per resource group that sends info to the hobbit server and you can define whatever checks you want based on the name of the resource group. To make it more complicated, I can specify extra scripts to run per resource group in the config file.

Stef

Buchan Milne wrote:

What I do is to do all the infrastructure tests on both nodes, the procs checks check only for the cluster intrastructure (besides local processes like say syslog), then a check using the cluster software on both nodes, and finally monitor the network services (and only them) on the clustered service's IP.

How are you monitoring only those, I assuming using the nodisk, noconn, etc tags?

And, there are a few reasons I avoid installing software on the SAN, this is one of them ... (the other is that if you screw up upgrading your clustered software, you have an outage, whereas if it's installed locally on each box, you could just fail back ... or in most cases the cluster middleware would have done it already).

I'm in total agreement about the SAN issue. Sadly in my case, it is not my choice. At least I have found that for the most part, Hobbit gracefully handles using a shared homedir, as it adds the hostname to the filename of the various files the clients generate.

-Charles