Hobbit client executing a script to be proactive if a problem occurs?
Hello,
I've been running hobbit for a few years, after converting from Big Brother. I actively use it to monitor hundreds of hosts, send alerts, etc. However, one thing I have yet to do is use any custom scripts for anything. Looking at docs, I see two ways that scripts are called:
an alert can call a script instead of email directly. This script is passed variables, and just about anything can be done *on the hobbit server* from here.
A script can be launched on the client, from ext/, when the hobbit clien starts. This script is generally used to add another column to hobbit.
What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
Thanks --Chris
Hi Chris,
I think it really depends on what you are testing? If you are using the standard hobbit client and the standard tests, most of the client side is pretty basic, I guess you could call it a dumb client in a way as it does a simple job of pulling the data out and sends it on without any intelligent decisions being made about thresholds etc.
To do what you want, you either have to do as you say (set up keys from the server etc and have the server perform an action after a threshold breach from a script initiated/configured in the hobbit-alerts file.
Or, which would be much simpler, put some code in your monitoring script to take action, but then you are starting to move away from the simplicity of hobbit. It decomes even harder if you want to take action based on something picked up in the standard tests (like CPU that you mentioned in your post). You may need to write your own test/new column that monitors the same metric but in a different light. In my view, an automated action based on a detected event probably does not belong in the monitoring system. If a failure can be expected and an automated action is known to fix the issue, perhaps that should be built into the startup process of the application (a watchdog process etc). Hobbit can then be used to monitor the log for a restart event, or a failed restart event etc. Actually, thinking about it more, building the intelligent action into the agent is an ok idea and you also have the opportunity of capturing and transmitting additional information about why something dies if you run an action to fix and it failed etc. I am waffling... You still need extra security based configuration steps on the client with sudo or ssh anyway to get around access permission to restart something anyway as your client is running as the hobbit userid so this brings the client configuration closer to an ssh setup on the server. I don't think either way is perfect but both would do what you want...
Perhaps this is something that belongs on a request for feature list of a future release of hobbit.
The hobbit client installation to configure sudo to allow it to run commands as other users (on admins acceptance during the installation of course). The ability of the hobbit server to send a series of actions to the hobbit client for execution via the hobbit communication channel. Sounds like something that could have lots of uses if done well...
Thoughts?
Cheers
Phil
2008/4/11 Chris Wopat <chrisw at supranet.net>:
Hello,
I've been running hobbit for a few years, after converting from Big Brother. I actively use it to monitor hundreds of hosts, send alerts, etc. However, one thing I have yet to do is use any custom scripts for anything. Looking at docs, I see two ways that scripts are called:
an alert can call a script instead of email directly. This script is passed variables, and just about anything can be done *on the hobbit server* from here.
A script can be launched on the client, from ext/, when the hobbit clien starts. This script is generally used to add another column to hobbit.
What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
Thanks --Chris
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Tel: 0400 466 952 Fax: 0433 123 226 email: philwild at gmail.com
Phil Wild wrote:
Hi Chris,
I think it really depends on what you are testing? If you are using the standard hobbit client and the standard tests, most of the client side is pretty basic, I guess you could call it a dumb client in a way as it does a simple job of pulling the data out and sends it on without any intelligent decisions being made about thresholds etc.
To do what you want, you either have to do as you say (set up keys from the server etc and have the server perform an action after a threshold breach from a script initiated/configured in the hobbit-alerts file.
Bummer, I was hoping there was perhaps a barely documented feature that would let you exec a script on the client to make my life easier.
Or, which would be much simpler, put some code in your monitoring script to take action, but then you are starting to move away from the simplicity of hobbit. It decomes even harder if you want to take action based on something picked up in the standard tests (like CPU that you mentioned in your post).
Indeed, my intent is for standard tests.
You may need to write your own test/new column that monitors the same metric but in a different light. In my view, an automated action based on a detected event probably does not belong in the monitoring system. If a failure can be expected and an automated action is known to fix the issue, perhaps that should be built into the startup process of the application (a watchdog process etc).
Indeed, a daemon shouldn't fail and it should run properly or be fixed natively. However, in the case of what I'm trying to monitor is an item that has a series of dependencies - Postfix, depending on Amavis, depending on p0f, depending on ClamAV, depending on greylist software, depending on database, etc. Under certain circumstances if one of these were to go down, it ends up snowballing to have high CPU, in my case.
The better way for me to handle this is to likely search logs for items, instead of relying on high CPU.
Hobbit can then be used to monitor the log for a restart event, or a failed restart event etc. Actually, thinking about it more, building the intelligent action into the agent is an ok idea and you also have the opportunity of capturing and transmitting additional information about why something dies if you run an action to fix and it failed etc.
For the scenario I laid out above, I intend to write a script that will restart the daemons properly in the correct order, but this is the "oh shit" script, and wouldn't be a system startup script, for example.
I am waffling... You still need extra security based configuration steps on the client with sudo or ssh anyway to get around access permission to restart something anyway as your client is running as the hobbit userid so this brings the client configuration closer to an ssh setup on the server. I don't think either way is perfect but both would do what you want...
Indeed. It sounds like generally the two scenarios I'd mentioned in my email are the way to get it to work, and whichever is most reliable/less hack-ish would be the best way to do it.
Perhaps this is something that belongs on a request for feature list of a future release of hobbit.
The hobbit client installation to configure sudo to allow it to run commands as other users (on admins acceptance during the installation of course). The ability of the hobbit server to send a series of actions to the hobbit client for execution via the hobbit communication channel. Sounds like something that could have lots of uses if done well...
I think this would be the perfect scenario. Something added to the hobbit client, that would go in 'localclient.cfg'. A simple 'SCRIPT' line that could be nested under a test, that would pass along whatever meaningful variables that could be useful, such as PID.
The hobbit server *already uses* 'setuid root' for some binaries, such as 'hobbitping'. The client would simply need to call some binary who's sole purpose is to launch scripts as root so essentially anything would be possible.
I can't think of any reason that some hook would have to exist for the script to tell the hobbit client anything back, I think it can just wait for the next poll period to see if it went back to green.
--Chris
David,
Sorry for your loss =( I know of many people that are being forced into Outlook.
I use puTTY and Gmail exclusively. I'm using rich text (Gmail will not send out HTML) which VI seems to like.
Anyways, just thought I'd help you eliminate issues - I think we can all agree Outlook is the cause of the copy/paste malfunction.
Josh
On Fri, Apr 11, 2008 at 1:17 PM, Chris Wopat <chrisw at supranet.net> wrote:
Phil Wild wrote:
Hi Chris,
I think it really depends on what you are testing? If you are using the standard hobbit client and the standard tests, most of the client side is pretty basic, I guess you could call it a dumb client in a way as it does a simple job of pulling the data out and sends it on without any intelligent decisions being made about thresholds etc.
To do what you want, you either have to do as you say (set up keys from the server etc and have the server perform an action after a threshold breach from a script initiated/configured in the hobbit-alerts file.
Bummer, I was hoping there was perhaps a barely documented feature that would let you exec a script on the client to make my life easier.
Or, which would be much simpler, put some code in your monitoring script
to take action, but then you are starting to move away from the simplicity of hobbit. It decomes even harder if you want to take action based on something picked up in the standard tests (like CPU that you mentioned in your post).
Indeed, my intent is for standard tests.
You may need to write your own test/new column that monitors the same
metric but in a different light. In my view, an automated action based on a detected event probably does not belong in the monitoring system. If a failure can be expected and an automated action is known to fix the issue, perhaps that should be built into the startup process of the application (a watchdog process etc).
Indeed, a daemon shouldn't fail and it should run properly or be fixed natively. However, in the case of what I'm trying to monitor is an item that has a series of dependencies - Postfix, depending on Amavis, depending on p0f, depending on ClamAV, depending on greylist software, depending on database, etc. Under certain circumstances if one of these were to go down, it ends up snowballing to have high CPU, in my case.
The better way for me to handle this is to likely search logs for items, instead of relying on high CPU.
Hobbit can then be used to monitor the log for a restart event, or a
failed restart event etc. Actually, thinking about it more, building the intelligent action into the agent is an ok idea and you also have the opportunity of capturing and transmitting additional information about why something dies if you run an action to fix and it failed etc.
For the scenario I laid out above, I intend to write a script that will restart the daemons properly in the correct order, but this is the "oh shit" script, and wouldn't be a system startup script, for example.
I am waffling... You still need extra security based configuration steps
on the client with sudo or ssh anyway to get around access permission to restart something anyway as your client is running as the hobbit userid so this brings the client configuration closer to an ssh setup on the server. I don't think either way is perfect but both would do what you want...
Indeed. It sounds like generally the two scenarios I'd mentioned in my email are the way to get it to work, and whichever is most reliable/less hack-ish would be the best way to do it.
Perhaps this is something that belongs on a request for feature list of a
future release of hobbit.
The hobbit client installation to configure sudo to allow it to run commands as other users (on admins acceptance during the installation of course). The ability of the hobbit server to send a series of actions to the hobbit client for execution via the hobbit communication channel. Sounds like something that could have lots of uses if done well...
I think this would be the perfect scenario. Something added to the hobbit client, that would go in 'localclient.cfg'. A simple 'SCRIPT' line that could be nested under a test, that would pass along whatever meaningful variables that could be useful, such as PID.
The hobbit server *already uses* 'setuid root' for some binaries, such as 'hobbitping'. The client would simply need to call some binary who's sole purpose is to launch scripts as root so essentially anything would be possible.
I can't think of any reason that some hook would have to exist for the script to tell the hobbit client anything back, I think it can just wait for the next poll period to see if it went back to green.
--Chris
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
Here is how I execute a remote "pkill" on a client. Replace "client_server" with your client hostname, and replace "client_userid" with the userid (on the client) that you want to run the script (pkill) under. Also, set up ssh pubkey authentication between the Hobbit server and client so that ssh does not prompt you for a password.
hobbit_alerts.cfg:
############################## $KILL_SCRIPT=/hobbit/server/ext/kill_script HOST=client_server SERVICE=procs COLOR=red SCRIPT $KILL_SCRIPT client_server,client_userid,grep_pattern ##############################
/hobbit/server/ext/kill_script:
##############################
#!/bin/perl
#$debug++; # Comment-out this line for normal operation
($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3);
exit unless ((defined $host) && (defined $userid) && (defined
$pattern));
$command = qq!ssh $userid\@$host "pkill -f '$pattern'"!;
if ($debug) {
$command =~ s/pkill/pgrep -l/;
print $command;
}
else {
$command;
}
##############################
And here is how I execute an arbitrary script on the client (in this example, "weblogics_kill_script"). It is a minor modification of the above:
hobbit_alerts.cfg:
############################## $REMOTE_SCRIPT=/hobbit/server/ext/remote_script HOST=ssdpws001 SERVICE=weblogics COLOR=red SCRIPT $REMOTE_SCRIPT client_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s cript ##############################
Note: weblogics_kill_script exists on the client - I use it to force a few thread dumps, archive logfiles, and then kill/restart weblogics
/hobbit/server/ext/remote_script:
##############################
#!/bin/perl
#$debug++; # Comment-out this line for normal operation
($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3);
exit unless ((defined $host) && (defined $userid) && (defined $script));
$command = qq!ssh $userid\@$host "$script"!;
if ($debug) {
print $command;
}
else {
$command;
}
##############################
-----Original Message----- From: Chris Wopat [mailto:chrisw at supranet.net] Sent: Friday, April 11, 2008 7:56 AM To: hobbit at hswn.dk Subject: [hobbit] Hobbit client executing a script to be proactive if a problem occurs?
Hello,
I've been running hobbit for a few years, after converting from Big Brother. I actively use it to monitor hundreds of hosts, send alerts, etc. However, one thing I have yet to do is use any custom scripts for anything. Looking at docs, I see two ways that scripts are called:
an alert can call a script instead of email directly. This script is passed variables, and just about anything can be done *on the hobbit server* from here.
A script can be launched on the client, from ext/, when the hobbit clien starts. This script is generally used to add another column to hobbit.
What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
Thanks --Chris
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Sorry about the formating of the code I sent. I don't know why I can't successfully cut-n-paste from vi on Linux into Outlook on Windows and not get screwed up format-wise. I guess if you know Hobbit alert syntax and PERL, they you can figure it out easy enough.
-----Original Message----- From: Haertig, David F (Dave) [mailto:haertig at avaya.com] Sent: Friday, April 11, 2008 10:30 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Hobbit client executing a script to be proactive if a problem occurs?
Here is how I execute a remote "pkill" on a client. Replace "client_server" with your client hostname, and replace "client_userid" with the userid (on the client) that you want to run the script (pkill) under. Also, set up ssh pubkey authentication between the Hobbit server and client so that ssh does not prompt you for a password.
hobbit_alerts.cfg:
############################## $KILL_SCRIPT=/hobbit/server/ext/kill_script HOST=client_server SERVICE=procs COLOR=red SCRIPT $KILL_SCRIPT client_server,client_userid,grep_pattern ##############################
/hobbit/server/ext/kill_script:
##############################
#!/bin/perl
#$debug++; # Comment-out this line for normal operation
($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3); exit unless
((defined $host) && (defined $userid) && (defined $pattern)); $command =
qq!ssh $userid\@$host "pkill -f '$pattern'"!; if ($debug) {
$command =~ s/pkill/pgrep -l/;
print $command;
}
else {
$command;
}
##############################
And here is how I execute an arbitrary script on the client (in this example, "weblogics_kill_script"). It is a minor modification of the above:
hobbit_alerts.cfg:
############################## $REMOTE_SCRIPT=/hobbit/server/ext/remote_script HOST=ssdpws001 SERVICE=weblogics COLOR=red SCRIPT $REMOTE_SCRIPT client_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s cript ##############################
Note: weblogics_kill_script exists on the client - I use it to force a few thread dumps, archive logfiles, and then kill/restart weblogics
/hobbit/server/ext/remote_script:
##############################
#!/bin/perl
#$debug++; # Comment-out this line for normal operation
($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3); exit unless
((defined $host) && (defined $userid) && (defined $script)); $command =
qq!ssh $userid\@$host "$script"!; if ($debug) {
print $command;
}
else {
$command;
}
##############################
-----Original Message----- From: Chris Wopat [mailto:chrisw at supranet.net] Sent: Friday, April 11, 2008 7:56 AM To: hobbit at hswn.dk Subject: [hobbit] Hobbit client executing a script to be proactive if a problem occurs?
Hello,
I've been running hobbit for a few years, after converting from Big Brother. I actively use it to monitor hundreds of hosts, send alerts, etc. However, one thing I have yet to do is use any custom scripts for anything. Looking at docs, I see two ways that scripts are called:
an alert can call a script instead of email directly. This script is passed variables, and just about anything can be done *on the hobbit server* from here.
A script can be launched on the client, from ext/, when the hobbit clien starts. This script is generally used to add another column to hobbit.
What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
Thanks --Chris
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Assuming you're using SSH - what SSH client are you pasting from? The formating would be put in the buffer from the client, not VI at all.
I use puTTY and the formatting works perfectly for me.
On Fri, Apr 11, 2008 at 12:51 PM, Haertig, David F (Dave) <haertig at avaya.com> wrote:
Sorry about the formating of the code I sent. I don't know why I can't successfully cut-n-paste from vi on Linux into Outlook on Windows and not get screwed up format-wise. I guess if you know Hobbit alert syntax and PERL, they you can figure it out easy enough.
-----Original Message----- From: Haertig, David F (Dave) [mailto:haertig at avaya.com] Sent: Friday, April 11, 2008 10:30 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Hobbit client executing a script to be proactive if a problem occurs?
Here is how I execute a remote "pkill" on a client. Replace "client_server" with your client hostname, and replace "client_userid" with the userid (on the client) that you want to run the script (pkill) under. Also, set up ssh pubkey authentication between the Hobbit server and client so that ssh does not prompt you for a password.
hobbit_alerts.cfg:
############################## $KILL_SCRIPT=/hobbit/server/ext/kill_script HOST=client_server SERVICE=procs COLOR=red SCRIPT $KILL_SCRIPT client_server,client_userid,grep_pattern ##############################
/hobbit/server/ext/kill_script:
############################## #!/bin/perl #$debug++; # Comment-out this line for normal operation ($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3); exit unless ((defined $host) && (defined $userid) && (defined $pattern)); $command = qq!ssh $userid\@$host "pkill -f '$pattern'"!; if ($debug) { $command =~ s/pkill/pgrep -l/; print
$command; } else {$command; } ##############################And here is how I execute an arbitrary script on the client (in this example, "weblogics_kill_script"). It is a minor modification of the above:
hobbit_alerts.cfg:
############################## $REMOTE_SCRIPT=/hobbit/server/ext/remote_script HOST=ssdpws001 SERVICE=weblogics COLOR=red SCRIPT $REMOTE_SCRIPT client_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s cript ##############################
Note: weblogics_kill_script exists on the client - I use it to force a few thread dumps, archive logfiles, and then kill/restart weblogics
/hobbit/server/ext/remote_script:
############################## #!/bin/perl #$debug++; # Comment-out this line for normal operation ($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3); exit unless ((defined $host) && (defined $userid) && (defined $script)); $command = qq!ssh $userid\@$host "$script"!; if ($debug) { print
$command; } else {$command; } ##############################-----Original Message----- From: Chris Wopat [mailto:chrisw at supranet.net] Sent: Friday, April 11, 2008 7:56 AM To: hobbit at hswn.dk Subject: [hobbit] Hobbit client executing a script to be proactive if a problem occurs?
Hello,
I've been running hobbit for a few years, after converting from Big Brother. I actively use it to monitor hundreds of hosts, send alerts, etc. However, one thing I have yet to do is use any custom scripts for anything. Looking at docs, I see two ways that scripts are called:
an alert can call a script instead of email directly. This script is passed variables, and just about anything can be done *on the hobbit server* from here.
A script can be launched on the client, from ext/, when the hobbit clien starts. This script is generally used to add another column to hobbit.
What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
Thanks --Chris
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
My local computer is Windows XP, using Putty to ssh into my Linux box (which runs Hobbit). VI on Linux, highlight, then cut-n-paste into an Outlook 2003 reply to the Hobbit mailing list. I have only noticed this formatting problem when posting code to this Hobbit list. It's probably some text-only email line-wrap formatting option I have set in Outlook. It's not an issue in HTML email (which I hate, but when in Rome, do as the Romans do, and HTML email is what most everybody in my company uses). I had to grudging give up on Thunderbird and move to Outlook so I could easily deal with all the other Windows-loving coworkers and their Outlook calendar meeting invites.
I'll check my Outlook text settings again. There's probably something in there that I missed.
From: Josh Luthman [mailto:josh at imaginenetworksllc.com] Sent: Friday, April 11, 2008 10:56 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Hobbit client executing a script to be proactive if a problem occurs?
Assuming you're using SSH - what SSH client are you pasting from? The formating would be put in the buffer from the client, not VI at all.
I use puTTY and the formatting works perfectly for me.
On Fri, Apr 11, 2008 at 12:51 PM, Haertig, David F (Dave) <haertig at avaya.com> wrote:
Sorry about the formating of the code I sent. I don't know whyI can't successfully cut-n-paste from vi on Linux into Outlook on Windows and not get screwed up format-wise. I guess if you know Hobbit alert syntax and PERL, they you can figure it out easy enough.
-----Original Message-----
From: Haertig, David F (Dave) [mailto:haertig at avaya.com]
Sent: Friday, April 11, 2008 10:30 AM
To: hobbit at hswn.dk
Subject: RE: [hobbit] Hobbit client executing a script to beproactive if a problem occurs?
Here is how I execute a remote "pkill" on a client. Replace
"client_server" with your client hostname, and replace"client_userid" with the userid (on the client) that you want to run the script (pkill) under. Also, set up ssh pubkey authentication between the Hobbit server and client so that ssh does not prompt you for a password.
hobbit_alerts.cfg:
##############################
$KILL_SCRIPT=/hobbit/server/ext/kill_script
HOST=client_server SERVICE=procs COLOR=red
SCRIPT $KILL_SCRIPTclient_server,client_userid,grep_pattern ##############################
/hobbit/server/ext/kill_script:
##############################
#!/bin/perl
#$debug++; # Comment-out this line for normal operation
($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3); exitunless
((defined $host) && (defined $userid) && (defined $pattern));
$command =
qq!ssh $userid\@$host "pkill -f '$pattern'"!; if ($debug) {
$command =~ s/pkill/pgrep -l/;
print $command;
}
else {
$command;
}
##############################
And here is how I execute an arbitrary script on the client (inthis example, "weblogics_kill_script"). It is a minor modification of the above:
hobbit_alerts.cfg:
##############################
$REMOTE_SCRIPT=/hobbit/server/ext/remote_script
HOST=ssdpws001 SERVICE=weblogics COLOR=red
SCRIPT $REMOTE_SCRIPTclient_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s cript ##############################
Note: weblogics_kill_script exists on the client - I use it toforce a few thread dumps, archive logfiles, and then kill/restart weblogics
/hobbit/server/ext/remote_script:
##############################
#!/bin/perl
#$debug++; # Comment-out this line for normal operation
($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3); exitunless
((defined $host) && (defined $userid) && (defined $script));
$command =
qq!ssh $userid\@$host "$script"!; if ($debug) {
print $command;
}
else {
$command;
}
##############################
-----Original Message-----
From: Chris Wopat [mailto:chrisw at supranet.net]
Sent: Friday, April 11, 2008 7:56 AM
To: hobbit at hswn.dk
Subject: [hobbit] Hobbit client executing a script to beproactive if a problem occurs?
Hello,
I've been running hobbit for a few years, after converting fromBig Brother. I actively use it to monitor hundreds of hosts, send alerts, etc. However, one thing I have yet to do is use any custom scripts for anything. Looking at docs, I see two ways that scripts are called:
1) an alert can call a script instead of email directly. Thisscript is passed variables, and just about anything can be done *on the hobbit server* from here.
2) A script can be launched on the client, from ext/, when thehobbit clien starts. This script is generally used to add another column to hobbit.
What I'd like to do is execute a script *on the client* when analert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.
Is there a cleaner way built in that will just say "if serviceFOO is red then on client run script ext/BAR.sh"?
Thanks
--Chris
To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe at hswn.dk-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
Haertig, David F (Dave) wrote:
Here is how I execute a remote "pkill" on a client. Replace "client_server" with your client hostname, and replace "client_userid" with the userid (on the client) that you want to run the script (pkill) under. Also, set up ssh pubkey authentication between the Hobbit server and client so that ssh does not prompt you for a password.
hobbit_alerts.cfg:
<snip>
Thanks, this is exactly what I needed to get started. I can un-wrap the lines no problem, only a few were anyway.
I'm assuming you're using ssh keys? My current hobbit server installation (from FreeBSD ports) has no home dir set, so it looks like I'll have to set one to store its side of the keys.
--Chris
Chris Wopat wrote:
Haertig, David F (Dave) wrote:
Here is how I execute a remote "pkill" on a client. Replace "client_server" with your client hostname, and replace "client_userid" with the userid (on the client) that you want to run the script (pkill) under. Also, set up ssh pubkey authentication between the Hobbit server and client so that ssh does not prompt you for a password.
hobbit_alerts.cfg:
<snip>
Thanks, this is exactly what I needed to get started. I can un-wrap the lines no problem, only a few were anyway.
I'm assuming you're using ssh keys? My current hobbit server installation (from FreeBSD ports) has no home dir set, so it looks like I'll have to set one to store its side of the keys.
d'oh, I just realized you already wrote that in oyur original reply. I've got everything set up, now to wait for it to go red!
--Chris
Yes, all of my Hobbit clients have ssh authorized_keys setup to allow the Hobbit server in without password. In the case where I need to run a script on the Hobbit client under a different userid than 'hobbit', that other userid also has the Hobbit servers pubkey in its authorized_keys file. Alternately, you gould use setuid scripts on the client (very unsecure), or use "expect" in your Hobbit server scripts and "sudo" on the client end to gain access to the userid you need. This second expect/sudo route is doable, but messy and requires you to have the 'hobbit' password from the client end stored on your server (not the most secure thing).
When coworkers ask me to use Hobbit to "fix" something on their client end I council them that Hobbit really is an alerting system and not a repairing system. But if they really want me to attempt an automated "repair", then they have to put my pubkey into their authorized_keys files, therefore giving me full access to their userid on the client machine. I also mandate that if they want me to restart a process, that I will only kill it. The restart must be their responsibility (using a local cronjob or whatever). This further insulates me from any political fallback regarding a failed automated repair attempt on an errant process. Further insolation for me is provided by me informing everyone (management too) that they can shut off my automated Hobbit repairs at any time, instantaneously, by simply removing my pubkey from their authorized_keys file(s). It's called "CYA" for when I am pressured to make Hobbit do something that it rally wasn't designed for.
-----Original Message----- From: Chris Wopat [mailto:chrisw at supranet.net] Sent: Friday, April 11, 2008 12:51 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Hobbit client executing a script to be proactive if a problem occurs?
Haertig, David F (Dave) wrote:
Here is how I execute a remote "pkill" on a client. Replace "client_server" with your client hostname, and replace "client_userid" with the userid (on the client) that you want to run the script (pkill) under. Also, set up ssh pubkey authentication between the Hobbit server and client so that ssh does not prompt you for a password.
hobbit_alerts.cfg:
<snip>
Thanks, this is exactly what I needed to get started. I can un-wrap the lines no problem, only a few were anyway.
I'm assuming you're using ssh keys? My current hobbit server installation (from FreeBSD ports) has no home dir set, so it looks like I'll have to set one to store its side of the keys.
--Chris
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On Fri, Apr 11, 2008 at 08:56:02AM -0500, Chris Wopat wrote:
What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
There is. You use the Hobbit "query" command to probe the status, and then act accordingly. To use your example with the "cpu" status going red:
#!/bin/sh
CPUCOLOR=$BB $BBDISP "query $MACHINE.cpu" | awk '{print $1}'
if test "$CPUCOLOR" = "red"
then
# Kill some cpu hogging processes
.... whatever it takes ....
fi
exit 0
The "query" command returns the first line of the current status, and the first word is the current color. You probably will want to put in some sort of sanity check so you don't trigger this too often. Or perhaps just log the fact that it triggered.
Regards, Henrik
Henrik Stoerner wrote:
On Fri, Apr 11, 2008 at 08:56:02AM -0500, Chris Wopat wrote:
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
There is. You use the Hobbit "query" command to probe the status, and then act accordingly. To use your example with the "cpu" status going red:
#!/bin/sh
CPUCOLOR=
$BB $BBDISP "query $MACHINE.cpu" | awk '{print $1}'if test "$CPUCOLOR" = "red" then # Kill some cpu hogging processes .... whatever it takes .... fi exit 0The "query" command returns the first line of the current status, and the first word is the current color. You probably will want to put in some sort of sanity check so you don't trigger this too often. Or perhaps just log the fact that it triggered.
Excellent, was unaware of this query feature. The script above looks like it would perfectly deal with my situation if I were to put it in cron on the client machine, which may be suitable.
Is there some way to tell the hobbit server to execute this script on the client, or would something in a SCRIPT line in a hobbit alert, likely using ssh keys, be the only way?
--Chris
Hi Chris,
As far as I know, you would need to use a SCRIPT line in hobbit-alerts. The example that Henrik gave is quite simple and I would be very careful about how you use it... For example, if you run a backup, or a one off job that does something CPU intensive that would not be considered a fault, yet it did trigger a CPU threshold breach, you may actually triger an event that kills and restarts a perfectly healthy process.
Your best approach is to see if there is something that is very unique about your problem and build a test around it, for example, use ps or top to give you the process using the most cpu, if that process is your usual suspect and cpu usage is greater than 80% and it has been that way for five minutes, then take an action and report up to hobbit that an action has been taken...
Cheers
Phil
2008/4/15 Chris Wopat <chrisw at supranet.net>:
Henrik Stoerner wrote:
On Fri, Apr 11, 2008 at 08:56:02AM -0500, Chris Wopat wrote:
Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?
There is. You use the Hobbit "query" command to probe the status, and then act accordingly. To use your example with the "cpu" status going red:
#!/bin/sh
CPUCOLOR=
$BB $BBDISP "query $MACHINE.cpu" | awk '{print $1}'if test "$CPUCOLOR" = "red" then # Kill some cpu hogging processes .... whatever it takes .... fi exit 0The "query" command returns the first line of the current status, and the first word is the current color. You probably will want to put in some sort of sanity check so you don't trigger this too often. Or perhaps just log the fact that it triggered.
Excellent, was unaware of this query feature. The script above looks like it would perfectly deal with my situation if I were to put it in cron on the client machine, which may be suitable.
Is there some way to tell the hobbit server to execute this script on the client, or would something in a SCRIPT line in a hobbit alert, likely using ssh keys, be the only way?
--Chris
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Tel: 0400 466 952 Fax: 0433 123 226 email: philwild AT gmail.com
participants (5)
-
chrisw＠supranet.net
-
haertig＠avaya.com
-
henrik＠hswn.dk
-
josh＠imaginenetworksllc.com
-
philwild＠gmail.com