Yup.
There's always the case that the process died for a good reason and it's going to die again as soon as it restarts. If you don't set up the restart/recovery routine with a bit of logic to quit after a while you can end up with a box that's spending all its cycles just restarting, dumping, and dying.
I learned that one the hard way :-)
Tom Kauffman NIBCO, Inc
-----Original Message----- From: Aiello, Steve (GE, Corporate, consultant) [mailto:steve.aiello at ge.com] Sent: Friday, July 21, 2006 12:11 PM To: hobbit at hswn.dk Subject: RE: [hobbit] bbfix functionality in hobbit
I agree completely with Henrik. Adding 'intelligence' into a script is rather difficult. But I do have a rather 'painful' IIS ASP application. And frequently IIS will die, hang, stop processing ASP. So I wrote a script that runs on each IIS server, that queries the HTTP status from the monitoring server and if the status is red all dllhost & inetinfo processes are killed & the IIS is started. My fear was that this restart script could be stuck in a loop, i.e. possibility that IIS can not be started. So I added the intelligence into my script that logs the date & time of each time the script restarts IIS. It will only restart, if it has not done it more than 3 times in the last hour. Lately I have been thinking of adding in more logic to check if a restart occured in the last 5 minutes (poling period). Becuase I have seen the restart script do it's job and fix IIS, but the monitoring server has not checked yet. Thus the restart script bounces IIS again...
-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Friday, July 21, 2006 11:07 AM To: hobbit at hswn.dk Subject: Re: [hobbit] bbfix functionality in hobbit
On Fri, Jul 21, 2006 at 10:21:40AM -0400, Gary B. wrote:
Is there any method with hobbit to have the client automatically restart services, like bbfix does for BB? We would like to be able to restart services that tend to fail often (such as SSH tunnels) automatically through Hobbit. Without writing a custom external script, I can't seem to find any information about doing this.
You will need to do some scripting, no doubt about that.
Whether it's a good idea or not ... it depends. From the Hobbit "design" perspective (whoa - that sounds expensive) I have a very firm belief that Hobbit should *monitor* things, not *fix* them. I have seen far too many "intelligent" systems get in the way of real problem-fixing because "intelligent" systems are usually pretty dumb, and cannot handle anything out of the ordinary. When they try, they often fail in spectacular ways.
And having things happen behind your back - because you forgot about that little automatic script someone else setup 2 years ago - is just plain frustrating.
With that little sermon as introduction, here's what you can do. On the host(s) where you want to restart these services, write a script to query the Hobbit server for the status of the service. If it's red, do the restart. You can use the Hobbit "query" command to tell what status the service has. E.g. if you want to reset the SSH tunnels when the "tunnels" status goes red, then this little script run from the Hobbit client's clientlaunch.cfg would do it:
#!/bin/sh
TUNNELSTATUS=
$BB $BBDISP "query $MACHINE.tunnels"|awk '{print $1}'if test "$TUNNELSTATUS" = "red"; then sudo /etc/init.d/sshtunnels stop sleep 5 sudo /etc/init.d/sshtunnels start echo "date: SSH tunnels restarted" fiexit 0
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
CONFIDENTIALITY NOTICE: This email and any attachments are for the
exclusive and confidential use of the intended recipient. If you are not
the intended recipient, please do not read, distribute or take action in
reliance upon this message. If you have received this in error, please
notify us immediately by return email and promptly delete this message
and its attachments from your computer system. We do not waive
attorney-client or work product privilege by the transmission of this
message.