July 2006 - Xymon - Xymon mailman web

[hobbit] bbfix functionality in hobbit
by KauffmanT＠nibco.com 21 Jul '06

21 Jul '06

Yup. There's always the case that the process died for a good reason and it's going to die again as soon as it restarts. If you don't set up the restart/recovery routine with a bit of logic to quit after a while you can end up with a box that's spending all its cycles just restarting, dumping, and dying. I learned that one the hard way :-) Tom Kauffman NIBCO, Inc -----Original Message----- From: Aiello, Steve (GE, Corporate, consultant) [mailto:steve.aiello at ge.com] Sent: Friday, July 21, 2006 12:11 PM To: hobbit at hswn.dk Subject: RE: [hobbit] bbfix functionality in hobbit I agree completely with Henrik. Adding 'intelligence' into a script is rather difficult. But I do have a rather 'painful' IIS ASP application. And frequently IIS will die, hang, stop processing ASP. So I wrote a script that runs on each IIS server, that queries the HTTP status from the monitoring server and if the status is red all dllhost & inetinfo processes are killed & the IIS is started. My fear was that this restart script could be stuck in a loop, i.e. possibility that IIS can not be started. So I added the intelligence into my script that logs the date & time of each time the script restarts IIS. It will only restart, if it has not done it more than 3 times in the last hour. Lately I have been thinking of adding in more logic to check if a restart occured in the last 5 minutes (poling period). Becuase I have seen the restart script do it's job and fix IIS, but the monitoring server has not checked yet. Thus the restart script bounces IIS again... > -----Original Message----- > From: Henrik Stoerner [mailto:henrik at hswn.dk] > Sent: Friday, July 21, 2006 11:07 AM > To: hobbit at hswn.dk > Subject: Re: [hobbit] bbfix functionality in hobbit > > > On Fri, Jul 21, 2006 at 10:21:40AM -0400, Gary B. wrote: > > Is there any method with hobbit to have the client automatically > > restart services, like bbfix does for BB? We would like to > be able to > > restart services that tend to fail often (such as SSH tunnels) > > automatically through Hobbit. Without writing a custom external > > script, I can't seem to find any information about doing this. > > You will need to do some scripting, no doubt about that. > > Whether it's a good idea or not ... it depends. From the > Hobbit "design" perspective (whoa - that sounds expensive) I > have a very firm belief that Hobbit should *monitor* things, > not *fix* them. I have seen far too many "intelligent" > systems get in the way of real problem-fixing because > "intelligent" systems are usually pretty dumb, and cannot > handle anything out of the ordinary. When they try, they > often fail in spectacular ways. > > And having things happen behind your back - because you > forgot about that little automatic script someone else setup > 2 years ago - is just plain frustrating. > > With that little sermon as introduction, here's what you can > do. On the host(s) where you want to restart these services, > write a script to query the Hobbit server for the status of > the service. If it's red, do the restart. You can use the > Hobbit "query" command to tell what status the service has. > E.g. if you want to reset the SSH tunnels when the "tunnels" > status goes red, then this little script run from the Hobbit > client's clientlaunch.cfg would do it: > > #!/bin/sh > > TUNNELSTATUS=`$BB $BBDISP "query $MACHINE.tunnels"|awk > '{print $1}'` > if test "$TUNNELSTATUS" = "red"; then > sudo /etc/init.d/sshtunnels stop > sleep 5 > sudo /etc/init.d/sshtunnels start > echo "`date`: SSH tunnels restarted" > fi > > exit 0 > > > Regards, > Henrik > > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > > > To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.

1 0

[hobbit] bbfix functionality in hobbit
by gmbfly98＠gmail.com 21 Jul '06

21 Jul '06

Is there any method with hobbit to have the client automatically restart services, like bbfix does for BB? We would like to be able to restart services that tend to fail often (such as SSH tunnels) automatically through Hobbit. Without writing a custom external script, I can't seem to find any information about doing this. Thanks

4 4

[hobbit] Sigsegv in Bbtest-net
by tschenk＠pironet-ndh.com 21 Jul '06

21 Jul '06

Thanks ;) Regards, Thomas > > This is a bug which gets triggered if you do a traceroute when the > ping test fails. Patch attached. > > > Regards, > Henrik > >

1 0

Clientupdate minimum version
by jeffrey.chandler＠gmail.com 21 Jul '06

21 Jul '06

Hello, I couldn't find any documentation regarding this, but is there a minimum version for the clientupdate to work properly? I have just upgraded my server to the RC for 4.2, but I am still using the BigBrother client and deploying the client to my servers step by step, and would prefer not to have to deploy clients again once a "stable" version is released. Thanks, Jeffrey Chandler

1 0

Suppressing tests?
by James.Gearing＠analog.com 21 Jul '06

21 Jul '06

Does anyone know of a simple way, maybe in the bb-hosts file to suppress tests? I'd like to suppress the "info" and "conn" and a few other custom test column names, for some, but not all clients. Im using hobbit server 4.2 with BB 1.9i clients.

3 2

[hobbit] clientupdate?
by David.Gore＠verizonbusiness.com 21 Jul '06

21 Jul '06

Henrik, What is the status (this release? next release?) of clientupdate to automatically update the client? We read the man page and configured it, hoping it would work. Obviously, it would save us a lot of time. Here is what we get on the client: tail -f hobbitclient.log 2006-07-19 13:30:59 connect to bbd failed - Connection refused 2006-07-19 13:30:59 Whoops ! bb failed to send message - Connection failed Cannot fetch new client tarfile tar: blocksize = 0 ls -al ../tmp -rw-r--r-- 1 nmsbb staff 0 Jul 19 13:36 .inprogress.update -r-xr-x--- 1 nmsbb staff 165540 Jul 19 13:30 .update.omzimp01.1153315859.tmp Anything we can do to help, or hack this to get it to work? ~David

3 9

Sigsegv in Bbtest-net
by tschenk＠pironet-ndh.com 21 Jul '06

21 Jul '06

Hi, bbtest-net version 4.2-RC-20060712 SSL library : OpenSSL 0.9.7e 25 Oct 2004 LDAP library: OpenLDAP 20320 Statistics: Hosts total : 444 Hosts with no tests : 28 Total test count : 665 Status messages : 640 Alert status msgs : 0 Transmissions : 9 Bbtest-net crashed the second time today, first time it was red for about 3 minutes, the second time it was down about 20 minutes before I restarted it manually. I am not sure what triggered the crash. (gdb) bt #0 0xb7cea83b in raise () from /lib/tls/libc.so.6 #1 0xb7cebfa2 in abort () from /lib/tls/libc.so.6 #2 0x08064361 in sigsegv_handler (signum=0) at sig.c:57 #3 <signal handler called> #4 0xb7d3c087 in memcpy () from /lib/tls/libc.so.6 #5 0x08065532 in strbuf_addtobuffer (buf=0x0, newtext=0x1 <Address 0x1 out of bounds>, newlen=137255296) at strfunc.c:113 #6 0x080655e1 in addtostrbuffer (buf=0x1, newtext=0x82e5980) at strfunc.c:124 #7 0x080629ca in addtostrstatus (p=0x1) at sendmsg.c:658 #8 0x0804eb86 in send_results (service=0x8080098, failgoesclear=1) at bbtest-net.c:1716 #9 0x0805096c in main (argc=4, argv=0xbfbe2d74) at bbtest-net.c:2261 Regards, Thomas

2 1

[hobbit] bbfix functionality in hobbit
by JMSmith＠stvincenthealth.com 21 Jul '06

21 Jul '06

My personal opinion is that having Hobbit (or Big Brother) take action other than sending an alert goes beyond the scope of what it is intended for...as a monitoring system. I had thought about the same thing at one point, but was talked out of it. -----Original Message----- From: Gary B. [mailto:gmbfly98 at gmail.com] Sent: Friday, July 21, 2006 9:22 AM To: hobbit at hswn.dk Subject: [hobbit] bbfix functionality in hobbit Is there any method with hobbit to have the client automatically restart services, like bbfix does for BB? We would like to be able to restart services that tend to fail often (such as SSH tunnels) automatically through Hobbit. Without writing a custom external script, I can't seem to find any information about doing this. Thanks To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk NOTICE: This email contains confidential or proprietary information which may be legally privileged. It is intended only for the named recipient(s). If an addressing error has misdirected the email, please notify the author by replying to this message. If you are not the named recipient, you are not authorized to use, disclose, distribute, copy, print or rely on this email, and should immediately delete it from your computer system.

4 3

[hobbit] bbfix functionality in hobbit
by JMSmith＠stvincenthealth.com 21 Jul '06

21 Jul '06

Basically, it was just the scope argument. And I know all about on-call weeks...my one backup-analyst and I rotate weeks so I'm on-call half the year! But my first focus would be to attack the issue on its home server. For instance, if it is a Unix server I would investigate creating some kind of script that could be put in /etc/inittab to restart a dead process. If all else fails, however, you gotta do what you gotta do to get some sleep at night! -----Original Message----- From: Gary B. [mailto:gmbfly98 at gmail.com] Sent: Friday, July 21, 2006 9:33 AM To: hobbit at hswn.dk Subject: Re: [hobbit] bbfix functionality in hobbit > My personal opinion is that having Hobbit (or Big Brother) take action > other than sending an alert goes beyond the scope of what it is intended > for...as a monitoring system. I had thought about the same thing at one > point, but was talked out of it. I would tend to agree myself, except 1) some people are asking about such functionality, and 2) it would be nice during oncall weeks to not have to get up and restart processes such as procallator, etc. I would be interested in hearing what talked you out of this, however. If nothing else, I can use it to talk other people out of it ;-) > -----Original Message----- > From: Gary B. [mailto:gmbfly98 at gmail.com] > Sent: Friday, July 21, 2006 9:22 AM > To: hobbit at hswn.dk > Subject: [hobbit] bbfix functionality in hobbit > > Is there any method with hobbit to have the client automatically > restart services, like bbfix does for BB? We would like to be able to > restart services that tend to fail often (such as SSH tunnels) > automatically through Hobbit. Without writing a custom external > script, I can't seem to find any information about doing this. > > > Thanks > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > > > NOTICE: This email contains confidential or proprietary information which may be legally privileged. It is intended only for the named recipient(s). If an addressing error has misdirected the email, please notify the author by replying to this message. If you are not the named recipient, you are not authorized to use, disclose, distribute, copy, print or rely on this email, and should immediately delete it from your computer system. > > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > > > To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk NOTICE: This email contains confidential or proprietary information which may be legally privileged. It is intended only for the named recipient(s). If an addressing error has misdirected the email, please notify the author by replying to this message. If you are not the named recipient, you are not authorized to use, disclose, distribute, copy, print or rely on this email, and should immediately delete it from your computer system.

1 0

[clientversion] section
by rsmrcina＠wi.rr.com 21 Jul '06

21 Jul '06

I noticed a [clientversion] section of the 'Client Data', is the actual version information supposed to appear there? If so, it isn't. This is SLES9 with hobbit-4.2-RC-20060712. -- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007

2 1