March 2008 - Xymon - Xymon mailman web

Hobbitd_client on server is crashing
by Martin.Ward＠colt.net 29 Mar '08

29 Mar '08

Hi All, I am running Hobbit 4.2.0 on a number of Sun servers and recently the hobbitd_client daemon has been crashing and leaving core files around. I ran gdb according to http://www.hswn.dk/hobbit/help/known-issues.html#bugreport and it came back with: hobbit at hbt0:/opt/hobbit/server>gdb bin/hobbitd_client tmp/core GNU gdb 6.6 ... Core was generated by `hobbitd_client'. Program terminated with signal 6, Aborted. #0 0xfee25687 in _lwp_kill () from /lib/libc.so.1 (gdb) bt #0 0xfee25687 in _lwp_kill () from /lib/libc.so.1 #1 0xfee22dee in thr_kill () from /lib/libc.so.1 #2 0xfedd11bb in raise () from /lib/libc.so.1 #3 0xfedb15d9 in abort () from /lib/libc.so.1 #4 0x08067172 in sigsegv_handler (signum=11) at sig.c:57 #5 0xfee24a4f in __sighndlr () from /lib/libc.so.1 #6 0xfee1ae72 in call_user_handler () from /lib/libc.so.1 #7 <signal handler called> #8 0x08063ef9 in get_ostype (osname=0x806c28d "") at misc.c:43 #9 0x0805b320 in main (argc=1, argv=0x80464ac) at hobbitd_client.c:1766 The really bizarre thing is that it only crashes when one particular machine connects to it and sends data. Other Hobbit clients connect in and transfer thir core data over with no issues, it's only this one Sun Ultra 60 that causes a problem. Any ideas where I might start looking? |\/|artin -- Martin Ward Network Systems Operations Specialist DDI: +44 (0) 20 7863 5218 Fax: +44 (0) 20 7863 5610 Mob: +44 (0) 7971 97 77 21 www.colt.net Data | Voice | Managed Services Help reduce your carbon footprint | Think before you print COLT Telecommunications, Beaufort House, 15 St Botolph Street, London, EC3A 7QN UK Registered in England and Wales, registered number 02452736, VAT number GB 645 4205 50 ************************************************************************************* The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security at colt.net and delete the message and any attachments without retaining any copies. Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party. Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

2 1

F5 BigIP [Devmon] Response to Large or cant Fragment or configuration?
by wrichardson＠llbean.com 28 Mar '08

28 Mar '08

I sent this to the devmon mailing list and thought it might be worth trying the hobbit list to see if anyone was using Hobbit and or Devmon to monitor F5's. Any thoughts/help? -----Original Message----- From: Bill Richardson Sent: Wednesday, March 26, 2008 7:46 AM To: 'Buchan Milne' Cc: devmon-support at lists.sourceforge.net Subject: RE: [Devmon] Response to Large or cant Fragment or configuration? Thanks for your thoughts... Just for fun I add a few of our 4500 Cisco switches that are fully populated and Devmon is working without any issues. I then took another set of bigip F5's that was working with Devmon and increased the number of Pool members one at a time. "The F5's that worked had 80 Pool memebers" When I got to 97 pool members thats when I see "No SNMP data found for PoolMemberStatusReason" in the Devmon log. I delete one of the pool members Devmon starts to work again. "The F5's that Devmon doesn't work has 128 pool members" So this is a test for anyone that is using Devmon to monitor bigip's is to see how many pool members they have. Seems like the braking point is around the 97 mark. I have tcpdumps of a good working "96 members" and the failure "97 members" I could send you but I would like to sanitize them first "some how?" -----Original Message----- From: Buchan Milne [mailto:bgmilne at staff.telkomsa.net] Sent: Tuesday, March 25, 2008 9:24 AM To: devmon-support at lists.sourceforge.net Cc: Bill Richardson Subject: Re: [Devmon] Response to Large or cant Fragment or configuration? On Tuesday 25 March 2008 14:24:03 Bill Richardson wrote: > I'm using devmon to monitor some bigip F5. When devmon is first > started "the first poll" all works. During every poll after on one of > my F5's under the pool "PoolMemberStatusReason" is not being populated > with data. "Agian on the first poll it is populted" > > Looking in the devmon.log is this this message over and over every > minute: > "No SNMP data found for PoolMemberStatusReason on bigip1500" > > Starting devmon with the "-f " I see this starting after the 1st poll > "First poll is fine" > ---------------------------------------------------------------------- > -- > --------------------- > ./devmon -f > > SNMP Error: > Error decoding response PDU: > Expected length 9855, got 7996 > %{%i%s%*{%i%i%i%{%@ > ^ > SNMPv2c_Session (remote host: "192.168.1.10" [192.168.1.10].161) > community: "MyPublic" > request ID: -1368482924 > PDU bufsize: 8000 bytes > timeout: 5s > retries: 3 > backoff: 1) > at /usr/local/devmon-0.3.0-rc1/modules/dm_snmp.pm line 540 Use of > uninitialized value in string ne at > /usr/lib/perl5/site_perl/5.8.8/SNMP_Session.pm line 871, <$__ANONIO__> > line 16. > ---------------------------------------------------------------------- > -- > ------------------------ > > Based on this I upgraded from SNMP_Session 1.08 to 1.12 and still see > the same issue. > > I did some tcpdumps and was thinking it may have been the size of the > response.... > > Looking at the dumps I noticed that the first poll response in the > trace has a total length of only 988, while the second poll response > has a length of 1514. > > Additionally I noticed that second poll packet has the fragment bit set. > meaning there should be more packets to follow. I wonder if the perl > script is having trouble handling a fragmented packet. This might > have something to do with the message that I'm seeing that was > complaining about the expected length of the packet. > > So, thinking it has something to do with the size of the response, as > a test I went in to the bigip box (the standby box) and I deleted a > bunch of the pools - like half of them or more. Once I did that, I > went in and checked devmon again. Once it refreshed it was able to > load the pool info for this box. This proves that it is definitely > related to the size of the response that the SNMP bulkget is receiving > that is causing the error. > > My guess is that is that the first request is working because of that > MAX REPETITION setting. If you look at that first response packet, it > returns 12 SNMP items. At some point between that first request and > the second one, devmon must be learning that there are really 129 > items here, not 12. Then when it tries on the second attempt to pull > in that many item responses they cause the error. Whether it is the > packet fragmenting that is the cause or something else related to the > size of the response, I'm not sure. > > Any help with this one... I have looked and looked dont see this > reported on the list in the past.. I'm currently running perl-SNMP_Session 1.08, and monitoring some cisco devices with more than 180 items per test. It might be best if you could send captured files from tcpdump or wireshark. However, this looks more like an issue with SNMP_Session, than with devmon. But, I'll try and investigate. Regards, Buchan

2 1

graphing questions
by philwild＠gmail.com 28 Mar '08

28 Mar '08

Hello hobbit users, Three items.... We monitor the size of oracle tables in a similar way as the base disk monitoring is performed. We use a custom script at the back end to create the rrd's as new tablespaces appear (i.e. one tablespace per rrd) The tablespace sizes are graphed in a similar way to the disk graphs as well with the standard hobbit method of dividing the tables into smaller groups (I think it is 7) and charting them on one graph. The issue we have is that some tables are very small (say 100MB and some are large 30GBytes). When these are plotted on the same graph, the plots for the small tables are of little value as they look like zero... I assume the only way to address this would be to plot each table individually. Anyone have any ideas how I can force this? Next.... I want to stack all the tables onto an area chart to show total database growth. However, the first value needs to be area and the rest of the values stack. How can I achieve this when a regex is used to pick up the rrd's? And last, I am toying with the idea of trending growth rate. It would seem that the rrd database would make a good source of historical data for this. Has anyone done something here? One idea I am playing with is if a filesystem breaches a capacity threshold out of hours, but the greatest of the last five minute and 30 minute growth rates would indicate that it is not going to run out of space before 10AM on a business day, then don't send me an sms to wake me up. Thoughts? Phil

2 1

history not updating
by stewartl42＠gmail.com 28 Mar '08

28 Mar '08

Not sure if this is related to my issue about Whoops errors or not, but it seems that the history does not always update correctly when a service goes down. Seems to be the same problem as here... http://www.hswn.dk/hobbiton/2008/03/msg00064.html My history.log shows no issues... Bunch of "Will not update <server.test> - color unchanged" messages. hobbitlaunch.log shows this. 2008-03-26 23:21:05 Whoops ! bb failed to send message - timeout 2008-03-26 23:56:17 Whoops ! bb failed to send message - timeout -- Stewart The revolution will not be televised. The revolution will be no re-run brothers; The revolution will be live.

2 2

Whoops errors
by stewartl42＠gmail.com 28 Mar '08

28 Mar '08

I've asked before and I've looked in the archives... I applied a patch from Henrik about non-blocking sockets that I was given last time I asked, but I'm still getting these errors... Any thoughts? bb-display.log:2008-03-26 20:34:23 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 20:44:26 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 21:09:40 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 21:29:39 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 21:34:44 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 21:54:48 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 22:20:04 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 22:25:03 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 22:35:15 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 22:40:19 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 23:05:29 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 23:25:37 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 23:35:39 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-26 23:55:49 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 00:00:47 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 00:05:54 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 01:11:11 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 02:16:24 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 03:52:08 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 04:22:14 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 04:52:23 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 05:17:34 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 05:57:47 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 06:33:07 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 07:18:22 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 07:23:17 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 07:48:29 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 07:53:29 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 07:58:34 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 08:03:40 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 08:38:53 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 10:24:38 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 10:29:32 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 11:04:40 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 13:55:34 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 14:30:59 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 15:51:26 Whoops ! bb failed to send message - timeout bb-display.log:2008-03-27 16:26:41 Whoops ! bb failed to send message - timeout hobbitlaunch.log:2008-03-26 23:21:05 Whoops ! bb failed to send message - timeout hobbitlaunch.log:2008-03-26 23:56:17 Whoops ! bb failed to send message - timeout -- Stewart The revolution will not be televised. The revolution will be no re-run brothers; The revolution will be live.

2 2

client compilation error
by Dirk.Kastens＠uni-osnabrueck.de 28 Mar '08

28 Mar '08

Hi, I just tried to compile the client of the latest snapshot under RedHat Enterprise Linux AS 4 (kernel 2.6.9-67.ELsmp). It failed with the following error message: hobbitd_worker.o(.text+0x20e): In function `net_worker_run': /usr/src/redhat/SOURCES/snapshot/hobbitd/hobbitd_worker.c:295: undefined reference to `locator_init' hobbitd_worker.o(.text+0x245):/usr/src/redhat/SOURCES/snapshot/hobbitd/hobbitd_worker.c:295: undefined reference to `locator_init' hobbitd_worker.o(.text+0x270):/usr/src/redhat/SOURCES/snapshot/hobbitd/hobbitd_worker.c:298: undefined reference to `locator_register_server' hobbitd_worker.o(.text+0x2e7):/usr/src/redhat/SOURCES/snapshot/hobbitd/hobbitd_worker.c:312: undefined reference to `locator_serverdown' hobbitd_worker.o(.text+0x568):/usr/src/redhat/SOURCES/snapshot/hobbitd/hobbitd_worker.c:317: undefined reference to `locator_serverdown' hobbitd_worker.o(.text+0x6c5):/usr/src/redhat/SOURCES/snapshot/hobbitd/hobbitd_worker.c:80: undefined reference to `locator_serverup' -- Regards, Dirk Kastens Universitaet Osnabrueck, Rechenzentrum (Computer Center) Albrechtstr. 28, 49069 Osnabrueck, Germany Tel.: +49-541-969-2347, FAX: -2470

1 0

[hobbit] I need help DESPERATELY!! roracle script....
by rebecca_henderson＠link.freedom.com 27 Mar '08

27 Mar '08

Ok, I did check the perms and that is fixed, here is what I get: $ sqlplus SQL*Plus: Release 10.2.0.3.0 - Production on Thu Mar 27 13:48:52 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Enter user-name: hobbit Enter password: Error accessing PRODUCT_USER_PROFILE Warning: Product user profile information not loaded! You may need to run PUPBLD.SQL as SYSTEM Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production With the Partitioning and Data Mining options SQL> But I am still red in the web page here is what I'm checking for LOCAL ADB1PLS1 hobbit aBEfijk LISTENER_MAC2 -----Original Message----- From: hobbit at hswn.dk [mailto:hobbit at hswn.dk] Sent: Thursday, March 27, 2008 1:16 PM To: Rebecca Henderson; hobbit at hswn.dk Subject: RE: [hobbit] I need help DESPERATELY!! roracle script.... Importance: Low Check and make sure the hobbit user can run sqlplus and tnsping, when oracle installs it sets permissions on some libraries that stops users from accessing the oracle binaries unless the user is in the same group as oracle, usually dba. Mike ________________________________ From: Rebecca Henderson [mailto:rebecca_henderson at link.freedom.com] Sent: Thu 27/03/2008 20:03 To: hobbit at hswn.dk Subject: [hobbit] I need help DESPERATELY!! roracle script.... Here is what I;m getting I have checked the path, the ORACLE_HOME I'm in dire need to get this going. Thank you in advance. Process Check (ADB1PLS1) ________________________________ green<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[h obbit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mu ltipart/image001.gif> Database ADB1PLS1 UP processes: ckpt dbw0 lgwr pmon reco smon Database Checks (ADB1PLS1) ________________________________ red<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[hob bit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mult ipart/image002.gif> Database check: ADB1PLS1 is down or in hung state FUNC_DATABASE_CHECK[16]: /oraclebin2/app/oracle/product/10.2: cannot execute red<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[hob bit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mult ipart/image002.gif> Tablespace check disabled since db is down or check is disabled red<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[hob bit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mult ipart/image002.gif> Extent check disabled since db is down or check is disabled ________________________________________________________________________ This email has been scanned for all viruses by the MessageLabs service. ________________________________________________________________________ This email has been scanned for all viruses by the MessageLabs service. ________________________________________________________________________ To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

[hobbit] I need help DESPERATELY!! roracle script....
by rebecca_henderson＠link.freedom.com 27 Mar '08

27 Mar '08

Thanks I'll check that. -----Original Message----- From: hobbit at hswn.dk [mailto:hobbit at hswn.dk] Sent: Thursday, March 27, 2008 1:16 PM To: Rebecca Henderson; hobbit at hswn.dk Subject: RE: [hobbit] I need help DESPERATELY!! roracle script.... Importance: Low Check and make sure the hobbit user can run sqlplus and tnsping, when oracle installs it sets permissions on some libraries that stops users from accessing the oracle binaries unless the user is in the same group as oracle, usually dba. Mike ________________________________ From: Rebecca Henderson [mailto:rebecca_henderson at link.freedom.com] Sent: Thu 27/03/2008 20:03 To: hobbit at hswn.dk Subject: [hobbit] I need help DESPERATELY!! roracle script.... Here is what I;m getting I have checked the path, the ORACLE_HOME I'm in dire need to get this going. Thank you in advance. Process Check (ADB1PLS1) ________________________________ green<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[h obbit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mu ltipart/image001.gif> Database ADB1PLS1 UP processes: ckpt dbw0 lgwr pmon reco smon Database Checks (ADB1PLS1) ________________________________ red<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[hob bit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mult ipart/image002.gif> Database check: ADB1PLS1 is down or in hung state FUNC_DATABASE_CHECK[16]: /oraclebin2/app/oracle/product/10.2: cannot execute red<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[hob bit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mult ipart/image002.gif> Tablespace check disabled since db is down or check is disabled red<https://lo-mail.rightmove.com/exchange/mike.rawell/Drafts/RE:%20[hob bit]%20I%20need%20help%20DESPERATELY!!%20roracle%20script.....EML/1_mult ipart/image002.gif> Extent check disabled since db is down or check is disabled ________________________________________________________________________ This email has been scanned for all viruses by the MessageLabs service. ________________________________________________________________________ This email has been scanned for all viruses by the MessageLabs service. ________________________________________________________________________ To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

I need help DESPERATELY!! roracle script....
by rebecca_henderson＠link.freedom.com 27 Mar '08

27 Mar '08

Here is what I;m getting I have checked the path, the ORACLE_HOME I'm in dire need to get this going. Thank you in advance. Process Check (ADB1PLS1) ________________________________ Database ADB1PLS1 UP processes: ckpt dbw0 lgwr pmon reco smon Database Checks (ADB1PLS1) ________________________________ Database check: ADB1PLS1 is down or in hung state FUNC_DATABASE_CHECK[16]: /oraclebin2/app/oracle/product/10.2: cannot execute Tablespace check disabled since db is down or check is disabled Extent check disabled since db is down or check is disabled

3 2

Alert HOST + EXHOST rules not working correctly
by gumby3203＠gmail.com 27 Mar '08

27 Mar '08

I must be missing something stupid as I've been staring at this and can't figure out what's wrong. I have the following alert rules set up: HOST=%(.*-stag-.*) EXHOST=%^(marketing-stag-.*) EXHOST=%^(infra-stag-.*) TIME=*:0800:2100 RECOVERED DURATION>11m SCRIPT $SMSPLUS $WLI_STAGMAN_EMAIL REPEAT=1h HOST=%^(marketing-stag-.*) TIME=*:0800:2100 RECOVERED SCRIPT $SMSPLUS $WLI_MKT_STAG_EMAIL REPEAT=1h HOST=%^(infra-stag-.*) TIME=*:0800:2100 RECOVERED SCRIPT $SMSPLUS $WLI_INFRA_STAG_EMAIL REPEAT=1h The problem is that the EXHOST doesn't appear to be working the way I expect it to. When I do a hobbitd_alert --test on "marketing-stag-1", it shows the first HOST line (the one with the EXHOSTs) as matching and alerting. $> bbcmd hobbitd_alert --test marketing-stag-1 Managed-Server --duration=800 . . . 12:50:51 Matching host:service:page 'marketing-stag-1:Managed-Server:wblx-staging' against rule line 388 12:50:51 *** Match with 'HOST=$WLI_MANAGED_STAG EXHOST=$WLI_MKTING_STAG EXHOST=$WLI_INFRA_STAG TIME=*:0800:2100 RECOVERED DURATION>11m' *** 12:50:51 Matching host:service:page 'marketing-stag-1:Managed-Server:wblx-staging' against rule line 388 12:50:51 *** Match with 'HOST=$WLI_MANAGED_STAG EXHOST=$WLI_MKTING_STAG EXHOST=$WLI_INFRA_STAG TIME=*:0800:2100 RECOVERED DURATION>11m' *** 12:50:51 Script alert with command '/var/hobbit/server/ext/smsplus' and recipient abc at mydomain.com 12:50:51 Matching host:service:page 'marketing-stag-1:Managed-Server:wblx-staging' against rule line 390 12:50:51 *** Match with 'HOST=$WLI_MKTING_STAG TIME=*:0800:2100 RECOVERED' *** 12:50:51 Matching host:service:page 'marketing-stag-1:Managed-Server:wblx-staging' against rule line 390 12:50:51 *** Match with 'HOST=$WLI_MKTING_STAG TIME=*:0800:2100 RECOVERED' *** 12:50:51 Script alert with command '/var/hobbit/server/ext/smsplus' and recipient def at mydomain.com 12:50:51 Matching host:service:page 'marketing-stag-1:Managed-Server:wblx-staging' against rule line 392 12:50:51 Failed 'HOST=$WLI_INFRA_STAG TIME=*:0800:2100 RECOVERED' (hostname not in include list) . . . I've tried the regex with and without the "^", and it doesn't seem to have an effect. Any ideas what I'm missing? Without the "--duration=800", the first HOST line doesn't match (as expected).

1 1