April 2005 - Xymon - Xymon mailman web

[hobbit] A few hobbit problems
by EES2Y＠hscmail.mcc.virginia.edu 04 Apr '05

04 Apr '05

>Could you try getting the call trace from the core file ? Assuming >the core file is in the current directory, you should do this: > > $ gdb /usr/local/hobbit/server/bin/bb-eventlog.cgi core > [messages from gdb] > gdb> bt > >The output from the "bt" command would be very helpful in narrowing >down the problem. Below is the output from gdb, I apoligize for the formatting, I'm using a rather awkward web client. Interestingly, I found that it runs fine as user 'hobbit', but users root and apache get a segfault. Core was generated by `/usr/local/hobbit/server/bin/bb-eventlog.cgi'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 #0 0x08049502 in do_eventlog (output=0x9145c0, maxcount=100, maxminutes=140, allowallhosts=1) at eventlog.c:170 170 fprintf(output, "<TD ALIGN=CENTER BGCOLOR=%s><FONT COLOR=black>%s</FONT></TD>\n", (gdb) bt #0 0x08049502 in do_eventlog (output=0x9145c0, maxcount=100, maxminutes=140, allowallhosts=1) at eventlog.c:170 #1 0x08049c5b in main (argc=160256136, argv=0x98d50a3) at eventlog.c:338

2 1

A few hobbit problems
by EES2Y＠hscmail.mcc.virginia.edu 03 Apr '05

03 Apr '05

Hi all, We're in the process of migrating from BB to hobbit, and, for the most part, the process has been quick and painless (hobbit seems to be handling our 800+ hosts much better than BB did). We're seeing a few oddities, though; my apoligies if someone else has already pointed these out, I've only been on the list since earlier today and don't know how to search the archives. 1. We see the occaisional yellow message from bbtest: Error output: dns_queue_run deadlock - loops=260 During this problem, ~70 seconds get added to our "Test setup" phase as reported by bb-test. This only happens once or twice an hour so far, so its not a showstopper. We are using the --dns=ip option when calling the bbtest-net binary from hobbitlaunch. 2. The bb-eventlog.sh script dumps core. I've run it successfully in the past, but at some point, as we added more hosts to our bb-hosts file, it began to fail. Calling it from the command line: % setenv QUERY_STRING 'MAXTIME=140&MAXCOUNT=&Send=View+log' % /usr/local/hobbit/bb-eventlog.sh Segmentation fault (core dumped) TIA, -Eric Schwimmer Network Engineer University of Virginia HSCS

2 1

[hobbit] hobbitd coredumping and purple trends
by tbarnes1＠hfhs.org 03 Apr '05

03 Apr '05

I experienced same thing after making some changes to hobbit - might be a longshot, but here is what caused this for me. After restarting hobbit and receiving the same as you, found that some hobbit processes were hung. If I stopped hobbit - could still see most processes were still running. Even after multiple attempt to do a ~/server/hobbit.sh stop, the processes continued to run. Killed those processes and restart hobbit - problem solved. Like I say - could be a longshot, but worth a look. Terry Barnes Siemens Com @ HFHS 248-853-4968 (Office) 586-405-8382 (Cellular) 248-844-3030 (Fax) 5864058382 at messaging.nextel.com (Text Pager) tbarnes1 at hfhs.org >>> rdeal at tigr.org 4/1/05 5:22:42 PM >>> My hobbitd is core dumping every so often and less often but still occasional the trends column turns purple. Looking through the makefile the only oddity is MAXMSG=32768 Were my old BBd was set to #define MAXLINE 11264 I have core files in /tmp from hobbitd Logs : > more bb-display.log 2005-04-01 15:47:59 Whoops ! bb failed to send message - timeout 2005-04-01 16:02:59 Whoops ! bb failed to send message - timeout 2005-04-01 16:03:00 connect to bbd failed - Connection refused 2005-04-01 16:03:00 Whoops ! bb failed to send message - Connection failed 2005-04-01 16:03:00 connect to bbd failed - Connection refused 2005-04-01 16:03:00 Whoops ! bb failed to send message - Connection failed 2005-04-01 16:03:00 connect to bbd failed - Connection refused 2005-04-01 16:03:00 Whoops ! bb failed to send message - Connection failed 2005-04-01 16:18:05 Whoops ! bb failed to send message - timeout 2005-04-01 17:03:08 Whoops ! bb failed to send message - timeout > more hobbitd.log 2005-04-01 15:32:47 Setup complete 2005-04-01 15:32:54 Setup complete 2005-04-01 15:48:01 Setup complete 2005-04-01 16:03:01 Setup complete 2005-04-01 16:33:03 Setup complete 2005-04-01 16:48:04 Setup complete I have a lot of these errors in larrd-data.log from various hosts. 2005-04-01 17:17:53 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/ray1.tigr.org/netstat.rrd from 172.17.10.20: expected 12 data source readings (got 16) from 1112393873:597496849:203665680:0:1400608:474490:380897:4323:190:65584910 3:2750185864:9271815:54370878:358842800:919424657:55608:57615:... 2005-04-01 17:18:15 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/akela.tigr.org/netstat.rrd from 172.17.10.87: expected 12 data source readings (got 16) from 1112393894:7278664:4601574:0:2187293:80558:15408:1028:18:3786687185:3319 9304:551592:3055134:392628802:534540232:12324:8938:... 2005-04-01 17:18:22 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/vader.tigr.org/netstat.rrd from 172.16.4.50: expected 12 data source readings (got 16) from 1112393902:844147:844153:0:173177:11681993:15774:1756237:109:2946405093: 1171800154:1508:44541250:1263968085:53592252:29:1305303:... 2005-04-01 17:18:49 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/invino.tigr.org/netstat.rrd from 172.17.10.29: expected 12 data source readings (got 16) from 1112393929:161474660:161355279:0:979032:1013326:8108:2751:26:3077107260: 3115145104:3779497608:1171327:3474031250:2366740414:176290878:15382:... I used the moverrd.sh . And these errors from lard-status.log: 005-04-01 17:18:10 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/IGR51RRTB.tigr.org/temperature .module_6_asic-.rrd from 172.17.10.16: illegal attempt to update using time 1112393889 when last update time is 1112393889 (minimum one second step) 2005-04-01 17:20:04 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/utah.tigr.org/disk.rrd from 172.17.10.79: illegal attempt to update using time 1112394004 when last update time is 1112394004 (minimum one second step) 2005-04-01 17:20:04 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/utah.tigr.org/disk.rrd from 172.17.10.79: illegal attempt to update using time 1112394004 when last update time is 1112394004 (minimum one second step) 2005-04-01 17:21:27 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/atlas.tigr.org/netstat.rrd from 172.17.10.80: expected 11 data source readings (got 16) from 1112394087:23501770:2904610:0:97558:26724:76:17:8:U:U:U:U:226801128:2976 62863:U:956:... any suggestions? Thanks To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk ============================================================================== CONFIDENTIALITY NOTICE: This email contains information from the sender that may be CONFIDENTIAL, LEGALLY PRIVILEGED, PROPRIETARY or otherwise protected from disclosure. This email is intended for use only by the person or entity to whom it is addressed. If you are not the intended recipient, any use, disclosure, copying, distribution, printing, or any action taken in reliance on the contents of this email, is strictly prohibited. If you received this email in error, please contact the sending party by replying in an email to the sender, delete the email from your computer system and shred any paper copies of the email you printed. Note to Patients: There are a number of risks you should consider before using e-mail to communicate with us. These risks are described in our Privacy Policy at http://henryford.com. Review that policy carefully before continuing to communicate with us by e-mail. For greater Internet security, our policy describes the Henry Ford MyHealth electronic communication process - you may register at http://henryford.com. If you do not believe that our policy gives you the privacy and security protection you need, do not send e-mail or Internet communications to us. ==============================================================================

2 1

And now?
by dhd733＠yahoo.se 03 Apr '05

03 Apr '05

So, what happens now, when "we" have come this far with hobbitserver. Is it time for the hobbitclient? It has been very fun and nice to take part of this, since I'm sitting all alone at home making it. Thanks Henrik and all others. Regards Lars Hobbithobbyist "If you run UNIX and you don't have a UPS, you should see a psychiatrist...." --Byte Magazine (years ago)

2 1

[hobbit] trends columns all turned purple
by rdeal＠tigr.org 02 Apr '05

02 Apr '05

I tried but I still don't get an actual list of hosts just the option for "ALL". The trends column has remained green though. It may have been from starting and stopping a few times when I was testing at the beginning that the history logs were created. -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Friday, April 01, 2005 1:34 AM To: hobbit at hswn.dk Subject: Re: [hobbit] trends columns all turned purple On Thu, Mar 31, 2005 at 06:04:14PM -0500, Deal, Richard wrote: [trends column goes purple] > Has anyone else notices maint.pl lists now hosts, only "ALL"? Neither of these show up on any of the hosts I've tested. sounds like there's some problem with the hobbit daemon on your box, could you try stopping Hobbit, then do a "ps" to make sure all of the tasks have stopped, then restarting it ? Regards, Henrik

3 9

hobbitd coredumping and purple trends
by rdeal＠tigr.org 02 Apr '05

02 Apr '05

My hobbitd is core dumping every so often and less often but still occasional the trends column turns purple. Looking through the makefile the only oddity is MAXMSG=32768 Were my old BBd was set to #define MAXLINE 11264 I have core files in /tmp from hobbitd Logs : > more bb-display.log 2005-04-01 15:47:59 Whoops ! bb failed to send message - timeout 2005-04-01 16:02:59 Whoops ! bb failed to send message - timeout 2005-04-01 16:03:00 connect to bbd failed - Connection refused 2005-04-01 16:03:00 Whoops ! bb failed to send message - Connection failed 2005-04-01 16:03:00 connect to bbd failed - Connection refused 2005-04-01 16:03:00 Whoops ! bb failed to send message - Connection failed 2005-04-01 16:03:00 connect to bbd failed - Connection refused 2005-04-01 16:03:00 Whoops ! bb failed to send message - Connection failed 2005-04-01 16:18:05 Whoops ! bb failed to send message - timeout 2005-04-01 17:03:08 Whoops ! bb failed to send message - timeout > more hobbitd.log 2005-04-01 15:32:47 Setup complete 2005-04-01 15:32:54 Setup complete 2005-04-01 15:48:01 Setup complete 2005-04-01 16:03:01 Setup complete 2005-04-01 16:33:03 Setup complete 2005-04-01 16:48:04 Setup complete I have a lot of these errors in larrd-data.log from various hosts. 2005-04-01 17:17:53 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/ray1.tigr.org/netstat.rrd from 172.17.10.20: expected 12 data source readings (got 16) from 1112393873:597496849:203665680:0:1400608:474490:380897:4323:190:65584910 3:2750185864:9271815:54370878:358842800:919424657:55608:57615:... 2005-04-01 17:18:15 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/akela.tigr.org/netstat.rrd from 172.17.10.87: expected 12 data source readings (got 16) from 1112393894:7278664:4601574:0:2187293:80558:15408:1028:18:3786687185:3319 9304:551592:3055134:392628802:534540232:12324:8938:... 2005-04-01 17:18:22 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/vader.tigr.org/netstat.rrd from 172.16.4.50: expected 12 data source readings (got 16) from 1112393902:844147:844153:0:173177:11681993:15774:1756237:109:2946405093: 1171800154:1508:44541250:1263968085:53592252:29:1305303:... 2005-04-01 17:18:49 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/invino.tigr.org/netstat.rrd from 172.17.10.29: expected 12 data source readings (got 16) from 1112393929:161474660:161355279:0:979032:1013326:8108:2751:26:3077107260: 3115145104:3779497608:1171327:3474031250:2366740414:176290878:15382:... I used the moverrd.sh . And these errors from lard-status.log: 005-04-01 17:18:10 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/IGR51RRTB.tigr.org/temperature .module_6_asic-.rrd from 172.17.10.16: illegal attempt to update using time 1112393889 when last update time is 1112393889 (minimum one second step) 2005-04-01 17:20:04 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/utah.tigr.org/disk.rrd from 172.17.10.79: illegal attempt to update using time 1112394004 when last update time is 1112394004 (minimum one second step) 2005-04-01 17:20:04 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/utah.tigr.org/disk.rrd from 172.17.10.79: illegal attempt to update using time 1112394004 when last update time is 1112394004 (minimum one second step) 2005-04-01 17:21:27 RRD error updating /local/packages/IT/HOBBIT/hobbit/data/rrd/atlas.tigr.org/netstat.rrd from 172.17.10.80: expected 11 data source readings (got 16) from 1112394087:23501770:2904610:0:97558:26724:76:17:8:U:U:U:U:226801128:2976 62863:U:956:... any suggestions? Thanks

3 3

External test with LAARD graph
by rsmrcina＠wi.rr.com 02 Apr '05

02 Apr '05

How do get Hobbit to trigger a LAARD graph on an external test that is coming in to Hobbit from a client? The results of the test is a number that I would like to have graphed over time. Thanks. -- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (866)569-7378 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2005 - Colorado Springs - May 20-24, 2005

2 2

[hobbit] trends columns all turned purple
by David.Gore＠mci.com 01 Apr '05

01 Apr '05

David Gore (v965-3670) Enhanced Technology Support (ETS) Network Management Systems (NMS) IMPACT Transport Team Lead - SCSA, SCNA Page: 1-800-PAG-eMCI pin 1406090 Vnet: 965-3676 David Gore (v965-3670) Enhanced Technology Support (ETS) Network Management Systems (NMS) IMPACT Transport Team Lead - SCSA, SCNA Page: 1-800-PAG-eMCI pin 1406090 Vnet: 965-3676 David Gore wrote: > > David Gore wrote: > >> Yes one core file: >> >> hobbit at hobbit ~/server> find . -name core >> ./tmp/core >> hobbit at hobbit ~/server> file tmp/core >> tmp/core: ELF 32-bit MSB core file SPARC Version 1, from 'hobbitd' >> hobbit at hobbit ~/server> ls -al tmp/core >> -rw------- 1 hobbit other 8322084 Apr 1 03:27 tmp/core >> >> I checked to make sure the lines were correct in maint.pl. I am >> reinstalling 4.0.1, again and will let you know how it goes. >> >> ~David >> >> Henrik Storner wrote: >> >>> OK, I think there are two completely un-related issues here. >>> >>> >>> In <424D7FE3.1040600 at mci.com> David Gore <David.Gore at mci.com> writes: >>> >>> >>>> I re-installed RC6 to get rid of the purple trends. >>> >>> >>> >>> >>> The "trends" don't go purple by themselves, they go purple because the >>> bb-larrdcolumn tool doesn't update them. >>> >>> I'd like you to check for unusual messages in the >>> /var/log/hobbit/bb-display.log file, and also for any core-files left >>> behind from bb-larrdcolumn. They should be in the ~/server/tmp/ >>> directory, but please check in the ~/data/rrd/ directory also. >>> >>> >>> >>> >>>> The maint.pl lists and 'ALL' may still be a problem, it does occur in >>>> both Firefox and IE. It can be alleviated by changing views. >>>> Typically we monitor bb2.html, but if we switch to the Main view and >>>> try enable/disable, it typically works, this only occurred since RC6, >>>> I think. >>> >>> >>> >>> >>> The missing hosts on the maint.pl display sounds like you're still >>> running the version of maint.pl that uses cookies to try and display >>> only the hosts from the page you were on. This was removed in the 4.0 >>> release precisely because it was causing problems. Could you check >>> your maint.pl script and see if lines 432-433 look like this: >>> >>> 432 # open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." >>> \"hobbitdboard ".$filter."\" |"); >>> 433 open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." hobbitdboard |"); >>> >>> >>> Thanks, >>> Henrik >>> >>> To unsubscribe from the hobbit list, send an e-mail to >>> hobbit-unsubscribe at hswn.dk >>> >>> >> > New core and the purples are back: > > find -namehobbit at hobbit ~/server> find -name core -ls > 141454 8024 -rw------- 1 hobbit other 8207396 Apr 1 22:08 > ./tmp/core > hobbit at hobbit ~/server> file ./tmp/core > ./tmp/core: ELF 32-bit MSB core file SPARC Version 1, from 'hobbitd' > One message in bb-display.log: 2005-04-01 22:07:55 Whoops ! bb failed to send message - timeout We do have close to a 1000 entries in bb-hosts, if it matters.

1 0

[hobbit] trends columns all turned purple
by rdeal＠tigr.org 01 Apr '05

01 Apr '05

Ok I found the problem to maint.pl It is hardcoded to use the default bb port of 1984 I changed line 433 from: open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." hobbitdboard |"); to : open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}.":".$BBENV{'BBPORT'}." hobbitdboard |"); and now it works. -----Original Message----- From: Henrik Storner [mailto:henrik at hswn.dk] Sent: Friday, April 01, 2005 12:19 PM To: hobbit at hswn.dk Subject: Re: [hobbit] trends columns all turned purple OK, I think there are two completely un-related issues here. In <424D7FE3.1040600 at mci.com> David Gore <David.Gore at mci.com> writes: >I re-installed RC6 to get rid of the purple trends. The "trends" don't go purple by themselves, they go purple because the bb-larrdcolumn tool doesn't update them. I'd like you to check for unusual messages in the /var/log/hobbit/bb-display.log file, and also for any core-files left behind from bb-larrdcolumn. They should be in the ~/server/tmp/ directory, but please check in the ~/data/rrd/ directory also. >The maint.pl lists and 'ALL' may still be a problem, it does occur in >both Firefox and IE. It can be alleviated by changing views. >Typically we monitor bb2.html, but if we switch to the Main view and >try enable/disable, it typically works, this only occurred since RC6, >I think. The missing hosts on the maint.pl display sounds like you're still running the version of maint.pl that uses cookies to try and display only the hosts from the page you were on. This was removed in the 4.0 release precisely because it was causing problems. Could you check your maint.pl script and see if lines 432-433 look like this: 432 # open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." \"hobbitdboard ".$filter."\" |"); 433 open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." hobbitdboard |"); Thanks, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

[hobbit] trends columns all turned purple
by rdeal＠tigr.org 01 Apr '05

01 Apr '05

My maint.pl lines are: # open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." \"hobbitdboard".$filter."\" |"); open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." hobbitdboard |"); I have a core from hobbitd in /tmp > file tmp/core tmp/core: ELF 32-bit MSB core file SPARC Version 1, from 'hobbitd' > more hobbitd.log 2005-04-01 12:52:47 Setup complete 2005-04-01 12:52:54 Setup complete > more larrd-data.log > more larrd-status.log > more bb-display.log I had it stopped for about two hours while I copied history and rrd over when I stared it back up trends and now info went purple. Right now I am running on port 201984 for testing but other than that it is pretty straight forward install. -----Original Message----- From: Henrik Storner [mailto:henrik at hswn.dk] Sent: Friday, April 01, 2005 12:19 PM To: hobbit at hswn.dk Subject: Re: [hobbit] trends columns all turned purple OK, I think there are two completely un-related issues here. In <424D7FE3.1040600 at mci.com> David Gore <David.Gore at mci.com> writes: >I re-installed RC6 to get rid of the purple trends. The "trends" don't go purple by themselves, they go purple because the bb-larrdcolumn tool doesn't update them. I'd like you to check for unusual messages in the /var/log/hobbit/bb-display.log file, and also for any core-files left behind from bb-larrdcolumn. They should be in the ~/server/tmp/ directory, but please check in the ~/data/rrd/ directory also. >The maint.pl lists and 'ALL' may still be a problem, it does occur in >both Firefox and IE. It can be alleviated by changing views. >Typically we monitor bb2.html, but if we switch to the Main view and >try enable/disable, it typically works, this only occurred since RC6, >I think. The missing hosts on the maint.pl display sounds like you're still running the version of maint.pl that uses cookies to try and display only the hosts from the page you were on. This was removed in the 4.0 release precisely because it was causing problems. Could you check your maint.pl script and see if lines 432-433 look like this: 432 # open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." \"hobbitdboard ".$filter."\" |"); 433 open (HOBBITDLIST, "bb ".$BBENV{'BBDISP'}." hobbitdboard |"); Thanks, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0