Hi there,
I am monitoring a FreeBSD 4.11 box with Hobbit, and it works greats except for an occasional hobbitd_client process crash. I have been noticing this because I'll see a purple icon on my monitoring web page. This shows up in the /var/log/hobbit/clientdata.log file on the FBSD box:
2006-03-05 12:01:12 Worker process died with exit code 134, terminating 2006-03-05 12:01:18 Worker process died with exit code 134, terminating 2006-03-07 18:16:05 Worker process died with exit code 134, terminating 2006-03-07 18:16:10 Worker process died with exit code 134, terminating
It looks to me that the hobbitd_client process is respawned, because all I have to do is reset the display with a command like:
./bb 127.0.0.1 "drop my.host.name hobbitd_client"
That gets rid of the purple icon.
I found a core file for the process, but it doesn't yield much:
-su-2.05b$ pwd /home/hobbit/server/tmp -su-2.05b$ gdb ../bin/hobbitd_client hobbitd_client.core GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf
Core was generated by `hobbitd_client'. Program terminated with signal 6, Abort trap. Reading symbols from /usr/local/lib/libpcre.so.0...done. Reading symbols from /usr/lib/libc.so.4...done. Reading symbols from /usr/libexec/ld-elf.so.1...done. #0 0x280c7748 in kill () from /usr/lib/libc.so.4 (gdb) bt #0 0x280c7748 in kill () from /usr/lib/libc.so.4 #1 0x2810903e in abort () from /usr/lib/libc.so.4 #2 0x80551a6 in sigsegv_handler (signum=11) at sig.c:57 #3 0xbfbfffac in ?? () #4 0x804d8b9 in main (argc=1, argv=0xbfbfeadc) at hobbitd_client.c:853 (gdb)
How serious is this problem? Should I just recompile hobbitd_client with debugging symbols so I can get a better backtrace? Is there anything else I should do?
Thank you, Greg
SourceHosting.net, LLC Ready. Set. Code. http://www.sourcehosting.net/
On Tue, Mar 07, 2006 at 09:03:13PM -0500, Greg Larkin wrote:
Hi there,
I am monitoring a FreeBSD 4.11 box with Hobbit, and it works greats except for an occasional hobbitd_client process crash. I have been noticing this because I'll see a purple icon on my monitoring web page.
How serious is this problem? Should I just recompile hobbitd_client with debugging symbols so I can get a better backtrace? Is there anything else I should do?
First thing I'd suggest is an upgrade to 4.1.2p1 which has a number of fixes for the client code.
Regards, Henrik
-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Wednesday, March 08, 2006 1:28 AM To: hobbit at hswn.dk Subject: Re: [hobbit] hobbitd_client crashing on FreeBSD 4.11
On Tue, Mar 07, 2006 at 09:03:13PM -0500, Greg Larkin wrote:
Hi there,
I am monitoring a FreeBSD 4.11 box with Hobbit, and it works greats except for an occasional hobbitd_client process crash. I have been noticing this because I'll see a purple icon on my monitoring web page.
How serious is this problem? Should I just recompile hobbitd_client with debugging symbols so I can get a better backtrace? Is there anything else I should do?
First thing I'd suggest is an upgrade to 4.1.2p1 which has a number of fixes for the client code.
Regards, Henrik
Hi Henrik,
Sorry, I should have clarified - my operating system is FreeBSD 4.11, and I am running Hobbit 4.1.2p1 on all of the client and server machines.
Regards, Greg
On Wed, Mar 08, 2006 at 08:51:44AM -0500, Greg Larkin wrote:
Sorry, I should have clarified - my operating system is FreeBSD 4.11, and I am running Hobbit 4.1.2p1 on all of the client and server machines.
Ah OK - I saw the "4.11" and thought you meant "Hobbit 4.1.1".
The gdb trace shows that this happens while processing a "client" message from a FreeBSD server.
I'd like to see the client message that triggers this. Since it only happens occasionally (not every 5 minutes), could you setup some tracing for me:
- Add the option "--trace=your.freebsd.box" to the hobbitd command in hobbbitlaunch.cfg. If you have more than one such box, just list the hostnames like "--trace=boxA,boxB,...."
- Wait until it crashes - i.e. look for the hobbitd_client status to show up.
- Send me the ~hobbit/server/tmp/*.trace files that were generated around the time that hobbitd_client crashed.
Regards, Henrik
-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Thursday, March 09, 2006 12:06 PM To: hobbit at hswn.dk Subject: Re: [hobbit] hobbitd_client crashing on FreeBSD 4.11
On Wed, Mar 08, 2006 at 08:51:44AM -0500, Greg Larkin wrote:
Sorry, I should have clarified - my operating system is FreeBSD 4.11,
and I
am running Hobbit 4.1.2p1 on all of the client and server machines.
Ah OK - I saw the "4.11" and thought you meant "Hobbit 4.1.1".
The gdb trace shows that this happens while processing a "client" message from a FreeBSD server.
I'd like to see the client message that triggers this. Since it only happens occasionally (not every 5 minutes), could you setup some tracing for me:
- Add the option "--trace=your.freebsd.box" to the hobbitd command in hobbbitlaunch.cfg. If you have more than one such box, just list the hostnames like "--trace=boxA,boxB,...."
- Wait until it crashes - i.e. look for the hobbitd_client status to show up.
- Send me the ~hobbit/server/tmp/*.trace files that were generated around the time that hobbitd_client crashed.
Regards, Henrik
Hi Henrik,
Ok, I have modified that line in hobbitlaunch.cfg and restarted Hobbit. I'll let you know if I see the crash again, but it may take a week or so, if it happens at all.
Thank you, Greg
participants (2)
-
glarkin＠sourcehosting.net
-
henrik＠hswn.dk