I just realized I left out the actual backtrace. Here's one I just ran again:
0xb7584416 in semop () from /lib/tls/libc.so.6 (gdb) bt #0 0xb7584416 in semop () from /lib/tls/libc.so.6 #1 0x08049f27 in posttochannel (channel=0xbfffc740, channelmarker=0x80538b2 "status", msg=0x8101868 "status uberbox.sys green Tue Jan 18 16:51:44 2005 \nSystem Description NetApp Release 6.5.2: Sun Jul 25 10:56:02 PDT 2004\n\nObject ID", ' ' <repeats 11 times>, ".1.3.6.1.4.1.789.2.1\n\nUptime", ' ' <repeats 14 times>, "143 days, 10:42"..., sender=0x8053bbc "hobbitd", hostname=0x81017f0 "uberbox", log=0x8101800, readymsg=0x0) at hobbitd.c:390 #2 0x0804ab22 in handle_status ( msg=0x8101868 "status uberbox.sys green Tue Jan 18 16:51:44 2005 \nSystem Description NetApp Release 6.5.2: Sun Jul 25 10:56:02 PDT 2004\n\nObject ID", ' ' <repeats 11 times>, ".1.3.6.1.4.1.789.2.1\n\nUptime", ' ' <repeats 14 times>, "143 days, 10:42"..., sender=0x8053bbc "hobbitd", hostname=0x81017f0 "uberbox", testname=0x7e6bf <Address 0x7e6bf out of bounds>, log=0x8101800, newcolor=3) at hobbitd.c:830 #3 0x0804df9a in check_purple_status () at hobbitd.c:2001 #4 0x0804ebff in main (argc=7, argv=0xbfffcb14) at hobbitd.c:2359
-- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer
-----Original Message----- From: Bruce Lysik Sent: Wednesday, January 19, 2005 9:35 AM To: hobbit at hswn.dk Subject: RE: [hobbit] hosts still disappear
Correct, beta-5, plus the patch.
Okay, hobbitd isn't responding at all. hobbitd.chk hasn't been updated since yesterday, and the hobbitlog command doesn't return anything.
Strangely, no pid file was located in /var/log/hobbit. However, a ps grepping for hobbit showed this:
bbuser 32296 32295 0 Jan18 ? 00:00:00 hobbitd --restart=/opt/bb/server/tmp/hobbitd.chk --checkpoint-file=/opt/bb/server/tmp/hobbitd.chk --checkpoint-interval=600 --purple-conn=conn --log=/var/log/hobbit/hobbitd.log --admin-senders=127.0.0.1 172.16.190.2
I used gdb and attached to that pid, resulting in:
(gdb) attach 32296 Attaching to process 32296 Reading symbols from /opt/bb/server/bin/hobbitd...done. Using host libthread_db library "/lib/tls/libthread_db.so.1". Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 0xb7584416 in semop () from /lib/tls/libc.so.6 (gdb)
-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Tue 1/18/2005 10:58 PM To: hobbit at hswn.dk Subject: Re: [hobbit] hosts still disappear
On Tue, Jan 18, 2005 at 03:17:51PM -0800, Bruce Lysik wrote:
Well, I spoke too soon about hosts disappearing. I'm still
seeing the same
problem. Is there anything I can do to help in debugging this?
This is with the beta-5 version PLUS the patch I posted yesterday ?
First, check if the hobbit daemon is still responding at all. You can see if the ~server/tmp/hobbitd.chk file is being updated - it should refresh every 10 minutes. You can also try this: If
~/server/bin/bb 127.0.0.1 "hobbitdlog some,server,name.conn"
responds with a status log, then it is responding. If not, proceed:
Next, I want to figure out what the hobbitd proces is doing. You'll need the gdb (GNU debugger) for that. Login as the hobbit user, Find the process ID of hobbitd - it's recorded in /var/log/hobbit/hobbitd.pid, or use "ps" - then:
gdb ~/server/bin/hobbitd [gdb messages] gdb> attach <hobbitd-processid> gdb> bt
Let me know what this shows.
Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk