Yes. Quite often. ---snip--- 2005-07-04 14:09:17 Whoops ! bb failed to send message - timeout 2005-07-04 14:09:17 Could not get the Hobbit statuslog-list 2005-07-04 14:09:50 Whoops ! bb failed to send message - timeout 2005-07-04 14:09:50 hobbitd status-board not available 2005-07-04 14:10:49 Whoops ! bb failed to send message - timeout 2005-07-04 14:10:49 hobbitd status-board not available 2005-07-04 14:11:49 Whoops ! bb failed to send message - timeout 2005-07-04 14:11:49 hobbitd status-board not available 2005-07-04 14:12:52 Whoops ! bb failed to send message - timeout 2005-07-04 14:12:52 hobbitd status-board not available 2005-07-04 14:13:50 Whoops ! bb failed to send message - timeout 2005-07-04 14:13:50 hobbitd status-board not available 2005-07-04 14:14:50 Whoops ! bb failed to send message - timeout 2005-07-04 14:14:50 hobbitd status-board not available 2005-07-04 14:16:22 Whoops ! bb failed to send message - timeout 2005-07-04 14:16:22 hobbitd status-board not available 2005-07-04 14:16:22 WARNING: Runtime 61 longer than BBSLEEP (60) 2005-07-04 14:16:52 Whoops ! bb failed to send message - timeout 2005-07-04 14:16:52 hobbitd status-board not available 2005-07-04 14:17:52 Whoops ! bb failed to send message - timeout 2005-07-04 14:17:52 hobbitd status-board not available 2005-07-04 14:18:52 Whoops ! bb failed to send message - timeout 2005-07-04 14:18:52 hobbitd status-board not available 2005-07-04 14:19:52 Whoops ! bb failed to send message - timeout 2005-07-04 14:19:52 hobbitd status-board not available 2005-07-04 14:21:26 Whoops ! bb failed to send message - timeout 2005-07-04 14:21:26 hobbitd status-board not available 2005-07-04 14:21:26 WARNING: Runtime 61 longer than BBSLEEP (60) 2005-07-04 14:21:59 Whoops ! bb failed to send message - timeout 2005-07-04 14:21:59 hobbitd status-board not available ---snip---
-----Original Message----- From: Stefan Loos [mailto:stefan_loos at hotmail.com] Sent: Monday, 4 July 2005 2:16 PM To: hobbit at hswn.dk Subject: RE: [hobbit] Status Unavailable
Hello Vernon,
can you tell me, if there is anything like "hobbitd status board not available" in the bb-display.log?
Regards,
Stefan
<br><br><br>>From: "Vernon Everett" <v.everett at afgonline.com.au><br>>Reply-To: hobbit at hswn.dk<br>>To: <hobbit at hswn.dk><br>>Subject: RE: [hobbit] Status Unavailable<br>>Date: Fri, 1 Jul 2005 16:56:38 +0800<br>><br>>Hi Henrik<br>><br>>It should be idle. All the system does is run hobbit. :-)<br>><br>>Hobbitd is currently dead in the water.<br>> [root at pengo log]# strace -p 3025<br>> Process 3025 attached - interrupt to quit<br>> futex(0x40141b20, FUTEX_WAIT, 2,
NULL<br>><br>>And it's been like this a while.<br>>When I did the kill -6 I got this.<br>> [root at pengo log]# strace -p 3025<br>> Process 3025 attached - interrupt to quit<br>> futex(0x40141b20, FUTEX_WAIT, 2, NULL) = -1 EINTR (Interrupted<br>>system call)<br>> --- SIGABRT (Aborted) @ 0 (0) ---<br>> Process 3025 detached<br>>Which I suppose was expected :-)<br>><br>>I restarted it, and got this.<br>> [root at pengo etc]# strace -p 9223<br>> Process 9223 attached
- interrupt to quit<br>> semop(32769, 0xbfffe3a0, 1<br>>Nope, there is nothing I forgot to cut and paste.<br>>That really was it.<br>><br>>And this shit just gets stranger and stranger.<br>>It isn't dumping core.<br>>I hit it with a kill -6 and nothing happens.<br>>I then thought maybe we were both mistaken, and had the command wrong or<br>>my linux was defaulted to not core, so I started vi in a session and did<br>>a kill -6 on that. That dumped?!<br>>Hobbit isn't dumping.<br>><br>>I rebooted and tried again.<br>>I managed to get a nice strace output - see attached
- but still no damn<br>>core.<br>><br>>OK, I added debug, and
restarted.<br>>When I went to check the logs, I found this in
hobbitlaunch.log.<br>>---snip---<br>>2005-07-01 16:37:21 Loading
tasklist configuration
from<br>>/usr/lib/hobbit/server/etc/hobbitlaunch.cfg<br>>2005-07-0
1
16:37:21 Loading hostnames<br>>2005-07-01 16:37:21 Loading saved
state<br>>2005-07-01 16:37:21 Setting up network listener on
0.0.0.0:1984<br>>2005-07-01 16:37:21 Cannot bind to listen socket
(Address already in<br>>use)<br>>2005-07-01 16:37:21 Task hobbitd
started with PID 4761<br>>2005-07-01 16:37:26 Task hobbitd
terminated, status 1<br>>2005-07-01 16:37:26 Loading
hostnames<br>>2005-07-01
16:37:26 Loading saved state<br>>2005-07-01 16:37:26 Task hobbitd
started with PID 4765<br>>2005-07-01 16:37:26 Setting up network
listener on
0.0.0.0:1984<br>>2005-07-01 16:37:26 Cannot bind to listen socket
(Address already in<br>>use)<br>>2005-07-01 16:37:26 Task hobbitd
terminated, status 1<br>>2005-07-01 16:37:31 Loading
hostnames<br>>2005-07-01 16:37:31 Loading saved
state<br>>2005-07-01
16:37:31 Task hobbitd started with PID 4770<br>>2005-07-01 16:37:31
Setting up network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:31
Cannot bind to listen socket (Address already
in<br>>use)<br>>2005-07-01 16:37:31 Task hobbitd terminated,
status
1<br>>2005-07-01 16:37:36 Task hobbitd started with PID
4774<br>>2005-07-01 16:37:36 Loading hostnames<br>>2005-07-01
16:37:36 Loading saved state<br>>2005-07-01 16:37:36 Setting up
network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:36 Cannot bind
to listen socket (Address already in<br>>use)<br>>2005-07-01
16:37:36 Task hobbitd terminated, status 1<br>>2005-07-01 16:37:41
Task hobbitd started with PID 4778<br>>2005-07-01 16:37:41 Loading
hostnames<br>>2005-07-01
16:37:41 Loading saved state<br>>2005-07-01 16:37:41 Setting up
network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:41 Cannot bind
to listen socket (Address already in<br>>use)<br>>2005-07-01
16:37:41 Task hobbitd terminated, status 1<br>>2005-07-01 16:37:46
Task hobbitd started with PID 4783<br>>2005-07-01 16:37:46 Loading
hostnames<br>>2005-07-01
16:37:46 Loading saved state<br>>2005-07-01 16:37:46 Setting up
network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:46 Cannot bind
to listen socket (Address already in<br>>use)<br>>2005-07-01
16:37:46 Task hobbitd terminated, status
1<br>>---snip---<br>><br>>Looks like a clue.<br>>I will add
the output of netstat -a<br>><br>>Got the hobbitd.log file for you
too.<br>><br>>Let me know if there is
anything else I can get you.<br>><br>>Regards<br>>
Vernon<br>><br>>P.S. Your cold one is quickly becoming many cold ones if you ever get to<br>>Perth<br>><br>><br>><br>><br>><br>>-----Orig inal Message-----<br>>From: Henrik Stoerner [mailto:henrik at hswn.dk]<br>>Sent: Friday, 1 July 2005 3:38 PM<br>>To: hobbit at hswn.dk<br>>Subject: Re: [hobbit] Status Unavailable<br>><br>>On Fri, Jul 01, 2005 at 03:25:30PM +0800, Vernon Everett wrote:<br>> > Thanks for helping on this.<br>> > I rebooted this morning. Could the memory leak still effect me in that<br>><br>> > short time?<br>><br>>Probably not. Just wanted to rule out this possibility.<br>><br>> > No "failed allocation" in dmesg output.<br>> > Do you want the full output?<br>><br>>No, I dont think that is necessary.<br>><br>> > [root at pengo log]# vmstat 4 20<br>><br>>And your system is mostly idle with no swap or disk activity.<br>><br>> > [hobbit at pengo hobbit]$ server/bin/bb 127.0.0.1 "hobbitdboard"<br>> > 2005-07-01 15:21:45 Whoops ! bb failed to send message - timeout<br>><br>>Could you try running "strace -p <process-ID of the hobbitd process>"<br>>for a minute or two and send me the output, then do a "kill -6<br>><process-id>" and mail me the core-file from ~hobbit/server/tmp/<br>>together with the ~hobbit/server/bin/hobbitd file ?<br>><br>>Also, after this try adding a "--debug" to the hobbitd commandline in<br>>hobbitlaunch.cfg. Let it run for a while and then mail me the<br>>hobbitd.log file.<br>><br>>This bug sounds a bit nasty, I think ....<br>><br>><br>>Regards,<br>>Henrik<br>><br>><br>&g t;To unsubscribe from the hobbit list, send an e-mail to<br>>hobbit-unsubscribe at hswn.dk<br>><br>><br>>_ _ _ _ _ _
_<br>><br>>NOTICE: This message and any attachments are confidential and may contain copyright material<br>>of Australian Finance Group Limited or a third party. It is intended solely for the purpose of the<br>>addressee and any other named recipient. If you are not the intended recipient, any use,<br>>distribution, disclosure or copying of this message is strictly prohibited. The confidentiality attached<br>>to this message is not waived or lost by reason of the mistaken transmission or delivery to any<br>>unintended party. If you have received this message in error, please notify the author immediately or<br>>contact Australian Finance Group on +61 8 9420 7888.<br>><br>><br>>To unsubscribe from the hobbit list, send an e-mail to<br>>hobbit-unsubscribe at hswn.dk<br>><br>><br>
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
NOTICE: This message and any attachments are confidential and may contain copyright material of Australian Finance Group Limited or a third party. It is intended solely for the purpose of the addressee and any other named recipient. If you are not the intended recipient, any use, distribution, disclosure or copying of this message is strictly prohibited. The confidentiality attached to this message is not waived or lost by reason of the mistaken transmission or delivery to any unintended party. If you have received this message in error, please notify the author immediately or contact Australian Finance Group on +61 8 9420 7888.
And can you stop the hobbit server with hobbit.sh or is one process still running after that?
<br><br><br>>From: "Vernon Everett" <v.everett at afgonline.com.au><br>>Reply-To: hobbit at hswn.dk<br>>To: <hobbit at hswn.dk><br>>Subject: RE: [hobbit] Status Unavailable<br>>Date: Mon, 4 Jul 2005 14:23:56 +0800<br>><br>>Yes.<br>>Quite often.<br>>---snip---<br>>2005-07-04 14:09:17 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:09:17 Could not get the Hobbit statuslog-list<br>>2005-07-04 14:09:50 Whoops ! bb failed to send message
- timeout<br>>2005-07-04 14:09:50 hobbitd status-board not
available<br>>2005-07-04 14:10:49 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:10:49 hobbitd status-board not
available<br>>2005-07-04 14:11:49 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:11:49 hobbitd status-board not
available<br>>2005-07-04 14:12:52 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:12:52 hobbitd status-board not
available<br>>2005-07-04 14:13:50 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:13:50 hobbitd status-board not
available<br>>2005-07-04 14:14:50 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:14:50 hobbitd status-board not
available<br>>2005-07-04 14:16:22 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:16:22 hobbitd status-board not
available<br>>2005-07-04 14:16:22 WARNING: Runtime 61 longer than BBSLEEP
(60)<br>>2005-07-04 14:16:52 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:16:52 hobbitd status-board not
available<br>>2005-07-04 14:17:52 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:17:52 hobbitd status-board not
available<br>>2005-07-04 14:18:52 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:18:52 hobbitd status-board not
available<br>>2005-07-04 14:19:52 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:19:52 hobbitd status-board not
available<br>>2005-07-04 14:21:26 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:21:26 hobbitd status-board not
available<br>>2005-07-04 14:21:26 WARNING: Runtime 61 longer than BBSLEEP
(60)<br>>2005-07-04 14:21:59 Whoops ! bb failed to send message -
timeout<br>>2005-07-04 14:21:59 hobbitd status-board not
available<br>>---snip---<br>><br>><br>>-----Original
Message-----<br>>From: Stefan Loos
[mailto:stefan_loos at hotmail.com]<br>>Sent: Monday, 4 July 2005 2:16
PM<br>>To: hobbit at hswn.dk<br>>Subject: RE: [hobbit] Status
Unavailable<br>><br>>Hello Vernon,<br>><br>>can you tell me, if
there is anything like "hobbitd status board not<br>>available"
in the
bb-display.log?<br>><br>>Regards,<br>><br>>Stefan<br>><br>><br><br><br>>From:
"Vernon
Everett"<br>><v.everett at afgonline.com.au><br>>Reply-To:<br>>hobbit at hswn.dk<br>>To:
<hobbit at hswn.dk><br>>Subject: RE:<br>>[hobbit]
Status Unavailable<br>>Date: Fri, 1 Jul 2005
16:56:38<br>>+0800<br>><br>>Hi
Henrik<br>><br>>It should be idle. All
the<br>>system does is run hobbit.
:-)<br>><br>>Hobbitd is currently
dead<br>>in<br>>the water.<br>> [root at pengo log]# strace
-p 3025<br>><br>>Process 3025<br>>attached - interrupt to
quit<br>> futex(0x40141b20, FUTEX_WAIT,
2,<br>><br>>NULL<br>><br>>And it's been like
this a while.<br>>When I did<br>>the<br>>kill -6 I got
this.<br>> [root at pengo log]# strace -p
3025<br>><br>>Process<br>>3025 attached - interrupt to
quit<br>> futex(0x40141b20,<br>>FUTEX_WAIT, 2,<br>>NULL)
= -1 EINTR (Interrupted<br>>system call)<br>> ---<br>>SIGABRT<br>>(Aborted) @ 0 (0) ---<br>> Process 3025 detached<br>>Which I<br>>suppose<br>>was expected :-)<br>><br>>I restarted it, and got<br>>this.<br>> [root at pengo etc]# strace -p 9223<br>> Process<br>>9223 attached<br>>- interrupt to quit<br>> semop(32769, 0xbfffe3a0, 1<br>>Nope,<br>>there is<br>>nothing I forgot to cut and paste.<br>>That really was<br>>it.<br>><br>>And this shit just gets stranger and<br>>stranger.<br>>It isn't dumping core.<br>>I hit it with a kill -6<br>>and nothing happens.<br>>I then thought maybe we were both mistaken,<br>>and had the command wrong or<br>>my linux was defaulted to not core,<br>>so I started vi in a session and did<br>>a kill -6 on that. That<br>>dumped?!<br>>Hobbit isn't dumping.<br>><br>>I rebooted and<br>>tried again.<br>>I managed to get a nice strace output - see attached<br>>- but still no damn<br>>core.<br>><br>>OK, I added debug, and<br>>restarted.<br>>When I went to check the logs, I found this in<br>>hobbitlaunch.log.<br>>---snip---<br>>2005-07-01 16:37:21 Loading<br>>tasklist configuration<br>>from<br>>/usr/lib/hobbit/server/etc/hobbitlaunch.cfg<br>>2005-07-0<br>>1<br>>16:37:21 Loading hostnames<br>>2005-07-01 16:37:21 Loading saved<br>>state<br>>2005-07-01 16:37:21 Setting up network listener on<br>>0.0.0.0:1984<br>>2005-07-01 16:37:21 Cannot bind to listen socket<br>>(Address already in<br>>use)<br>>2005-07-01 16:37:21 Task hobbitd<br>>started with PID 4761<br>>2005-07-01 16:37:26 Task hobbitd<br>>terminated, status 1<br>>2005-07-01 16:37:26 Loading<br>>hostnames<br>>2005-07-01<br>>16:37:26 Loading saved state<br>>2005-07-01 16:37:26 Task hobbitd<br>>started with PID 4765<br>>2005-07-01 16:37:26 Setting up network<br>>listener on<br>>0.0.0.0:1984<br>>2005-07-01 16:37:26 Cannot bind to listen socket<br>>(Address already in<br>>use)<br>>2005-07-01 16:37:26 Task hobbitd<br>>terminated, status 1<br>>2005-07-01 16:37:31 Loading<br>>hostnames<br>>2005-07-01 16:37:31 Loading saved<br>>state<br>>2005-07-01<br>>16:37:31 Task hobbitd started with PID 4770<br>>2005-07-01 16:37:31<br>>Setting up network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:31<br>>Cannot bind to listen socket (Address already<br>>in<br>>use)<br>>2005-07-01 16:37:31 Task hobbitd terminated,<br>>status<br>>1<br>>2005-07-01 16:37:36 Task hobbitd started with PID<br>>4774<br>>2005-07-01 16:37:36 Loading hostnames<br>>2005-07-01<br>>16:37:36 Loading saved state<br>>2005-07-01 16:37:36 Setting up<br>>network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:36 Cannot bind<br>>to listen socket (Address already in<br>>use)<br>>2005-07-01<br>>16:37:36 Task hobbitd terminated, status 1<br>>2005-07-01 16:37:41<br>>Task hobbitd started with PID 4778<br>>2005-07-01 16:37:41 Loading<br>>hostnames<br>>2005-07-01<br>>16:37:41 Loading saved state<br>>2005-07-01 16:37:41 Setting up<br>>network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:41 Cannot bind<br>>to listen socket (Address already in<br>>use)<br>>2005-07-01<br>>16:37:41 Task hobbitd terminated, status 1<br>>2005-07-01 16:37:46<br>>Task hobbitd started with PID 4783<br>>2005-07-01 16:37:46 Loading<br>>hostnames<br>>2005-07-01<br>>16:37:46 Loading saved state<br>>2005-07-01 16:37:46 Setting up<br>>network listener on 0.0.0.0:1984<br>>2005-07-01 16:37:46 Cannot bind<br>>to listen socket (Address already in<br>>use)<br>>2005-07-01<br>>16:37:46 Task hobbitd terminated, status<br>>1<br>>---snip---<br>><br>>Looks like a clue.<br>>I will add<br>>the output of netstat -a<br>><br>>Got the hobbitd.log file for you<br>>too.<br>><br>>Let me know if there is<br>>anything else I can get you.<br>><br>>Regards<br>><br>>Vernon<br>><br>>P.S. Your cold one is quickly becoming many cold<br>>ones if you ever get<br>>to<br>>Perth<br>><br>><br>><br>><br>><br>>-----Orig<br>>inal<br>>Message-----<br>>From: Henrik Stoerner<br>>[mailto:henrik at hswn.dk]<br>>Sent: Friday, 1 July 2005 3:38<br>>PM<br>>To:<br>>hobbit at hswn.dk<br>>Subject: Re: [hobbit] Status<br>>Unavailable<br>><br>>On Fri, Jul 01, 2005 at 03:25:30PM +0800,<br>>Vernon Everett wrote:<br>> > Thanks for helping on this.<br>><br>>> I rebooted this morning. Could the memory leak still effect me in<br>>that<br>><br>> > short time?<br>><br>>Probably not. Just<br>>wanted to rule out this possibility.<br>><br>> > No<br>>"failed allocation" in dmesg output.<br>> > Do you want<br>>the full output?<br>><br>>No, I dont think that is<br>>necessary.<br>><br>> > [root at pengo log]# vmstat 4<br>>20<br>><br>>And your system is mostly idle with no swap or disk<br>>activity.<br>><br>> > [hobbit at pengo hobbit]$ server/bin/bb<br>>127.0.0.1 "hobbitdboard"<br>> ><br>>2005-07-01 15:21:45 Whoops ! bb failed to send message -<br>>timeout<br>><br>>Could you try running "strace -p<br>><process-ID of the hobbitd process>"<br>>for a minute or<br>>two and send me the output, then do a "kill<br>>-6<br>><process-id>" and mail me the core-file from<br>>~hobbit/server/tmp/<br>>together with the ~hobbit/server/bin/hobbitd<br>>file ?<br>><br>>Also, after this try adding a "--debug"<br>>to the hobbitd commandline in<br>>hobbitlaunch.cfg.<br>>Let it run for a while and then mail me the<br>>hobbitd.log<br>>file.<br>><br>>This bug sounds a bit nasty, I think<br>>....<br>><br>><br>>Regards,<br>>Henrik<br>><br>><br>&g<br>>t;To<br>>unsubscribe from the hobbit list, send an e-mail<br>>to<br>>hobbit-unsubscribe at hswn.dk<br>><br>><br>>_ _ _ _ _ _<br>>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _<br>>_<br>><br>>NOTICE: This message and any attachments are<br>>confidential and may contain copyright material<br>>of Australian<br>>Finance Group Limited or a third party. It is intended solely for the<br>>purpose of the<br>>addressee and any other named recipient. If you<br>>are not the intended recipient, any use,<br>>distribution, disclosure<br>>or copying of this message is strictly prohibited. The confidentiality<br>>attached<br>>to this message is not waived or lost by reason of the<br>>mistaken transmission or delivery to any<br>>unintended party. If you<br>>have received this message in error, please notify the author<br>>immediately or<br>>contact Australian Finance Group on +61 8 9420<br>>7888.<br>><br>><br>>To unsubscribe from the hobbit list, send<br>>an e-mail to<br>>hobbit-unsubscribe at hswn.dk<br>><br>><br><br>><br>><br>><br>>To unsubscribe from the hobbit list, send an e-mail to<br>>hobbit-unsubscribe at hswn.dk<br>><br>><br>>_ _ _ _ _ _ _ _
_<br>><br>>NOTICE: This message and any attachments are confidential and may contain copyright material<br>>of Australian Finance Group Limited or a third party. It is intended solely for the purpose of the<br>>addressee and any other named recipient. If you are not the intended recipient, any use,<br>>distribution, disclosure or copying of this message is strictly prohibited. The confidentiality attached<br>>to this message is not waived or lost by reason of the mistaken transmission or delivery to any<br>>unintended party. If you have received this message in error, please notify the author immediately or<br>>contact Australian Finance Group on +61 8 9420 7888.<br>><br>><br>>To unsubscribe from the hobbit list, send an e-mail to<br>>hobbit-unsubscribe at hswn.dk<br>><br>><br>
participants (2)
-
stefan_loos@hotmail.com
-
v.everett@afgonline.com.au