Hi All,
My allevents file was nearing the 2GB mark but was 95% full of lines from two custom tests that changed state every 30 seconds or so. These were removed manually as trim history could not be used as we would loose valuable historic data. allevents file cannot exceed 2GB.
The file was cat'ed out with a grep -v of the unwated tests, completed fine and now new allevents file is 141mb.
Hobbit was stopped before the process started, a ps showed no hobbit processes running, above change was made, permissions were checked in all events to ensure they were the same, hobbit was then started.
about 500 hosts are monitored with most of them running client side custom scripts, very shortly after startup core files were generated and hobbit either crashed or stopped itself.
Below is the debug output from the core files and relevant log line entries round this time. The only way to get hobbit started back up and the hobbit status page displayed, was to firewall off the server so that all status messages were ignored (iptables used), stop hobbit, ensure all processes are gone and started again, waited a few minutes so that hobbit could do what it wanted, then the iptables rule was removed. After 10 mins all returned to normal and no more core files have been generated.
Not sure if whilst hobbit is starting up, if it receives too many status updates, or any during startup it causes it to core dump as it is not ready?
Any help would be greatly appreciated.
CORE FILES
-rw------- 1 hobbit hobbit 23M Jan 22 09:36 core.27740 -rw------- 1 hobbit hobbit 23M Jan 22 09:37 core.28187 -rw------- 1 hobbit hobbit 23M Jan 22 09:38 core.28385
DEBUG OUTPUT
gdb ../bin/hobbitd core.27740
GNU gdb Red Hat Linux (6.3.0.0-1.138.el3rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `hobbitd --pidfile=/var/log/hobbit/hobbitd.pid --restart=/var/hobbit/server/tmp/'. Program terminated with signal 6, Aborted. Reading symbols from /lib/libpcre.so.0...done. Loaded symbols for /lib/libpcre.so.0 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 #0 0x00434eff in raise () from /lib/tls/libc.so.6 (gdb) bt #0 0x00434eff in raise () from /lib/tls/libc.so.6 #1 0x00436705 in abort () from /lib/tls/libc.so.6 #2 0x0805a006 in sigsegv_handler (signum=11) at sig.c:57 #3 <signal handler called> #4 0x0048701c in mempcpy () from /lib/tls/libc.so.6 #5 0x0047a6a2 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6 #6 0x004535c7 in vfprintf () from /lib/tls/libc.so.6 #7 0x0046f4ec in vsprintf () from /lib/tls/libc.so.6 #8 0x0045a0dd in sprintf () from /lib/tls/libc.so.6 #9 0x0804df40 in generate_outbuf (outbuf=0xbffee4c4, outpos=0xbffee4c8, outsz=0xbffee4cc, hwalk=0x89793b0, lwalk=0x897b350, acklevel=-1) at hobbitd.c:2227 #10 0x0805074f in do_message (msg=0x8df5d10, origin=0x805c4fc "") at hobbitd.c:2848 #11 0x08053472 in main (argc=8, argv=0xbfff18e4) at hobbitd.c:4377
gdb /var/hobbit/server/bin/hobbitd /var/hobbit/server/tmp/core.28187
GNU gdb Red Hat Linux (6.3.0.0-1.138.el3rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `hobbitd --pidfile=/var/log/hobbit/hobbitd.pid --restart=/var/hobbit/server/tmp/'. Program terminated with signal 6, Aborted. Reading symbols from /lib/libpcre.so.0...done. Loaded symbols for /lib/libpcre.so.0 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 #0 0x0093aeff in raise () from /lib/tls/libc.so.6 (gdb) bt #0 0x0093aeff in raise () from /lib/tls/libc.so.6 #1 0x0093c705 in abort () from /lib/tls/libc.so.6 #2 0x0805a006 in sigsegv_handler (signum=11) at sig.c:57 #3 <signal handler called> #4 0x0098d01c in mempcpy () from /lib/tls/libc.so.6 #5 0x009806a2 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6 #6 0x009595c7 in vfprintf () from /lib/tls/libc.so.6 #7 0x009754ec in vsprintf () from /lib/tls/libc.so.6 #8 0x009600dd in sprintf () from /lib/tls/libc.so.6 #9 0x0804df40 in generate_outbuf (outbuf=0xbfff0f74, outpos=0xbfff0f78, outsz=0xbfff0f7c, hwalk=0x9352838, lwalk=0x93557a0, acklevel=-1) at hobbitd.c:2227 #10 0x0805074f in do_message (msg=0x9c27bb0, origin=0x805c4fc "") at hobbitd.c:2848 #11 0x08053472 in main (argc=8, argv=0xbfff4394) at hobbitd.c:4377 (gdb) quit
gdb /var/hobbit/server/bin/hobbitd /var/hobbit/server/tmp/core.28385
GNU gdb Red Hat Linux (6.3.0.0-1.138.el3rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `hobbitd --pidfile=/var/log/hobbit/hobbitd.pid --restart=/var/hobbit/server/tmp/'. Program terminated with signal 6, Aborted. Reading symbols from /lib/libpcre.so.0...done. Loaded symbols for /lib/libpcre.so.0 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 #0 0x00abfeff in raise () from /lib/tls/libc.so.6 (gdb) bt #0 0x00abfeff in raise () from /lib/tls/libc.so.6 #1 0x00ac1705 in abort () from /lib/tls/libc.so.6 #2 0x0805a006 in sigsegv_handler (signum=11) at sig.c:57 #3 <signal handler called> #4 0x00b1201c in mempcpy () from /lib/tls/libc.so.6 #5 0x00b056a2 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6 #6 0x00ade5c7 in vfprintf () from /lib/tls/libc.so.6 #7 0x00afa4ec in vsprintf () from /lib/tls/libc.so.6 #8 0x00ae50dd in sprintf () from /lib/tls/libc.so.6 #9 0x0804df40 in generate_outbuf (outbuf=0xbfff3774, outpos=0xbfff3778, outsz=0xbfff377c, hwalk=0x9c863b0, lwalk=0x9c88350, acklevel=-1) at hobbitd.c:2227 #10 0x0805074f in do_message (msg=0xa678100, origin=0x805c4fc "") at hobbitd.c:2848 #11 0x08053472 in main (argc=8, argv=0xbfff6b94) at hobbitd.c:4377 (gdb) quit
LOG FILES
hobbitlaunch.log:2008-01-22 09:36:05 hobbitlaunch starting hobbitlaunch.log:2008-01-22 09:36:05 Loading tasklist configuration from /var/hobbit/server/etc/hobbitlaunch.cfg hobbitlaunch.log:2008-01-22 09:36:05 Loading hostnames hobbitlaunch.log:2008-01-22 09:36:05 Loading saved state hobbitlaunch.log:2008-01-22 09:36:06 Setting up network listener on 0.0.0.0:1984 hobbitlaunch.log:2008-01-22 09:36:06 Setting up signal handlers hobbitlaunch.log:2008-01-22 09:36:06 Setting up hobbitd channels hobbitlaunch.log:2008-01-22 09:36:06 Setting up logfiles hobbitlaunch.log:2008-01-22 09:36:10 Task hobbitd terminated by signal 6 hobbitlaunch.log:2008-01-22 09:36:10 Loading hostnames hobbitlaunch.log:2008-01-22 09:36:10 Task bbdisplay terminated by signal 15 hobbitlaunch.log:2008-01-22 09:36:10 Task bbnet terminated by signal 15 hobbitlaunch.log:2008-01-22 09:36:10 Task hobbitclient terminated by signal 15 hobbitlaunch.log:2008-01-22 09:36:10 Loading saved state hobbitlaunch.log:2008-01-22 09:36:11 Setting up network listener on 0.0.0.0:1984 hobbitlaunch.log:2008-01-22 09:36:11 Setting up signal handlers hobbitlaunch.log:2008-01-22 09:36:11 Setting up hobbitd channels hobbitlaunch.log:2008-01-22 09:36:11 Setting up logfiles hobbitlaunch.log:2008-01-22 09:37:12 Task hobbitd terminated by signal 6 hobbitlaunch.log:2008-01-22 09:37:12 Task bbdisplay terminated by signal 15 hobbitlaunch.log:2008-01-22 09:37:12 Loading hostnames hobbitlaunch.log:2008-01-22 09:37:12 Loading saved state hobbitlaunch.log:2008-01-22 09:37:12 Setting up network listener on 0.0.0.0:1984 hobbitlaunch.log:2008-01-22 09:37:12 Setting up signal handlers hobbitlaunch.log:2008-01-22 09:37:12 Setting up hobbitd channels hobbitlaunch.log:2008-01-22 09:37:12 Setting up logfiles hobbitlaunch.log:2008-01-22 09:38:14 Task hobbitd terminated by signal 6 hobbitlaunch.log:2008-01-22 09:38:14 Task bbdisplay terminated by signal 15 hobbitlaunch.log:2008-01-22 09:45:27 hobbitlaunch starting hobbitlaunch.log:2008-01-22 09:45:27 Loading tasklist configuration from /var/hobbit/server/etc/hobbitlaunch.cfg hobbitlaunch.log:2008-01-22 09:45:27 Loading hostnames hobbitlaunch.log:2008-01-22 09:45:27 Loading saved state hobbitlaunch.log:2008-01-22 09:45:27 Setting up network listener on 0.0.0.0:1984 hobbitlaunch.log:2008-01-22 09:45:27 Setting up signal handlers hobbitlaunch.log:2008-01-22 09:45:27 Setting up hobbitd channels hobbitlaunch.log:2008-01-22 09:45:27 Setting up logfiles
bb-display.log:2008-01-22 09:37:12 2008-01-22 09:38:14 2008-01-22 09:45:47 Whoops ! bb failed to send message - timeout
bb-display.log:2008-01-22 09:45:47 hobbitd status-board not available
bb-display.log:2008-01-22 09:46:34 Whoops ! bb failed to send message - timeout
bb-display.log:2008-01-22 09:47:21 Whoops ! bb failed to send message - timeout
bb-display.log:2008-01-22 09:47:21 hobbitd status-board not available
bb-display.log:2008-01-22 09:48:08 Whoops ! bb failed to send message - timeout
bb-display.log:2008-01-22 09:48:55 Whoops ! bb failed to send message - timeout
bb-display.log:2008-01-22 09:48:55 hobbitd status-board not available
hobbitclient.log:2008-01-22 09:46:25 Whoops ! bb failed to send message - timeout
hobbitclient.log:2008-01-22 09:51:26 Whoops ! bb failed to send message - timeout
hobbitclient.log:2008-01-22 09:56:26 Whoops ! bb failed to send message - timeout
hobbitd.log:2008-01-22 09:27:47 BOARDBUSY locked at 1, GETNCNT is 0, GETPID is 0, 0 clients
hobbitd.log:2008-01-22 09:36:06 Setup complete
hobbitd.log:2008-01-22 09:36:11 Setup complete
hobbitd.log:2008-01-22 09:37:12 Setup complete
hobbitd.log:2008-01-22 09:45:27 Setup complete
Thanks
Regards
Dominic
__________________________________________________________Sent from Yahoo! Mail - a smarter inbox http://uk.mail.yahoo.com