after a recent restart of a stable hobbit server (v4.2.0) instance (>
1 year without issues), hobbitd_client is now repeatedly crashing. a
sanitized version of what gdb reports follows:
GNU gdb Red Hat Linux (6.3.0.0-1.143.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host
libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `hobbitd_client'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libpcre.so.0...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0 0x003d17a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) bt
#0 0x003d17a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x004127a5 in raise () from /lib/tls/libc.so.6
#2 0x00414209 in abort () from /lib/tls/libc.so.6
#3 0x0805d4e6 in sigsegv_handler (signum=11) at sig.c:57
#4 <signal handler called>
#5 rbtFind (h=0x0, key=0x8f25178) at rbtr.c:379
#6 0x08054cb8 in getrule (hostname=0x8f25178 "hostname.foo.com",
pagename=0x8f246e0 "oldprod", classname=0xb7e7b04f "linux",
ruletype=C_LOAD) at client_config.c:237
#7 0x08054f32 in get_cpu_thresholds (hinfo=0x8f246e0,
classname=0xb7e7b04f "linux", loadyellow=0xbfee0654,
loadred=0xbfee0658,
recentlimit=0xbfee065c, ancientlimit=0xbfee0660,
maxclockdiff=0xbfee0664)
at client_config.c:1209
#8 0x08049fde in unix_cpu_report (
hostname=0xb7e7b033 "hostname.foo.com",
clientclass=0xb7e7b04f "linux", os=OS_LINUX, hinfo=0x8f25110,
fromline=0xbfee1810 "\nStatus message received from
xxx.xxx.xxx.xxx\n",
timestr=0xb7e7b085 "Tue Oct 30 23:19:57 PDT 2007",
uptimestr=0xb7e7b128 " 23:19:57 up 71 days, 1:14, 0 users,
load average: 0.29, 0.23, 0.19",
clockstr=0xb7e94e94 "epoch: 1193811603.588122\nlocal: 2007-10-30
23:20:03 PDT\nUTC: 2007-10-31 06:20:03 GMT\n", msgcachestr=0x0,
whostr=0xb7e7b175 "",
psstr=0xb7e7eca3 " PID PPID USER STARTED S PRI %CPU
TIME %MEM RSZ VSZ CMD\n 1 0 root Aug 20 S 24 0.0
00:00:49 0.0 504 1540 init\n 2 0 root Aug 20 S 139
0.0 00:00:00 0.0 0"...,
topstr=0xb7e8afca "\n\n 23:19:58 up 71 days, 1:14, 0 users,
load average: 0.29, 0.23, 0.19\n526 processes: 525 sleeping, 1
running, 0 zombie, 0 stopped\nCPU states: cpu user nice
system irq softirq iowait "...)
at hobbitd_client.c:263
#9 0x0804d035 in handle_linux_client (
hostname=0xb7e7b033 "hostname.foo.com",
clienttype=0xb7e7b04f "linux", os=OS_LINUX, hinfo=0x8f25110,
sender=0xb7e7b025 "xxx.xxx.xxx.xxx", timestamp=1193811604,
clientdata=0xb7e7b055 "client hostname,foo,com.linux linux\n[date")
at client/linux.c:55
#10 0x080514ed in main (argc=1, argv=0xbfee1e24) at hobbitd_client.c:
1788