Hi Guys,
today my hobbit server started throwing up purple alerts for all my network (conn) and various other tcp connection tests. I noticed that a core file was being created every 5 minutes. Here is what gdb has to say -
[hobbit at monitor server]$ gdb bin/hobbitd core.4115 GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/hobbit/server/bin/hobbitd...done.
warning: core file may not match specified executable file. Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x00e3d402 in __kernel_vsyscall () (gdb) bt #0 0x00e3d402 in __kernel_vsyscall () #1 0x005f2040 in ?? () #2 0x00720ff4 in ?? () #3 0xb7f356d0 in ?? () #4 0xbf8057a8 in ?? () #5 0x005f3a21 in ?? () #6 0x00000006 in ?? () #7 0xbf80571c in ?? () #8 0x00000000 in ?? ()
This is the latest output from the bb-network.log file -
xstrdup: Cannot dup NULL string
not sure how to find out what version of Hobbit I'm running.
Any help would be greatly appreciated.
Regards
Brian Daly.
On 19/01/11 5:42 AM, Brian Daly wrote:
Hi Guys,
today my hobbit server started throwing up purple alerts for all my network (conn) and various other tcp connection tests. I noticed that a core file was being created every 5 minutes. Here is what gdb has to say -
[hobbit at monitor server]$ gdb bin/hobbitd core.4115 GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/hobbit/server/bin/hobbitd...done.
warning: core file may not match specified executable file. Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x00e3d402 in __kernel_vsyscall () (gdb) bt #0 0x00e3d402 in __kernel_vsyscall () #1 0x005f2040 in ?? () #2 0x00720ff4 in ?? () #3 0xb7f356d0 in ?? () #4 0xbf8057a8 in ?? () #5 0x005f3a21 in ?? () #6 0x00000006 in ?? () #7 0xbf80571c in ?? () #8 0x00000000 in ?? ()
This is the latest output from the bb-network.log file -
xstrdup: Cannot dup NULL string
not sure how to find out what version of Hobbit I'm running.
To show the version:
$ bbcmd bbtest-net --version 2011-01-19 09:03:48 Using default environment file /usr/lib/hobbit/server/etc/hobbitserver.cfg bbtest-net version 4.2.3 SSL library : OpenSSL 0.9.8e-rhel5 01 Jul 2008 LDAP library: OpenLDAP 20343
Try running the gdb again against the bbtest-net binary at /usr/local/hobbit/server/bin/bbtest-net on your system.
David.
-- David Baldwin - IT Unit Australian Sports Commission www.ausport.gov.au Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616 david.baldwin at ausport.gov.au Leverrier Street Bruce ACT 2617
Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au
This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.
Hi David,
here is the results of gdb agains bbtest-net
gdb bin/bbtest-net core.9941 GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/hobbit/server/bin/bbtest-net...done.
warning: .dynamic section for "/lib/libnss_files.so.2" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations Reading symbols from /usr/lib/libldap-2.3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libldap-2.3.so.0 Reading symbols from /usr/lib/liblber-2.3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/liblber-2.3.so.0 Reading symbols from /lib/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libssl.so.6 Reading symbols from /lib/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.6 Reading symbols from /lib/i686/nosegneg/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i686/nosegneg/libc.so.6 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /lib/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libselinux.so.1 Reading symbols from /lib/libsepol.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libsepol.so.1 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_dns.so.2 Core was generated by `bbtest-net --report --ping --checkresponse --no-ares'. Program terminated with signal 6, Aborted. #0 0x00ea0402 in __kernel_vsyscall () (gdb) bt #0 0x00ea0402 in __kernel_vsyscall () #1 0x005f2040 in raise () from /lib/i686/nosegneg/libc.so.6 #2 0x005f3a21 in abort () from /lib/i686/nosegneg/libc.so.6 #3 0x0805d70c in xstrdup (s=0x0) at memory.c:169 #4 0x08051da0 in setup_ssl (item=0x918d8b8) at contest.c:626 #5 0x080531e9 in do_tcp_tests (timeout=10, concurrency=246) at contest.c:998 #6 0x080502b7 in main (argc=5, argv=0xbf980a84) at bbtest-net.c:2284
[root at monitor bin]# ./bbcmd bbtest-net --version 2011-01-18 23:13:47 Using default environment file /usr/local/hobbit/server/etc/hobbitserver.cfg bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.8b 04 May 2006 LDAP library: OpenLDAP 20327
thanks
Brian
On 18 Jan 2011, at 22:09, David Baldwin wrote:
On 19/01/11 5:42 AM, Brian Daly wrote:
Hi Guys,
today my hobbit server started throwing up purple alerts for all my network (conn) and various other tcp connection tests. I noticed that a core file was being created every 5 minutes. Here is what gdb has to say -
[hobbit at monitor server]$ gdb bin/hobbitd core.4115 GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/hobbit/server/bin/hobbitd...done.
warning: core file may not match specified executable file. Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x00e3d402 in __kernel_vsyscall () (gdb) bt #0 0x00e3d402 in __kernel_vsyscall () #1 0x005f2040 in ?? () #2 0x00720ff4 in ?? () #3 0xb7f356d0 in ?? () #4 0xbf8057a8 in ?? () #5 0x005f3a21 in ?? () #6 0x00000006 in ?? () #7 0xbf80571c in ?? () #8 0x00000000 in ?? ()
This is the latest output from the bb-network.log file -
xstrdup: Cannot dup NULL string
not sure how to find out what version of Hobbit I'm running.
To show the version:
$ bbcmd bbtest-net --version 2011-01-19 09:03:48 Using default environment file /usr/lib/hobbit/server/etc/hobbitserver.cfg bbtest-net version 4.2.3 SSL library : OpenSSL 0.9.8e-rhel5 01 Jul 2008 LDAP library: OpenLDAP 20343
Try running the gdb again against the bbtest-net binary at /usr/local/hobbit/server/bin/bbtest-net on your system.
David.
-- David Baldwin - IT Unit Australian Sports Commission www.ausport.gov.au Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616 david.baldwin at ausport.gov.au Leverrier Street Bruce ACT 2617
Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au
This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.
Brian Daly Unix Administrator
Critical Path, Inc. 42-47, Lower Mount St., Dublin 2, Ireland
Phone: +353 1 241 5000 Fax: +353 1 241 5170 www.criticalpath.net
In <6D830853-74A9-4EF2-BA1F-415236D80A0E at criticalpath.net> Brian Daly <brian.daly at criticalpath.net> writes:
here is the results of gdb agains bbtest-net
gdb bin/bbtest-net core.9941 Program terminated with signal 6, Aborted. #0 0x00ea0402 in __kernel_vsyscall () (gdb) bt #0 0x00ea0402 in __kernel_vsyscall () #1 0x005f2040 in raise () from /lib/i686/nosegneg/libc.so.6 #2 0x005f3a21 in abort () from /lib/i686/nosegneg/libc.so.6 #3 0x0805d70c in xstrdup (s=0x0) at memory.c:169 #4 0x08051da0 in setup_ssl (item=0x918d8b8) at contest.c:626 #5 0x080531e9 in do_tcp_tests (timeout=10, concurrency=246) at contest.c:998 #6 0x080502b7 in main (argc=5, argv=0xbf980a84) at bbtest-net.c:2284
[root at monitor bin]# ./bbcmd bbtest-net --version bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.8b 04 May 2006 LDAP library: OpenLDAP 20327
4.2.0 is a pretty old version.
Based on the traceback, I would think this is the "bbtest-net segfaults on SSL cert expire 2054" bug reported by the Debian folks in http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=503111
This bugreport also includes a patch that you can use if you must remain at version 4.2.0. But I would really recommend that you update to at least 4.2.3, which is the last 4.2.x release.
Regards, Henrik
In <A0F1E85D-F28A-4251-BEC9-865533002350 at criticalpath.net> Brian Daly <brian.daly at criticalpath.net> writes:
today my hobbit server started throwing up purple alerts for all my = network (conn) and various other tcp connection tests. I noticed that a = core file was being created every 5 minutes. Here is what gdb has to say =
[hobbit at monitor server]$ gdb bin/hobbitd core.4115
You're feeding gdb the wrong program. It says
Core was generated by `bbtest-net --report --ping --checkresponse'.
So instead of "bin/hobbitd", use "bin/bbtest-net". That should (hopefully) give us a much more usable gdb output.
This is the latest output from the bb-network.log file - xstrdup: Cannot dup NULL string
Hmm, pretty generic that one. Anything else in that log?
You can run bbtest-net by hand - with $ bbcmd bbtest-net HOSTNAME where HOSTNAME is the name of one of your hosts. It would be interesting to know if it crashes if you test only one host - that would point to the problem being in global configuration. If it runs fine with one host, then the problem is probably in the configuration of one particular host in bb-hosts.
I suspect it is a mis-configuration of one network test, there have been some problems with error handling if network tests (especially URL's in complex web- or ldap-checks) were not specified correctly.
not sure how to find out what version of Hobbit I'm running.
bin/bbtest-net --version
Regards, Henrik
participants (3)
-
brian.daly＠criticalpath.net
-
david.baldwin＠ausport.gov.au
-
henrik＠hswn.dk