Hi Henrik
You may recall, a while ago I posted a question to the list about "passive tests" no longer working. See thread from here http://www.hswn.dk/hobbiton/2008/10/msg00150.html Since then, it has happened a few times, but in all cases we have not had a chance to sit down and examine the core. Until Now.
Below are our 5 most recent cores. And the corresponding entry in the bb-network.log file where available. (Not all generate the entry in the log file)
Colin, our Linux man, and part-time C hacker, had this to say
---snip--- [root at las006 tmp]# gdb ~hobbit/server/bin/bbtest-net GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
(gdb) l *0x42493a 0x42493a is in ares_free_hostent (ares_free_hostent.c:35). 30 free(host->h_name); 31 for (p = host->h_aliases; *p; p++) 32 free(*p); 33 free(host->h_aliases); 34 free(host->h_addr_list[0]); 35 free(host->h_addr_list); 36 free(host); 37 } (gdb)
line 31: Make 'p' equal to the memory address of host->h_aliases, check that pointer p is not null and increment p line 32: free the memory pointed at by 'p'
The problem here is that you check p then free p++. My C is quite rusty but I'd hazard a guess and say line 31 should read like: 31 for (p = host->h_aliases; *(p++);)
You need to be checking the addess you're going to free, not the one next to it.
Hope this a) makes sense and b) is vaguely accurate...
---snip---
I am no C expert, but looking at the grep output below, it does look like a memory issue.
Cheers Vernon
$ grep glibc bb-network.log *** glibc detected *** bbtest-net: malloc(): memory corruption: 0x000000001f9408c0 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x000000001e44ec70 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x000000000a96dd20 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x00000000120ca000 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x0000000007178440 *** *** glibc detected *** bbtest-net: free(): invalid next size (fast): 0x000000000fe240a0 *** *** glibc detected *** bbtest-net: munmap_chunk(): invalid pointer: 0x00000000164be700 *** *** glibc detected *** bbtest-net: munmap_chunk(): invalid pointer: 0x000000001981fe90 *** *** glibc detected *** bbtest-net: munmap_chunk(): invalid pointer: 0x000000000a005e90 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x0000000009fef8c0 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x00000000149dbf30 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x000000000e655e60 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x0000000002c46920 ***
gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.31727
GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x0000000000424a57 in qcallback (arg=0x1bc753c0, status=0, abuf=0x7fff461c1c50 "\201\200", alen=140) at ares_query.c:108 #5 0x00000000004216c2 in end_query (channel=0x1bc5e4d0, query=0x1bcf41c0, status=0, abuf=0x0, alen=1176247376) at ares_process.c:627 #6 0x0000000000421ebd in process_answer (channel=0x1bc5e4d0, abuf=0x7fff461c1c50 "\201\200", alen=140, whichserver=0, tcp=<value optimized out>, now=1226882897) at ares_process.c:359 #7 0x00000000004221ff in ares_process (channel=0x1bc5e4d0, read_fds=0x7fff461c1f20, write_fds=0x7fff461c1ea0) at ares_process.c:273 #8 0x000000000040f064 in dns_queue_run (channel=0x1bc5e4d0) at dns.c:228 #9 0x000000000040f464 in flush_dnsqueue () at dns.c:259 #10 0x00000000004088e6 in main (argc=4, argv=0x7fff461c3428) at bbtest-net.c:2099 #11 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #12 0x00000000004039e9 in _start () (gdb) quit
*** glibc detected *** bbtest-net: double free or corruption (out): 0x000000000e655e60 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3db7a71634] /lib64/libc.so.6(cfree+0x8c)[0x3db7a74c5c] bbtest-net[0x422424] bbtest-net[0x422515] bbtest-net[0x424a59] bbtest-net[0x4216c2] bbtest-net[0x421ebd] bbtest-net[0x4221ff] bbtest-net[0x40f064] bbtest-net[0x40f464] bbtest-net[0x4088e6] /lib64/libc.so.6(__libc_start_main+0xf4)[0x3db7a1d8b4] bbtest-net[0x4039e9] ======= Memory map: ======== 00400000-00430000 r-xp 00000000 fd:01 389476 /usr/lib/hobbit/server/bin/bbtest-net 00630000-00631000 rw-p 00030000 fd:01 389476 /usr/lib/hobbit/server/bin/bbtest-net 00631000-00637000 rw-p 00631000 00:00 0 00830000-00832000 rw-p 00030000 fd:01 389476 /usr/lib/hobbit/server/bin/bbtest-net 0e533000-0e6bf000 rw-p 0e533000 00:00 0 3224600000-3224638000 r-xp 00000000 fd:01 68061 /usr/lib64/libldap-2.3.so.0.2.15 3224638000-3224838000 ---p 00038000 fd:01 68061 /usr/lib64/libldap-2.3.so.0.2.15 3224838000-322483a000 rw-p 00038000 fd:01 68061 /usr/lib64/libldap-2.3.so.0.2.15 3366400000-3366443000 r-xp 00000000 fd:00 75879 /lib64/libssl.so.0.9.8b 3366443000-3366643000 ---p 00043000 fd:00 75879 /lib64/libssl.so.0.9.8b 3366643000-3366649000 rw-p 00043000 fd:00 75879 /lib64/libssl.so.0.9.8b 3368000000-3368125000 r-xp 00000000 fd:00 75876 /lib64/libcrypto.so.0.9.8b 3368125000-3368325000 ---p 00125000 fd:00 75876 /lib64/libcrypto.so.0.9.8b 3368325000-3368344000 rw-p 00125000 fd:00 75876 /lib64/libcrypto.so.0.9.8b 3368344000-3368348000 rw-p 3368344000 00:00 0 385c600000-385c63b000 r-xp 00000000 fd:00 75779 /lib64/libsepol.so.1 385c63b000-385c83b000 ---p 0003b000 fd:00 75779 /lib64/libsepol.so.1 385c83b000-385c83c000 rw-p 0003b000 fd:00 75779 /lib64/libsepol.so.1 385c83c000-385c846000 rw-p 385c83c000 00:00 0 385ca00000-385ca15000 r-xp 00000000 fd:00 75786 /lib64/libselinux.so.1 385ca15000-385cc15000 ---p 00015000 fd:00 75786 /lib64/libselinux.so.1 385cc15000-385cc17000 rw-p 00015000 fd:00 75786 /lib64/libselinux.so.1 385cc17000-385cc18000 rw-p 385cc17000 00:00 0 385ce00000-385ce8f000 r-xp 00000000 fd:01 68057 /usr/lib64/libkrb5.so.3.3 385ce8f000-385d08e000 ---p 0008f000 fd:01 68057 /usr/lib64/libkrb5.so.3.3 385d08e000-385d092000 rw-p 0008e000 fd:01 68057 /usr/lib64/libkrb5.so.3.3 385d600000-385d608000 r-xp 00000000 fd:01 68055 /usr/lib64/libkrb5support.so.0.1 385d608000-385d807000 ---p 00008000 fd:01 68055 /usr/lib64/libkrb5support.so.0.1 385d807000-385d808000 rw-p 00007000 fd:01 68055 /usr/lib64/libkrb5support.so.0.1 385da00000-385da24000 r-xp 00000000 fd:01 68056 /usr/lib64/libk5crypto.so.3.1 385da24000-385dc23000 ---p 00024000 fd:01 68056 /usr/lib64/libk5crypto.so.3.1 385dc23000-385dc25000 rw-p 00023000 fd:01 68056 /usr/lib64/libk5crypto.so.3.1 385de00000-385de2c000 r-xp 00000000 fd:01 68058 /usr/lib64/libgssapi_krb5.so.2.2 385de2c000-385e02c000 ---p 0002c000 fd:01 68058 /usr/lib64/libgssapi_krb5.so.2.2 385e02c000-385e02e000 rw-p 0002c000 fd:01 68058 /usr/lib64/libgssapi_krb5.so.2.2 3af4400000-3af440d000 r-xp 00000000 fd:01 68095 /usr/lib64/liblber-2.3.so.0.2.15 3af440d000-3af460d000 ---p 0000d000 fd:01 68095 /usr/lib64/liblber-2.3.so.0.2.15 3af460d000-3af460e000 rw-p 0000d000 fd:01 68095 /usr/lib64/liblber-2.3.so.0.2.15 3db7600000-3db761a000 r-xp 00000000 fd:00 75791 /lib64/ld-2.5.so 3db781a000-3db781b000 r--p 0001a000 fd:00 75791 /lib64/ld-2.5.so 3db781b000-3db781c000 rw-p 0001b000 fd:00 75791 /lib64/ld-2.5.so 3db7a00000-3db7b4a000 r-xp 00000000 fd:00 75797 /lib64/libc-2.5.so 3db7b4a000-3db7d49000 ---p 0014a000 fd:00 75797 /lib64/libc-2.5.so 3db7d49000-3db7d4d000 r--p 00149000 fd:00 75797 /lib64/libc-2.5.so 3db7d4d000-3db7d4e000 rw-p 0014d000 fd:00 75797 /lib64/libc-2.5.so 3db7d4e000-3db7d53000 rw-p 3db7d4e000 00:00 0 3db7e00000-3db7e02000 r-xp 00000000 fd:00 75840 /lib64/libdl-2.5.so 3db7e02000-3db8002000 ---p 00002000 fd:00 75840 /lib64/libdl-2.5.so 3db8002000-3db8003000 r--p 00002000 fd:00 75840 /lib64/libdl-2.5.so 3db8003000-3db8004000 rw-p 00003000 fd:00 75840 /lib64/libdl-2.5.so 3db8200000-3db8218000 r-xp 00000000 fd:01 67958 /usr/lib64/libsasl2.so.2.0.22 3db8218000-3db8418000 ---p 00018000 fd:01 67958 /usr/lib64/libsasl2.so.2.0.22 3db8418000-3db8419000 rw-p 00018000 fd:01 67958 /usr/lib64/libsasl2.so.2.0.22 3db8600000-3db8614000 r-xp 00000000 fd:01 67257 /usr/lib64/libz.so.1.2.3 3db8614000-3db8813000 ---p 00014000 fd:01 67257 /usr/lib64/libz.so.1.2.3 3db8813000-3db8814000 rw-p 00013000 fd:01 67257 /usr/lib64/libz.so.1.2.3 3db9a00000-3db9a09000 r-xp 00000000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9a09000-3db9c08000 ---p 00009000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9c08000-3db9c09000 r--p 00008000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9c09000-3db9c0a000 rw-p 00009000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9c0a000-3db9c38000 rw-p 3db9c0a000 00:00 0 3dba600000-3dba611000 r-xp 00000000 fd:00 76082 /lib64/libresolv-2.5.so 3dba611000-3dba811000 ---p 00011000 fd:00 76082 /lib64/libresolv-2.5.so 3dba811000-3dba812000 r--p 00011000 fd:00 76082 /lib64/libresolv-2.5.so 3dba812000-3dba813000 rw-p 00012000 fd:00 76082 /lib64/libresolv-2.5.so 3dba813000-3dba815000 rw-p 3dba813000 00:00 0 3dbaa00000-3dbaa02000 r-xp 00000000 fd:00 76083 /lib64/libcom_err.so.2.1 3dbaa02000-3dbac01000 ---p 00002000 fd:00 76083 /lib64/libcom_err.so.2.1 3dbac01000-3dbac02000 rw-p 00001000 fd:00 76083 /lib64/libcom_err.so.2.1 3dbba00000-3dbba02000 r-xp 00000000 fd:00 76080 /lib64/libkeyutils-1.2.so 3dbba02000-3dbbc01000 ---p 00002000 fd:00 76080 /lib64/libkeyutils-1.2.so 3dbbc01000-3dbbc02000 rw-p 00001000 fd:00 76080 /lib64/libkeyutils-1.2.so 3dbc200000-3dbc20d000 r-xp 00000000 fd:00 75803 /lib64/libgcc_s-4.1.2-20080102.so.1 3dbc20d000-3dbc40d000 ---p 0000d000 fd:00 75803 /lib64/libgcc_s-4.1.2-20080102.so.1 3dbc40d000-3dbc40e000 rw-p 0000d000 fd:00 75803 /lib64/libgcc_s-4.1.2-20080102.so.1 2b7f796eb000-2b7f796ec000 rw-p 2b7f796eb000 00:00 0 2b7f796f8000-2b7f79700000 rw-p 2b7f796f8000 00:00 0 2b7f79700000-2b7f7970a000 r-xp 00000000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f7970a000-2b7f79909000 ---p 0000a000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f79909000-2b7f7990a000 r--p 00009000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f7990a000-2b7f7990b000 rw-p 0000a000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f7c000000-2b7f7c021000 rw-p 2b7f7c000000 00:00 0 2b7f7c021000-2b7f80000000 ---p 2b7f7c021000 00:00 0 7fff313a9000-7fff313bf000 rw-p 7fff313a9000 00:00 0 [stack] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
[root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.31042 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x0000000000424a57 in qcallback (arg=0x163b9ee0, status=0, abuf=0x7fff9d536fc0 "\201\200", alen=140) at ares_query.c:108 #5 0x00000000004216c2 in end_query (channel=0x163a34d0, query=0x163d6e50, status=0, abuf=0x0, alen=-1655476288) at ares_process.c:627 #6 0x0000000000421ebd in process_answer (channel=0x163a34d0, abuf=0x7fff9d536fc0 "\201\200", alen=140, whichserver=0, tcp=<value optimized out>, now=1226882805) at ares_process.c:359 #7 0x00000000004221ff in ares_process (channel=0x163a34d0, read_fds=0x7fff9d537290, write_fds=0x7fff9d537210) at ares_process.c:273 #8 0x000000000040f064 in dns_queue_run (channel=0x163a34d0) at dns.c:228 #9 0x000000000040f464 in flush_dnsqueue () at dns.c:259 #10 0x00000000004088e6 in main (argc=4, argv=0x7fff9d538798) at bbtest-net.c:2099 #11 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #12 0x00000000004039e9 in _start () (gdb) quit [root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.15643 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x0000003db7a79f10 in strcasecmp () from /lib64/libc.so.6 #5 0x0000000000418cda in rbtFind (h=0x8d4b680, key=0x8e22310) at rbtr.c:381 #6 0x000000000040f4d8 in find_dnscache (hostname=0x0) at dns.c:86 #7 0x000000000040f546 in dnsresolve (hostname=0x8e22310 "DRPIR101") at dns.c:275 #8 0x0000000000406848 in ip_to_test (h=0x8e22150) at bbtest-net.c:790 #9 0x00000000004069ca in start_ping_service (service=<value optimized out>) at bbtest-net.c:1099 #10 0x000000000040891f in main (argc=4, argv=0x7fffefcdff38) at bbtest-net.c:2105 #11 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #12 0x00000000004039e9 in _start () (gdb) quit [root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.13374 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x00000000004069b7 in start_ping_service (service=<value optimized out>) at bbtest-net.c:1098 #5 0x000000000040891f in main (argc=4, argv=0x7fff8f61e4d8) at bbtest-net.c:2105 #6 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #7 0x00000000004039e9 in _start () (gdb) quit
[root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.7801 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 ares_process (channel=0x157137c0, read_fds=0x7fff91e5dbb0, write_fds=0x7fff91e5db30) at ares_process.c:285 #5 0x000000000040f064 in dns_queue_run (channel=0x157137c0) at dns.c:228 #6 0x000000000040f464 in flush_dnsqueue () at dns.c:259 #7 0x00000000004088e6 in main (argc=4, argv=0x7fff91e5f0b8) at bbtest-net.c:2099 #8 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #9 0x00000000004039e9 in _start () (gdb) quit
NOTICE: This email and any attachments are confidential. They may contain legally privileged information or copyright material. You must not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages and all attachments.