Hi Henrik
You may recall, a while ago I posted a question to the list about "passive tests" no longer working. See thread from here http://www.hswn.dk/hobbiton/2008/10/msg00150.html Since then, it has happened a few times, but in all cases we have not had a chance to sit down and examine the core. Until Now.
Below are our 5 most recent cores. And the corresponding entry in the bb-network.log file where available. (Not all generate the entry in the log file)
Colin, our Linux man, and part-time C hacker, had this to say
---snip--- [root at las006 tmp]# gdb ~hobbit/server/bin/bbtest-net GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
(gdb) l *0x42493a 0x42493a is in ares_free_hostent (ares_free_hostent.c:35). 30 free(host->h_name); 31 for (p = host->h_aliases; *p; p++) 32 free(*p); 33 free(host->h_aliases); 34 free(host->h_addr_list[0]); 35 free(host->h_addr_list); 36 free(host); 37 } (gdb)
line 31: Make 'p' equal to the memory address of host->h_aliases, check that pointer p is not null and increment p line 32: free the memory pointed at by 'p'
The problem here is that you check p then free p++. My C is quite rusty but I'd hazard a guess and say line 31 should read like: 31 for (p = host->h_aliases; *(p++);)
You need to be checking the addess you're going to free, not the one next to it.
Hope this a) makes sense and b) is vaguely accurate...
---snip---
I am no C expert, but looking at the grep output below, it does look like a memory issue.
Cheers Vernon
$ grep glibc bb-network.log *** glibc detected *** bbtest-net: malloc(): memory corruption: 0x000000001f9408c0 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x000000001e44ec70 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x000000000a96dd20 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x00000000120ca000 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x0000000007178440 *** *** glibc detected *** bbtest-net: free(): invalid next size (fast): 0x000000000fe240a0 *** *** glibc detected *** bbtest-net: munmap_chunk(): invalid pointer: 0x00000000164be700 *** *** glibc detected *** bbtest-net: munmap_chunk(): invalid pointer: 0x000000001981fe90 *** *** glibc detected *** bbtest-net: munmap_chunk(): invalid pointer: 0x000000000a005e90 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x0000000009fef8c0 *** *** glibc detected *** bbtest-net: corrupted double-linked list: 0x00000000149dbf30 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x000000000e655e60 *** *** glibc detected *** bbtest-net: double free or corruption (out): 0x0000000002c46920 ***
gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.31727
GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x0000000000424a57 in qcallback (arg=0x1bc753c0, status=0, abuf=0x7fff461c1c50 "\201\200", alen=140) at ares_query.c:108 #5 0x00000000004216c2 in end_query (channel=0x1bc5e4d0, query=0x1bcf41c0, status=0, abuf=0x0, alen=1176247376) at ares_process.c:627 #6 0x0000000000421ebd in process_answer (channel=0x1bc5e4d0, abuf=0x7fff461c1c50 "\201\200", alen=140, whichserver=0, tcp=<value optimized out>, now=1226882897) at ares_process.c:359 #7 0x00000000004221ff in ares_process (channel=0x1bc5e4d0, read_fds=0x7fff461c1f20, write_fds=0x7fff461c1ea0) at ares_process.c:273 #8 0x000000000040f064 in dns_queue_run (channel=0x1bc5e4d0) at dns.c:228 #9 0x000000000040f464 in flush_dnsqueue () at dns.c:259 #10 0x00000000004088e6 in main (argc=4, argv=0x7fff461c3428) at bbtest-net.c:2099 #11 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #12 0x00000000004039e9 in _start () (gdb) quit
*** glibc detected *** bbtest-net: double free or corruption (out): 0x000000000e655e60 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3db7a71634] /lib64/libc.so.6(cfree+0x8c)[0x3db7a74c5c] bbtest-net[0x422424] bbtest-net[0x422515] bbtest-net[0x424a59] bbtest-net[0x4216c2] bbtest-net[0x421ebd] bbtest-net[0x4221ff] bbtest-net[0x40f064] bbtest-net[0x40f464] bbtest-net[0x4088e6] /lib64/libc.so.6(__libc_start_main+0xf4)[0x3db7a1d8b4] bbtest-net[0x4039e9] ======= Memory map: ======== 00400000-00430000 r-xp 00000000 fd:01 389476 /usr/lib/hobbit/server/bin/bbtest-net 00630000-00631000 rw-p 00030000 fd:01 389476 /usr/lib/hobbit/server/bin/bbtest-net 00631000-00637000 rw-p 00631000 00:00 0 00830000-00832000 rw-p 00030000 fd:01 389476 /usr/lib/hobbit/server/bin/bbtest-net 0e533000-0e6bf000 rw-p 0e533000 00:00 0 3224600000-3224638000 r-xp 00000000 fd:01 68061 /usr/lib64/libldap-2.3.so.0.2.15 3224638000-3224838000 ---p 00038000 fd:01 68061 /usr/lib64/libldap-2.3.so.0.2.15 3224838000-322483a000 rw-p 00038000 fd:01 68061 /usr/lib64/libldap-2.3.so.0.2.15 3366400000-3366443000 r-xp 00000000 fd:00 75879 /lib64/libssl.so.0.9.8b 3366443000-3366643000 ---p 00043000 fd:00 75879 /lib64/libssl.so.0.9.8b 3366643000-3366649000 rw-p 00043000 fd:00 75879 /lib64/libssl.so.0.9.8b 3368000000-3368125000 r-xp 00000000 fd:00 75876 /lib64/libcrypto.so.0.9.8b 3368125000-3368325000 ---p 00125000 fd:00 75876 /lib64/libcrypto.so.0.9.8b 3368325000-3368344000 rw-p 00125000 fd:00 75876 /lib64/libcrypto.so.0.9.8b 3368344000-3368348000 rw-p 3368344000 00:00 0 385c600000-385c63b000 r-xp 00000000 fd:00 75779 /lib64/libsepol.so.1 385c63b000-385c83b000 ---p 0003b000 fd:00 75779 /lib64/libsepol.so.1 385c83b000-385c83c000 rw-p 0003b000 fd:00 75779 /lib64/libsepol.so.1 385c83c000-385c846000 rw-p 385c83c000 00:00 0 385ca00000-385ca15000 r-xp 00000000 fd:00 75786 /lib64/libselinux.so.1 385ca15000-385cc15000 ---p 00015000 fd:00 75786 /lib64/libselinux.so.1 385cc15000-385cc17000 rw-p 00015000 fd:00 75786 /lib64/libselinux.so.1 385cc17000-385cc18000 rw-p 385cc17000 00:00 0 385ce00000-385ce8f000 r-xp 00000000 fd:01 68057 /usr/lib64/libkrb5.so.3.3 385ce8f000-385d08e000 ---p 0008f000 fd:01 68057 /usr/lib64/libkrb5.so.3.3 385d08e000-385d092000 rw-p 0008e000 fd:01 68057 /usr/lib64/libkrb5.so.3.3 385d600000-385d608000 r-xp 00000000 fd:01 68055 /usr/lib64/libkrb5support.so.0.1 385d608000-385d807000 ---p 00008000 fd:01 68055 /usr/lib64/libkrb5support.so.0.1 385d807000-385d808000 rw-p 00007000 fd:01 68055 /usr/lib64/libkrb5support.so.0.1 385da00000-385da24000 r-xp 00000000 fd:01 68056 /usr/lib64/libk5crypto.so.3.1 385da24000-385dc23000 ---p 00024000 fd:01 68056 /usr/lib64/libk5crypto.so.3.1 385dc23000-385dc25000 rw-p 00023000 fd:01 68056 /usr/lib64/libk5crypto.so.3.1 385de00000-385de2c000 r-xp 00000000 fd:01 68058 /usr/lib64/libgssapi_krb5.so.2.2 385de2c000-385e02c000 ---p 0002c000 fd:01 68058 /usr/lib64/libgssapi_krb5.so.2.2 385e02c000-385e02e000 rw-p 0002c000 fd:01 68058 /usr/lib64/libgssapi_krb5.so.2.2 3af4400000-3af440d000 r-xp 00000000 fd:01 68095 /usr/lib64/liblber-2.3.so.0.2.15 3af440d000-3af460d000 ---p 0000d000 fd:01 68095 /usr/lib64/liblber-2.3.so.0.2.15 3af460d000-3af460e000 rw-p 0000d000 fd:01 68095 /usr/lib64/liblber-2.3.so.0.2.15 3db7600000-3db761a000 r-xp 00000000 fd:00 75791 /lib64/ld-2.5.so 3db781a000-3db781b000 r--p 0001a000 fd:00 75791 /lib64/ld-2.5.so 3db781b000-3db781c000 rw-p 0001b000 fd:00 75791 /lib64/ld-2.5.so 3db7a00000-3db7b4a000 r-xp 00000000 fd:00 75797 /lib64/libc-2.5.so 3db7b4a000-3db7d49000 ---p 0014a000 fd:00 75797 /lib64/libc-2.5.so 3db7d49000-3db7d4d000 r--p 00149000 fd:00 75797 /lib64/libc-2.5.so 3db7d4d000-3db7d4e000 rw-p 0014d000 fd:00 75797 /lib64/libc-2.5.so 3db7d4e000-3db7d53000 rw-p 3db7d4e000 00:00 0 3db7e00000-3db7e02000 r-xp 00000000 fd:00 75840 /lib64/libdl-2.5.so 3db7e02000-3db8002000 ---p 00002000 fd:00 75840 /lib64/libdl-2.5.so 3db8002000-3db8003000 r--p 00002000 fd:00 75840 /lib64/libdl-2.5.so 3db8003000-3db8004000 rw-p 00003000 fd:00 75840 /lib64/libdl-2.5.so 3db8200000-3db8218000 r-xp 00000000 fd:01 67958 /usr/lib64/libsasl2.so.2.0.22 3db8218000-3db8418000 ---p 00018000 fd:01 67958 /usr/lib64/libsasl2.so.2.0.22 3db8418000-3db8419000 rw-p 00018000 fd:01 67958 /usr/lib64/libsasl2.so.2.0.22 3db8600000-3db8614000 r-xp 00000000 fd:01 67257 /usr/lib64/libz.so.1.2.3 3db8614000-3db8813000 ---p 00014000 fd:01 67257 /usr/lib64/libz.so.1.2.3 3db8813000-3db8814000 rw-p 00013000 fd:01 67257 /usr/lib64/libz.so.1.2.3 3db9a00000-3db9a09000 r-xp 00000000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9a09000-3db9c08000 ---p 00009000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9c08000-3db9c09000 r--p 00008000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9c09000-3db9c0a000 rw-p 00009000 fd:00 76086 /lib64/libcrypt-2.5.so 3db9c0a000-3db9c38000 rw-p 3db9c0a000 00:00 0 3dba600000-3dba611000 r-xp 00000000 fd:00 76082 /lib64/libresolv-2.5.so 3dba611000-3dba811000 ---p 00011000 fd:00 76082 /lib64/libresolv-2.5.so 3dba811000-3dba812000 r--p 00011000 fd:00 76082 /lib64/libresolv-2.5.so 3dba812000-3dba813000 rw-p 00012000 fd:00 76082 /lib64/libresolv-2.5.so 3dba813000-3dba815000 rw-p 3dba813000 00:00 0 3dbaa00000-3dbaa02000 r-xp 00000000 fd:00 76083 /lib64/libcom_err.so.2.1 3dbaa02000-3dbac01000 ---p 00002000 fd:00 76083 /lib64/libcom_err.so.2.1 3dbac01000-3dbac02000 rw-p 00001000 fd:00 76083 /lib64/libcom_err.so.2.1 3dbba00000-3dbba02000 r-xp 00000000 fd:00 76080 /lib64/libkeyutils-1.2.so 3dbba02000-3dbbc01000 ---p 00002000 fd:00 76080 /lib64/libkeyutils-1.2.so 3dbbc01000-3dbbc02000 rw-p 00001000 fd:00 76080 /lib64/libkeyutils-1.2.so 3dbc200000-3dbc20d000 r-xp 00000000 fd:00 75803 /lib64/libgcc_s-4.1.2-20080102.so.1 3dbc20d000-3dbc40d000 ---p 0000d000 fd:00 75803 /lib64/libgcc_s-4.1.2-20080102.so.1 3dbc40d000-3dbc40e000 rw-p 0000d000 fd:00 75803 /lib64/libgcc_s-4.1.2-20080102.so.1 2b7f796eb000-2b7f796ec000 rw-p 2b7f796eb000 00:00 0 2b7f796f8000-2b7f79700000 rw-p 2b7f796f8000 00:00 0 2b7f79700000-2b7f7970a000 r-xp 00000000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f7970a000-2b7f79909000 ---p 0000a000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f79909000-2b7f7990a000 r--p 00009000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f7990a000-2b7f7990b000 rw-p 0000a000 fd:00 75873 /lib64/libnss_files-2.5.so 2b7f7c000000-2b7f7c021000 rw-p 2b7f7c000000 00:00 0 2b7f7c021000-2b7f80000000 ---p 2b7f7c021000 00:00 0 7fff313a9000-7fff313bf000 rw-p 7fff313a9000 00:00 0 [stack] ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
[root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.31042 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x0000000000424a57 in qcallback (arg=0x163b9ee0, status=0, abuf=0x7fff9d536fc0 "\201\200", alen=140) at ares_query.c:108 #5 0x00000000004216c2 in end_query (channel=0x163a34d0, query=0x163d6e50, status=0, abuf=0x0, alen=-1655476288) at ares_process.c:627 #6 0x0000000000421ebd in process_answer (channel=0x163a34d0, abuf=0x7fff9d536fc0 "\201\200", alen=140, whichserver=0, tcp=<value optimized out>, now=1226882805) at ares_process.c:359 #7 0x00000000004221ff in ares_process (channel=0x163a34d0, read_fds=0x7fff9d537290, write_fds=0x7fff9d537210) at ares_process.c:273 #8 0x000000000040f064 in dns_queue_run (channel=0x163a34d0) at dns.c:228 #9 0x000000000040f464 in flush_dnsqueue () at dns.c:259 #10 0x00000000004088e6 in main (argc=4, argv=0x7fff9d538798) at bbtest-net.c:2099 #11 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #12 0x00000000004039e9 in _start () (gdb) quit [root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.15643 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x0000003db7a79f10 in strcasecmp () from /lib64/libc.so.6 #5 0x0000000000418cda in rbtFind (h=0x8d4b680, key=0x8e22310) at rbtr.c:381 #6 0x000000000040f4d8 in find_dnscache (hostname=0x0) at dns.c:86 #7 0x000000000040f546 in dnsresolve (hostname=0x8e22310 "DRPIR101") at dns.c:275 #8 0x0000000000406848 in ip_to_test (h=0x8e22150) at bbtest-net.c:790 #9 0x00000000004069ca in start_ping_service (service=<value optimized out>) at bbtest-net.c:1099 #10 0x000000000040891f in main (argc=4, argv=0x7fffefcdff38) at bbtest-net.c:2105 #11 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #12 0x00000000004039e9 in _start () (gdb) quit [root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.13374 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 0x00000000004069b7 in start_ping_service (service=<value optimized out>) at bbtest-net.c:1098 #5 0x000000000040891f in main (argc=4, argv=0x7fff8f61e4d8) at bbtest-net.c:2105 #6 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #7 0x00000000004039e9 in _start () (gdb) quit
[root at las006 tmp]# gdb /usr/lib/hobbit/server/bin/bbtest-net -c core.7801 GNU gdb Red Hat Linux (6.5-37.el5_2.1rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /usr/lib64/libldap-2.3.so.0...done. Loaded symbols for /usr/lib64/libldap-2.3.so.0 ** snip ** Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `bbtest-net --report --ping --checkresponse'. Program terminated with signal 6, Aborted. #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003db7a30155 in raise () from /lib64/libc.so.6 #1 0x0000003db7a31bf0 in abort () from /lib64/libc.so.6 #2 0x000000000041d7e3 in sigsegv_handler (signum=<value optimized out>) at sig.c:57 #3 <signal handler called> #4 ares_process (channel=0x157137c0, read_fds=0x7fff91e5dbb0, write_fds=0x7fff91e5db30) at ares_process.c:285 #5 0x000000000040f064 in dns_queue_run (channel=0x157137c0) at dns.c:228 #6 0x000000000040f464 in flush_dnsqueue () at dns.c:259 #7 0x00000000004088e6 in main (argc=4, argv=0x7fff91e5f0b8) at bbtest-net.c:2099 #8 0x0000003db7a1d8b4 in __libc_start_main () from /lib64/libc.so.6 #9 0x00000000004039e9 in _start () (gdb) quit
NOTICE: This email and any attachments are confidential. They may contain legally privileged information or copyright material. You must not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages and all attachments.
In <A3D12FAD74FC8B46991703F40C182BABA20D5B4F at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
--_000_A3D12FAD74FC8B46991703F40C182BABA20D5B4Fpermls102wdewoo_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Below are our 5 most recent cores. And the corresponding entry in the bb-network.log file where available. (No= t all generate the entry in the log file)
It seems the problems occur in the C-ARES library which is used for hostname lookups (via DNS).
Hobbit 4.2.0 uses version 1.2.1 of C-ARES; the current version is 1.5.3. Unfortunately there's been a small change of the C-ARES API, so we cannot just drop version 1.5.3 of the library into Hobbit 4.2.0. But it would be nice to know if the problem is still there in the latest version.
If you want to try that, you can grab version 1.4.0 from http://c-ares.haxx.se/c-ares-1.4.0.tar.gz and then change Hobbit's build/Makefile.rules: The "ARESVER" setting should be "1.4.0". Then "make clean; rm -rf bbnet/c-ares/* bbnet/libcares.*; make" and copy bbnet/bbtest-net to ~hobbit/server/bin/
If you want to try out the 1.5.3 version, drop me a mail and I'll send you the patches needed for this to compile with Hobbit 4.2.0.
Colin, our Linux man, and part-time C hacker, had this to say
---snip--- [root at las006 tmp]# gdb ~hobbit/server/bin/bbtest-net (gdb) l *0x42493a 0x42493a is in ares_free_hostent (ares_free_hostent.c:35). 30 free(host->h_name); 31 for (p =3D host->h_aliases; *p; p++) 32 free(*p); 33 free(host->h_aliases); 34 free(host->h_addr_list[0]); 35 free(host->h_addr_list); 36 free(host); 37 } (gdb)
line 31: Make 'p' equal to the memory address of host->h_aliases, check tha= t pointer p is not null and increment p line 32: free the memory pointed at by 'p'
The problem here is that you check p then free p++. My C is quite rusty bu= t I'd hazard a guess and say line 31 should read like: 31 for (p =3D host->h_aliases; *(p++);)
Nope, sorry - Colin needs to revisit his C programming manual. The "for" loop construct in line 31-32 will do the "p++" AFTER doing the "free(*p)".
Another way of writing the same code would be
for (i=0; *(host->h_aliases[i]); i++) free(host->h_aliases[i]);
where it is perhaps clearer that this goes through an array of host aliases until there's an empty alias (this indicates the end of the array), and frees the memory used to record each alias. But again, the counter (i) is incremented AFTER the "free" is done.
The code is a bit obscure, but I think it's correct (didn't write it myself, though ...) I'm afraid the data structures used for hostname lookups weren't written to be easy to understand.
Regards, Henrik
-----Original Message----- From: Henrik "Størner [mailto:henrik at hswn.dk] Sent: Tuesday, November 18, 2008 9:06 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Hobbit bbtest-net crashing
In <A3D12FAD74FC8B46991703F40C182BABA20D5B4F at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
--_000_A3D12FAD74FC8B46991703F40C182BABA20D5B4Fpermls102wdewoo_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Below are our 5 most recent cores. And the corresponding entry in the bb-network.log file where available. (No= t all generate the entry in the log file)
It seems the problems occur in the C-ARES library which is used for hostname lookups (via DNS).
Hobbit 4.2.0 uses version 1.2.1 of C-ARES; the current version is 1.5.3. Unfortunately there's been a small change of the C-ARES API, so we cannot just drop version 1.5.3 of the library into Hobbit 4.2.0. But it would be nice to know if the problem is still there in the latest version.
If you want to try that, you can grab version 1.4.0 from http://c-ares.haxx.se/c-ares-1.4.0.tar.gz and then change Hobbit's build/Makefile.rules: The "ARESVER" setting should be "1.4.0". Then "make clean; rm -rf bbnet/c-ares/* bbnet/libcares.*; make" and copy bbnet/bbtest-net to ~hobbit/server/bin/
If you want to try out the 1.5.3 version, drop me a mail and I'll send you the patches needed for this to compile with Hobbit 4.2.0.
I've been using version 1.3.2 for over a year now without any problems.
Greg
Hi Henrik
Does 4.2.2 RC? use C-ARES 1.2.1, 1.4.0 or 1.5.3? Or have you not made any changes that drastic?
We got a test server up and running 4.2 with patches and C-ARES 1.4.0, and we will see if it also crashes.
Cheers V
-----Original Message----- From: Henrik Størner [mailto:henrik at hswn.dk] Sent: Tuesday, 18 November 2008 11:06 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Hobbit bbtest-net crashing
In <A3D12FAD74FC8B46991703F40C182BABA20D5B4F at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
--_000_A3D12FAD74FC8B46991703F40C182BABA20D5B4Fpermls102wdewoo_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Below are our 5 most recent cores. And the corresponding entry in the bb-network.log file where available. (No= t all generate the entry in the log file)
It seems the problems occur in the C-ARES library which is used for hostname lookups (via DNS).
Hobbit 4.2.0 uses version 1.2.1 of C-ARES; the current version is 1.5.3. Unfortunately there's been a small change of the C-ARES API, so we cannot just drop version 1.5.3 of the library into Hobbit 4.2.0. But it would be nice to know if the problem is still there in the latest version.
If you want to try that, you can grab version 1.4.0 from http://c-ares.haxx.se/c-ares-1.4.0.tar.gz and then change Hobbit's build/Makefile.rules: The "ARESVER" setting should be "1.4.0". Then "make clean; rm -rf bbnet/c-ares/* bbnet/libcares.*; make" and copy bbnet/bbtest-net to ~hobbit/server/bin/
If you want to try out the 1.5.3 version, drop me a mail and I'll send you the patches needed for this to compile with Hobbit 4.2.0.
NOTICE: This email and any attachments are confidential. They may contain legally privileged information or copyright material. You must not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages and all attachments.
In <A3D12FAD74FC8B46991703F40C182BABAB31681A at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
Does 4.2.2 RC? use C-ARES 1.2.1, 1.4.0 or 1.5.3? Or have you not made any changes that drastic?
4.2.2 still has the 1.2.1 version of c-ares in it.
I've added the patch needed for it to compile with the current version of C-ARES (1.5.x), so if need be we can easily drop that into the code if people have problems with the old version. This is not in RC1, but in the repository.
Regards, Henrik
Henrik StXXrner wrote:
In <A3D12FAD74FC8B46991703F40C182BABAB31681A at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
Does 4.2.2 RC? use C-ARES 1.2.1, 1.4.0 or 1.5.3? Or have you not made any changes that drastic?
4.2.2 still has the 1.2.1 version of c-ares in it.
I've added the patch needed for it to compile with the current version of C-ARES (1.5.x), so if need be we can easily drop that into the code if people have problems with the old version. This is not in RC1, but in the repository.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
FWIW, using the ARES resolver on out CentOS4 x86_64 systems has been a general disaster, running much slower than disabling it. -Alan
Disabling it? Are you saying we do not need to use this module? How would I go about changing my installation to not use C-ARES? Bbtest.net is crashing at least twice a day in our environment.
Any advice appreciated.
Cheers V
-----Original Message----- From: Alan Sparks [mailto:asparks at doublesparks.net] Sent: Thursday, 4 December 2008 9:17 AM To: Hobbit List Subject: Re: [hobbit] Hobbit bbtest-net crashing
Henrik StXXrner wrote:
In <A3D12FAD74FC8B46991703F40C182BABAB31681A at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
Does 4.2.2 RC? use C-ARES 1.2.1, 1.4.0 or 1.5.3? Or have you not made any changes that drastic?
4.2.2 still has the 1.2.1 version of c-ares in it.
I've added the patch needed for it to compile with the current version of C-ARES (1.5.x), so if need be we can easily drop that into the code if people have problems with the old version. This is not in RC1, but in the repository.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
FWIW, using the ARES resolver on out CentOS4 x86_64 systems has been a general disaster, running much slower than disabling it. -Alan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
NOTICE: This email and any attachments are confidential. They may contain legally privileged information or copyright material. You must not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages and all attachments.
In <A3D12FAD74FC8B46991703F40C182BABAB316B3C at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
Disabling it? Are you saying we do not need to use this module? How would I go about changing my installation to not use C-ARES?
Run bbtest-net with the "--no-ares" option.
Regards, Henrik
Hmmm Found the answer. Thanks Alan. And a little caveat.
"The system resolver function does not provide a mechanism for controlling timeouts of the hostname lookups, so if your DNS or NIS server is down, bbtest-net can take a very long time to run. The --dns-timeout option is effectively disabled when using this option."
Amazing what you can learn when you sit down and carefully RTFM :-)
A little worried about implementing it just yet, but we will see how it goes.
Of course, as is always the case, since I started getting somewhere with trying to resolve this, it stopped crashing. No crashes since last night - touch wood.
Cheers V
-----Original Message----- From: Everett, Vernon Sent: Thursday, 4 December 2008 9:25 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Hobbit bbtest-net crashing
Disabling it? Are you saying we do not need to use this module? How would I go about changing my installation to not use C-ARES? Bbtest.net is crashing at least twice a day in our environment.
Any advice appreciated.
Cheers V
-----Original Message----- From: Alan Sparks [mailto:asparks at doublesparks.net] Sent: Thursday, 4 December 2008 9:17 AM To: Hobbit List Subject: Re: [hobbit] Hobbit bbtest-net crashing
Henrik StXXrner wrote:
In <A3D12FAD74FC8B46991703F40C182BABAB31681A at permls102.wde.woodside.com.au> "Everett, Vernon" <Vernon.Everett at woodside.com.au> writes:
Does 4.2.2 RC? use C-ARES 1.2.1, 1.4.0 or 1.5.3? Or have you not made any changes that drastic?
4.2.2 still has the 1.2.1 version of c-ares in it.
I've added the patch needed for it to compile with the current version of C-ARES (1.5.x), so if need be we can easily drop that into the code if people have problems with the old version. This is not in RC1, but in the repository.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
FWIW, using the ARES resolver on out CentOS4 x86_64 systems has been a general disaster, running much slower than disabling it. -Alan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
NOTICE: This email and any attachments are confidential. They may contain legally privileged information or copyright material. You must not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages and all attachments.
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
NOTICE: This email and any attachments are confidential. They may contain legally privileged information or copyright material. You must not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages and all attachments.
participants (4)
-
asparks＠doublesparks.net
-
henrik＠hswn.dk
-
shea_greg＠emc.com
-
Vernon.Everett＠woodside.com.au