buffer overflow detected in xymongen (4.3.21)
Hi,
today our xymongen check went purple for about an hour. In the xymongen.log I found tons of crash reports like this one:
*** buffer overflow detected ***: xymongen terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x731ff)[0x7f31a13fc1ff] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f31a147f4c7] /lib/x86_64-linux-gnu/libc.so.6(+0xf46e0)[0x7f31a147d6e0] xymongen[0x40d526] xymongen[0x403b72] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f31a13aab45] xymongen[0x40510c] ======= Memory map: ======== 00400000-00445000 r-xp 00000000 fd:00 151583 /usr/lib/xymon/server/bin/xymongen 00644000-00645000 r--p 00044000 fd:00 151583 /usr/lib/xymon/server/bin/xymongen 00645000-00647000 rw-p 00045000 fd:00 151583 /usr/lib/xymon/server/bin/xymongen 00647000-00658000 rw-p 00000000 00:00 0 01ea3000-022fe000 rw-p 00000000 00:00 0 [heap] 7f31a0c26000-7f31a0c3c000 r-xp 00000000 fd:00 524307 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f31a0c3c000-7f31a0e3b000 ---p 00016000 fd:00 524307 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f31a0e3b000-7f31a0e3c000 rw-p 00015000 fd:00 524307 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f31a0e3c000-7f31a0f68000 rw-p 00000000 00:00 0 7f31a0f68000-7f31a0f80000 r-xp 00000000 fd:00 535294 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f31a0f80000-7f31a117f000 ---p 00018000 fd:00 535294 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f31a117f000-7f31a1180000 r--p 00017000 fd:00 535294 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f31a1180000-7f31a1181000 rw-p 00018000 fd:00 535294 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f31a1181000-7f31a1185000 rw-p 00000000 00:00 0 7f31a1185000-7f31a1188000 r-xp 00000000 fd:00 535286 /lib/x86_64-linux-gnu/libdl-2.19.so 7f31a1188000-7f31a1387000 ---p 00003000 fd:00 535286 /lib/x86_64-linux-gnu/libdl-2.19.so 7f31a1387000-7f31a1388000 r--p 00002000 fd:00 535286 /lib/x86_64-linux-gnu/libdl-2.19.so 7f31a1388000-7f31a1389000 rw-p 00003000 fd:00 535286 /lib/x86_64-linux-gnu/libdl-2.19.so 7f31a1389000-7f31a1528000 r-xp 00000000 fd:00 535301 /lib/x86_64-linux-gnu/libc-2.19.so 7f31a1528000-7f31a1728000 ---p 0019f000 fd:00 535301 /lib/x86_64-linux-gnu/libc-2.19.so 7f31a1728000-7f31a172c000 r--p 0019f000 fd:00 535301 /lib/x86_64-linux-gnu/libc-2.19.so 7f31a172c000-7f31a172e000 rw-p 001a3000 fd:00 535301 /lib/x86_64-linux-gnu/libc-2.19.so 7f31a172e000-7f31a1732000 rw-p 00000000 00:00 0 7f31a1732000-7f31a179e000 r-xp 00000000 fd:00 524394 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f31a179e000-7f31a199e000 ---p 0006c000 fd:00 524394 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f31a199e000-7f31a199f000 r--p 0006c000 fd:00 524394 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f31a199f000-7f31a19a0000 rw-p 0006d000 fd:00 524394 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f31a19a0000-7f31a1b6b000 r-xp 00000000 fd:00 131288 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f31a1b6b000-7f31a1d6b000 ---p 001cb000 fd:00 131288 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f31a1d6b000-7f31a1d88000 r--p 001cb000 fd:00 131288 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f31a1d88000-7f31a1d98000 rw-p 001e8000 fd:00 131288 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f31a1d98000-7f31a1d9b000 rw-p 00000000 00:00 0 7f31a1d9b000-7f31a1df1000 r-xp 00000000 fd:00 133362 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 7f31a1df1000-7f31a1ff1000 ---p 00056000 fd:00 133362 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 7f31a1ff1000-7f31a1ff4000 r--p 00056000 fd:00 133362 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 7f31a1ff4000-7f31a1ffb000 rw-p 00059000 fd:00 133362 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 7f31a1ffb000-7f31a201b000 r-xp 00000000 fd:00 535260 /lib/x86_64-linux-gnu/ld-2.19.so 7f31a220b000-7f31a2210000 rw-p 00000000 00:00 0 7f31a2217000-7f31a221b000 rw-p 00000000 00:00 0 7f31a221b000-7f31a221c000 r--p 00020000 fd:00 535260 /lib/x86_64-linux-gnu/ld-2.19.so 7f31a221c000-7f31a221d000 rw-p 00021000 fd:00 535260 /lib/x86_64-linux-gnu/ld-2.19.so 7f31a221d000-7f31a221e000 rw-p 00000000 00:00 0 7fff1cf05000-7fff1cf2d000 rw-p 00000000 00:00 0 [stack] 7fff1cffc000-7fff1cffe000 r-xp 00000000 00:00 0 [vdso] 7fff1cffe000-7fff1d000000 r--p 00000000 00:00 0 [vvar] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
After ca. one hour it went yellow again with the following error output: "symlink nongreen.xml->index.wml failed: Transport endpoint is not connected". Shortly thereafter it went green again, but the runtime is about 1.5 times as high as before.
xymon is installed from Debian's xymon packages.
Feel free to tell me what else could be helpful to track down this bufferflow. I've found no (recent) core dump.
Kind regards, Axel Beckert-- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/
Hi again,
On Thu, Jul 02, 2015 at 03:49:17PM +0200, Axel Beckert wrote:
today our xymongen check went purple for about an hour. In the xymongen.log I found tons of crash reports like this one:
*** buffer overflow detected ***: xymongen terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x731ff)[0x7f31a13fc1ff] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f31a147f4c7] /lib/x86_64-linux-gnu/libc.so.6(+0xf46e0)[0x7f31a147d6e0] xymongen[0x40d526] xymongen[0x403b72] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f31a13aab45] xymongen[0x40510c] ======= Memory map: ======== [...] After ca. one hour it went yellow again [...]
In the meanwhile I had cases where it took like 15 hours or so to recover.
Feel free to tell me what else could be helpful to track down this bufferflow. I've found no (recent) core dump.
In the meanwhile I gathered hundreds of core dumps, but the backtrace generated from them is not that helpful except for the exact commandline:
[New LWP 104939] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `xymongen --recentgifs --subpagecolumns=2 --wml --rss --nongreen-ignorecolumns=l'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f96ef76a107 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
... because the error message...
"symlink nongreen.xml->index.wml failed: Transport endpoint is not connected".
... comes from xymongen/wmlgen.c, i.e. I assume the buffer overflow is related to the generation of the WML view of Xymon as it would explain why nobody else suffered from it so far as this feature is probably rarely used nowadays.
I'll also disable it for now in our setup to see if I'm right with my assumption about the source of the buffer overflow. But I still think it should be fixed if it's indeed in there.
Kind regards, Axel Beckert-- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/
Hi Axel,
Would you be able to do a full BT on one of those core dumps? I'm not certain if Debian splits the debuginfo off into a separate package (like is done on the RH side), but that might need to be installed first.
Having the specific line this is coming from would make tracking down the root easier.
WML is indeed probably one of the more rarely used features nowadays, so it's quite possible there's a latent bug in there.
-jc
On 9/29/2015 10:27 AM, Axel Beckert wrote:
Hi again,
On Thu, Jul 02, 2015 at 03:49:17PM +0200, Axel Beckert wrote:
today our xymongen check went purple for about an hour. In the xymongen.log I found tons of crash reports like this one:
*** buffer overflow detected ***: xymongen terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x731ff)[0x7f31a13fc1ff] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f31a147f4c7] /lib/x86_64-linux-gnu/libc.so.6(+0xf46e0)[0x7f31a147d6e0] xymongen[0x40d526] xymongen[0x403b72] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f31a13aab45] xymongen[0x40510c] ======= Memory map: ======== [...] After ca. one hour it went yellow again [...] In the meanwhile I had cases where it took like 15 hours or so to recover.
Feel free to tell me what else could be helpful to track down this bufferflow. I've found no (recent) core dump. In the meanwhile I gathered hundreds of core dumps, but the backtrace generated from them is not that helpful except for the exact commandline:
[New LWP 104939] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `xymongen --recentgifs --subpagecolumns=2 --wml --rss --nongreen-ignorecolumns=l'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f96ef76a107 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
... because the error message...
"symlink nongreen.xml->index.wml failed: Transport endpoint is not connected". ... comes from xymongen/wmlgen.c, i.e. I assume the buffer overflow is related to the generation of the WML view of Xymon as it would explain why nobody else suffered from it so far as this feature is probably rarely used nowadays.
I'll also disable it for now in our setup to see if I'm right with my assumption about the source of the buffer overflow. But I still think it should be fixed if it's indeed in there.
Kind regards, Axel Beckert
Hi,
On Tue, Sep 29, 2015 at 10:40:32AM -0700, Japheth Cleaver wrote:
Would you be able to do a full BT on one of those core dumps?
Only with recompiling the packages.
I'm not certain if Debian splits the debuginfo off into a separate package (like is done on the RH side), but that might need to be installed first.
We currently don't build debug packages for Debian's xymon package. They're (currently) optional in Debian and only present if the package's maintainer adds them explicitly. And we didn't need them for xymon so far.
Debug packages in Debian will change generally in the near future as automatically built debug packages will be available in Debian Unstable soon-ish. So I won't manually add debug packages to the official Debian package as they will become obsolete soon. And because adding them requires some additional review process.
So I'll try to add debug packages to my backports the classical way by specifying them manually.
P.S.: No new crashes since I disabled the WML generation yesterday, but that's probably not yet a significant amount of time. ;-)
Kind regards, Axel (one of Debian's xymon package maintainers)
Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/
participants (2)
-
beckert＠phys.ethz.ch
-
cleaver＠terabithia.org