buffer overflow detected in xymongen (4.3.21)
Hi,
On Tue, Sep 29, 2015 at 10:40:32AM -0700, Japheth Cleaver wrote:
Would you be able to do a full BT on one of those core dumps?
Today it finally crashed again after I deployed the debug packages. (It's seldom that I'm so happy to see a core dump. ;-)
Here's the most recent backtrace:
Reading symbols from /usr/lib/xymon/server/bin/xymongen...done. Reading symbols from /usr/lib/debug/.build-id/c8/bc967d123c1a308a9973d19da8c2357af16a6d.debug...done. [New LWP 154856] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `xymongen --recentgifs --subpagecolumns=2 --wml --rss --nongreen-ignorecolumns=l'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f29044aa107 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007f29044aa107 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007f29044ab4e8 in __GI_abort () at abort.c:89 #2 0x00007f29044e8204 in __libc_message (do_abort=do_abort at entry=2, fmt=fmt at entry=0x7f29045d8a2b "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f290456b4c7 in __GI___fortify_fail (msg=msg at entry=0x7f29045d89c2 "buffer overflow detected") at fortify_fail.c:31 #4 0x00007f29045696e0 in __GI___chk_fail () at chk_fail.c:28 #5 0x000000000040d526 in strcpy (__src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:104 #6 generate_wml_statuscard (host=<optimized out>, entry=<optimized out>) at wmlgen.c:150 #7 do_wml_cards (webdir=0x7f29045d0200 <_itoa_lower_digits> "0123456789abcdefghijklmnopqrstuvwxyz") at wmlgen.c:296 #8 0x0000000000403b72 in main (argc=4404274, argv=0x25ce8) at xymongen.c:678 (gdb)
And here's the oldest one:
Reading symbols from /usr/lib/xymon/server/bin/xymongen...done. Reading symbols from /usr/lib/debug/.build-id/c8/bc967d123c1a308a9973d19da8c2357af16a6d.debug...done. [New LWP 122254] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `xymongen --recentgifs --subpagecolumns=2 --wml --rss --nongreen-ignorecolumns=l'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f12adf18107 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007f12adf18107 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007f12adf194e8 in __GI_abort () at abort.c:89 #2 0x00007f12adf56204 in __libc_message (do_abort=do_abort at entry=2, fmt=fmt at entry=0x7f12ae046a2b "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f12adfd94c7 in __GI___fortify_fail (msg=msg at entry=0x7f12ae0469c2 "buffer overflow detected") at fortify_fail.c:31 #4 0x00007f12adfd76e0 in __GI___chk_fail () at chk_fail.c:28 #5 0x000000000040d526 in strcpy (__src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:104 #6 generate_wml_statuscard (host=<optimized out>, entry=<optimized out>) at wmlgen.c:150 #7 do_wml_cards (webdir=0x7f12ae03e200 <_itoa_lower_digits> "0123456789abcdefghijklmnopqrstuvwxyz") at wmlgen.c:296 #8 0x0000000000403b72 in main (argc=4404274, argv=0x1dd8e) at xymongen.c:678 (gdb)
They look more or less identical to me.
It's indeed in the WML generation code, should be this line here: https://sources.debian.net/src/xymon/4.3.21-1/xymongen/wmlgen.c/#L150
HTH
Kind regards, Axel Beckert-- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/
Hi,
On Fri, Oct 02, 2015 at 01:32:51PM +0200, Axel Beckert wrote:
Today it finally crashed again after I deployed the debug packages. (It's seldom that I'm so happy to see a core dump. ;-)
Here's the [...] oldest one: [...] It's indeed in the WML generation code, should be this line here: https://sources.debian.net/src/xymon/4.3.21-1/xymongen/wmlgen.c/#L150
Looking at what status changed in the five minutes before the crash I found two status reports which had each line lengths of 6576 and 6897 characters, but both far shorter than the MAX_LINE_LEN of 16384 characters
Another thing I could imagine are lines with no trailing newline at the end. But then again I have no idea where they could come from nor how I should look for them.
Kind regards, Axel Beckert-- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/
A malformed message could be returned as a result of a truncated network connection to xymond, or other sorts of interesting cases.
p = strchr(nextline, '\n'); if (p) *p = '\0';
strcpy(l, nextline);
if (p) nextline = p+1; else nextline = NULL;There does seem to be something a little odd there.
I suppose the below patch would at least keep us from writing past the end on 'l', but in a proper situation we shouldn't need something like it.
-jc
On Fri, October 2, 2015 4:58 am, Axel Beckert wrote:
Hi,
On Fri, Oct 02, 2015 at 01:32:51PM +0200, Axel Beckert wrote:
Today it finally crashed again after I deployed the debug packages. (It's seldom that I'm so happy to see a core dump. ;-)
Here's the [...] oldest one: [...] It's indeed in the WML generation code, should be this line here: https://sources.debian.net/src/xymon/4.3.21-1/xymongen/wmlgen.c/#L150
Looking at what status changed in the five minutes before the crash I found two status reports which had each line lengths of 6576 and 6897 characters, but both far shorter than the MAX_LINE_LEN of 16384 characters
Another thing I could imagine are lines with no trailing newline at the end. But then again I have no idea where they could come from nor how I should look for them.
Kind regards, Axel Beckert-- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Hi,
On Fri, Oct 02, 2015 at 11:58:28AM -0700, J.C. Cleaver wrote:
A malformed message could be returned as a result of a truncated network connection to xymond,
Hrm. Wouldn't expect that here. Not _that_ often. :-)
or other sorts of interesting cases.
It's probably one of these cases. ;-)
I suppose the below patch would at least keep us from writing past the end on 'l', but in a proper situation we shouldn't need something like it.
The patch seems to help. Until installing xymon with the patch applied I had segfaults every minute and since then, no segfault has happened anymore for at least 10 minutes.
Thanks!
Kind regards, Axel Beckert-- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/
On Mon, October 5, 2015 2:36 am, Axel Beckert wrote:
I suppose the below patch would at least keep us from writing past the end on 'l', but in a proper situation we shouldn't need something like it.
The patch seems to help. Until installing xymon with the patch applied I had segfaults every minute and since then, no segfault has happened anymore for at least 10 minutes.
Hmm. I didn't realize it was crashing that often for you. :/
Would it be possible to run the previous copy of xymongen for a cycle or two with --debug mode enabled? In theory, that should give us a good idea of the trigger.
Regards,
-jc
participants (2)
-
beckert＠phys.ethz.ch
-
cleaver＠terabithia.org