Hi.
We have a large xymon setup that averages 25 messages/sec for xymond. I am in the process of migrating to a new xymon server and was planning on using a front end xymonproxy process on a third box to keep traffic going to both old and new boxes while I migrated across the old data (primarily the rrd data as that requires conversion from 32bit to 64bit). The xymonproxy is working fine under light load (20 or so client machines) but when I modify the firewall port forward to point the incoming port 1984 traffic to it it crashes after a few seconds.
2016-05-31 16:08:19.793720 xymonproxy version 4.3.27-1.el7.terabithia starting 2016-05-31 16:08:19.793791 Listening on 0.0.0.0:1984 2016-05-31 16:08:19.793796 Sending to 2 Xymon server(s): 172.16.100.103:1984 172.16.100.101:1984 2016-05-31 16:09:23.010861 conn 0: server 1 (172.16.100.101:1984) not responding, message lost 2016-05-31 16:09:23.011363 conn 0, socket -1: write shutdown failed: Bad file descriptor 2016-05-31 16:09:24.012798 conn 0: server 1 (172.16.100.101:1984) not responding, message lost 2016-05-31 16:09:25.014274 conn 0, socket -1: write shutdown failed: Bad file descriptor *** buffer overflow detected ***: xymonproxy terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f234c396597] /lib64/libc.so.6(+0x10c750)[0x7f234c394750] /lib64/libc.so.6(+0x10e507)[0x7f234c396507] xymonproxy[0x403e11] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f234c2a9b15] xymonproxy[0x406579] ======= Memory map: ======== 00400000-0041d000 r-xp 00000000 fd:00 25459501 /usr/libexec/xymon/xymonproxy 0061d000-0061e000 r--p 0001d000 fd:00 25459501 /usr/libexec/xymon/xymonproxy 0061e000-0061f000 rw-p 0001e000 fd:00 25459501 /usr/libexec/xymon/xymonproxy 0061f000-00646000 rw-p 00000000 00:00 0 02472000-025f4000 rw-p 00000000 00:00 0 [heap] 7f234a874000-7f234a889000 r-xp 00000000 fd:00 18879360 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f234a889000-7f234aa88000 ---p 00015000 fd:00 18879360 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f234aa88000-7f234aa89000 r--p 00014000 fd:00 18879360 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f234aa89000-7f234aa8a000 rw-p 00015000 fd:00 18879360 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f234aa8a000-7f234aaae000 r-xp 00000000 fd:00 16856000 /usr/lib64/liblzma.so.5.0.99 7f234aaae000-7f234acad000 ---p 00024000 fd:00 16856000 /usr/lib64/liblzma.so.5.0.99 7f234acad000-7f234acae000 r--p 00023000 fd:00 16856000 /usr/lib64/liblzma.so.5.0.99 7f234acae000-7f234acaf000 rw-p 00024000 fd:00 16856000 /usr/lib64/liblzma.so.5.0.99 7f234acaf000-7f234acd0000 r-xp 00000000 fd:00 16915048 /usr/lib64/libselinux.so.1 7f234acd0000-7f234aed0000 ---p 00021000 fd:00 16915048 /usr/lib64/libselinux.so.1 7f234aed0000-7f234aed1000 r--p 00021000 fd:00 16915048 /usr/lib64/libselinux.so.1 7f234aed1000-7f234aed2000 rw-p 00022000 fd:00 16915048 /usr/lib64/libselinux.so.1 7f234aed2000-7f234aed4000 rw-p 00000000 00:00 0 7f234aed4000-7f234aeea000 r-xp 00000000 fd:00 16855963 /usr/lib64/libresolv-2.17.so 7f234aeea000-7f234b0ea000 ---p 00016000 fd:00 16855963 /usr/lib64/libresolv-2.17.so 7f234b0ea000-7f234b0eb000 r--p 00016000 fd:00 16855963 /usr/lib64/libresolv-2.17.so 7f234b0eb000-7f234b0ec000 rw-p 00017000 fd:00 16855963 /usr/lib64/libresolv-2.17.so 7f234b0ec000-7f234b0ee000 rw-p 00000000 00:00 0 7f234b0ee000-7f234b0f1000 r-xp 00000000 fd:00 17026913 /usr/lib64/libkeyutils.so.1.5 7f234b0f1000-7f234b2f0000 ---p 00003000 fd:00 17026913 /usr/lib64/libkeyutils.so.1.5 7f234b2f0000-7f234b2f1000 r--p 00002000 fd:00 17026913 /usr/lib64/libkeyutils.so.1.5 7f234b2f1000-7f234b2f2000 rw-p 00003000 fd:00 17026913 /usr/lib64/libkeyutils.so.1.5 7f234b2f2000-7f234b2ff000 r-xp 00000000 fd:00 18086631 /usr/lib64/libkrb5support.so.0.1 7f234b2ff000-7f234b4ff000 ---p 0000d000 fd:00 18086631 /usr/lib64/libkrb5support.so.0.1 7f234b4ff000-7f234b500000 r--p 0000d000 fd:00 18086631 /usr/lib64/libkrb5support.so.0.1 7f234b500000-7f234b501000 rw-p 0000e000 fd:00 18086631 /usr/lib64/libkrb5support.so.0.1 7f234b501000-7f234b517000 r-xp 00000000 fd:00 16855957 /usr/lib64/libpthread-2.17.so 7f234b517000-7f234b717000 ---p 00016000 fd:00 16855957 /usr/lib64/libpthread-2.17.so 7f234b717000-7f234b718000 r--p 00016000 fd:00 16855957 /usr/lib64/libpthread-2.17.so 7f234b718000-7f234b719000 rw-p 00017000 fd:00 16855957 /usr/lib64/libpthread-2.17.so 7f234b719000-7f234b71d000 rw-p 00000000 00:00 0 7f234b71d000-7f234b720000 r-xp 00000000 fd:00 17867018 /usr/lib64/libdl-2.17.so 7f234b720000-7f234b91f000 ---p 00003000 fd:00 17867018 /usr/lib64/libdl-2.17.so 7f234b91f000-7f234b920000 r--p 00002000 fd:00 17867018 /usr/lib64/libdl-2.17.so 7f234b920000-7f234b921000 rw-p 00003000 fd:00 17867018 /usr/lib64/libdl-2.17.so 7f234b921000-7f234b950000 r-xp 00000000 fd:00 17149883 /usr/lib64/libk5crypto.so.3.1 7f234b950000-7f234bb4f000 ---p 0002f000 fd:00 17149883 /usr/lib64/libk5crypto.so.3.1 7f234bb4f000-7f234bb51000 r--p 0002e000 fd:00 17149883 /usr/lib64/libk5crypto.so.3.1 7f234bb51000-7f234bb52000 rw-p 00030000 fd:00 17149883 /usr/lib64/libk5crypto.so.3.1 7f234bb52000-7f234bb53000 rw-p 00000000 00:00 0 7f234bb53000-7f234bb56000 r-xp 00000000 fd:00 16915067 /usr/lib64/libcom_err.so.2.1 7f234bb56000-7f234bd55000 ---p 00003000 fd:00 16915067 /usr/lib64/libcom_err.so.2.1 7f234bd55000-7f234bd56000 r--p 00002000 fd:00 16915067 /usr/lib64/libcom_err.so.2.1 7f234bd56000-7f234bd57000 rw-p 00003000 fd:00 16915067 /usr/lib64/libcom_err.so.2.1 7f234bd57000-7f234be2c000 r-xp 00000000 fd:00 17166500 /usr/lib64/libkrb5.so.3.3 7f234be2c000-7f234c02c000 ---p 000d5000 fd:00 17166500 /usr/lib64/libkrb5.so.3.3 7f234c02c000-7f234c039000 r--p 000d5000 fd:00 17166500 /usr/lib64/libkrb5.so.3.3 7f234c039000-7f234c03c000 rw-p 000e2000 fd:00 17166500 /usr/lib64/libkrb5.so.3.3 7f234c03c000-7f234c085000 r-xp 00000000 fd:00 17177116 /usr/lib64/libgssapi_krb5.so.2.2 7f234c085000-7f234c285000 ---p 00049000 fd:00 17177116 /usr/lib64/libgssapi_krb5.so.2.2 7f234c285000-7f234c286000 r--p 00049000 fd:00 17177116 /usr/lib64/libgssapi_krb5.so.2.2 7f234c286000-7f234c288000 rw-p 0004a000 fd:00 17177116 /usr/lib64/libgssapi_krb5.so.2.2 7f234c288000-7f234c43f000 r-xp 00000000 fd:00 16855941 /usr/lib64/libc-2.17.so 7f234c43f000-7f234c63f000 ---p 001b7000 fd:00 16855941 /usr/lib64/libc-2.17.so 7f234c63f000-7f234c643000 r--p 001b7000 fd:00 16855941 /usr/lib64/libc-2.17.so 7f234c643000-7f234c645000 rw-p 001bb000 fd:00 16855941 /usr/lib64/libc-2.17.so 7f234c645000-7f234c64a000 rw-p 00000000 00:00 0 7f234c64a000-7f234c6aa000 r-xp 00000000 fd:00 18095822 /usr/lib64/libpcre.so.1.2.0 7f234c6aa000-7f234c8a9000 ---p 00060000 fd:00 18095822 /usr/lib64/libpcre.so.1.2.0 7f234c8a9000-7f234c8aa000 r--p 0005f000 fd:00 18095822 /usr/lib64/libpcre.so.1.2.0 7f234c8aa000-7f234c8ab000 rw-p 00060000 fd:00 18095822 /usr/lib64/libpcre.so.1.2.0 7f234c8ab000-7f234ca69000 r-xp 00000000 fd:00 17026905 /usr/lib64/libcrypto.so.1.0.1e 7f234ca69000-7f234cc69000 ---p 001be000 fd:00 17026905 /usr/lib64/libcrypto.so.1.0.1e 7f234cc69000-7f234cc83000 r--p 001be000 fd:00 17026905 /usr/lib64/libcrypto.so.1.0.1e 7f234cc83000-7f234cc8f000 rw-p 001d8000 fd:00 17026905 /usr/lib64/libcrypto.so.1.0.1e 7f234cc8f000-7f234cc93000 rw-p 00000000 00:00 0 7f234cc93000-7f234ccf6000 r-xp 00000000 fd:00 18093381 /usr/lib64/libssl.so.1.0.1e 7f234ccf6000-7f234cef5000 ---p 00063000 fd:00 18093381 /usr/lib64/libssl.so.1.0.1e 7f234cef5000-7f234cef9000 r--p 00062000 fd:00 18093381 /usr/lib64/libssl.so.1.0.1e 7f234cef9000-7f234cf00000 rw-p 00066000 fd:00 18093381 /usr/lib64/libssl.so.1.0.1e 7f234cf00000-7f234cf0b000 r-xp 00000000 fd:00 17867002 /usr/lib64/liblz4.so.1.7.1 7f234cf0b000-7f234d10a000 ---p 0000b000 fd:00 17867002 /usr/lib64/liblz4.so.1.7.1 7f234d10a000-7f234d10b000 r--p 0000a000 fd:00 17867002 /usr/lib64/liblz4.so.1.7.1 7f234d10b000-7f234d10c000 rw-p 00000000 00:00 0 7f234d10c000-7f234d121000 r-xp 00000000 fd:00 16915045 /usr/lib64/libz.so.1.2.7 7f234d121000-7f234d320000 ---p 00015000 fd:00 16915045 /usr/lib64/libz.so.1.2.7 7f234d320000-7f234d321000 r--p 00014000 fd:00 16915045 /usr/lib64/libz.so.1.2.7 7f234d321000-7f234d322000 rw-p 00015000 fd:00 16915045 /usr/lib64/libz.so.1.2.7 7f234d322000-7f234d343000 r-xp 00000000 fd:00 18043068 /usr/lib64/ld-2.17.so 7f234d52e000-7f234d538000 rw-p 00000000 00:00 0 7f234d540000-7f234d543000 rw-p 00000000 00:00 0 7f234d543000-7f234d544000 r--p 00021000 fd:00 18043068 /usr/lib64/ld-2.17.so 7f234d544000-7f234d545000 rw-p 00022000 fd:00 18043068 /usr/lib64/ld-2.17.so 7f234d545000-7f234d546000 rw-p 00000000 00:00 0 7ffdfcb9d000-7ffdfcbbf000 rw-p 00000000 00:00 0 [stack] 7ffdfcbdb000-7ffdfcbdd000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] 2016-05-31 16:09:25.092174 xymonproxy version 4.3.27-1.el7.terabithia starting 2016-05-31 16:09:25.092364 Listening on 0.0.0.0:1984 2016-05-31 16:09:25.092379 Sending to 2 Xymon server(s): 172.16.100.103:1984 172.16.100.101:1984
Xymonproxy is running on CentOS 7 with all the latest patches installed. 172.16.100.103 is xymon 4.3.16 on CentOS 5, 172.16.100.101 is xymon 4.3.27-1.el7.terabithia on CentOS 7.
Any ideas on how to troubleshoot this further?
Thanks, JT