Looking at the logs below, it appears something might be broken in the [telnet] definition. Here's the first few lines from my protocols.cfg file [ftp] send "quit\r\n" expect "220" options banner port 21 [ftps] send "quit\r\n" expect "220" options ssl,banner port 990 [ssh|ssh1|ssh2] send "SSH-2.0-OpenSSH_4.1\r\n" expect "SSH" options banner port 22 [telnet] options banner,telnet port 23
From rrd-data.log 5614 2015-03-11 08:29:07 Opening file /opt/local/xymon/server/etc/rrddefinitions.cfg 5614 2015-03-11 08:29:07 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=2101247 5614 2015-03-11 08:29:07 Got 403 bytes 5614 2015-03-11 08:29:07 xymond_rrd: Got message 849930 @@data#849930/transfer|1426033746.620045|172.21.219.192||transfer|iostatdisk|sunos|unix/UnixInf 5614 2015-03-11 08:29:07 startpos 403, fillpos 403, endpos -1 5614 2015-03-11 08:29:07 Transport setup is: 5614 2015-03-11 08:29:07 xymondportnumber = 1984 5614 2015-03-11 08:29:07 xymonproxyhost = NONE 5614 2015-03-11 08:29:07 xymonproxyport = 0 5614 2015-03-11 08:29:07 Recipient listed as '172.21.219.192' 5614 2015-03-11 08:29:07 Standard protocol on port 1984 5614 2015-03-11 08:29:07 Will connect to address 172.21.219.192 port 1984 5614 2015-03-11 08:29:07 Connect status is 0 5614 2015-03-11 08:29:07 Sent 16 bytes 5614 2015-03-11 08:29:07 Read 8192 bytes 5614 2015-03-11 08:29:07 Read 7657 bytes 5614 2015-03-11 08:29:07 Closing connection 5614 2015-03-11 08:29:07 Opening file /opt/local/xymon/server/etc/analysis.cfg 5614 2015-03-11 08:29:07 Compiling regex ctrldom[A-D] 5614 2015-03-11 08:29:07 Compiling regex infdom[A-B]1 5614 2015-03-11 08:29:07 Compiling regex .+dom.. 5614 2015-03-11 08:29:07 Compiling regex infdom[A-B]1|notesdomD1 5614 2015-03-11 08:29:07 Compiling regex [.:]80$ 5614 2015-03-11 08:29:07 Compiling regex syn.*,%oradom.* 5614 2015-03-11 08:29:07 Compiling regex syn.*|oradom.*|SINFV.* 5614 2015-03-11 08:29:07 Compiling regex [.:]514$ 5614 2015-03-11 08:29:07 Compiling regex [.:]3389$ 5614 2015-03-11 08:29:07 Compiling regex energise.* 5614 2015-03-11 08:29:07 Compiling regex [.:]8989$ 5614 2015-03-11 08:29:07 Compiling regex [.:]80$ 5614 2015-03-11 08:29:07 Compiling regex [.:]4848$ 5614 2015-03-11 08:29:07 Compiling regex [.:]8080$ 5614 2015-03-11 08:29:07 Compiling regex ^/var/crash/.* 5614 2015-03-11 08:29:07 Compiling regex ^/var/core/.* 5614 2015-03-11 08:29:07 Compiling regex [.:]22$ 5614 2015-03-11 08:29:07 Compiling regex sinfct.*|domino.* 5614 2015-03-11 08:29:07 Compiling regex [.:]80$ 5614 2015-03-11 08:29:07 Compiling regex [.:]25$ 5614 2015-03-11 08:29:07 Compiling regex sinfct.*|domino.*|SINFV.*|hermes|sirocco 5614 2015-03-11 08:29:07 Compiling regex SINFV.* 5614 2015-03-11 08:29:07 Compiling regex .* 5614 2015-03-11 08:29:07 Compiling regex ^error.* 5614 2015-03-11 08:29:07 Compiling regex ^error.* 5614 2015-03-11 08:29:07 Compiling regex .*restart/shutdown.*|.*Citrix 5614 2015-03-11 08:29:07 Compiling regex .* 5614 2015-03-11 08:29:07 Compiling regex ^warning.* 5614 2015-03-11 08:29:07 Compiling regex ^warning.* 5614 2015-03-11 08:29:07 Compiling regex .*MetaFrameEvents.*|.*restart/shutdown.*|.*Citrix-Multimedia-Flash.*|.*Printer 5614 2015-03-11 08:29:07 Compiling regex .* 5614 2015-03-11 08:29:07 Compiling regex ^failure.* 5614 2015-03-11 08:29:07 Compiling regex ^failure.* 5614 2015-03-11 08:29:07 Compiling regex ^/var/crash/.* 5614 2015-03-11 08:29:07 Compiling regex ^/var/core/.* 5614 2015-03-11 08:29:07 Compiling regex [.:]22$ 5614 2015-03-11 08:29:07 Compiling regex ^/var/crash/.* 5614 2015-03-11 08:29:07 Compiling regex ^/var/core/.* 5614 2015-03-11 08:29:07 Compiling regex [.:]22$ 5614 2015-03-11 08:29:07 Opening file /opt/local/xymon/server/etc/protocols.cfg 5614 2015-03-11 08:29:07 Service list dump 5614 2015-03-11 08:29:07 Name : ftp 5614 2015-03-11 08:29:07 Sendtext: quit\r\n 5614 2015-03-11 08:29:07 Sendlen : 6 5614 2015-03-11 08:29:07 Exp.text: 220 5614 2015-03-11 08:29:07 Exp.len : 3 5614 2015-03-11 08:29:07 Exp.ofs : 0 5614 2015-03-11 08:29:07 Flags : 1 5614 2015-03-11 08:29:07 Port : 21 5614 2015-03-11 08:29:07 Name : ftps 5614 2015-03-11 08:29:07 Sendtext: quit\r\n 5614 2015-03-11 08:29:07 Sendlen : 6 5614 2015-03-11 08:29:07 Exp.text: 220 5614 2015-03-11 08:29:07 Exp.len : 3 5614 2015-03-11 08:29:07 Exp.ofs : 0 5614 2015-03-11 08:29:07 Flags : 5 5614 2015-03-11 08:29:07 Port : 990 5614 2015-03-11 08:29:07 Name : ssh 5614 2015-03-11 08:29:07 Sendtext: SSH-2.0-OpenSSH_4.1\r\n 5614 2015-03-11 08:29:07 Sendlen : 21 5614 2015-03-11 08:29:07 Exp.text: SSH 5614 2015-03-11 08:29:07 Exp.len : 3 5614 2015-03-11 08:29:07 Exp.ofs : 0 5614 2015-03-11 08:29:07 Flags : 1 5614 2015-03-11 08:29:07 Port : 22 5614 2015-03-11 08:29:07 Name : ssh1 5614 2015-03-11 08:29:07 Sendtext: SSH-2.0-OpenSSH_4.1\r\n 5614 2015-03-11 08:29:07 Sendlen : 21 5614 2015-03-11 08:29:07 Exp.text: SSH 5614 2015-03-11 08:29:07 Exp.len : 3 5614 2015-03-11 08:29:07 Exp.ofs : 0 5614 2015-03-11 08:29:07 Flags : 1 5614 2015-03-11 08:29:07 Port : 22 5614 2015-03-11 08:29:07 Name : ssh2 5614 2015-03-11 08:29:07 Sendtext: SSH-2.0-OpenSSH_4.1\r\n 5614 2015-03-11 08:29:07 Sendlen : 21 5614 2015-03-11 08:29:07 Exp.text: SSH 5614 2015-03-11 08:29:07 Exp.len : 3 5614 2015-03-11 08:29:07 Exp.ofs : 0 5614 2015-03-11 08:29:07 Flags : 1 5614 2015-03-11 08:29:07 Port : 22 5614 2015-03-11 08:29:07 Name : telnet 5614 2015-03-11 08:29:07 2015-03-11 08:29:08 Child process 5614 died: Signal 6
From rrd-status.log 5376 2015-03-11 08:29:06 Opening file /opt/local/xymon/server/etc/rrddefinitions.cfg 5376 2015-03-11 08:29:06 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=2101247 5376 2015-03-11 08:29:06 Got 6075 bytes 5376 2015-03-11 08:29:06 xymond_rrd: Got message 621101 @@status#621101/sinfctap01|1426033745.004291|172.21.198.73||sinfctap01|svcs|1426035545|green||green|1417272435|0||0||1426033594|sunos|unix/appzones|0| 5376 2015-03-11 08:29:06 startpos 6075, fillpos 6075, endpos -1 5376 2015-03-11 08:29:06 Transport setup is: 5376 2015-03-11 08:29:06 xymondportnumber = 1984 5376 2015-03-11 08:29:06 xymonproxyhost = NONE 5376 2015-03-11 08:29:06 xymonproxyport = 0 5376 2015-03-11 08:29:06 Recipient listed as '172.21.219.192' 5376 2015-03-11 08:29:06 Standard protocol on port 1984 5376 2015-03-11 08:29:06 Will connect to address 172.21.219.192 port 1984 5376 2015-03-11 08:29:06 Connect status is 0 5376 2015-03-11 08:29:06 Sent 16 bytes 5376 2015-03-11 08:29:06 Read 8192 bytes 5376 2015-03-11 08:29:06 Read 7657 bytes 5376 2015-03-11 08:29:06 Closing connection 5376 2015-03-11 08:29:06 Opening file /opt/local/xymon/server/etc/analysis.cfg 5376 2015-03-11 08:29:06 Compiling regex ctrldom[A-D] 5376 2015-03-11 08:29:06 Compiling regex infdom[A-B]1 5376 2015-03-11 08:29:06 Compiling regex .+dom.. 5376 2015-03-11 08:29:06 Compiling regex infdom[A-B]1|notesdomD1 5376 2015-03-11 08:29:06 Compiling regex [.:]80$ 5376 2015-03-11 08:29:06 Compiling regex syn.*,%oradom.* 5376 2015-03-11 08:29:06 Compiling regex syn.*|oradom.*|SINFV.* 5376 2015-03-11 08:29:06 Compiling regex [.:]514$ 5376 2015-03-11 08:29:06 Compiling regex [.:]3389$ 5376 2015-03-11 08:29:06 Compiling regex energise.* 5376 2015-03-11 08:29:06 Compiling regex [.:]8989$ 5376 2015-03-11 08:29:06 Compiling regex [.:]80$ 5376 2015-03-11 08:29:06 Compiling regex [.:]4848$ 5376 2015-03-11 08:29:06 Compiling regex [.:]8080$ 5376 2015-03-11 08:29:06 Compiling regex ^/var/crash/.* 5376 2015-03-11 08:29:06 Compiling regex ^/var/core/.* 5376 2015-03-11 08:29:06 Compiling regex [.:]22$ 5376 2015-03-11 08:29:06 Compiling regex sinfct.*|domino.* 5376 2015-03-11 08:29:06 Compiling regex [.:]80$ 5376 2015-03-11 08:29:06 Compiling regex [.:]25$ 5376 2015-03-11 08:29:06 Compiling regex sinfct.*|domino.*|SINFV.*|hermes|sirocco 5376 2015-03-11 08:29:06 Compiling regex SINFV.* 5376 2015-03-11 08:29:06 Compiling regex .* 5376 2015-03-11 08:29:06 Compiling regex ^error.* 5376 2015-03-11 08:29:06 Compiling regex ^error.* 5376 2015-03-11 08:29:06 Compiling regex .*restart/shutdown.*|.*Citrix 5376 2015-03-11 08:29:06 Compiling regex .* 5376 2015-03-11 08:29:06 Compiling regex ^warning.* 5376 2015-03-11 08:29:06 Compiling regex ^warning.* 5376 2015-03-11 08:29:06 Compiling regex .*MetaFrameEvents.*|.*restart/shutdown.*|.*Citrix-Multimedia-Flash.*|.*Printer 5376 2015-03-11 08:29:06 Compiling regex .* 5376 2015-03-11 08:29:06 Compiling regex ^failure.* 5376 2015-03-11 08:29:06 Compiling regex ^failure.* 5376 2015-03-11 08:29:06 Compiling regex ^/var/crash/.* 5376 2015-03-11 08:29:06 Compiling regex ^/var/core/.* 5376 2015-03-11 08:29:06 Compiling regex [.:]22$ 5376 2015-03-11 08:29:06 Compiling regex ^/var/crash/.* 5376 2015-03-11 08:29:06 Compiling regex ^/var/core/.* 5376 2015-03-11 08:29:06 Compiling regex [.:]22$ 5376 2015-03-11 08:29:06 Opening file /opt/local/xymon/server/etc/protocols.cfg 5376 2015-03-11 08:29:06 Service list dump 5376 2015-03-11 08:29:06 Name : ftp 5376 2015-03-11 08:29:06 Sendtext: quit\r\n 5376 2015-03-11 08:29:06 Sendlen : 6 5376 2015-03-11 08:29:06 Exp.text: 220 5376 2015-03-11 08:29:06 Exp.len : 3 5376 2015-03-11 08:29:06 Exp.ofs : 0 5376 2015-03-11 08:29:06 Flags : 1 5376 2015-03-11 08:29:06 Port : 21 5376 2015-03-11 08:29:06 Name : ftps 5376 2015-03-11 08:29:06 Sendtext: quit\r\n 5376 2015-03-11 08:29:06 Sendlen : 6 5376 2015-03-11 08:29:06 Exp.text: 220 5376 2015-03-11 08:29:06 Exp.len : 3 5376 2015-03-11 08:29:06 Exp.ofs : 0 5376 2015-03-11 08:29:06 Flags : 5 5376 2015-03-11 08:29:06 Port : 990 5376 2015-03-11 08:29:06 Name : ssh 5376 2015-03-11 08:29:06 Sendtext: SSH-2.0-OpenSSH_4.1\r\n 5376 2015-03-11 08:29:06 Sendlen : 21 5376 2015-03-11 08:29:06 Exp.text: SSH 5376 2015-03-11 08:29:06 Exp.len : 3 5376 2015-03-11 08:29:06 Exp.ofs : 0 5376 2015-03-11 08:29:06 Flags : 1 5376 2015-03-11 08:29:06 Port : 22 5376 2015-03-11 08:29:06 Name : ssh1 5376 2015-03-11 08:29:06 Sendtext: SSH-2.0-OpenSSH_4.1\r\n 5376 2015-03-11 08:29:06 Sendlen : 21 5376 2015-03-11 08:29:06 Exp.text: SSH 5376 2015-03-11 08:29:06 Exp.len : 3 5376 2015-03-11 08:29:06 Exp.ofs : 0 5376 2015-03-11 08:29:06 Flags : 1 5376 2015-03-11 08:29:06 Port : 22 5376 2015-03-11 08:29:06 Name : ssh2 5376 2015-03-11 08:29:06 Sendtext: SSH-2.0-OpenSSH_4.1\r\n 5376 2015-03-11 08:29:06 Sendlen : 21 5376 2015-03-11 08:29:06 Exp.text: SSH 5376 2015-03-11 08:29:06 Exp.len : 3 5376 2015-03-11 08:29:06 Exp.ofs : 0 5376 2015-03-11 08:29:06 Flags : 1 5376 2015-03-11 08:29:06 Port : 22 5376 2015-03-11 08:29:06 Name : telnet 5376 2015-03-11 08:29:06 2015-03-11 08:29:07 Peer at 0.0.0.0:0 failed: Broken pipe 2015-03-11 08:29:07 Child process 5376 died: Signal 6
On 11 March 2015 at 09:52, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
On 11 March 2015 at 11:37, Vernon Everett <everett.vernon at gmail.com> wrote:
And even with --no-cache, I am still getting these corrupted rrd files.
:-(
I tried again with --debug (and --no-cache) and it core dumps.
Here's the backtrace.
libc.so.1`vfprintf+0xec(6c3d0, 514c0, ffbfb3e8, 0, a0ba4, 33e1c) dbgprintf+0xa4(514c0, 0, 51400, 6c3f0, bf, 2ab388) dump_tcp_services+0x74(a0, 1c00, fef37940, 0, 51400, 51400)
So dump_tcp_services() calls dbgprintf() (both on lib/netservices.c) which in turn calls vprintf() from libc, but with bad parameters. I've had a look through the code in dump_tcp_services() and I don't know enough C to recognize any problems. But it might be useful to know which call to dbgprintf() is causing the problem.
Does the log file for xymond_rrd show any debug output at all? If so, what's the last line that is shown.
It might be helpful if you can recompile xymond_rrd with dump_tcp_services() modified. Initially, I would simply try it with "return" added after the first call to dbgprintf(). That is, dump_tcp_services() will output "Service list dump" and return. This might stop the core dumps so that we can get debug output for other parts of the xymond_rrd processing.
If adding "return" at that point fixes this core dump, more diagnostic lines would be useful to determine what the problem is. For example, there's a global array called svcinfo that is iterated over, but if the array is empty, it might cause the core dump. So adding a line that checks whether the array is empty and displays the result would help to pin this down.
Note that "svcinfo" appears to be populated from the protocols.cfg file and/or XYMONNETSVCS. Is it possible that your protocols.cfg file is empty, or has some syntax error that causes it to be unparseable? The same for XYMONNETSVCS (in xymonserver.cfg)?
J
-- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton