On 2020-05-22 07:16, Mark O. Stitson wrote:
I have seen similar behaviour last week on one of our Solaris servers with many mounts where trends fails.
Interestingly we run 2 Xymon servers for redundancy and it works flawlessly on the secondary with exactly the same data.
When I run scvstatus.sh manually with strace a working machine all the RRDs and then continues, the broken monitored machine throws a segfault: stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 0 getdents(3, /* 0 entries */, 32768) = 0 close(3) = 0 brk(0x282d000) = 0x282d000 brk(0x2828000) = 0x2828000 brk(0x2849000) = 0x2849000 open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0 read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276 close(3) ...
Whereas the broken one segfaults: stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 0 getdents(3, /* 0 entries */, 32768) = 0 close(3) = 0 brk(0x266b000) = 0x266b000 brk(0x2666000) = 0x2666000 mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e2f29000 brk(0x2661000) = 0x2661000 brk(0x265f000) = 0x265f000 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8}
+++ killed by SIGSEGV +++
It is on my list to investigate, but I haven't had time yet.
Mark
-----Original Message----- From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger Sent: 21 May 2020 16:47 To: xymon at xymon.com Subject: [Xymon] Too much data for svcstatus.sh?
Good morning.
Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server.
This morning, I encountered an interesting issue.
I have a monitored system that has 197 discrete filesystems/mounts.
When I attempt to enter this server's "Trends" section, I receive an internal server error (500).
My httpd error log shows this when I attempt to access the trends link for the server in question:
[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
This does not happen for any other monitored system in my environment.
This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.
Henrik, et al:
Any thoughts?
Mike Burger http://www.bubbanfriends.org
"It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1