newer
MSGS Alert question

Too much data for svcstatus.sh?

older
"noconn" configuration feature...

mburger＠bubbanfriends.org

21 May 2020 21 May '20

3:47 p.m.

Good morning.

Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server.

This morning, I encountered an interesting issue.

I have a monitored system that has 197 discrete filesystems/mounts.

When I attempt to enter this server's "Trends" section, I receive an internal server error (500).

My httpd error log shows this when I attempt to access the trends link for the server in question:

[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906
/apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906
/apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html

This does not happen for any other monitored system in my environment.

This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.

-- Mike Burger http://www.bubbanfriends.org

"It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1

Show replies by date

mark＠stitson.com

22 May 22 May

11:16 a.m.

I have seen similar behaviour last week on one of our Solaris servers with many mounts where trends fails.

Interestingly we run 2 Xymon servers for redundancy and it works flawlessly on the secondary with exactly the same data.

When I run scvstatus.sh manually with strace a working machine all the RRDs and then continues, the broken monitored machine throws a segfault: stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 0 getdents(3, /* 0 entries */, 32768) = 0 close(3) = 0 brk(0x282d000) = 0x282d000 brk(0x2828000) = 0x2828000 brk(0x2849000) = 0x2849000 open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0 read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276 close(3)
...

Whereas the broken one segfaults: stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 0 getdents(3, /* 0 entries */, 32768) = 0 close(3) = 0 brk(0x266b000) = 0x266b000 brk(0x2666000) = 0x2666000 mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e2f29000 brk(0x2661000) = 0x2661000 brk(0x265f000) = 0x265f000 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8} --- +++ killed by SIGSEGV +++

It is on my list to investigate, but I haven't had time yet.

Mark

-----Original Message----- From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger Sent: 21 May 2020 16:47 To: xymon at xymon.com Subject: [Xymon] Too much data for svcstatus.sh?

Good morning.

Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server.

This morning, I encountered an interesting issue.

I have a monitored system that has 197 discrete filesystems/mounts.

When I attempt to enter this server's "Trends" section, I receive an internal server error (500).

My httpd error log shows this when I attempt to access the trends link for the server in question:

This does not happen for any other monitored system in my environment.

This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.

-- Mike Burger http://www.bubbanfriends.org

"It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1

Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

mburger＠bubbanfriends.org

28 May 28 May

2:58 p.m.

On 2020-05-22 07:16, Mark O. Stitson wrote:

...

I have seen similar behaviour last week on one of our Solaris servers with many mounts where trends fails.

Interestingly we run 2 Xymon servers for redundancy and it works flawlessly on the secondary with exactly the same data.

When I run scvstatus.sh manually with strace a working machine all the RRDs and then continues, the broken monitored machine throws a segfault: stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 0 getdents(3, /* 0 entries */, 32768) = 0 close(3) = 0 brk(0x282d000) = 0x282d000 brk(0x2828000) = 0x2828000 brk(0x2849000) = 0x2849000 open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0 read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276 close(3) ...

Whereas the broken one segfaults: stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 0 getdents(3, /* 0 entries */, 32768) = 0 close(3) = 0 brk(0x266b000) = 0x266b000 brk(0x2666000) = 0x2666000 mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e2f29000 brk(0x2661000) = 0x2661000 brk(0x265f000) = 0x265f000 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8}

+++ killed by SIGSEGV +++

It is on my list to investigate, but I haven't had time yet.

Mark

-----Original Message----- From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger Sent: 21 May 2020 16:47 To: xymon at xymon.com Subject: [Xymon] Too much data for svcstatus.sh?

Good morning.

Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server.

This morning, I encountered an interesting issue.

I have a monitored system that has 197 discrete filesystems/mounts.

When I attempt to enter this server's "Trends" section, I receive an internal server error (500).

My httpd error log shows this when I attempt to access the trends link for the server in question:

[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html

This does not happen for any other monitored system in my environment.

This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.

Henrik, et al:

Any thoughts?

Mike Burger http://www.bubbanfriends.org

"It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1

marco.avvisano＠regione.toscana.it

12 Oct 12 Oct

8:43 a.m.

New subject: [ANTISPAMRT:]Re: Too much data for svcstatus.sh?

Hi,

i have the same problem on Red Hat Enterprise Linux Server release 6.3,

running xymon 4.3.30 compiled from source

Best Regards

Marco

Il 28/05/2020 16:58, Mike Burger ha scritto:

...

On 2020-05-22 07:16, Mark O. Stitson wrote:

...
I have seen similar behaviour last week on one of our Solaris servers with many mounts where trends fails.

Interestingly we run 2 Xymon servers for redundancy and it works flawlessly on the secondary with exactly the same data.

When I run scvstatus.sh manually with strace a working machine all the RRDs and then continues, the broken monitored machine throws a segfault: stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 0 getdents(3, /* 0 entries */, 32768)???? = 0 close(3)??????????????????????????????? = 0 brk(0x282d000)????????????????????????? = 0x282d000 brk(0x2828000)????????????????????????? = 0x2828000 brk(0x2849000)????????????????????????? = 0x2849000 open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0 read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276 close(3) ...

Whereas the broken one segfaults: stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 0 getdents(3, /* 0 entries */, 32768)???? = 0 close(3)??????????????????????????????? = 0 brk(0x266b000)????????????????????????? = 0x266b000 brk(0x2666000)????????????????????????? = 0x2666000 mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e2f29000 brk(0x2661000)????????????????????????? = 0x2661000 brk(0x265f000)????????????????????????? = 0x265f000 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8} --- +++ killed by SIGSEGV +++

It is on my list to investigate, but I haven't had time yet.

Mark

-----Original Message----- From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger Sent: 21 May 2020 16:47 To: xymon at xymon.com Subject: [Xymon] Too much data for svcstatus.sh?

Good morning.

Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server.

This morning, I encountered an interesting issue.

I have a monitored system that has 197 discrete filesystems/mounts.

When I attempt to enter this server's "Trends" section, I receive an internal server error (500).

My httpd error log shows this when I attempt to access the trends link for the server in question:

[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html

This does not happen for any other monitored system in my environment.

This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.

Henrik, et al:

Any thoughts?

henrik＠hswn.dk

2:01 p.m.

Hi,

the best way to debug this is to get a proper stack trace of when the program crashes. For this, you first need to make sure you have a binary of svcstatus.cgi which includes debug information - if you look at the Xymon sources, the build/Makefile.Linux file has a commented-out line setting the CFLAGS variable for debugging:

Compile flags for debugging

CFLAGS = -g -DDEBUG -Wall -D_REENTRANT $(LFSDEF) $(OSDEF)

Uncomment the last line, run "make clean; make" and then use the web/svcstatus.cgi binary from that.

To run the program in the gdb debugger, I would bypass the web server CGI and run it by hand like this:

$ export SCRIPT_NAME="svcstatus.sh" $ export REQUEST_METHOD="GET" $ export QUERY_STRING="HOST=claudio.hswn.dk&SERVICE=disk" $ gdb ./web/svcstatus.cgi (gdb) run

The value for the QUERY_STRING should be taken from the URL for your status-page - the part after the '?' sign. The example here is from the xymon.com site, but you really just need to modify the hostname to that of your own.

Note that you must run it on the Xymon server, because it reads the file stored there.

Regards, Henrik

Den 21.05.2020 kl. 17.47 skrev Mike Burger:

...

Good morning.

Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server.

This morning, I encountered an interesting issue.

I have a monitored system that has 197 discrete filesystems/mounts.

When I attempt to enter this server's "Trends" section, I receive an internal server error (500).

My httpd error log shows this when I attempt to access the trends link for the server in question:

[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906????????????????????????????? /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906????????????????????????????? /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html

This does not happen for any other monitored system in my environment.

This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.

2081

Age (days ago)

2225

Last active (days ago)

List overview

Download

4 comments

4 participants

participants (4)

henrik＠hswn.dk
marco.avvisano＠regione.toscana.it
mark＠stitson.com
mburger＠bubbanfriends.org

Too much data for svcstatus.sh?

mburger＠bubbanfriends.org

mark＠stitson.com

mburger＠bubbanfriends.org

Any thoughts?

marco.avvisano＠regione.toscana.it

henrik＠hswn.dk

Compile flags for debugging

CFLAGS = -g -DDEBUG -Wall -D_REENTRANT $(LFSDEF) $(OSDEF)

tags

participants (4)