November 2023 - Xymon - Xymon mailman web

Possible bug in xymon svcstatus causing segfault (buffer overrun?)
by jeremy＠laidman.org 03 Jun '24

03 Jun '24

Hi all I'm not a C programmer by any stretch, so I'm seeking confirmation that my working is sound. At our site we run 4.3.30-1 from Terabithia RPM. For most hosts, the "trends" page displays just fine, but for some hosts, the webserver gives a CGI error, and when I manually run the CGI directly it crashes with a seg fault. This seems to be more likely for hosts with a lot of entries on the trends page, but some with more trends graphs are OK than others with fewer. I believe I've tracked the issue down to the last few lines of code in svcstatus-trends.c, where a URL is constructed for one graph entry, before being added to the list of trends in a buffer. My reading of the code suggests that it first pre-allocates (using malloc()) a 16k buffer for the trends data, then sequentially appends trend strings (which include a URL) into the buffer, until the length of a string would exceed the size of the buffer. In that case, an extra 4k of memory is added to the buffer (using realloc()), and then the string is added to the buffer. The problem with the code, as I see it, is that the string might be more than 4k long, in which case, adding string would cause a buffer overrun. I don't really know what's in each string, so I don't know if it's ever likely to be 4k, but we're seeing the process dumping core immediately after this bit of code, so that's the suspicion. Here's the bit of code, which is attempting to add the string "onelink" to the buffer, after seeing if it's going to exceed the buffer, and extending the buffer if required: if ((buflen + strlen(onelink)) >= allrrdlinks_buflen) { SBUF_REALLOC(allrrdlinks, allrrdlinks_buflen+4096); allrrdlinksend = allrrdlinks + buflen; } allrrdlinksend += snprintf(allrrdlinksend, (allrrdlinks_buflen - (allrrdlinksend - allrrdlinks)), "%s", onelink); Elsewhere in other code, the SBUF_REALLOC macro is used to add extra bufferspace by calculating a new buffer size of [existing buffer size plus 4k plus the length of the string being added]. So if the string to be added had 20 bytes, SBUF_REALLOC would be used to add 4k+20 bytes to the buffer size. Notably, the size of the string is included in the calculation. For example this bit of code elsewhere in the same file does a similar buffer extension: if ((strlen(rrdlink) + strlen(partlink) + 1) >= rrdlink_buflen) { SBUF_REALLOC(rrdlink, rrdlink_buflen + strlen(partlink) + 4096); } strncat(rrdlink, partlink, (rrdlink_buflen - strlen(rrdlink))); Note the extra "+ strlen(<stringvar>)" bit in the extra memory being allocated that isn't there in the first block of code above. So my thinking is that the SBUF_REALLOC() needs to take the string's size into account, like so: SBUF_REALLOC(allrrdlinks, allrrdlinks_buflen+strlen(onelink)+4096); Is anyone with C skills able to confirm that this does indeed look like a bug, and also that my proposed change is a suitable fix? After applying this change and recompiling, we haven't yet seen a seg fault from this bit of code, so I'm hoping it's all fixed, and without negative side effects. Cheers Jeremy

3 6

Update on 4.4 (Alpha1 release)
by cleaver＠terabithia.org 15 May '24

15 May '24

Hello all! Just wanted to update everyone on a few things. Going through the 4.x codebase re-familiarizing myself again with the state of everything, I recalled that I'd actually built a release artifact before development was paused, including a testing RPM that has actually been running unbeknownst on a VM for quite a while. Unfortunately, while validating that compiles from source worked I ran into what looked like showstopper bugs. These weren't showing up in the package, so I had to pore through a lot more code* than I expected to try to figure out where the package was accidentally fixing it. (*message-sending code, which changes a lot in 4.4 to support compression, TLS, HTTP, and size-prefixed messages) Fortunately, it ended up being a relatively simple issue (isn't it always?), with easy-enough validation and workaround (given below). Although more fixes could be put in (and I'm sure will be needed!), I believe there's some benefit to having a stable starting point using a build artifact from 2019. As a result, I'm pleased to announce the first alpha for Xymon 4.4! As this is an alpha release, this is absolutely NOT suitable for production use and is intended mainly for smoketesting by source users. (And I expect there will be smoke.) The release tarball is available now at: https://sourceforge.net/projects/xymon/files/Xymon/4.4-alpha/xymon-4.4-alph… Major improvements are too many to list here but are noted in the Changes and RELEASENOTES files. README documents for compression, TLS, and IPv6 are in progress. Some of the changes and improvements other than TLS support have been in the Terabithia packages (to some extent) for a while, so there will be less of a delta for those users to this alpha. For the moment, I'd like to focus on source tarball users and others running xymon from scratch to help fix underlying issues and any smoketest failures, which I expect may crop up particularly on untested non-Linux platforms. Once the lower-hanging fruit is fixed and more patches are upstreamed and cleaned up, I'll release an alpha2 release at which point packages may be released to help with testing iteration and downstream distributors. Please flag bug reports or patches as related to "alpha1" in the subject line to help keep things clear. Thank you to everyone in the community for remaining part of it -- and xymon users! -- during this interregnum. Restarting development has been a process, but I'm glad to take this important first step to our next release. Regards, -jc *** KNOWN ISSUES -- IMPORTANT *** The configure script is likely setting XYMONHOME/XYMONCLIENTHOME and XYMSRV/XYMSERVERS/XYMONSERVERS incorrectly, which will prevent xymonlaunch from correctly launching anything. Double check the xymon environment files (xymonserver/xymonclient.cfg) before starting either the server or client. The tools/xymon-client.default and tools/xymonlaunch files may be placed into /etc/{sysconfig,default}/{xymon-client,xymonlaunch} to more easily set $XYMONSERVERS Revealed by the above issue is a bug causing a crash in the client (or anywhere) when given an empty string as the destination argument (instead of 0, 127.0.0.1 or no variable set). Don't do this --> [ xymon '' "ping" ] The client and logfetch binary may be being re-compiled during 'make install' Compilation may fail if c-ares and at least one compression library is not present (lzo/lz4/zlib are all options). xxHash libraries are also highly recommended as the plan is for xxhash to replace md5 for checksumming functionality. libtirpc may be necessary on some platforms, and it's possible the build code is still expecting traditional (Sun) RPC

9 13

Fwd: Re. Is this thing on?
by bruno.manzoni＠ubi-network.ch 29 Jan '24

29 Jan '24

Hi. Seem that Ye-Fee is unable to email to the mailing list? *Can you check that if there is a problem ( I think Henrik) ?* By to way, I remember that we have to migrate to another service. *What about: https://mailman3.com/ which is free for opensource ?* (comment to do or not to do are welcome!) *Can someone take the job ? * Thing to do: - Ask malman3 if possible to have a freemailing list - The we can see how to do it! (That will need for sure to contact Henrik), but I can help also. Bruno -------- Forwarded Message -------- Subject: Re. [Xymon] Is this thing on? Date: Thu, 28 Sep 2023 16:22:00 +0000 From: Liang, Ye-Fee [PHSA] <yefee.liang at phsa.ca> To: bruno.manzoni at ubi-network.ch <bruno.manzoni at ubi-network.ch> Hi Bruno, ?? I was able to email xymon at xymon.com long time ago, and now find myself unable to email anymore.? I have used xymon at different organizations over the decades, and would love to help the tool/platform to continue for decades more.? Simply there is no other product that is more versatile, easy to manage, and light server+client requirement.? Many more people are eager to contribute, but can only just read the xymon list email, from the side line. ?? Thanks for your summary of Xymon non-technical and technical needs. Just to keep the enthusiasm going, I would suggest a section on "making the better use of Xymon as now" I am only a sys admin, not a coder, so my focus is on management and usability areas. 1. enhanced cpu utilization graph (adds wait IO; better colors); see 1a. 2. menu script (see 2a) to clean up of accumulating rrd, hostdata and histlogs, eg. due to short-term ???ad-hoc metrics like nfs file systems, container nic devices. 3. menu script (see 3a) to manage clients easier Keep up the great work! Ye-Fee 1a 2a ------------------------------------------------------ c) change Client:?????? yyy ct) change days to trim: 30 1) summary???? 1a) list pings?? 1b) list cpu clients 1bbb) list to scp file?????????? 1bb) list in one column 1c) list non-pings tests?????? ??1d) list non-green cpu tests 2)? duplicate report 3)? Info on client?????????????? 3a) formatted 4)? Search client in hosts.cfg+ 6)? find clients with too many rrd's???? 6z) list -5 RRD's 6a) list top 10?? 6af) find top 10?? 6at) trim hostdata's ?6b) list top 10?? 6bf) find top 10?? 6bt) trim histlogs's 6c) list top 10?? 6cf) find top 10?? 6ct) trim Select from top 10 to trim??? 6ta) RRD? 6tb) hostdata? 6tc) histlogs 7a) diff host_cfg files????????? 7u) update host_cfg files ------------------------------------------------------ 3a ----------------------------------------------------------- c) Change Client: vag7 c1) check ssh???????????????? c2) ps -ef | grep xym c3) copy-id?????????????????? c4) check client os version 2a) info xymonclient/logs???? 3a) log: logs/xymonclient.log 2b) info xymonclient/tmp????? 3i) log info? 3t) log tail 2c) info xymonclient dir????? 3u) log util 3s) scan xymonclient.log 5a) start xymonclient???????? 5c) status 5b) stop t1) curl nag1 (10.248.224.149) 1984 test???? t3) telnet t2) curl nag2 (142.71.43.51)?? 1984 test???? t4) telnet t8) rm known_hosts entry dc) compare config file?????? dx) cp? + extract client file d2) diff vmstat2.sh?????????? d3) cp vmstat2.sh to client d6) client file: xymonclient_v4_3_28_RH7.gz d7) config file: xymonclient.cfg d9) copy config file to client -----------------------------------------------------------

4 12

Xymon Server on RHEL9
by colin.coe＠gmail.com 29 Nov '23

29 Nov '23

Hi all I'm getting ready to replace our venerable RHEL6 Xymon server with one running RHEL 9. We use the terabithia.org RPMs. NTP tests are all returning red. I've updated xymonserver.cfg (and restarted the xymon service) by changing NTPDATE="/usr/bin/chronyc" and adding NTPDATEOPTS="tracking". When I run "/usr/bin/chronyc tracking <IP HERE>" from the shell, it works fine however the Xymon page is red. Service ntp on <CENSORED> is not OK : Service unavailable Command: /usr/bin/chronyc tracking <CENSORED> 2>&1 Any ideas?

5 5

'Invalid timespec' in xymond.log
by john.thurston＠alaska.gov 27 Nov '23

27 Nov '23

I'm seeing messages like this > Invalid timespec - expected 4 digits, got: '*:*' > Invalid timespec - expected 4 digits, got: '*' > Invalid timespec - expected 4 digits, got: '*:*' > Invalid timespec - expected 4 digits, got: '*' pop up in my xymond.log. They've been there for ages, so this is nothing new. I've poked at them before, to no satisfaction. I think this came out of the 'minutes' function in lib/timefunc.c But what was the input which was passed through that function, to generate the error? I have 'DOWNTIME' specifications like > DOWNTIME=60:*:*;*:1630:0800 > But there are 21 of those in hosts.cfg, and I don't have that many complaints in the log. Does anyone else have a suggestion where to look? or how to interactively run some parsing commands to tease out the source? Maybe that function is used for more than parsing the configs? Could the source of the errors be from messages passed into the server from clients? -- -- Do things because you should, not just because you can. John Thurston 907-465-8591 John.Thurston at alaska.gov Department of Administration State of Alaska

2 2

Acknowledgements and page color.
by cleaver＠terabithia.org 09 Nov '23

09 Nov '23

On Thu, November 9, 2023 11:35, Grant Taylor wrote: > On 11/8/23 11:46???AM, J.C. Cleaver wrote: >> This isn't something currently there, but I could see adding it in >> in the future as an option or toggle. > > That would be nice to have. :-) > >> For the moment, the distinction here between one Ack'd and one >> a page-viewer "doesn't need to care about" would be achieved by >> sending a "Disabled until OK" message, which sets the color to the >> non-alerting Blue. > > Yep, that's what we've been doing. > > But I figured I'd stop and ask instead of doing the same thing has > always been done out of ignorance. > > I should clarify that I was referring to the individual status pages for a host (pulled live via svcstatus.cgi, or a static HTML generation if enabled). If you want to prevent an entire *page* from reflecting the worst status if that status has been acked, you can pass the --nopropack=* option to xymongen in your tasks.cfg entry. See https://xymon.sourceforge.io/xymon/help/manpages/man1/xymongen.1.html -jc

1 0

Acknowledgements and page color.
by cleaver＠terabithia.org 09 Nov '23

09 Nov '23

On Wed, November 8, 2023 09:24, Grant Taylor wrote: > Hi, > > Is there any way to have the page color go back to green when a yellow / > red / purple is acknowledged? > > Or will the page color always and forever reflect the worst color > (purple -> yellow -> red) on the page? > > This isn't something currently there, but I could see adding it in in the future as an option or toggle. For the moment, the distinction here between one Ack'd and one a page-viewer "doesn't need to care about" would be achieved by sending a "Disabled until OK" message, which sets the color to the non-alerting Blue. -jc

2 1

Missing support for HTTP/2 ?
by jeremy＠laidman.org 08 Nov '23

08 Nov '23

On Wed, 8 Nov 2023 at 23:14, Ingeborg Hellemo via Xymon <xymon at xymon.com> wrote: > > ~/server/bin/xymonnet --debug --no-update <servername> > > 35422 2023-11-08 13:06:06.168385 1 status messages merged into 1 > transmissions > Address=[IP]:443, open=1, res=0, err=5, connecttime=0.000303, > totaltime=0.002631, > httpstatus = -5, open=1, errcode=5, parsestatus=0 > Response: > (no headers) > URL : https://<servername>/ > HTTP status : -5 > I'm surprised there isn't some kind of error message being displayed. err=5 is "CONTEST_ESSL" meaning something went wrong in the SSL comms, or setup. In most cases where the error code is set to CONTEST_ESSL, there's an "errprintf()" that describes the error. I'd have thought "--debug" would show these on STDERR. The missing error message is likely going to help narrow down the problem. These are all of the error messages in contest.c where err is set to CONTEST_ESSL: errprintf("SSL test, but xymonnet was built without SSL support\n"); errprintf("Failed to find enough entropy on your system"); errprintf("Cannot create SSL context - IP %s, service %s: %s\n", inet_ntoa(item->addr.sin_addr), item->svcinfo->svcname, sslerrmsg); errprintf("Cannot load SSL client certificate/key %s: %s\n", item->ssloptions->clientcert, sslerrmsg); errprintf("SSL_new failed - IP %s, service %s: %s\n", inet_ntoa(item->addr.sin_addr), item->svcinfo->svcname, sslerrmsg); errprintf("Private/public key mismatch for certificate %s\n", item->ssloptions->clientcert); errprintf("Could not initiate SSL on connection - IP %s, service %s: %s\n", inet_ntoa(item->addr.sin_addr), item->svcinfo->svcname, sslerrmsg); errprintf("IO error in SSL_connect to %s on host %s: %s\n", portinfo, inet_ntoa(item->addr.sin_addr), sslerrmsg); errprintf("Unspecified SSL error in SSL_connect to %s on host %s: %s\n", portinfo, inet_ntoa(item->addr.sin_addr), sslerrmsg); errprintf("Unknown error %d in SSL_connect to %s on host %s: %s\n", err, portinfo, inet_ntoa(item->addr.sin_addr), sslerrmsg); errprintf("Cannot get peer certificate for %s on host %s\n", portinfo, inet_ntoa(item->addr.sin_addr)); Perhaps these can give you some clues about what might be going wrong. I also suggest taking a look at the webserver logs for any errors, warnings or notice messages associated with the xymonnet connection. J

1 0

Acknowledgements and page color.
by gtaylor＠tnetconsulting.net 08 Nov '23

08 Nov '23

Hi, Is there any way to have the page color go back to green when a yellow / red / purple is acknowledged? Or will the page color always and forever reflect the worst color (purple -> yellow -> red) on the page? -- Grant. . . . unix || die

1 0

Missing support for HTTP/2 ?
by tom＠4schmidts.com 08 Nov '23

08 Nov '23

Ingeborg, HTTP/2 support is currently not in xymon, including in the 4.4alpha1 development tree. I did a little testing, and sites like google that support HTTP/2 still pass the xymonnet checks as seen below: $ curl --head https://www.google.com | head -1 HTTP/2 200 $ grep google ~/server/etc/hosts.cfg 0.0.0.0 www.google.com # conn https://www.google.com/ $ ~/server/bin/xymonnet --no-update www.google.com |grep -B2 HTTP &green https://www.google.com/ - OK HTTP/1.1 200 OK $ ~/server/bin/xymonnet --no-update www.google.com |grep -i ssl status+30 www,google,com.sslcert green Wed Nov 1 08:38:21 2023 &green SSL certificate for https://www.google.com/ expires in 2252 days Perhaps the website you are testing is configured to support HTTP/2 and not support HTTP/1? Try running the xymonnet command manually similar to what I did here. Tom Schmidt On Wed, Nov 1, 2023 at 7:28?AM Ingeborg Hellemo via Xymon <xymon at xymon.com> wrote: > > > > ---------- Forwarded message ---------- > From: Ingeborg Hellemo <ingeborg.hellemo at uit.no> > To: xymon at xymon.com > Cc: > Bcc: > Date: Wed, 01 Nov 2023 14:21:04 +0100 > Subject: Missing support for HTTP/2 ? > I have a webserver which works ok when you visit it in a browser or when > you > use curl, but Xymon http test shows "SSL error". Tests with "openssl > s_client" > from the command line works as expected. > > Could the culprit be that xymonnet does not support HTTP/2 ? > > > /#curl --head https://<servername>/ > HTTP/2 200 > cache-control: private > content-length: 4423 > content-type: text/html > strict-transport-security: max-age=31536000 > date: Wed, 01 Nov 2023 13:15:44 GMT > > > > > --Ingeborg > -- > Ingeborg ?strem Hellemo -- ingeborg.hellemo at uit.no > Dep. of Information Technology --- Univ. of Troms? > > > > > > ---------- Forwarded message ---------- > From: Ingeborg Hellemo via Xymon <xymon at xymon.com> > To: xymon at xymon.com > Cc: > Bcc: > Date: Wed, 01 Nov 2023 14:21:04 +0100 > Subject: [Xymon] Missing support for HTTP/2 ? > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon >

2 1