November 2015 - Xymon - Xymon mailman web

Missing client memory test!?
by David.Mills＠hhsc.state.tx.us 17 Nov '15

17 Nov '15

Hey, all... Have a puzzle to solve: we have a host (Solaris 10) running our tried-and-true version of Xymon (4.3.2) which is functioning just fine except for one test: the standard "memory" test. It has worked fine in the past (I'm pretty sure!) but at some point we noticed Xymon isn't posting the "status" results for this test on the web page (it's a '-'). Debugging I've done so far: * On the client side, found the output file corresponding to "$MSGTMPFILE" / "$MSGFILE" used in the "xymonclient.sh" script. It _does_ contain valid data for the "[memory]" stanza: [memory] 0 8 0 39549560 48447184 1039 3853 3934 14 14 0 0 0 0 0 0 20276 71323 29065 57 16 28 * From this client, manually ran the command to post the data to the server: $ ~xymon/client/bin/xymon <server IP> "@" < msg.hostname.txt log:/var/adm/messages:10240 ignore %nxge0|nxge1|nxge4|nxge5|nxge0|107833 * On the server, ran: $ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname " hostname|trends|green||0|0|0|0|0||0| hostname|info|green||0|0|0|0|0||0| hostname|files|clear||1434028386|1446841258|1446843058|0|0|10.235.130.17|-1|clear Files status at Fri Nov 6 14:20:43 CST 2015 hostname|msgs|green||1434028386|1446841258|1446843058|0|0|10.235.130.17|-1|green System logs at Fri Nov 6 14:20:43 CST 2015 hostname|ports|red||1446840042|1446841258|1446843058|0|0|10.235.130.17|27225|red Fri Nov 6 14:20:43 CST 2015 - Ports NOT ok ... (The timestamps are current, matching the most recent version of the client file sent) * However, the "memory" test status is simply missing! For example, on the server: $ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname fields=msg" | perl -pe 's!\\n!\n!gm' > out $ grep '^status' out status hostname.files clear Files status at Fri Nov 6 14:25:46 CST 2015 status hostname.msgs green System logs at Fri Nov 6 14:25:46 CST 2015 status/group:unix-sadm hostname.ports red Fri Nov 6 14:25:46 CST 2015 - Ports NOT ok status/group:unix-sadm hostname.procs red Fri Nov 6 14:25:46 CST 2015 - Processes NOT ok status hostname.disk green Fri Nov 6 14:25:46 CST 2015 - Filesystems OK status hostname.cpu green Fri Nov 6 14:25:46 CST 2015 up: 249 days, 10 users, 956 procs, load=21.63 status hostname.prtdiag green Fri Nov 6 14:25:48 CST 2015 status hostname.raid green Fri Nov 6 14:25:46 CST 2015 Veritas Volume Manager OK status hostname.meta green Fri Nov 6 14:25:47 CST 2015 status hostname.zones green Fri Nov 6 14:25:46 CST 2015 status+30 hostname.conn green  Fri Nov 6 14:23:37 2015 conn ok status hostname.memUse GREEN status hostname.iostat GREEN status hostname.sar green Fri Nov 6 14:25:50 CST 2015 status+30 hostname.ssh green  Fri Nov 6 14:19:58 2015 ssh ok status+30 hostname.ntp green  Fri Nov 6 14:19:58 2015 ntp ok Note: no "memory" status line! Help? Thanks! david ~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 873-6665

3 4

Purple after reboot
by markconnor64＠gmail.com 17 Nov '15

17 Nov '15

I have approximately 500 Linux servers managed my xymon. My managed clients reboot monthly. A small percentage come up purple and stay purple until the client software is restarted. Sound familiar?

2 1

xymonfetch
by Paul.Root＠CenturyLink.com 16 Nov '15

16 Nov '15

I'm having issues with xymonfetch. I have 2 xymon proxies, that also have the xymon server running as a backup to the main server on port 1985. The proxy sends to the main server on port 1984 and to itself on port 1985. I'm trying to add 2 new machines as clients of these proxy servers. They are outside a firewall that I don't want to open, so I'm doing a fetch. Xymon-client is running on 1984, and msgcache is running. Client clientlaunch.cfg: [msgcache] # DISABLED ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg CMD $XYMONCLIENTHOME/bin/msgcache --no-daemon --pidfile=$XYMONCLIENTLOGS/msgcache.pid LOGFILE $XYMONCLIENTLOGS/msgcache.log Proxy tasks.cfg: [xymonfetch] ENVFILE /etc/xymon/xymonserver.cfg #CMD $XYMONHOME/bin/xymonfetch --server=YOUR.XYMON.SERVER.IP --no-daemon --pidfile=$XYMONSERVERLOGS/xymonfetch.pid CMD $XYMONHOME/bin/xymonfetch --server=127.0.0.1:1985 --log-interval=60 --id=1 --pidfile=$XYMONSERVERLOGS/xymonfetch.pid LOGFILE $XYMONSERVERLOGS/xymonfetch.log Proxy hosts.cfg: 10.6.0.14 remoteclient # pulldata=10.6.0.14:1984 http://198.36.155.243 10.6.0.15 remoteclient2 # pulldata=10.6.0.15:1984 I have a third xymon server, the development server, which is not a proxy, and the server runs on 1984. This pulls the data just fine from the remoteclients. The xymonfetch.log file gets these errors: 2015-11-11 15:56:24 Connection lost during connect/write to 10.6.0.14:1985 (req 27): Connection refused 2015-11-11 15:56:41 Connection lost during connect/write to 10.6.0.15:1985 (req 28): Connection refused 2015-11-11 15:57:27 Connection lost during connect/write to 10.6.0.14:1985 (req 29): Connection refused 2015-11-11 15:57:44 Connection lost during connect/write to 10.6.0.15:1985 (req 30): Connection refused The proxy servers are 4.3.10. The development server is 4.3.21. The main server is also 4.3.10, and it runs a fetch without issue, to other clients. The clients are 4.3.10 (CentOS 5) and 4.3.21 (CentOS 6). I am working on upgrading the proxy servers in OS (CentOS 5 -> 6) and Xymon (to 4.3.21). But not quite there yet. Another subject, but sending data from a proxy to another proxy does not work. So, when I have everything else, I may end up putting it in place, and hoping the proxy works.... Any ideas here? It seems like such a simple thing. Paul Root Lead Engineer CenturyLink Network Reliability Operations Center 390 Commerce Dr Woodbury, MN 55125 Direct: (651)312-5207 Paul.Root at centurylink.com This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.

3 9

Demotool
by dewelker＠gmail.com 16 Nov '15

16 Nov '15

I am interested in using this to create a testbed. From what I can tell, it only sends a message that contains the [date], [uptime], and [free] sections of the clientlog to the server, so how are the rest of the columns created, by the xymond daemon? Is there any simple way to add ones own columns to this structure and have them show up on the server? Thanks, David

2 3

Xymon 4.3.23 Released
by cleaver＠terabithia.org 13 Nov '15

13 Nov '15

Hello all, Xymon 4.3.23 has been released and is available on SF. This fixes the issue with TRACK-ing of process counts and port statuses in RRD that was introduced in v 4.3.22. (Note that thresholding and alerting (other than DS entries) for the procs and ports tests were unaffected.) It also corrects an error where the logfetch client would exit after reporting on a log file that was being tracked with 'deltacount' and was missing on the file system. All users are encouraged to upgrade. Regards, Japheth "J.C." Cleaver Xymon v4 Maintainer

2 2

Alert issue
by dewelker＠gmail.com 12 Nov '15

12 Nov '15

I admit, this is my first attempt at alerting, so this may or may not be user error. It also could be an issue that is fixed in the latest upgrade (I'm still on 4.3.21), but before I resort to that (which will be soon enough), I wanted to make sure I was at least thinking along the right line... The cpu column is reporting green. The entry in alerts.cfg includes a COLOR=yellow parameter. Running the xymond_alert --test reports (as expected)... Failed 'MAIN me at here COLOR=yellow EXSERVICE=conn,disk' (color) If, however, I change the COLOR to red, or even if I just add it (COLOR=yellow,red), I get the "Mail alert with command 'mail -s...'" line rather than the failed. Did I do something wrong? Funny though, either way I don't actually get the email. Thanks, David

3 5

Two procs/processes graph issues after server upgrade from 4.3.21 to 4.3.22-rc2
by beckert＠phys.ethz.ch 12 Nov '15

12 Nov '15

Hi, [TL;DR: See Summary at the end.] I'm slowly running out of ideas with the following issue which has been noticed after I rolled out 4.3.22-rc2 on our two monitoring servers (still running the servers on 4.3.22-rc2 at the moment): The graph on https://xymon.phys.ethz.ch/xymon-cgi/svcstatus.sh?HOST=zwoelfi&SERVICE=procs is no more there, because https://xymon.phys.ethz.ch/xymon-cgi/showgraph.sh?host=zwoelfi&service=proc… returns only an 1x1 pixel PMG. The same happens on the second (independent, not slave) server, too. (No version changes on the affected clients. Those I checked have either 4.3.0-beta2 from Debian 7 or 4.3.17 from Debian 8.) I've found the following messages upon reloading the above URL in Apache's error.log: 2015-11-11 12:32:38.839801 Sendto failed: Connection refused 2015-11-11 12:32:38.839853 Sendto failed: Connection refused 2015-11-11 12:32:38.839871 Sendto failed: Connection refused I've found http://lists.xymon.com/archive/2015-February/041189.html with these messages, stopped the xymon service, removed all left over rrdctl.* files from /var/lib/xymon/tmp/ and started the xymon service again. Result is: I still only get an 1x1 pixel PNG, but the error messages are gone, i.e. the issues are likely unrelated as they were in the mailing list posting above. Then again on https://xymon.phys.ethz.ch/xymon-cgi/svcstatus.sh?HOST=zwoelfi&SERVICE=tren… the "Process counts" graph is there (but seems not working): https://xymon.phys.ethz.ch/xymon-cgi/showgraph.sh?host=zwoelfi&service=proc… The difference between this and the first URL are (besides the time stamps): The first URL has nostale (without value) and color=green as additional query string parameters, and the second URL has instead first=1 and count=4 as query string parameters. As soon as I remove the "nostale" without a value or add a value like e.g. "nostale=1", the graph is back again (but still no more working). So while the (reduced to the minimum parameters) URL https://xymon.phys.ethz.ch/xymon-cgi/showgraph.sh?host=zwoelfi&service=proc… shows (an empty) graph, https://xymon.phys.ethz.ch/xymon-cgi/showgraph.sh?host=zwoelfi&service=proc… gives a 1x1 pixel. With regards to the empty graph, https://xymon.phys.ethz.ch/xymon-cgi/showgraph.sh?host=zwoelfi&service=proc… is not empty, it just shows that there is no more data since the 4th of November (when I updated the servers from 4.3.21 to 4.3.22-rc2). And indeed, in /var/lib/xymon/rrd/zwoelfi/, not all files have been updated anymore since 4th of November: # ls -l *proc* -rw-r--r-- 1 xymon xymon 19640 Nov 4 15:40 processes.apache2.rrd -rw-r--r-- 1 xymon xymon 19640 Nov 4 15:40 processes.automount.rrd -rw-r--r-- 1 xymon xymon 19640 Nov 4 15:40 processes.stress.rrd -rw-r--r-- 1 xymon xymon 19640 Nov 11 13:09 procs.rrd # Summary ======= So there seem to be two issues with 4.3.22: * The graph in the procs check's page isn't displayed properly. Either + "nostale" should get a value in the page/template, + or the parsing of the "nostale" parameter without value in the showgraph CGI should be fixed. This sounds rather easy, but I'm not sure which variant is the expected one. * For some reason the processes.*.rrd files defined by "TRACK=" in analysis.cfg no more get updated. Here I currently have no good idea where this comes from. Maybe from one of the NCV-related changes. At least I found no configuration change (be it local or in the defaults/templates) which could have triggered this issue. Kind regards, Axel Beckert -- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/

2 6

SSL Certificate test failure
by Scot.Kreienkamp＠la-z-boy.com 11 Nov '15

11 Nov '15

Hi there, I am testing a site in Xymon that is testing OK, but throws an SSL error in the browser. Wondering why that was, I looked at the certificate for the site... it doesn't match the domain name of the site that's serving it, which causes the browser to display an SSL error. I was expecting Xymon to do the same. Apparently Xymon doesn't check to make sure the certificate matches the URL. Thanks! Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * 7349151444 | * Scot.Kreienkamp at la-z-boy.com<mailto:%7BE-mail%7D> www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> | facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy> | twitter.com/lazboy<https://twitter.com/lazboy> | youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy> [cid:lzbVertical_hres.jpg] This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

8 10

bug in xymon-client-4.3.22-2.el6.x86_64
by Scot.Kreienkamp＠la-z-boy.com 11 Nov '15

11 Nov '15

Hi JC, In the just released RPM for 4.3.22, the xymon client for Linux ports command was changed to netstat -antuW. The "W" option is invalid for RHEL6 and causes the ports output to be missing. Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * 7349151444 | * Scot.Kreienkamp at la-z-boy.com<mailto:%7BE-mail%7D> www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> | facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy> | twitter.com/lazboy<https://twitter.com/lazboy> | youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy> [cid:lzbVertical_hres.jpg] This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

4 8

Monitoring Old Servers for historical information
by bruno＠redix.com.br 10 Nov '15

10 Nov '15

Hi, I have some servers that are monitored in xymon and work fine. Some of this server have do be shutdown and i remove then from the hosts.cfg. I need to see this old server on the Xymon page to see some historical information. if i add this server again on hosts.cfg, the server does not show up on the page. I think that the server will only show if it sends some data There is a way that i can see the historical of this server from the xymon page? I still have the histdata and histlog folders for this server Thanks

2 2