January 2016 - Xymon - Xymon mailman web

log monitoring hostname matching trouble
by EDSchminke＠Hormel.com 14 Jan '16

14 Jan '16

Hi Jeremy, Thanks for your reply. First, I am aware of the 10 minute interval. I did observe that when working with an AIX system. I mention that because I am able to get log monitoring to work there, but not with a Linux system. That is to say, I do get a "logfetch.[hostname].cfg" file in $XYMONHOME/tmp, and changes come in approx ten minutes after being made on the XYMON server at $XYMONHOME/etc/client-local.cfg Is there anything I can do to reduce the 10 minute interval for testing purposes? ----- Entering "xymoncmd" [root at lpcobwas2p ~]# su - xymon -bash-4.1$ export XYMONCLIENTHOME=/home/xymon/client -bash-4.1$ client/bin/xymoncmd sh-4.1$ whoami xymon sh-4.1$ pwd /home/xymon sh-4.1$ ---- All files in /home/xymon are owned by xymon, group xymon and are writable sh-4.1$ find . ! -writable sh-4.1$ find . ! -user xymon sh-4.1$ find . ! -group xymon sh-4.1$ $XYMONTMP/msg.$(hostname).txt is getting updated every five minutes, as expected: sh-4.1$ ls -la client/tmp/msg.$(hostname).txt -rw-rw-r--. 1 xymon xymon 152566 Jan 13 08:00 client/tmp/msg.lpcobwas2p.hfc.ad.txt sh-4.1$ ls -la client/tmp/msg.$(hostname).txt -rw-rw-r--. 1 xymon xymon 148917 Jan 13 08:05 client/tmp/msg.lpcobwas2p.hfc.ad.txt sh-4.1$ ----- I sent this file manually to the xymon server, but got no output sh-4.1$ $XYMON $XYMSRV "@" < $XYMONTMP/msg.lpcobwas2p.hfc.ad.txt sh-4.1$ ----- Sending your suggested test string yields the following: sh-4.1$ $XYMON $XYMSRV "@" << EOT client $MACHINE.linux linux EOT log:/var/log/messages:10240 ignore MARK sh-4.1$ echo $MACHINE lpcobwas2p,hfc,ad sh-4.1$ Interesting... but a) why doesn't that come back with the message file and b) it still doesn't match the hostname ----- Lastly.. I'm still not clear on the correct format of a hostname definition for client/etc/client-local.cfg Should it be lpcobwas2p (just hostname) lpcobwas2p.hfc.ad (period) lpcobwas2p,hfc,ad (commas) Thanks, Erik D. Schminke | Associate Systems Programmer Hormel Foods Corporation | One Hormel Way | Austin, MN 55912 Phone: (507) 434-6817 edschminke at hormel.com | www.hormelfoods.com From: Jeremy Laidman <jlaidman at rebel-it.com.au> To: EDSchminke at hormel.com Date: 01/12/2016 08:03 PM Subject: Re: [Xymon] log monitoring hostname matching trouble Erik Hmm. It's hard to know what might be wrong. First, note that it can take some time (10 minutes maybe) for the cfg file to appear. Check your ~xymon/client/tmp/ directory and ensure that the xymon user can write there (probably should have owner and group both as "xymon"). Then, see if you have a client message temp file there with something like ls -l ~xymon/client/tmp/msg.*.txt This should show the hostname that the Xymon client is using, as part of the filename. It will also show the permissions, which should make the file writeable by the xymon user. Also, check the first line of the file (eg: head -1 ~xymon/client/tmp/msg.*.txt) and see if the hostname listed there is what you're expecting. Try sending the msg.*.txt file via the xymon client message, and see if you get the cfg file as output: $ ~xymon/client/bin/xymoncmd $ $XYMON $XYMSRV @ < $XYMONTMP/msg.*.txt $ exit Or even just: $ ~xymon/client/bin/xymoncmd $ echo "client $MACHINE.linux linux" | $XYMON $XYMSRV @ (Substitute "linux" for your OS.) Cheers Jeremy On Sat, Jan 9, 2016 at 3:10 AM <EDSchminke at hormel.com> wrote: Hello, I'm trying to get log monitoring to work on a new implementation of Xymon (v4.3.24) For whatever reason, my client doesn't seem to be receiving a logfetch.cfg file from the server after it checks in. I've tried a number of combinations of hostname and none of them seem to work: [lpcobwas2p] (hostname only) [lpcobwas2p.hfc.ad] (FQDN, periods) [lpcobwas2p,hfc,ad] (FQDN, commas) None of them seem to work. Any ideas? Server side: [root at lxcobxymonp server]# pwd /home/xymon/server [root at lxcobxymonp server]# hostname lxcobxymonp.hfc.ad [root at lxcobxymonp server]# [root at lxcobxymonp server]# tail -2 etc/client-local.cfg [lpcobwas2p] log:/var/log/messages:20480 [root at lxcobxymonp server]# [root at lxcobxymonp server]# grep lpcobwas2p etc/hosts.cfg 0.0.0.0 lpcobwas2p # CLIENT:lpcobwas2p.hfc.ad [root at lxcobxymonp server]# Client side: [root at lpcobwas2p client]# pwd /home/xymon/client [root at lpcobwas2p client]# hostname lpcobwas2p.hfc.ad [root at lpcobwas2p client]# [root at lpcobwas2p client]# head -2 tmp/msg.lpcobwas2p.hfc.ad.txt @@client#1|1|127.0.0.1|lpcobwas2p.hfc.ad|linux client lpcobwas2p,hfc,ad.linux linux [root at lpcobwas2p client]# Erik D. Schminke | Associate Systems Programmer Hormel Foods Corporation | One Hormel Way | Austin, MN 55912 Phone: (507) 434-6817 edschminke at hormel.com | www.hormelfoods.com _______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

3 2

where does xymon client get tcp packet data?
by john.thurston＠alaska.gov 14 Jan '16

14 Jan '16

The below relates to linux, xymon 4.3.17 I was going to write a script to collect and report numbers of TCP packet retransmissions. But when I look at the 'trends' for this host, I see there is already a chart for "TCP/IP statistics" which includes "In/Out/Retrans" I'm specifically looking for TCP retransmission count. This counter does not seem to line up with information being reported by nstat. What is this built-in test reporting, and from where is it collected? -- Do things because you should, not just because you can. John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska

3 3

Temperature plugin for xymon
by A.Chervonets＠cominder.eu 13 Jan '16

13 Jan '16

Some time ago I had found the lm_sensors project. There were some instructions for many hardware. Sorry to say, but right now the web-site http://lm-sensors.org/ have no more content about this. However the lm-sensors was included in many Linux distributions (at least on CentOS, RHEL 6.x and similar), so it was possible to: # install yum install lm_sensors # run to detect devices sensors-detect # and if OK (it has drivers for your hardware) then can get the output: sensors acpitz-virtual-0 Adapter: Virtual device temp1: +8.3 C (crit = +31.3 C) coretemp-isa-0000 Adapter: ISA adapter Physical id 0: +57.0 C (high = +78.0 C, crit = +88.0 C) Core 0: +54.0 C (high = +78.0 C, crit = +88.0 C) Core 1: +52.0 C (high = +78.0 C, crit = +88.0 C) Core 2: +51.0 C (high = +78.0 C, crit = +88.0 C) Core 3: +54.0 C (high = +78.0 C, crit = +88.0 C) power_meter-acpi-0 Adapter: ACPI interface power1: 0.00 nW (interval = 300.00 s) coretemp-isa-0004 Adapter: ISA adapter Physical id 1: +50.0 C (high = +78.0 C, crit = +88.0 C) Core 0: +50.0 C (high = +78.0 C, crit = +88.0 C) Core 1: +49.0 C (high = +78.0 C, crit = +88.0 C) Core 2: +45.0 C (high = +78.0 C, crit = +88.0 C) Core 3: +49.0 C (high = +78.0 C, crit = +88.0 C) Detailed information is available at: https://www.kernel.org/doc/Documentation/thermal/ Please, drop me a mail if You will find anything useful. I am interested too. Best regards, Andrey Chervonets ---------------------- CoMinder Support http://www.cominder.eu/ mobile: +371 26517848

2 1

How many times does xymonnet retry?
by glauber.ribeiro＠experian.com 13 Jan '16

13 Jan '16

Hello, all, I didn’t find this information in the man pages for xymonnet and xymonnet-again. How many times does xymonnet retry a failed test? Does it go red on the first failure (assuming default configuration), or only after all the retries failed? Thanks, glauber

4 6

Able to generate RRD but graphs do not show up
by kothari.raunak＠gmail.com 12 Jan '16

12 Jan '16

Hi all, I have a check in xymon to plot the time taken for several reports on the system to be run. For each report, I have xymon monitors that are collecting the data to the rrds but I am unable to populate the graph for these checks. Below is the output of rrdtool <info> on one of the rrds - filename = "SecondsElapsed.rrd" rrd_version = "0003" step = 300 last_update = 1446514863 header_size = 1216 ds[SecondsElapsed].index = 0 ds[SecondsElapsed].type = "GAUGE" ds[SecondsElapsed].minimal_heartbeat = 900 ds[SecondsElapsed].min = 0.0000000000e+00 ds[SecondsElapsed].max = NaN ds[SecondsElapsed].last_ds = "5" ds[SecondsElapsed].value = NaN ds[SecondsElapsed].unknown_sec = 63 rra[0].cf = "AVERAGE" rra[0].rows = 576 rra[0].cur_row = 258 rra[0].pdp_per_row = 1 rra[0].xff = 5.0000000000e-01 rra[0].cdp_prep[0].value = NaN rra[0].cdp_prep[0].unknown_datapoints = 0 rra[1].cf = "AVERAGE" rra[1].rows = 576 rra[1].cur_row = 481 rra[1].pdp_per_row = 6 rra[1].xff = 5.0000000000e-01 rra[1].cdp_prep[0].value = 0.0000000000e+00 rra[1].cdp_prep[0].unknown_datapoints = 2 rra[2].cf = "AVERAGE" rra[2].rows = 576 rra[2].cur_row = 382 rra[2].pdp_per_row = 24 rra[2].xff = 5.0000000000e-01 rra[2].cdp_prep[0].value = 0.0000000000e+00 rra[2].cdp_prep[0].unknown_datapoints = 20 rra[3].cf = "AVERAGE" rra[3].rows = 576 rra[3].cur_row = 338 rra[3].pdp_per_row = 288 rra[3].xff = 5.0000000000e-01 rra[3].cdp_prep[0].value = 0.0000000000e+00 rra[3].cdp_prep[0].unknown_datapoints = 20 Below is my graph definition in graphs.cfg (The column name in Xymon is "Status") [Status] FNPATTERN SecondsElapsed.rrd TITLE Report Execution time in seconds YAXIS Seconds DEF:SecondsElapsed=SecondsElapsed.rrd:SecondsElapsed:AVERAGE GPRINT:SecondsElapsed:LAST:Minutes\: %5.0lf (cur) Any clues why my graphs are not working ? Thanks! Regards, Raunak

2 1

Alerts
by john＠executech.com 11 Jan '16

11 Jan '16

I am getting strange behavior that I can't solve. I have specific users monitoring a subpage of servers. When I move a set of servers from one user (subpage) to another user (a different subpage), the new user is getting the alerts but the alerts are also going to previous user as well. One thing that helped in the past was to delete any data related to the set of servers in the subpage folder inside www. But recently that hasn't helped. Alerts are based on the page so that all subpages under them will alert them by default: PAGE=%.*UserPage/ SCRIPT=/custom-alerts/alert.sh user at domain.com RECOVERED REPEAT=720 DURATION>10 If I check the info page of any of the servers, the new user is listed but nothing is listed for the old user. Has anyone seen this before? John Tullis

1 0

Couple of questions on client data
by cleaver＠terabithia.org 11 Jan '16

11 Jan '16

Glad it could help :) There should be a README-local file in the client home directory (or documentation directory, depending on what install method you used). By default, the client will look in $XYMONHOME/local/ for anything executable. If anything is found, it's run (xymonclient.sh:65) and the output is added to the client report near the end, under the heading "[local:<filename>]". This allows a server admin to easily add new things on a one-off basis. Xymon v4.4 will have a similar directory named /sections/ with does the same thing, except without the "local:" prefix on the output. (It's intended for packaged site or distro/vendor-specific scripts.) Ryan: Yes, this is one area where it definitely needs more highlighting. A lot of users miss that this happens, but it's really an invaluable resource for forensics. Remember, though, that this requires xymon to be running in server-config mode, not --local client mode. In --local client mode, this raw data never leaves the original box and so can't be saved. Regards, -jc On Sun, January 10, 2016 3:51 pm, David Boyer wrote: > JC, > Yes, it does! Thanks for the background.. I've written/modified > several tests for my environment while we were still in the "big brother" > era. I'm still > learning the ins/outs of xymon. I'm intrigued by the use of the "local" > directory on the clientside. Where might I find more information about > using this feature. > > Dave > > On Sun, Jan 10, 2016 at 3:35 PM, Novosielski, Ryan > <novosirj at ca.rutgers.edu> > wrote: > >> I just wanted to thank you, JC, for this information. I wanted to know >> whether it would be possible to check the kernel version back some time >> even though that is something I did not test. I figured the info might >> show >> something, but that isn't kept in history or anything. If I had realized >> that a client data snapshot is kept when the status changes, I would >> have >> had the answer (which I still need, so it's very helpful). >> >> ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* >> || \\UTGERS |---------------------*O*--------------------- >> ||_// Biomedical | Ryan Novosielski - Senior Technologist >> || \\ and Health | novosirj at rutgers.edu- 973/972.0922 (2x0922) >> || \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark >> `' >> >> On Jan 10, 2016, at 06:48, J.C. Cleaver <cleaver at terabithia.org> wrote: >> >> Hi David, >> >> >> On Sat, January 9, 2016 3:15 pm, David Boyer wrote: >> >> I see that there is data in the client data that is not turned into >> >> columns? Can I turn this data into columns to report? I'm using xymon >> >> 4.3.20 on both the client/server. >> >> >> On linux, I see that this data is not turned into any columns: >> >> [who] >> >> [route] >> >> [netstat] >> >> [ifstat] >> >> >> Not that I'd use all of them, just wondering what the reasoning is >> behind >> >> it. >> >> >> This is correct. The client data is a full raw text of various bits of >> information that can be used to create status messages out of, but there >> isn't a mandated 1:1 correspondence between each section and each test. >> By >> keeping them independent (raw data and central processing), you have the >> flexibility to write up new tests based off of already-existing incoming >> data, and/or add new "raw" data without having a specific test in mind. >> >> For the 'who' data in particular, there's a sample processor in the >> source >> tarball that can demonstrate how easy it is to add simple new tests at >> >> https://sourceforge.net/p/xymon/code/HEAD/tree/branches/4.3.24/xymond/xymon… >> >> You can easily add new sections to the client data by adding files to a >> "/local/" directory on the client machine, or editing the >> xymonclient-${OS}.sh shell script by hand, and running any command that >> the unprivileged 'xymon' user can execute. >> >> What's the benefit to all this additional data if it's not used? >> Primarily >> forensics and triage. As the xymon(7) man page puts it: >> >> The Xymon user-interface is simple, but engineers will also find lots of >> >> relevant information. E.g. the data that clients report to Xymon contain >> the raw output from a number of system commands. That information is >> available directly in Xymon, so an administrator no longer needs to >> login to a server to get an overview of how it is behaving - the very >> commands they would normally run have already been performed, and the >> results are on-line in Xymon. >> >> https://www.xymon.com/help/manpages/man7/xymon.7.html >> >> >> >> This becomes even more relevant when you consider snapshoting. When a >> status goes "red", a snapshot of the client data at that time is kept. >> So >> if you went back later to try to figure out why (e.g.) CPU was rising, >> the >> output of the '[who]' section tells you who might have been doing >> something then, even if the data wasn't used for making a test out of at >> that time. >> >> >> >> >> Also, I notice that getting the data via xymondboard is not 100% >> either... >> >> >> [xymon at ztest bin]$ ./xymon 192.168.1.230 "clientlog yumlist section=who" >> >> *snip* >> >> >> As you can see below, requesting a status on all the tests, there is no >> >> "who" being reported. >> >> But just a moment ago, asking for just that data it works. >> >> >> [xymon at ztest bin]$ ./xymon 192.168.1.230 "xymondboard host=yumlist" >> >> *snip* >> >> >> Correct. The "clientlog" command retrieves the most recent raw client >> data, while the "xymondboard" command retrieves just the status messages >> (the tests that you see on the webpages). >> >> The "clientlog" column on the web page, much like the "info" and >> "trends" >> columns, isn't a real test... it's just present to provide easy access >> to >> the most recent data. You can also access it from the "Client data" link >> on the bottom of any of that host's status pages. >> >> >> Hope that helps! >> >> -jc >> >> _______________________________________________ >> Xymon mailing list >> Xymon at xymon.com >> http://lists.xymon.com/mailman/listinfo/xymon >> >> >

2 2

Couple of questions on client data
by novosirj＠ca.rutgers.edu 11 Jan '16

11 Jan '16

This is the info I need but it sounded like I can get it from historical periods, which is what I'm after. ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \\UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \\ and Health | novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>- 973/972.0922 (2x0922) || \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `' On Jan 10, 2016, at 18:56, David Boyer <davieb at gmail.com<mailto:davieb at gmail.com>> wrote: Ryan, What specific data are you looking for? If it's the kernel version, this could pull the info: [xymon at xytest bin]$ ./xymon localhost "clientlog yumlist section=uname" [uname] Linux yumlist 2.6.32-504.12.2.el6.x86_64 x86_64 Or more detailed info: [xymon at xytest bin]$ ./xymon localhost "clientlog yumlist section=osversion" [osversion] CentOS 6.6 LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: CentOS Description: CentOS release 6.6 (Final) Release: 6.6 Codename: Final and grab what you need... Dave On Sun, Jan 10, 2016 at 6:31 PM, Novosielski, Ryan <novosirj at ca.rutgers.edu<mailto:novosirj at ca.rutgers.edu>> wrote: Actually: how do you get at this? I can't see a way after hunting around a bit. I thought maybe in a client data link at the bottom of a historical page, but none is present. ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \\UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \\ and Health | novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>- 973/972.0922<tel:973%2F972.0922> (2x0922) || \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `' On Jan 10, 2016, at 06:48, J.C. Cleaver <cleaver at terabithia.org<mailto:cleaver at terabithia.org>> wrote: This becomes even more relevant when you consider snapshoting. When a status goes "red", a snapshot of the client data at that time is kept. So if you went back later to try to figure out why (e.g.) CPU was rising, the output of the '[who]' section tells you who might have been doing something then, even if the data wasn't used for making a test out of at that time.

3 3

Couple of questions on client data
by davieb＠gmail.com 10 Jan '16

10 Jan '16

I see that there is data in the client data that is not turned into columns? Can I turn this data into columns to report? I'm using xymon 4.3.20 on both the client/server. On linux, I see that this data is not turned into any columns: [who] [route] [netstat] [ifstat] Not that I'd use all of them, just wondering what the reasoning is behind it. Also, I notice that getting the data via xymondboard is not 100% either... [xymon at ztest bin]$ ./xymon 192.168.1.230 "clientlog yumlist section=who" [who] davieb tty1 May 11 21:01 (:0) davieb pts/1 May 11 21:02 (:0.0) davieb pts/2 May 11 21:40 (:0.0) As you can see below, requesting a status on all the tests, there is no "who" being reported. But just a moment ago, asking for just that data it works. [xymon at ztest bin]$ ./xymon 192.168.1.230 "xymondboard host=yumlist" yumlist|trends|green||0|0|0|0|0||| yumlist|info|green||0|0|0|0|0||| yumlist|uptime|green||1436045636|1452380534|1452382334|0|0|192.168.1.9||green Sat Jan 9 18:02:08 EST 2016 Uptime OK yumlist|memory|green||1431806850|1452380534|1452382334|0|0|192.168.1.9||green Sat Jan 9 18:02:08 EST 2016 - Memory OK yumlist|files|clear||1451081789|1452380534|1452382334|0|0|192.168.1.9||clear Sat Jan 9 18:02:08 EST 2016 - Files ok yumlist|msgs|green||1431807151|1452380534|1452382334|0|0|192.168.1.9||green Sat Jan 9 18:02:08 EST 2016 - System logs ok yumlist|ports|clear||1431806850|1452380534|1452382334|0|0|192.168.1.9||clear Sat Jan 9 18:02:08 EST 2016 - Ports ok yumlist|procs|green||1436043229|1452380534|1452382334|0|0|192.168.1.9||green Sat Jan 9 18:02:08 EST 2016 - Processes ok yumlist|inode|green||1431806850|1452380534|1452382334|0|0|192.168.1.9||green Sat Jan 9 18:02:08 EST 2016 - Filesystems ok yumlist|disk|green||1431806850|1452380534|1452382334|0|0|192.168.1.9||green Sat Jan 9 18:02:08 EST 2016 - Filesystems ok yumlist|cpu|green||1431806850|1452380534|1452382334|0|0|192.168.1.9||green Sat Jan 9 18:02:08 EST 2016 up: 289 days, 3 users, 261 procs, load=0.02 yumlist|hostinfo|clear||1432774838|1452380534|1452382334|0|0|192.168.1.9||clear Sat Jan 9 18:02:14 EST 2016 yumlist|cpu2|green||1431806845|1452380529|1452382329|0|0|192.168.1.9||green Sat Jan 9 18:02:09 EST 2016 yumlist|smart|green||1431806845|1452380528|1452382328|0|0|192.168.1.9||green SMART health check yumlist|fyi|green||1431806845|1452380719|1452382519|0|0|192.168.1.9||green Sat Jan 9 18:05:19 EST 2016 yumlist|rpc|green||1431805465|1452380560|1452382360|0|0|192.168.1.230||green Sat Jan 9 18:02:32 2016 rpc ok, yumlist|conn|green|OrdAsTLe|1431805465|1452380560|1452382360|0|0|192.168.1.230||green  Sat Jan 9 18:02:32 2016 conn ok

3 4

Merging hosts
by thomleavitt＠gmail.com 09 Jan '16

09 Jan '16

Is there an easy (or even medium hard) way to merge the content of two hosts? I updated the entry for a server in Xymon to a fully qualified hostname, but the server is identifying itself with an unqualified hostname, and putting the CLIENT: tag fixed it, but for some reason, it seems to have lost track of the prior history. :/ Regards, Thomas Leavitt

2 1