July 2015 - Xymon - Xymon mailman web

Display Host IP on Hover
by oakesbz＠gmail.com 23 Jul '15

23 Jul '15

For some reason this used to work and now it doesn't. I have HOSTPOPUP="CDI" in my xymonserver.cfg but none of my ip's are displaying when I hover over a host. Adding a COMMENT next to my host just displays it in parenthesis next to my host name. How can I get this working? Thanks.

1 1

BUG 'New' style interface names not recognised
by estassen＠gmail.com 21 Jul '15

21 Jul '15

On Centos 7, Xymon does not recognise interface names like p3p1, etc. This is the interface naming according to the 'Consistent Device Naming' standard ( http://linux.dell.com/files/whitepapers/consistent_network_device_naming_in… ) A simple fix would be to change the regex on line 22, do_ifstat.c from: "^([a-z]+[0123456789.:]*|lo:?)\\s", to "^([a-z0-9]+(_[0-9]+)?:*|lo:?)\\s", Regards, Eddie

2 1

NCV Bug-Fix / Feature Request
by everett.vernon＠gmail.com 20 Jul '15

20 Jul '15

Hi all Not sure if this is a bug or a feature request. Split_NCV is doing exactly what it says on the tin, so it's not really a bug. But, it makes things not work right, so it looks like a bug. Maybe I should call it an annoyance? Once again, I bashed heads with the amazingly useful Split_NCV feature of taking your data points and converting them to rrds graphs. Don't get me wrong, this is probably one of the coolest things in Xymon, because you can have an undefined number of data sets, and add more dynamically. However, it looks at your status output, and anything of that looks like a *description:value* it sticks into an associated rrd file. Including *Sun Jul 19 08:46:06 WST 2015* Which happens to be the first line of my test status, and is generally used as the first line of any test. So I switched to using status messages for display and data messages for graphing, which makes logical sense. And it still grabs the date and makes an rrd file of it. Which is wrong, and annoying. My request is that this be fixed. And I can think of a bunch of really simple ways to fix it, and bunch of more interesting ways. Some of the simpler ones I can think of are 1. Include metadata tag in your status. A simple #NoNCV which gets removed server-side for display, and the status doesn't get parsed by the NCV module. This means only the data messages will get parsed by the NCV module. (This will preserve backward compatibility) 2. Include metadata tags to indicate where the NCV data starts and stops in a status message. A #StartNC and #StopNCV or #NCV #/NCV. Of course there's nothing stopping us from using HTML like tags either. <NoNCV> or <NCV> and </NCV> pick a format. (Depending on how it's implemented, this might break backward compatibility) 3. Change/Add a server config tag. Instead of simply Split_NCV, we have Split_NCV that works like it does now. And a Split_NCVD tag, that will only parse the data messages. (This will preserve backward compatibility) There are many ways to achieve the result I am looking for, but please, we need to fix this. It's been an annoyance for way too long. Regards Vernon -- "Accept the challenges so that you can feel the exhilaration of victory" - General George Patton

3 2

CPU trend graph NaN - log shows RRD Error Updating , unknown DS name 'mem_swpd'
by betsy.schwartz＠gmail.com 20 Jul '15

20 Jul '15

I'm seeing a few RHEL6 servers with blank CPU trend graphs . the rrd files just contain NaN, sample line: /usr/bin/rrdtool dump /usr/local/xymon/data/rrd/ ext-dns1.example.com/vmstat.rrd  <row><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v><v> NaN </v></row> rrd-data.log has errors of the form /rrd-data.log:2015-07-18 14:08:42.443123 RRD error updating /usr/local/xymon/data/rrd/ ext-dns1.example.com/vmstat.rrd from 10.100.5.42: unknown DS name 'mem_swpd' So,why do only a few of my servers have this error? What do I need to do to make the CPU trends graphs work? Do I add mem_swpd in xymonservers.cfg and hobbitgraph.cfg and graphs.cfg, and can I do that without breaking the other hundreds of servers? Does anyone have the correct invocation for this? Thank you! (I am also assuming I'll have to blow away the old rrd file to get this to work...?)

2 1

Acknowledging an alert
by Paul.Root＠CenturyLink.com 17 Jul '15

17 Jul '15

Hi, I'm working on a script that works with a company internal callout (paging) system. The system requires a confirmation of a page when an engineer receives it. In the event of a recovery of the alert, the script sees that the message is a recovery, and sends an automatic confirmation. However, in the event of an acknowledgement in xymon, we just get another pageout. So there are two things to acknowledge. Is there a way in the script to know that the page out is an acknowledgement? I just noticed ACKCODE is an environment variable that is set. Does an acknowledge page have a particular code? I'll go try to see what it sends. Thanks, Paul. Paul Root Lead Engineer CenturyLink Network Reliability Operations Center 390 Commerce Dr Woodbury, MN 55125 Direct: (651)312-5207 Paul.Root at centurylink.com This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.

1 1

Turning Hostname into Hyperlink Changes Graph Titles
by oakesbz＠gmail.com 17 Jul '15

17 Jul '15

Hi all, I’m using the below to turn my hostnames into hyperlinks to login to each server (which works great): 10.10.10.10 Hostname # NAME:"<a href=ssh://root at 10.10.10.10>Displayed Hostname</a>" Problem is, it now uses this ugly html string as the hostname in the title of my graphs. So it displays "<a href=ssh://root… > Disk Utilization”, etc. as the title… which is obviously not pretty. Any thoughts on how to fix this? By the way, it also changes the displayed hostname in several other places which I managed to fix by changing the variable &XYMWEBHOST to &XMH_HOSTNAME in several of the $xymon/server/web/*_header files… Thank you!

3 3

gaps in gaps after server upgrade to Xymon 4.3.18
by nskyrca＠syr.edu 17 Jul '15

17 Jul '15

Hello, I recently upgraded my xymon servers (primary and failover) from Xymon 4.2.3 to 4.3.18. After the upgrade, we noticed gaps in such as CPU Utilization, TCP/IP Statistics, and Block IO, but only for clients that are running the xymon 4.2.3 client, not for those running the Big Brother client. I built a new 4.3.18 client but we still see gaps. We are running the old vmstat-bf.sh and netstat-bf.sh extension scripts in the xymon client. I turned on debugging for rrdstatus and rrddata on our failover server, and saw errors such as the ones below during the times when there was a gap. Does anyone have ideas on how to troubleshoot and fix this? Why was the xymon client working ok with xymon server 4.2.3 and not 4.3.18? from rrdstatus.log 12793 2015-07-15 14:24:20 xymond_rrd: Got message 73002 @@status#73002/its-splun k-log1.syr.edu|1436984660.571622|128.230.1.66||its-splunk-log1.syr.edu|netstat|1 436986460|green||green|1435596907|0||0||1436984369|linux|NSM/UNIX/Misc24x7|0| 12793 2015-07-15 14:24:20 startpos 770321, fillpos 770321, endpos -1 12793 2015-07-15 14:24:20 Flushing '/its-splunk-log1.syr.edu/netstat.rrd' with 4 updates pending, template 'udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActive Opens:tcpPassiveOpens:tcpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataByte s:tcpInInorderBytes:tcpInUnorderBytes:tcpRetransBytes:tcpOutDataPackets:tcpInIno rderPackets:tcpInUnorderPackets:tcpRetransPackets' 12793 2015-07-15 14:24:20 RRD error updating /usr/local/xymon/xymon-current/data /rrd/its-splunk-log1.syr.edu/netstat.rrd from 128.230.1.66: /usr/local/xymon/xym on-current/data/rrd/its-splunk-log1.syr.edu/netstat.rrd: illegal attempt to upda te using time 1436983760 when last update time is 1436983769 (minimum one second step) 12793 2015-07-15 14:24:20 Flush, but xymonmsg is empty from rrddata.log 12806 2015-07-15 16:27:30 Flushing '/its-splunk-log1.syr.edu/netstat.rrd' with 4 updates pending, template 'udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActive Opens:tcpPassiveOpens:tcpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataByte s:tcpInInorderBytes:tcpInUnorderBytes:tcpRetransBytes:tcpOutDataPackets:tcpInIno rderPackets:tcpInUnorderPackets:tcpRetransPackets' 12806 2015-07-15 16:27:30 Flushing cache '/its-splunk-log1.syr.edu/vmstat.rrd' 12806 2015-07-15 16:27:30 Flushing '/its-splunk-log1.syr.edu/vmstat.rrd' with 10 updates pending, template 'cpu_r:cpu_b:cpu_w:mem_swpd:mem_free:mem_buff:mem_cac h:mem_si:mem_so:dsk_bi:dsk_bo:cpu_int:cpu_csw:cpu_usr:cpu_sys:cpu_idl:cpu_wait' 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Flush of '/its-splunk-log1.syr.edu/' skipped, too soon 12806 2015-07-15 16:27:30 Want msg 38335, startpos 431139, fillpos 431139, endpo s -1, usedbytes=0, bufleft=3767260 12806 2015-07-15 16:27:30 Got 1121 bytes Thanks ! Nicole

1 0

query-frontend (CLI) for `clientlog`, `xymondboard` and `xymondlog`
by thomas.eckert＠it-eckert.de 17 Jul '15

17 Jul '15

> On 23 Jun 2015, at 22:12, J.C. Cleaver <cleaver at terabithia.org> wrote: > > On Tue, June 23, 2015 12:50 pm, Mark Felder wrote: >> >> >> On Tue, Jun 23, 2015, at 13:57, Thomas Eckert wrote: >>> Brothers, >>> >>> currently Iâm building a fronted for the various query-commands that >>> can >>> be sent to `xymond`. It is still in itâs early stages but I found >>> myself >>> using it daily already. So others may benefit too. >>> >>> Itâs named `xymonq` (âxymon queryâ). >>> >>> Two examples may give a basic idea: >>> >>> 1. Get a list of hosts on page `dc1`: >>> Traditionally this looks like this: >>> >>> $ xymon 127.0.0.1 "xymondboard page=dc1 test=info fields=hostname" >>> >>> With xymonq >>> >>> $ xymonq -P dc1 -L >>> >>> 2. Get disk information this group of hosts: >>> >>> $ for my_host in $(xymon 127.0.0.1 "xymondboard page=dc1 test=info >>> fields=hostname"); do >>> echo âHOST=$my_host" >>> xymon 127.0.0.1 "xymondlog $my_host.disk" >>> done >>> >>> And again the short version with xymonq: >>> >>> $ xymonq -q clientlog -P dc1 -s df >>> >>> >>> More information and download here: >>> http://www.it-eckert.com/software/xymonq/ >>> <http://www.it-eckert.com/software/xymonq/> >>> >>> Feedback welcome! >>> >> >> >> This looks really useful. Thanks!! > > > Nifty! > > One thing that might be useful is a sort of large-volume mode, optionally > shifting some of the processing from separate TCP xymond queries to a > local grep string, at the expense of network transfer instead. > > > We often use a perl one-liner (normally in a scriptlet, but on the CL > here) to re-wrap the resulting test message into newlines with the > hostname pre-pended on each line. Since it's a pipe, it's quick, and this > lets us grep lines after the fact: > > ]$ xymon localhost "xymondboard test=disk fields=hostname,msg" | perl -ne > 'm/^(\S+)|/ and $meh=$1 and s/(\\n|<br\/>)/\n$meh: /g and print;' | grep > /boot > > server01.example.com|status: /dev/sda1 495844 39330 430914 > 9% /boot > server02.example.com|status: /dev/sda1 99150 62675 31355 > 67% /boot > server03.example.com|status: /dev/sda1 99150 60628 33402 > 65% /boot > server04.example.com|status: /dev/sda1 99150 62856 31174 > 67% /boot > > etc... (The "<br>" check is so we can use this for HTML content returned > in an HTTP check too.) Hi J.C., hi Mark, thank you for your feedback! Right, a volume-mode does make sense for larger setups for sure. Some testing shows that the speed improvements are really significant. Only `xymondboard` is usable for such an optimisation (as it can return data for multiple hosts/tests matching given CRITERIA) I've implemented it as a special case for now: If no "HOST="-separator is requested (`-S` option) for a xymondboard query "volume-mode" kicks in automatically: // slow version w/ one tcp-connection per host: root at bb:~# /vagrant/xymonq -q xymondboard -H bb.local -T disk -f hostname,line1,msg -S HOST=bb.local bb.local|green Fri Jul 17 11:18:03 MSK 2015 - Filesystems ok|status bb,local.disk green Fri Jul 17 11:18:03 MSK 2015 - Filesystems ok\nFilesystem 1024-blocks Used Available Capacity Mounted on\n/dev/sda1 40558164 1614424 36883504 5% /\n/dev/disk/by-uuid/8769c421-b3e7-42b8-979a-28a8add5d0f3 40558164 1614424 36883504 5% /\n // volume-mode (no `-S` here): root at bb:~# /vagrant/xymonq -q xymondboard -H bb.local -T disk -f hostname,line1,msg bb.local:line1:green Fri Jul 17 11:23:04 MSK 2015 - Filesystems ok bb.local:msg:status bb,local.disk green Fri Jul 17 11:23:04 MSK 2015 - Filesystems ok bb.local:msg:Filesystem 1024-blocks Used Available Capacity Mounted on bb.local:msg:/dev/sda1 40558164 1614424 36883504 5% / bb.local:msg:/dev/disk/by-uuid/8769c421-b3e7-42b8-979a-28a8add5d0f3 40558164 1614424 36883504 5% / The post-processing is handled by an external awk-script (`unescape-xymonq.awk`, currently has to be stored in the same directory as `xymonq`) that - prefixes each line with "[hostname]:[fieldname]:", is the hostname is not printed is is left empty. - rewrites the escapes in "msg" The awk-script is tested w/ gawk, mawk and busybox-awk. A general post-processing implementation might be the best, enabling for something like xymonq .... -c unescape-xymonq.awk -c filter2.pl -c filter3.py All filters stored in a cont-able place. These filters can be re-used easily and even shared with others. The data provided by `xymondlog` could be emulated by `xymondboard fields=line1,msg` and minor post processing. I see no reason to use `xymondlog` at all -- it might be the best to remove this completely from `xymonq`. I assume that `xymondlog` existed before the more capable `xymondboard` and now serves as a quick "get the status for host.test" from the shell without worrying about the escaping required with `xymondboard`. For `clientlog` there is no other way to get the data but using a loop with one tcp-connection to xymond for each host (no other way to using the slow "looping mode" in this case). The described version is available here: http://www.it-eckert.com/downloads/download/beta/xymonq-HEAD-2015-07-17.tgz Any feedback much appreciated. All the best Thomas > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon

1 0

analysis.cfg PORT LOCAL syntax not working for powershell client netstat output?
by g.stone-tolcher＠its.uq.edu.au 17 Jul '15

17 Jul '15

Hi, has anyone got "ports" working with "LOCAL" criteria with client data from a powershell or bbwin client? It does not seem to work for me, but it works fine for unix clients: analysis.cfg line for windows host: HOST=windows.host PORT "LOCAL=%([.:]80)$" state=LISTEN TEXT=http PORT "LOCAL=%([.:]443)$" state=LISTEN TEXT=https PORT STATE=LISTENING MIN=0 TRACK=Listen TEXT=Listen Display output: [red] http (found 0, req. 1 or more) [red] https (found 0, req. 1 or more) [green] Listen (found 43, req. none) Active Connections Proto Local Address Foreign Address State TCP 0.0.0.0:80 0.0.0.0:0 LISTENING TCP 0.0.0.0:135 0.0.0.0:0 LISTENING TCP 0.0.0.0:443 0.0.0.0:0 LISTENING TCP 0.0.0.0:445 0.0.0.0:0 LISTENING ... TCP [::]:80 [::]:0 LISTENING TCP [::]:135 [::]:0 LISTENING TCP [::]:443 [::]:0 LISTENING UDP 0.0.0.0:123 *:* UDP 0.0.0.0:500 *:* UDP 0.0.0.0:4500 *:* UDP 0.0.0.0:5355 *:* UDP 0.0.0.0:11211 *:* ... analysis.cfg lines for unix host HOST=unixhost.blah PORT "LOCAL=%([.:]80)$" state=LISTEN TEXT=http PORT "LOCAL=%([.:]22)$" state=LISTEN TEXT=ssh Display output: [green] http (found 3, req. 1 or more) [green] ssh (found 2, req. 1 or more) tcp4 0 0 127.0.0.1.80 127.0.0.1.54675 TIME_WAIT tcp4 0 0 *.1984 *.* LISTEN tcp4 0 0 *.22 *.* LISTEN tcp4 0 0 *.* *.* CLOSED tcp4 0 0 *.80 *.* LISTEN tcp4 0 0 10.0.1.1.80 *.* LISTEN tcp6 0 0 *.22 *.* LISTEN ..... A cursory glance at "xymond/xymond_client.c" found a "localcol" being defined as 4: 1989: int localcol = 4, remotecol = 5, statecol = 6, portcolor = COL_GREEN; I am not sure how it is handling the different number of columns from windows "netstat -an" which does not have the Recv-Q Send-Q columns present? Any help/advice appreciated! Cheers Gavin Stone-Tolcher, IT Support Officer, Network Operations and Incident Response Information Technology Services The University of Queensland Level 4, Prentice Building, St Lucia 4072 T: +61 7 334 66645, M: +61 401 140 838 E: g.stone-tolcher at its.uq.edu.au<mailto:g.stone-tolcher at its.uq.edu.au> W: www.its.uq.edu.au<http://www.its.uq.edu.au> ITS: Service. Team. Accountability. Results. IMPORTANT: This email and any attachments are intended solely for the addressee(s), contain copyright material and are confidential. We do not waive any legal privilege or rights in respect of copyright or confidentiality. Except as intended addressees are otherwise permitted, you do not have permission to use, disclose, reproduce or communicate any part of this email or its attachments. Statements, opinions and information not related to the official business of The University of Queensland are neither given nor endorsed by us. By using this email (including accessing any attachments or links) you agree we are not liable for any loss or damage of any kind arising in connection with any electronic defect, virus or other malicious code we did not intentionally include. Please consider the environment before printing this email. CRICOS Code 00025B

2 2

Two basic questions
by mshort＠corelogic.com 17 Jul '15

17 Jul '15

Hello All, I'm a new subscriber, moving from an old BB setup to Xymon. I had two questions that I hadn't been able to find the answer to. 1) How do you setup an external script to run only on certain hosts? I didn't see anything like the bb-bbexttab file. It seems if a script isn't present in the $XYMHOME/ext/ directory, it can't run. But that's not as clean a solution; makes it had to have one set of files to push to all clients. So what's the preferred method? 2) How do you read system logfiles on Linux hosts, when the files are root-only and the monitoring runs as user xymon? This is for Red Hat systems, where things like /var/log/messages are perm 700. I could set the perms to 744, but that's likely to get changed back when the logs rotate or the systems is patched. Do you run Xymon as root user on Linux systems? I didn't think so, or the install instructions wouldn't have you create a xymon user and group. So is there a better way? Thanks in advance for your help. ****************************************************************************************** This message may contain confidential or proprietary information intended only for the use of the addressee(s) named above or may contain information that is legally privileged. If you are not the intended addressee, or the person responsible for delivering it to the intended addressee, you are hereby notified that reading, disseminating, distributing or copying this message is strictly prohibited. If you have received this message by mistake, please immediately notify us by replying to the message and delete the original message and any copies immediately thereafter. Thank you. ****************************************************************************************** CLLD

3 5