January 2017 - Xymon - Xymon mailman web

Capture and send data To Elasticsearch and Kibana
by francois.outteryck＠soprasteria.com 27 Sep '22

27 Sep '22

Hi Peter, Examples of task.cfg configuration with status and stachg events : [capture_status] ENVFILE /home/xymon/server/etc/xymonserver.cfg NEED xymond CMD xymond_channel --channel=status /home/xymon/server/ext/capture.sh --log=$XYMONSERVERLOGS/xymond_task_capture.log LOGFILE /var/log/xymon/lunch_capture.log [capture_stachg] ENVFILE /home/xymon/server/etc/xymonserver.cfg NEED xymond CMD xymond_channel --channel=stachg /home/xymon/server/ext/capture.sh --log=$XYMONSERVERLOGS/xymond_task_capture_stachg.log LOGFILE /var/log/xymon/lunch_capture_stachg.log Capture.sh script simple made to call a php script : #!/bin/sh /usr/bin/php /var/www/kibana/capture.php $@ </dev/stdin >>/var/log/xymon/xymond_capture_php.log 2>&1 And capture.php make all job : <?php date_default_timezone_set('Europe/Paris'); $TODO=FALSE; $ARRAY_COLLECTOR=array(); $COLLECTOR=null; while (FALSE !== ($line = fgets(STDIN))) if ($line!='') { $tab=reduce_array($line); if (sizeof($tab)==0) continue; if ($line=="@@\n") { if ($TODO) { $metadata['Message']=implode('',$ARRAY_COLLECTOR); SendData($id,$metadata,$COLLECTOR); $TODO=FALSE; $COLLECTOR=null; } } else if (substr($tab[0],0,8)=='@@stachg') { if ($TODO) { $metadata['Message']=implode('',$ARRAY_COLLECTOR); SendData($id,$metadata,$COLLECTOR); $TODO=FALSE; $COLLECTOR=null; } /* 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 /* @@stachg#seq|timestamp|sender|origin|hostname|testname|expiretime|color|prevcolor|changetime|disabletime|disablemsg|downtimeactive|clienttstamp|modifiers */ $COLLECTOR='stachg'; $metadata=explode('|',$tab[0]); $sender=$metadata[2]; $hostname=$metadata[3]; $metadata=array( 'Collector'=>$COLLECTOR, 'Sender'=>$metadata[2], 'ServerName'=>$metadata[4], 'Type'=>$metadata[5], 'Status'=>$metadata[7], 'PrevStatus'=>$metadata[8], 'LogTime'=>$metadata[1], 'ValidTime'=>$metadata[6], 'LastChange'=>$metadata[9], 'DisableTime'=>$metadata[10], 'DisableMSg'=>$metadata[11], 'Message'=>'' ); $id=$metadata['ServerName'].'_'.$metadata['Type'].'_'.$meta['LastChange']; $ARRAY_COLLECTOR=null; $ARRAY_COLLECTOR=array(); $TODO=TRUE; } else if (substr($tab[0],0,8)=='@@status') { if ($TODO) { $metadata['Message']=implode('',$ARRAY_COLLECTOR); SendData($id,$metadata,$COLLECTOR); $TODO=FALSE; $COLLECTOR=null; } /* 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 /* @@status|timestamp|sender|origin|hostname|testname|expiretime|color|testflags|prevcolor|changetime|ackexpiretime|ackmessage|disableexpiretime|disablemessage|clientmsgtstamp|flapping */ $COLLECTOR='status'; $metadata=explode('|',$tab[0]); $sender=$metadata[2]; $hostname=$metadata[3]; $metadata=array( 'Collector'=>$COLLECTOR, 'Sender'=>$metadata[2], 'ServerName'=>$metadata[4], 'Type'=>$metadata[5], 'Status'=>$metadata[7], 'LogTime'=>$metadata[1], 'ValidTime'=>$metadata[6], 'LastChange'=>$metadata[10], 'DisableTime'=>$metadata[13], 'AckTime'=>$metadata[11], 'DisableMSg'=>$metadata[14], 'Message'=>'' ); $id=$metadata['ServerName'].'.'.$metadata['Type']; $ARRAY_COLLECTOR=null; $ARRAY_COLLECTOR=array(); $TODO=TRUE; } else { if (substr($tab[0],0,2)=='@@') { if ($TODO) { $metadata['Message']=implode('',$ARRAY_COLLECTOR); SendData($id,$metadata,$COLLECTOR); $TODO=FALSE; $COLLECTOR=null; } } if ($TODO) $ARRAY_COLLECTOR[]=$line; } } function CallAPI($method, $url, $data = false) { $curl = curl_init(); switch ($method) { case "POST": curl_setopt($curl, CURLOPT_POST, 1); if ($data) curl_setopt($curl, CURLOPT_POSTFIELDS, $data); break; case "PUT": curl_setopt($curl, CURLOPT_PUT, 1); break; default: if ($data) $url = sprintf("%s?%s", $url, http_build_query($data)); } curl_setopt($curl, CURLOPT_URL, $url); curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); $result = curl_exec($curl); curl_close($curl); return $result; } function reduce_array($line) { $tab=explode(' ',$line); $nr=array(); for ($i=0;$i<sizeof($tab);$i++) if (trim($tab[$i])!='') $nr[]=trim($tab[$i]); return $nr; } function SendData($id,$cc,$COLLECTOR=null) { if ($COLLECTOR==null) return; $d=new DateTime(); if ($cc['LastChange']!='') {$d->setTimestamp($cc['LastChange']);$cc['LastChange']=$d->format('c');} else {$cc['LastChange']='NaN';} if ($cc['LogTime']!='') {$d->setTimestamp($cc['LogTime']);$cc['LogTime']=$d->format('c');} else {$cc['LogTime']='NaN';} if ($cc['ValidTime']!='') {$d->setTimestamp($cc['ValidTime']);$cc['ValidTime']=$d->format('c');} else {$cc['ValidTime']='NaN';} if (isset($cc['AckTime'])) { if ($cc['AckTime']!='') {$d->setTimestamp($cc['AckTime']);$cc['AckTime']=$d->format('c');} else {$cc['AckTime']='NaN';} } if ($cc['DisableTime']!='') {$d->setTimestamp($cc['DisableTime']);$cc['DisableTime']=$d->format('c'); } else {$cc['DisableTime']='NaN';} $data = json_encode($cc); $data=str_replace('"NaN"','null',$data); echo $id.'=>'.$data."\n"; $url="http://127.0.0.1:9200/xymon/".$COLLECTOR."/".$id; $res=CallAPI("POST",$url,$data); } ?> De : Peter Welter [mailto:peter.welter at gmail.com] Envoyé : vendredi 23 décembre 2016 14:45 À : OUTTERYCK Francois Cc : xymon at xymon.com Objet : Re: [Xymon] Capture and send data To Elasticsearch and Kibana Hi Francois, Looks good, your sample. I am very interested in your integration efforts. How do you select what is going to be inserted in the ELK stack? All status messages that Xymon receives? And could you provide an example of the xymond_channel block in you task.cfg and the capture.sh? Tasks.cfg probably is something like: [es] ... CMD xymond_channel --channel=status /home/xymon/server/ext/capture.sh ... And the capture.sh contains some curl script to the ELK stack? Regards, Peter 2016-12-14 18:15 GMT+01:00 OUTTERYCK Francois <francois.outteryck at soprasteria.com<mailto:francois.outteryck at soprasteria.com>>: Hello, I just do a small work with Xymon in order to send status data to Elasticsearch. And with Kibana reporting is easy. How it work ? Simple : With capture of status message (xymond_channel --channel=status /home/xymon/server/ext/capture.sh ) script encode status message in json and send as document in elasticSearch. Example of Mapping document status in elasticsearch : "status"=> array( "properties"=>array( "ServerName"=> array("type"=> "string"), "Type"=> array("type"=> "string"), "Status"=> array("type"=> "string"), "LastChange"=> array("type"=> "date","format"=> "strict_date_optional_time"), "LogTime"=> array("type"=> "date","format"=> "strict_date_optional_time"), "ValidTime"=> array("type"=> "date","format"=> "strict_date_optional_time"), "AckTime"=> array("type"=> "date","format"=> "strict_date_optional_time"), "DisableTime"=> array("type"=> "date","format"=> "strict_date_optional_time"), "Sender"=> array("type"=> "string"), "DisableMSg"=> array("type"=> "string"), "Message"=> array("type"=> "string") ) ) All is done with a 130 lines script. So simple to integrate in existing environment and so powerfull usage of Xymon/elasticsearch/Kibana For those who don’t know kibana : https://www.elastic.co/products/kibana [cid:image001.png at 01D25EA3.F74489F0] _______________________________________________ Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon

2 1

Need help with upgrade from 4.3.17 to 4.3.20
by vadud3＠gmail.com 17 Jul '18

17 Jul '18

I am not sure what I need to after the upgrade for the columns to show data instead of something like this ELF �� >� �� @��@�� Þ ��@�8� �@�%�"� �� @��@�@��@�@��ø ��ø �� 8 ��8 @��8 @�� @��@��ää��ää�� I removed FollowSymLinks from xymon-apache.conf file under /xymon-cgi/ and /xymon-seccgi/ and restarted. I am using ubuntu -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

9 29

Recovery notifications not generated when test goes red->yellow->green?
by hobbit＠elyograg.org 13 Dec '17

13 Dec '17

It appears that if a test goes from the red state, to the yellow state, and then to the green state, no recovery notification is sent. If it goes directly from red to green, then we get a notification. Initial notifications are only sent on RED and PURPLE. We have recovery notifications configured for most tests. Xymon server is 4.3.21, built using makerpm.sh and installed with yum on CentOS 6. I do not know whether our previous version (4.3.14) behaved similarly. It was almost an accident that I discovered the problem at all. Is this expected behavior? If it is, can I change it? Thanks, Shawn

4 3

CLASS not working as expected
by steve＠opendium.com 11 Dec '17

11 Dec '17

Under Xymon 4.3.25, I'm trying to use the class attribute to group hosts together by what monitoring and alerting they need. So for each host I have a hosts.cfg line like: 192.0.2.1 somehost.example.com # CLASS:Foo trace ssh Then in analysis.cfg I have: CLASS=Foo PROC this PROC that And alerts.cfg has: CLASS=Foo SCRIPT=blah However, the tests in analysis.cfg are not being applied. If I replace CLASS=Foo with HOST=somehost.example.com they work as expected. Also, CLASS=linux seems to work, which implies that the CLASS: directive in the hosts.cfg is being ignored. As far as I can tell, the messages on xymon's "client" channel don't contain the class name that was specified in hosts.cfg, so xymond_client is using the class that the client sent instead. Other channels, such as "status" do include the class name that was specified in hosts.cfg. Am I misunderstanding how the class is supposed to be used, or is this a bug? -- - Steve Hill Technical Director Opendium Limited http://www.opendium.com Direct contacts: Instant messager: xmpp:steve at opendium.com Email: steve at opendium.com Phone: sip:steve at opendium.com Sales / enquiries contacts: Email: sales at opendium.com Phone: +44-1792-824568 / sip:sales at opendium.com Support contacts: Email: support at opendium.com Phone: +44-1792-825748 / sip:support at opendium.com

7 9

DNS failures causing "runtime longer than time limit"
by jlaidman＠rebel-it.com.au 11 Sep '17

11 Sep '17

Hi I'm running Xymon v4.3.10 on Linux, and I'm quite sure it's compiled with c-ares support. I have 12 new DNS servers that were added to Xymon about one month ago. All of my server entries in hosts.cfg have "testip". The tasks.cfg runs xymonet with "--dns-timeout=3". The hosts entries look like so: 10.10.10.1 dnshost1.example.com # testip dns=NS:example.com,SOA: example.com About a week ago, connectivity to all of these servers failed, and at the same time, the xymonnet run time jumped from less than 15 seconds to about 330 seconds, so about 315 seconds extra. The xymonnet page says 295 seconds is taken up by DNS tests. If the increase in time taken is about 315 and is entirely due to the 12 servers failing, then each failed server is adding about 26 seconds to the total run time. I don't think this should be happening like this. With two DNS checks per server, the DNS checks should be taking 6 seconds each to time-out, not 26. If I run xymonnet with "--timing --no-update" and specify only one hostname, I can view the results and the timing. This shows that the ping check gets reported after about 3 seconds, and then the DNS tests are executed and take 26 seconds total. My naiive assumption was that when a server failed a ping (and didn't have "noclear" defined in hosts.cfg) then the network checks would be skipped. On re-reading the man page for hosts.cfg, it dawned on me that a failed ping simply suppresses failed test /results/, but doesn't stop the tests from being run. So the real problem is that the "--dns-timeout=3" is not being taken into consideration by xymonnet. If I run xymonnet with "--debug" it tells me: 1900 2015-06-03 12:02:20 ares_search: tlookup='example.com', class=1, type=2 1900 2015-06-03 12:02:20 ares_search: tlookup='example.com', class=1, type=6 1900 2015-06-03 12:02:20 Processing 0 DNS lookups with ARES 1900 2015-06-03 12:02:46 Finished ARES queue after loop 423 This is peculiar. Why would it say "processing 0 DNS lookups" when there are two lookups to test? Could this be because xymonnet hasn't actually been built with ARES support and I didn't know it? Is there a good way to tell? If I add "--no-ares" I get the same results perhaps suggesting a lack of ARES support. On the other hand, if I add "timeout:3" and "attempts:1" into resolv.conf, I also get the same results. If I run "nm /path/to/xymonnet | grep gethostby" it returns "ares_gethostbyname". Just for fun, I compiled Xymon v4.3.21 and ran the xymonnet binary from there, with no change in behaviour. I also tried removing the "--dns-timeout" option so that it defaults to 30 seconds, but still no change - 26 seconds for two DNS tests. So, I'm not really sure what the problem is, but xymonnet certainly isn't behaving as I would expect. Cheers Jeremy

2 3

TCP Connection Issues?
by skadz＠skadz.com 15 Feb '17

15 Feb '17

Anyone ever seen this? I have a few hundred servers in my xymon setup and have about 200 http checks, as well as some other server side scripts that check things via tcp. Recently all of my http and telnet connections will start to flap for 5-10 mins and cause all/most of my http and telnet checks to go red. I added a delayred to stop the paging, but I am still getting a wonderful Christmas tree a few times a day, just yellow now. It feels like an issue with the number of file handles or maybe the number of tcp connections, but I can't seem to get the right formula to fix it up. Anyone have any suggestions? Skadz PS - Running 4.3.17. Haven't had a time to upgrade in a while (and had some issues with some of the web changes last time I tried)

2 2

Automate Daily Report : CSV Output
by Brad.Radjoo＠mtn.com 07 Feb '17

07 Feb '17

G'Day, Is there a way to generate an availability report of the past 24 hours in csv format and set that to e-mail myself on a daily basis ? I notice that it uses the xymon-cgi/report.sh script Kind Regards, - - - - Bradley "This email is confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person as to do so could be a breach of confidentiality. Thank you for your co-operation. " "The information contained in this message is intended solely for the individual to whom it is specifically and originally addressed. This message and its contents may contain confidential or privileged information from MTN Group. If you are not the intended recipient, you are hereby notified that any disclosure or distribution, is strictly prohibited. If you receive this email in error, please notify MTN Group immediately and delete it. MTN Group does not accept any liability or responsibility if action is taken in reliance on the contents of this information . Note that all personal emails are not authorized by MTN Group. "

2 4

Absolute free space value for DISK gone rogue?
by David.Mills＠hhsc.state.tx.us 03 Feb '17

03 Feb '17

All - I have a file system on a host which is alarming "red" even though the file system is brand new / empty. In the notes below, the mis-reporting filesystem is "/ora_data_estdmprd" The stanza in analysis.cfg that covers it is: HOST=pwdu008 # Stand-alone prod DB hosts: won't have shared FS's DISK /ora_bin 85 95 # 85%/95% = CRQ000000013038 DISK /ora_data 85 95 # 85%/95% = CRQ000000013038 DISK /ora_admin 85 95 # 85%/95% = CRQ000000013038 DISK /ora_temp 85 95 # 85%/95% = CRQ000000013038 DISK /ora_redoa 85 95 # 85%/95% = CRQ000000013038 DISK /ora_redob 85 95 # 85%/95% = CRQ000000013038 DISK /ora_arch 85 95 # 85%/95% = CRQ000000013038 DISK /ora_data_finprod1 85 95 # 85%/95% = CRQ000000013038 DISK /ora_finbin 85 95 # 85%/95% = CRQ000000013038 DISK %/ora_data_perpr 85 95 # 85%/95% = CRQ000000013038 DISK %/ora_ 262144000U 209715200U # I.e. Yellow: <= 256GB; Red <= 204.8GB The output from the "Disk" detail page for that host was: status pwdu008.disk yellow Tue Jan 31 14:06:20 CST 2017 - Filesystems NOT ok &red /ora_data_estdmprd (3045500329 units free) has reached the CRITICAL level (209715200 units) Filesystem kbytes used avail capacity Mounted on /dev/md/dsk/d1 16525754 7879972 8480525 49% / /dev/md/dsk/d4 20655025 2723628 17724847 14% /var /dev/md/dsk/d7 17197130 2708305 14316854 16% /export /dev/odm 0 0 0 0% /dev/odm sharefs 0 0 0 0% /etc/dfs/sharetab swap 147391344 1936 147389408 1% /etc/svc/volatile swap 147396208 6840 147389368 1% /tmp swap 147389344 0 147389344 0% /dev/vx/dmp swap 147384744 0 147384744 0% /dev/vx/rdmp /dev/vx/dsk/ORA_DATA/ora_data 249925632 5610862 229045230 3% /ora_data /dev/vx/dsk/ORA_ARCH/ora_arch 249958400 685078 233695152 1% /ora_arch /dev/vx/dsk/ORA_BIN/ora_bin 124930048 54292389 66222935 46% /ora_bin /dev/vx/dsk/ORA_ADMIN/ora_admin 62465024 202884 58405611 1% /ora_admin /dev/vx/dsk/ORA_REDOA/ora_redoa 124962816 833493 116371374 1% /ora_redoa /dev/vx/dsk/ORA_ARCH_ESTDMPRD/ora_arch_estdmprd 1749710848 966260000 777330408 56% /ora_arch_estdmprd /dev/vx/dsk/ORA_TEMP/ora_temp 999961600 521070 936975632 1% /ora_temp /dev/vx/dsk/ORA_REDOB/ora_redob 124962816 423857 116755408 1% /ora_redob /dev/vx/dsk/ORA_DATA_DMSTGPRD/ora_data_dmstgprd 4999072768 1751121 4728890415 1% /ora_data_dmstgprd /dev/vx/dsk/ORA_DATA_ESTDMPRD/oradata_estdmprd01 3249462272 929825 3045500329 1% /ora_data_estdmprd In the report above, we see that the "critical" threshold (in KB on Solaris) is 209,715,200 free space (about 256 GB), but the actual free space for that file system is 3,045,500,329 (around 3 TB). Something is clearly not right! Also, note the file system second-from-the bottom: /ora_data_dmstgprd. It has _even more_ free space and should be triggered by the same rule in analysis.cfg (DISK %/ora_ ...) yet it is not flagged as having a problem. As a stop-gap to keep the filesystem from alarming, I've had to change the "DISK" rule in analysis.cfg to percentages (85 95) which makes everything go green. Particulars: Xymon v. 4.3.3 running on Solaris 10 host. (Have also tried restarting the server without impact.) Any thoughts? Thanks! david ~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 595-1238 (mobile)

1 1

linux client disk warning levels
by kspringer＠innovateteam.com 02 Feb '17

02 Feb '17

An HTML attachment was scrubbed... URL: <http://lists.xymon.com/pipermail/xymon/attachments/20170130/6cf79fc2/attach…>

7 8

Graphs not showing
by Wim.Nelis＠nlr.nl 31 Jan '17

31 Jan '17

Hello, my previous post was mangled because there were some snippets of HTML in it. Therefore the message is posted again, this time without HTML. the status of audio streams through an Open Digital Radio multiplexer is monitored in Xymon. A Perl script retrieves the status and statistics from the multiplexer, and sent them to Xymon. The statistics are to be shown in graphs. However, while the RR databases are build, no graphs are shown. I am using debian 8.6 with xymon 4.3.17. (An update to version 4.3.27 failed, as the configure script does not recognise rrdtool 1.4.8, resulting in a version of xymon which does not support trend graphs.) A perl script retrieves the status of (the input channels of) the multiplexer, and sends the message to xymon. The statistics are included in the format which is also used by devmon. The last part of the status message of test odr_mux contains the following as HTML comment: linecount=2 DEVMON RRD: odr 0 0 DS:PeakLeft:GAUGE:600:-100:100 DS:PeakRight:GAUGE:600:-100:100 \ DS:Underrun:DERIVE:600:0:U DS:Overrun:DERIVE:600:0:U \ DS:BufferMin:GAUGE:600:-1:U DS:BufferMax:GAUGE:600:0:U sub-ozo -90:-90:176972:0:-1:0 sub-urk -90:-90:176972:0:-1:0 In xymonserver.cfg two lines are adapted: TEST2RRD=".... ,odr_mux=devmon" GRAPHS=" .... ,odr_mux::1" The statistics are entered in RRD's: root at debian:/var/lib/xymon/rrd/OzoNop# ls -l odr* -rw-r--r-- 1 xymon xymon 114560 Jan 8 19:04 odr.sub-ozo.rrd -rw-r--r-- 1 xymon xymon 114560 Jan 8 19:04 odr.sub-urk.rrd root at debian:/var/lib/xymon/rrd/OzoNop# The graph definition is: [odr_mux] FNPATTERN ^odr\.(.+)\.rrd$ TITLE , Frame loss rate YAXIS Rate [/s] DEF:ur at RRDIDX@=@RRDFN@:Underrun:AVERAGE DEF:or at RRDIDX@=@RRDFN@:Overrun:AVERAGE LINE1:ur at RRDIDX@#FF0000:@RRDPARAM@ underrun GPRINT:ur at RRDIDX@:MIN:Min \: %5.1lf %s GPRINT:ur at RRDIDX@:MAX:Max \: %5.1lf %s GPRINT:ur at RRDIDX@:AVERAGE:Avg \: %5.1lf %s GPRINT:ur at RRDIDX@:LAST:Cur \: %5.1lf %s\n LINE1:or at RRDIDX@#00FF00:@RRDPARAM@ overrun GPRINT:or at RRDIDX@:MIN:Min \: %5.1lf %s GPRINT:or at RRDIDX@:MAX:Max \: %5.1lf %s GPRINT:or at RRDIDX@:AVERAGE:Avg \: %5.1lf %s GPRINT:or at RRDIDX@:LAST:Cur \: %5.1lf %s\n The web page of this test shows the two tables, but no graphs. Only the magnifying glasses, which should be on the right of a graph, are shown. When clicking on the magnifying glass, an empty page is shown. The source code of this page is two lines long, the first one specifying the title to be "Invalid request", the second on showing an empty body. Removing the graph definition results in a link to be displayed, and the magnifying glass shows an appropriate error message. Further experiments showed that if all multi-graph stuff is removed (that is, FNPATTERN and the @RRD...@ parameters) a single graph is shown. Extending the graph definition with FNPATTERN causes to problem to surface again. The logfiles of xymon do not contain messages related to this problem, as far as I can see. Enabling debugging for xymond_rrd, the one spawn by the status channel reader, does not give any helpful information. All the other graphs, including one multi-graph (test memory), of this host are displayed. The problem is only visible in test odr_mux. Can anyone give me a hint on how to proceed to solve this issue? Regards, Wim Nelis.

3 4