August 2015 - Xymon - Xymon mailman web

Syntax checking function
by john.thurston＠alaska.gov 27 Aug '15

27 Aug '15

I've been looking in the man pages and can't find what I'm looking for. I hope I'm just looking with my eyes closed. Some applications (ISC BIND is one that comes to mind) provide a dry-run or syntax-checking function for their config and data files. For BIND, I can run named-checkconf to test the syntax of the application configuration files. I also have named-checkzone which will confirm a zone file passes muster. Is there such a function available for the Xymon tasks or hosts? -- Do things because you should, not just because you can. John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska

2 1

xymon alert frequency
by snowolfex＠yahoo.com 27 Aug '15

27 Aug '15

Testing a script - and it seems that the email alerts only come out once every ten minutes. Am I seeing this wrong? Is there somewhere to adjust this?

1 0

Xymon WinPSClient - questions and comments
by spah＠syntec.co.uk 27 Aug '15

27 Aug '15

Hi, 1. Xymon WinPSClient - can it send custom status messages like BBNT and BBWin? In BBNT, we use the Saved Logs Location together with the Logs Timer to send custom status messages to the Xymon server. This is defined with different language in the readme file: PluginTimer: Waiting period between plugins status files loading Default is 60 seconds This means that BBNT waits XXX seconds before checking if any status files were written to the plugin directory (the ExternalPath variable) by plugin programs In BBWin, this also works with the: <setting name="logstimer" value="60s" /> Although you have to set the tmppath in the BBWin registry folder. BBNT more info: ExternalPath: Directory where external programs can save BB status messages to be sent by BBNT. This key is set with the value entered in the "Saved Log Location" field using the bbntcfg utility. This acts as a plugin facility where external program create their own status messages. Files that have no extensions (the file name should be the service name) are sent to BB. So first create your file with an extension and when it's ready to be sent over then rename it to the service name only. After processing, the status file is removed. ... The full description is in the attached readme file. Is there an equivalent in the XymonPSClient? Note that the client does not need to run anything, just look in a given folder for files (without extensions) and send any it finds to the Xymon server (preferably with an interval that is configurable separately to the normal interval). 2. Also some points about making the installation process clearer and easier... In the README.txt, in the 'Installing as service' and 'Client configuration' it is not clear if those commands should be run from a Command Prompt, Elevated Command Prompt, PowerShell prompt or elevated PowerShell prompt. This is much clearer (for installation anyway) in the XymonPSClient.doc file. Maybe the README.txt should say that fuller and more up to date instructions will be kept in the XymonPSClient.doc file? A note specifying that XymonPSClient.exe is actually ABC_SrvAny.exe and not used anymore might be helpful too. Also, I was originally trying to install this on a 32-bit version of Windows. So the instructions didn't work as only a 64-bit version of nssm.exe was provided. This wasted a lot of our time when we first tried this. Either a note about this, or putting both 32-bit and 64-bit versions in subversion would be helpful here. I think it was also necessary to unblock some files. When you download files in Windows, they start off blocked and you need to go into their file properties and unblock them. A note about this would be helpful. 3. Xymon WinPSClient - can it monitor services like BBNT and BBWin? The documentation has 'servicecheck:SERVICENAME:DURATION', but it only talks about restarting the service. I don't want services restarted automatically, I just want a red or yellow alarm. (And for some Windows services, they can't be monitored using procs.) In BBNT and BBWin it was much more configurable. In BBWin: name value alarmcolor You can set the default alarm color for the svcs rules. It should be set to red or yellow. Default is yellow. alwaysgreen Possible values are "true" or "false". If true, it will always send green status. Default is false. autoreset Possible values are "true" or "false". If true, it will try to restart each service in automatic startup state which would be stopped. A report with the default alarm color is sent when it try to restart one of these service. Default is false comment A simple string which appear in the report to add a small information about the services monitored if its name is not easy to understand. This parameter is optionnal delay Svcs agent will wait the period defined in the delay setting before doing monitoring if the computer has restarted recently. (It may be useful for services which would start later than BBWin or for services which take a lot of time to start). You can specify a unit after the numeric value : "m" for minutes, "h" for hours, "d" for days. Default is 5 minutes testname You can set the testname that will be sent to the monitoring server (column name in the bbdisplay) Default is "svcs" The BBNT config is like this (so it is more similar to PS and more succinct, but the extra BBWin parameters are also helpful): Terminal Services:R:R:N;Automatic Updates:R:R:N - documentation in attached readme file. Thanks for all your hard work! It would be really nice if all the functionality of BBWin is replicated in XymonPSClient as then the decision over which to go with is almost a no-brainer! Kind regards, SebA

2 1

New to Xymon
by jlaidman＠rebel-it.com.au 27 Aug '15

27 Aug '15

(Please keep the List in your replies.) I haven't installed a recent package on a Xymon server. I think you only need the server package, and it includes the client components, so you should be OK, but I just wanted to be sure. The Xymon server package installs an Apache config snippet to be included. I think it will be /etc/httpd/conf.d/xymon-apache.conf or similar. But it won't get activated until Apache is restarted, and I'm not sure if the package restarts Apache for you. On 27 August 2015 at 21:13, Steve Lamb <slamb at tyreman.co.uk> wrote: > apache is ok, I get a directory listing if i go to http://<host>/xymon > > Installed packages, I just did a dnf list xymon and installed the only > xymon package available. That then downloaded and installed other > dependencies. > > Thats basically all I have done though, are further steps required ? > > > On 27/08/2015 12:06, Jeremy Laidman wrote: > > Your Apache works OK otherwise? Do you get any error message on your > browser when you go to <http://xymon.server.name/xymon/> > http://xymon.server.name/xymon/? Perhaps have a look at the Apache > error_log file to see if it's complaining about something? > > What packages did you install? > > On 27 August 2015 at 20:38, Steve Lamb <slamb at tyreman.co.uk> wrote: > >> I've installed xymon on fedora 22 using the terabithia.repo. The >> installation completed without errors. >> >> Do I need to take further steps to get it working ? I can't seem to bring >> up a xymon webpage. >> >> >> _______________________________________________ >> Xymon mailing list >> Xymon at xymon.com >> http://lists.xymon.com/mailman/listinfo/xymon >> > > >

1 0

Custom 60 second interval test still has 5 min interval graph
by Scot.Kreienkamp＠la-z-boy.com 27 Aug '15

27 Aug '15

Hey everyone, I created a custom test and RRD definition because I wanted to reduce the interval time on a few graphs from 5 minute resolution to 1 minute. I got the data to be logged once every minute, which I verified is hitting the RRD. The problem is that when I view the graph it's still got a resolution of 5 minutes. Here's the partial RRDTool dump:  <row><v> 1.0033333333e+01 </v></row>  <row><v> 1.0550000000e+01 </v></row>  <row><v> 5.6333333333e+00 </v></row>  <row><v> 7.3166666667e+00 </v></row>  <row><v> 7.5000000000e+00 </v></row>  <row><v> 6.2500000000e+00 </v></row>  <row><v> 7.5500000000e+00 </v></row>  <row><v> 9.3333333333e+00 </v></row>  <row><v> 5.2666666667e+00 </v></row>  <row><v> 7.0666666667e+00 </v></row>  <row><v> 6.4000000000e+00 </v></row>  <row><v> 6.4000000000e+00 </v></row>  <row><v> 1.3000000000e+01 </v></row>  <row><v> 9.2000000000e+00 </v></row>  <row><v> 3.4650000000e+01 </v></row>  <row><v> 5.7200000000e+01 </v></row>  <row><v> 5.0283333333e+01 </v></row>  <row><v> 4.4100000000e+01 </v></row>  <row><v> 4.1500000000e+01 </v></row>  <row><v> 2.4500000000e+01 </v></row>  <row><v> 1.2000000000e+01 </v></row> So the data is getting logged every 60 seconds. It's also rolling up to the 5 second, etc, values. Here's the line from xymonserver.cfg: SPLITNCV_cpustat="*:GAUGE:180:0:100" #Configure for 60 second intervals, range of 0-100 Here's the lines from rrddefinitions.cfg: [cpustat] -s 60 RRA:AVERAGE:0.1:1:120 RRA:AVERAGE:0.5:5:576 RRA:AVERAGE:0.5:30:576 RRA:AVERAGE:0.5:120:576 RRA:AVERAGE:0.5:1440:576 RRA:MAX:0.1:1:120 RRA:MAX:0.5:5:576 RRA:MAX:0.5:30:576 RRA:MAX:0.5:120:576 RRA:MAX:0.5:1440:576 Here's the graph definition: [cpustat] FNPATTERN ^cpustat.(.+).rrd TITLE % CPU usage YAXIS Percent DEF:p at RRDIDX@=@RRDFN@:lambda:MAX LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@ GPRINT:p at RRDIDX@:LAST: \: %5.0lf (cur) GPRINT:p at RRDIDX@:MAX: \: %5.0lf (max) GPRINT:p at RRDIDX@:MIN: \: %5.0lf (min) GPRINT:p at RRDIDX@:AVERAGE: \: %5.0lf (avg)\n I had read that RRDTool should use the lowest resolution available, which is 1 minute for this RRD. So why are the graphs not going less than 5 minute intervals when zooming in? Any ideas? Thanks! Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * 7349151444 | * Scot.Kreienkamp at la-z-boy.com<mailto:%7BE-mail%7D> www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> | facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy> | twitter.com/lazboy<https://twitter.com/lazboy> | youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy> [cid:lzbVertical_hres.jpg] This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

4 6

Question: alternative(s) to sms/mms gateways
by glauber.ribeiro＠experian.com 26 Aug '15

26 Aug '15

Question for the Xymon list: What have you used, other than email-to-sms/mms gateway, to send xymon alerts to cell phones? I've been hit a few times with unreliable email to mms gateway (TMobile). I know about pushover.net. Are there any others? Thanks, glauber

6 5

Thoughts on Usefulness/Reliability of Purple Alerts
by mvanderw＠nd.edu 26 Aug '15

26 Aug '15

This is primarily for Henrik and J.C., but anyone else is free to chime in their thoughts on this as well! *Background: We have a Xymon server (the latest Terabithia RPM on RHEL 7) in production that monitors around 1950 hosts (and consistently growing). About a week ago, we experienced some pretty bad purple alert storms in the middle of the night that were all false-positive alerts (over 300 alerts one night). For most of the tests that went purple, they went back to green at the next update interval. At this point, we've been unable to figure out a root cause behind this issue, but it hasn't happened again since early last week (all the easy, understandable possible causes has been ruled out: network load/bandwidth, CPU load of Xymon server and of Xymon clients affected, etc.). We have been using purple alerts for some time now, find them fairly reliable for the most part, and think they are useful, as machines hang or something similar (causing the Xymon client on the machine to stop being able to report to the Xymon server) and we don't get any red or yellow alerts for any other tests (sometimes a machine can hang but still have a network connection that can be successfully pinged by Xymon, we have found). We haven't had any major issues with false-positive purple alerts (for the most part), or any purple alert storms, since we started using them consistently for all our machines a couple years ago. I understand that when Xymon was first forked from Big Brother a long while back, it may have been noted that one big change from Big Brother was that you didn't need to do purple alerts (or something like that) and that it was discouraged to use purple alerts, as they were seen as widely unreliable. (I'm hearing this from a coworker of mine, who set up our original Xymon server some 5 years ago, but have been unable to find what he's referring to.) But from what I can see from the current documentation and the mailing list archives, I'm not seeing any place where the use of purple alerts is discouraged due to them being unreliable. *Question(s): So, I wanted to see what the current thinking/view regarding purple alerts and the use of purple alerts was by both the original main maintainer, Henrik, and the more current main maintainer, J.C. (at least of the current release). Are purple alerts still considered wholly unreliable, or even somewhat unreliable (or were they ever)? Are they discouraged in any way or fashion from being used? Have they caused issues for any of you on this list? Or vice versa: Have they worked well for you? I'm fully aware that this purple alert storm issue we had is just a one-off occurrence and we could have not more additional issues in the future with purple alerts. I understand that purple alerts are different than other alerts, like red and yellow alerts, in that it is an indication that the Xymon client has stopped working/reporting (on a per-test basis) to the Xymon server for some reason, rather than an issue from a specific test (e.g. with the CPU load, memory, etc.). *(Possible) Feature Request: In addition, I'd be interested if there was a way that you could only get one alert for a machine if say all the tests for that machine go purple, instead of an alert for each purple test. I don't believe this is possible currently, correct? Is this something that could possibly be implemented in the future? I understand if it's not or if it wouldn't be very easy. I appreciate your time in answering my questions and look forward to your input! (And apologies for the long-winded e-mail!) Thanks very much in advance!! -- Matt Vander Werf

3 3

xymongrep
by dewelker＠gmail.com 26 Aug '15

26 Aug '15

Anybody know how to xymongrep via a different port? I don't see options allowing for a port change. Thanks, David

2 2

Thoughts on Usefulness/Reliability of Purple Alerts
by cleaver＠terabithia.org 26 Aug '15

26 Aug '15

On Mon, August 24, 2015 10:15 am, Sean MacGuire wrote: > OK, I'll chime in... I wrote Big Brother so know something about > purple alerts. > > The purple alert was something no other monitoring system did, and > it took care of the problem of a BB (and Xymon) client dropping dead > and/or machines being in a zombie state (i.e. responding to pings but > otherwise hung). > > They're useful as a indication of something being wrong, vs a red or > yellow alert which provide a clear and actionable problem ststus. > > So purple alerts are awesome, unless your server has lost contact > with the clients reporting in and everyone goes purple at the same > time resulting in the "Massive Purple Explosion". > > Background explanation - the idea was to timestamp reports into the > future, and if a client doesn't report in by then, the validity of > the last report is in question - it's works the exact same way as > the expiration date on a carton of milk - the milk might not have > gone bad, yet, but you might want to check it before drinking it. > > > Matt Vander Werf wrote: >> This is primarily for Henrik and J.C., but anyone else is free to chime >> in their thoughts on this as well! >> >> *Background: >> We have a Xymon server (the latest Terabithia RPM on RHEL 7) in >> production that monitors around 1950 hosts (and consistently growing). >> About a week ago, we experienced some pretty bad purple alert storms in >> the middle of the night that were all false-positive alerts (over 300 >> alerts one night). For most of the tests that went purple, they went >> back to green at the next update interval. At this point, we've been >> unable to figure out a root cause behind this issue, but it hasn't >> happened again since early last week (all the easy, understandable >> possible causes has been ruled out: network load/bandwidth, CPU load of >> Xymon server and of Xymon clients affected, etc.). >> >> We have been using purple alerts for some time now, find them fairly >> reliable for the most part, and think they are useful, as machines hang >> or something similar (causing the Xymon client on the machine to stop >> being able to report to the Xymon server) and we don't get any red or >> yellow alerts for any other tests (sometimes a machine can hang but >> still have a network connection that can be successfully pinged by >> Xymon, we have found). We haven't had any major issues with >> false-positive purple alerts (for the most part), or any purple alert >> storms, since we started using them consistently for all our machines a >> couple years ago. >> >> I understand that when Xymon was first forked from Big Brother a long >> while back, it may have been noted that one big change from Big Brother >> was that you didn't need to do purple alerts (or something like that) >> and that it was discouraged to use purple alerts, as they were seen as >> widely unreliable. (I'm hearing this from a coworker of mine, who set up >> our original Xymon server some 5 years ago, but have been unable to find >> what he's referring to.) But from what I can see from the current >> documentation and the mailing list archives, I'm not seeing any place >> where the use of purple alerts is discouraged due to them being >> unreliable. >> >> *Question(s): >> So, I wanted to see what the current thinking/view regarding purple >> alerts and the use of purple alerts was by both the original main >> maintainer, Henrik, and the more current main maintainer, J.C. (at least >> of the current release). Are purple alerts still considered wholly >> unreliable, or even somewhat unreliable (or were they ever)? Are they >> discouraged in any way or fashion from being used? Have they caused >> issues for any of you on this list? Or vice versa: Have they worked well >> for you? I'm fully aware that this purple alert storm issue we had is >> just a one-off occurrence and we could have not more additional issues >> in the future with purple alerts. >> >> I understand that purple alerts are different than other alerts, like >> red and yellow alerts, in that it is an indication that the Xymon client >> has stopped working/reporting (on a per-test basis) to the Xymon server >> for some reason, rather than an issue from a specific test (e.g. with >> the CPU load, memory, etc.). >> >> *(Possible) Feature Request: >> In addition, I'd be interested if there was a way that you could only >> get one alert for a machine if say all the tests for that machine go >> purple, instead of an alert for each purple test. I don't believe this >> is possible currently, correct? Is this something that could possibly be >> implemented in the future? I understand if it's not or if it wouldn't be >> very easy. >> >> >> I appreciate your time in answering my questions and look forward to >> your input! (And apologies for the long-winded e-mail!) >> >> >> Thanks very much in advance!! >> >> -- >> Matt Vander Werf Matt, Generally speaking, a purple alert should be seen first and foremost as an indication of a failure in the monitoring *system*... where "system" includes the client pushing data up from the various servers you're paying attention to. By having a calculation made on each message's receipt of how long that message is good for (receipt time + [default, or specified]), we have a "fail safe" for an unknown issue occurring that requires attention. The proximate cause of the purple is the failure to receive a message. Whether that's caused by a hang or death of the usual originator, a bug in a xymonproxy, a cut network cable, or xymond being unable to handle all of the traffic sent to it before it times out, is left somewhat as an exercise for the administrator. Because purple alerts are generated from xymond's own view of its internal state (calculated once a minute) and are never sent IN to xymond, purple alerts should be a reliable indicator that... some other type of unreliability is going on :) Because of the wide possibility of different configurations, it's a little dangerous to create a one-size-fits-all strategy for purples. In a typical xymon installation with xymonnet and xymond_client running locally on the same machine, with no proxies or network segments in the middle, and with clients reporting directly in as well, you really shouldn't see any purple alerts outside of clients dying... And if the client is dying because the box is dying, by default you'll only get the 'conn' test red alert instead of the various xymond_client and xymonnet-generated ones (unless you're using the 'noclear' line in hosts.cfg). Your suggestion to have only a single 'purple' come through would *typically* work, but you'd have to ask yourself which test would be the representative one. In our case, we found it easiest to nominate a specific xymond_client test -- "memory" -- and only send purple notifications for that out to our alert team. That takes care of xymond_client, while leaving esoteric situations caused by the failure of different sharded xymonnet's, xymonproxy's, or custom independent tests free to fail in their own way. Again, it's the non-typical cases where it gets tricky. What about custom tests that aren't being generated by xymond_client that are still functioning? Perhaps you have xymonnet running on a different machine that's reporting back to xymond (or to a xymonproxy that's reporting back to xymond!) that has failed in some way. And of course, it could be that xymond is under heavy load and is unable to keep up with incoming messages generally (something we experienced in both the TCP and BFQ configs as we were scaling out). Sort of along these lines, however, I'd been considering having a more "host-wide" way of defining certain failure states directly within xymond, which would allow some of this override logic to happen centrally (and more reliably). Imagine a 'conn' being red optionally causing *all* tests to fail-to-clear, removing the need for this calculation from the remainder of xymonnet tests. Or a true host-wide "disable" that gets applied to all tests, even new ones, as a xymond flag. A host-wide "purple-state" could be conceptualized as well. That's just a thought, though, and it kind of depends on whether people would find such a feature useful. Anyway, I hope that's answered some of your questions! Regards, -jc

1 0

Upgrade to 4.3.21, https tests seem to no longer work.
by hobbit＠elyograg.org 25 Aug '15

25 Aug '15

I upgraded to 4.3.21 today, from 4.3.14. It was built using this command: build/makerpm.sh 4.3.21 After working my way through recovering my configs that were pushed out of the way as .rpmsave files, I started it. Immediately I got http alarms failing. The most interesting one was this. I have xymon monitoring its own URL, which is secured with SSL: red Tue Aug 25 11:38:40 2015: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> &red https://xymon.REDACTED.com/xymon/ - <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> Instead use the HTTPS scheme to access this URL, please.<br /> <blockquote>Hint: <a href="https://xymon.REDACTED.com/"><b>https://xymon.REDACTED.com/</b></a></blockquote></p> <hr> <address>Apache/2.2.15 (CentOS) Server at xymon.REDACTED.com Port 443</address> </body></html> It looks like what's happening is that xymon's http test has forgotten how to speak HTTPS. It actually looks like it has forgotten how to speal SSL entirely -- imaps and pop3s are showing yellow status with "Unexpected service response" messages. I am now also seeing purple alarms from sslcert, because none of the SSL stuff is working right. How can I fix this? Thanks, Shawn

5 9