April 2019 - Xymon - Xymon mailman web

Yet another flap
by rcohen55＠gmail.com 26 Apr '19

26 Apr '19

Hi have one server which keeps flapping on msgs. Spent hours to look for the obvious culprit (another feeder) until it dawn on me that it must be the logfetch, since there are no issues with other columns. And indeed, after running logfetch with --debug=stderr, I discovered that it is not always running - I mean not triggered at all - which triggered the flapping status. I dug here and there, to no avail. Any bright ideas are welcome.. Cheers Ron

2 2

MRTG graphs are in bytes instead of bits
by scarville＠lereta.com 25 Apr '19

25 Apr '19

I started using mrtg to graph traffic over a 1 Gbps point-to-point link to a DR site. It is currently being monitored using cricket. When comparing the mrtg graph to the cricket graphs, the mrtg graph seems to be in bytes/second instead of bits/second. The y-axis label says bits/second but the values are way to small. This is the configuration for the switch port. ## CA to AZ P2P Directory[mrtg.ca-coresw.gi_1_1_1]: ca2az-p2p.lereta.net Target[mrtg.ca-coresw.gi_1_1_1]: 1:ROcommunity at ca-coresw.lereta.net Options[mrtg.ca-coresw.gi_1_1_1]: bits Title[mrtg.ca-coresw.gi_1_1_1]: P2P to AZ MaxBytes[mrtg.ca-coresw.gi_1_1_1]: 125000000 I haven't used mrtg in years so I may be just screwing up the configuration. Any help is appreciated. -- Stephen Carville

2 3

strange behavior of xymon client in openSUSE
by jeremy＠laidman.org 25 Apr '19

25 Apr '19

On Fri, 22 Mar 2019 at 02:00, George Michalopoulos <metron6 at gmail.com> wrote: > From an opensuse linux, i'm trying to connect to my xymon servers... > take a look at this.. > > in /etc/default/xymon-client > > This works XYMONSERVERS="10.129.13.2" > This works XYMONSERVERS="10.10.100.100" > This doesn't XYMONSERVERS="10.10.100.100 10.129.13.2" > George, in what way is this not working? > This works > XYMONSERVERS="10.10.100.100" > XYMONSERVERS="10.129.13.2" > This is likely to work inasmuch as it sends messages to the server in the last entry. You can check how a variable is set from the client's perspective by running xymoncmd and then "echo $VARNAME". However, I think the main problem you have is the variable name, which should be XYMSERVERS rather than XYMONSERVERS. J

1 0

How to debug "SSL error" on https test(s)?
by brodie＠mcw.edu 24 Apr '19

24 Apr '19

Hey everyone- So a recent web server I have configured is causing Xymon to get an "SSL Error". Even with DEBUG enabled in xymonnet, I don't see anything useful. All I see is, 32183 2019-04-23 15:14:54.869387 Calc http color host morn.rgd.mcw.edu : 32183 2019-04-23 15:14:54.869389 https://scge.mcw.edu/(red) 32183 2019-04-23 15:14:54.869391 --> red 32183 2019-04-23 15:14:54.869393 Adding to combo msg: status+30 morn,rgd,mcw,edu.http red Tue Apr 23 15:14:52 2019: SSL error In other words, even with debug enabled, all I see is "SSL error". When I connect to the site with any browser, there are no SSL or certificate issues. In fact, I have dozens of apache web sites all with certs. But this one is causing me fits and I don't know why. ANY help getting "more" debug info would be super appreciated! Thanks all --Kent

3 4

Xymon server post-migration blues pt. 2 - "phantom" reports apparently coming from 127.0.0.1
by earle＠isolar.DynDNS.ORG 24 Apr '19

24 Apr '19

Another thing that happened after my recent migration/Xymon upgrade is that I started getting phantom "disk" alerts purporting to be from the Xymon server itself. They looked like this: -- To: sysadmins at my.do.main Subject: Xymon [556665157] mgmt:disk CRITICAL (RED) Message-Id: <20190318195008.53EF4635153 at mgmt.my.do.main> From: xymon at mgmt.my.do.main (xymon Monitor (client)) red Mon Mar 18 12:50:04 PDT 2019 - Filesystems NOT ok &red /export/bkd05d (98% used) has reached the PANIC level (98%) [...] -- The thing is, the partition "/export/bkd05d" does not exist on the Xymon server host "mgmt". It exists on a completely different system (and I know exactly which one it is). I've seen other alerts like this where the disk partitions mentioned are from other systems, too. In short, the Xymon server is getting the reports from the clients but somehow they are getting mangled into looking like they are coming from 127.0.0.1 instead and thus are local to itself, and so it generates red alerts from itself as a result. In many cases they are filesystems where I already had exception clauses in "analysis.cfg" for them already, so I never get alarms from the actual client host. So to suddenly get "back from the dead" red alarms for them was a surprise, to say the least. I've kludged around it by making a special pseudo-clause in "analysis.cfg" for the Xymon server for all of these disk partition exceptions: -- # XXX - KLUDGE dummy entries to prevent Xymon from reporting false # XXX - "red" disk alerts for systems with faulty "127.0.0.1" reports HOST=mgmt DISK /export/brick1 101 101 DISK /export/data 101 101 DISK /export/data1 100 100 DISK /export/data2 99 100 DISK /export/work 99 100 DISK %(?-i)^.*/Volumes/Time 101 101 DISK /media/Oracle_Solaris-11_3-Text-SPARC 101 101 DISK /media/Solaris-11_3_28_4_0-Boot-SPARC 101 101 [... more here ...] -- but obviously I would prefer to solve the problem so I can remove this. What changed between Xymon 4.3.12 and 4.3.28 to cause this? How do I debug it? - Greg

2 2

Generic ssh was RE: Xymon server post-migration blues pt. 1 - "brown-outs"
by frankk74＠gmail.com 24 Apr '19

24 Apr '19

Hi, Maybe this might help....(it fixed a problem I was having on some Raspbian based systems).. I don't think it is xymon specifically related... probably due to the OS upgrade.... Although the link below is debian related I think it might be an issue with new/older versions of ssh(d)... the intergooglenet isn't clear, but it fixed my issue... (setting IPQoS 0x00) https://wiki.debian.org/SSH#SSH_hangs Hope this helps... Frank

1 0

Xymon server post-migration blues pt. 1 - "brown-outs"
by earle＠isolar.DynDNS.ORG 24 Apr '19

24 Apr '19

[Apologies in advance for the too-wordy message. Part 1 of 2 .] Recently I violated one of the prime rules of being a SysAdmin - don't ever change two things at once. At my work we were forced to migrate our data center to a new facility, so we bought a new monitoring PC to replace a RHEL 6.10 system that ran Xymon 4.3.12. The new monitoring PC runs RHEL 7.6 and Xymon 4.3.28 (using the Terabithia RPMs). Ever since then, I have been running into two big problems that did not exist before. This is the first problem ("brown-outs"); I'll describe the 2nd in part 2. Randomly, a group of systems (or some subset of the group) will report in as CRITICAL/RED due to failed xymonnet tests. Mostly SSH, but some SMTP and FTP as well. The hosts/services are all actually fine and the red alerts are incorrect/false positives. The problem is getting worse - I'm now seeing several hundred red alerts a day from these "brown-outs". The hosts involved are more-or-less random - different buildings/OSes, etc. Sometimes all of them provoke alerts; most of the time it's just a subset of the list. When they fail, the alert message is always -- Service <service> on <host> is not OK : Service listening but unavailable (connect timeout) -- To try and catch it in the act, I ran this test in a loop: [root at mgmt xymon]# while true; do ( echo "["`date`"]" ; xymonnet --report --ping --checkresponse --timing --debug --no-update 2>&1 > /tmp/xymonnet.out ; grep 'err=[^0]' /tmp/xymonnet.out ); done A couple of times I think I did catch it; here's an example: -- Address=192.168.1.26:22, open=1, res=0, err=1, connecttime=0.004546, totaltime=11.653128, Address=192.168.1.25:22, open=1, res=0, err=1, connecttime=0.004510, totaltime=11.653092, Address=192.168.1.219:22, open=1, res=0, err=1, connecttime=0.003163, totaltime=11.651745, Address=192.168.1.151:22, open=1, res=0, err=1, connecttime=0.002923, totaltime=11.651505, Address=192.168.1.50:22, open=1, res=0, err=1, connecttime=0.002906, totaltime=11.651488, Address=137.78.80.38:22, open=1, res=0, err=1, connecttime=0.002819, totaltime=11.651401, [... another 10 elided ...] Address=192.168.1.184:22, open=1, res=0, err=1, connecttime=0.001098, totaltime=12.393879, Address=192.168.1.174:25, open=1, res=0, err=1, connecttime=0.000426, totaltime=12.364234, Address=192.168.1.182:25, open=1, res=0, err=1, connecttime=0.000418, totaltime=12.364226, Address=192.168.1.25:25, open=1, res=0, err=1, connecttime=0.000411, totaltime=12.364219, Address=192.168.1.25:21, open=1, res=0, err=1, connecttime=0.022773, totaltime=12.364044, -- Notice the non-zero connecttime, but the exceeded-the-timeout totaltime values. The services always immediately recover in the next test pass. Are there any knobs I can turn on to help debug this problem? I'm assuming it's network/router/switch-related, but I need a smoking gun. Failing that, is there any way in a .cfg file setting to turn these particular "Service listening but unavailable" statuses into a Yellow alert rather than Red? (I'd rather not have to resort to this but as a stop-gap, I would.) - Greg

2 1

AUTO: Jason Jones is out of the office (returning 29/04/2019)
by jason.jones＠codelsoftware.com 24 Apr '19

24 Apr '19

An HTML attachment was scrubbed... URL: <http://lists.xymon.com/pipermail/xymon/attachments/20190424/16840de9/attach…>

1 0

Own RRD graph
by jonas.hultstrom＠scania.com 24 Apr '19

24 Apr '19

Hi, I try to create a custom graph in xymon. I have process which returns lines like: server1: 8 processes . serverX: 4 processes There are different numbers of line for different hosts. I want to have these numbers in a graph, so firstly I defined in xymonserver.cfg: TEST2RRD=..., sapprocs=ncv" SPLITNCV_sapprocs="server:GAUGE" Then in graphs.cfg: [sapprocs] FNPATTERN ^sapprocs(.*).rrd TITLE Processes YAXIS Processes DEF:p at RRDIDX@=@RRDFN@:lambda:AVERAGE LINE1:p at RRDIDX@#@COLOR@:@RRDPARAM@ GPRINT:p at RRDIDX@:LAST: \: %5.1lf (cur); GPRINT:p at RRDIDX@:MAX: \: %5.1lf (max); GPRINT:p at RRDIDX@:MIN: \: %5.1lf (min); GPRINT:p at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n I get a graph, but for some reason it doesn't look like I want. The values below the graph ( cur, max, min, avg) are all 0.0. When I do a rrdtool dump on one of the rrd files, I get the type is "DERIVE" and not GAUGE. The values in "last_ds" are correct. <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd">  <rrd> <version>0003</version> <step>300</step>  <lastupdate>1556000353</lastupdate>  <ds> <name> lambda </name> <type> DERIVE </type> <minimal_heartbeat>600</minimal_heartbeat> <min>NaN</min> <max>NaN</max>  <last_ds>1</last_ds> <value>-2.5215946844e+00</value> <unknown_sec> 0 </unknown_sec> </ds> What did I do wrong? Or isn't this possible? In the end I would like to have these values "stacked" in the graph... Thanks! Jonas

2 2

xymond_channel
by neilsimmonds1808＠gmail.com 18 Apr '19

18 Apr '19

I'm running 6 scripts from tasks.cfg as the following, CMD xymond_channel --channel=client --log=$XYMONSERVERLOGS/script.log $XYMONHOME/ext/script.sh The reason for doing this is that we have some custom monitors written on a mainframe but they just send data as part of the general Xymon data and Xymon doesn't understand the data so ignores it. These scripts process that data and do the alerting sending the data to a specific host.test on Xymon. These scripts have been observed to use quite a bit of CPU and I assume that is due to them reading all client data. Is there a way to narrow it down so it will only read data coming from a specific host? Regards, Neil Simmonds

2 3