June 2016 - Xymon - Xymon mailman web

Is SNMP a solution
by jlaidman＠rebel-it.com.au 16 Jun '16

16 Jun '16

I don't think SNMP can be used to monitor log files. At least I can't think of a way to do ir without using a script. If you can SSH from another server that can run the xymon client, then you could setup a reverse tunnel for syslog-over-tcp, and then your task becomes monitoring the logs on the other server. On Fri, 17 Jun 2016, 03:27 usa ims via Xymon <xymon at xymon.com> wrote: > It would be nice to use 'xymon-rclient.sh' on the '192.168.1.51' node > which is not exposed on the Internet. I wanted 'xymon-rclient.sh' to > monitor log files and alert if there is a problem. > > But I just found out that I can't even use ssh scripts because of > compliance. Would SNMP be a good solution if I can't use ssh scripts? > > > On Thursday, June 16, 2016 1:23 AM, Adam Goryachev < > adam at websitemanagers.com.au> wrote: > > > On 16/06/16 11:37, Bruce Ferrell wrote: > > If i understand you correctly, there is no direct path between the > sister server, on .51, and your monitor station. That would seem, on the > surface, to make use of SNMP and the > > devmon add in for xymon very difficult. > > Personally, I'd add a custom ext to the server you DO have access to, > which will ping/test connectivity to your second server, and then report > the results back along with the rest of the results. > > Or, did you want to run a client on the second machine? > > Regards, > Adam > > -- > Adam Goryachev > Website Managers > P: +61 2 8304 0000 adam at websitemanagers.com.au > F: +61 2 8304 0001 www.websitemanagers.com.au > > > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon > > > > ---------- Forwarded message ---------- > From: usa ims via Xymon <xymon at xymon.com> > To: Adam Goryachev <adam at websitemanagers.com.au>, "xymon at xymon.com" < > xymon at xymon.com> > Cc: > Date: Thu, 16 Jun 2016 19:27:36 +0200 (CEST) > Subject: Re: [Xymon] Is SNMP a solution > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon >

2 1

Is SNMP a solution
by bferrell＠baywinds.org 16 Jun '16

16 Jun '16

If i understand you correctly, there is no direct path between the sister server, on .51, and your monitor station. That would seem, on the surface, to make use of SNMP and the devmon add in for xymon very difficult. The simplest solutions that occurs to me would be to do port forwarding via ssh client configuration and /etc/hosts name resolution to create an ssh tunnel to the sister server. I'm assuming you ARE using the ssh method and have at lest part of the ssh client configuration done. Because xymon-rclient would run as the xymon user the ssh configuration might look something like this: ~xymon/.ssh/config: host 192.169.100.50 user xymon IdentityFile ~xymon/.ssh/id_rsa_50 host 192.169.100.51 user xymon2 IdentityFile ~xymon/.ssh/id_rsa_51 LocalForward 127.0.0.1:1985:192.169.100.51:1984 The above configuration MAY need (probably will) adjustment and experimentation to get xymon-rclient connecting to the sister server. On 06/15/2016 12:56 PM, usa ims via Xymon wrote: I have a xymon server located out in the Internet that uses custom scripts for monitoring. I monitor this one server which is NATed behind a firewall. I use 'xymon-rclient.sh' script to fetch the data. This server is only one that is exposed on the internet, it has an IP of 192.169.100.50. But I want to monitor it's sister server '192.168.100.51' which is not exposed on the Internet. Somebody said to me that SNMP might be a solution. If that is a solution, can somebody give me a high level explanation on how this might work? Thanks in advance. > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon

3 2

Xymon server (4.3.27) occasionally assigns reports to the wrong host
by beckert＠phys.ethz.ch 16 Jun '16

16 Jun '16

Hi, in the past few months I found more and more indices for a strange bug in (at least) Xymon 4.3.27 which occasionally mixes up hosts when handling reports: One of our two active Xymon servers occasionally assigns test reports to the wrong hosts, also causing false positives in alerting. (Actually I had the notion that this is happening for quite a while now but we today found a false positve caused by that and could nail down the real origin of the report to be clearly from a different host.) Not sure where I should start with describing the details, so here's just a bunch of facts: * Happens at least for the tests "disk", "ports", "cpu" and our self-written test "raid", but maybe more. * Some of the wrong status pages show "Status message received from 127.0.0.1" despite they should have come from some client IP address ("raid" test), others show the expected IP of the host for which the test is said to belong as sender ("disk" test). * Machines with a single disk (e.g. VMs) occassional report status of a "raid" test which is not deployed to them -- and then (for obvious reasons) went purple on it. On that server, there's only one machine in having a RAID, but its "raid" reports have been misassigned to at least three other hosts, all host which have rather many tests (compared to a bunch of sensors which send in only very few tests per host). * All the so far noticed wrongly assigned raid reports were originating from the same machine and all the so far noticed wrongly assigned disk reports were originating from a different, but always the same machine. * For about a month one machine's service status page (as well as non-green.html) occassionally (approx. 1-3 times a day) show the disk report for another machine. At least all the cases we looked at due to different "yellow" thresholds for disk usage levels were coming from the same other machine. Examples take from the history of the "disk" test of the same host: + An expected, "green" report: Wed Jun 15 17:39:49 CEST 2016 - Filesystems ok Filesystem 1024-blocks Used Available Capacity Mounted on udev 2013012 0 2013012 0% /dev tmpfs 405736 46412 359324 12% /run /dev/sda1 75620456 2599176 69942852 4% / tmpfs 2028668 0 2028668 0% /dev/shm tmpfs 5120 4 5116 1% /run/lock tmpfs 2028668 0 2028668 0% /sys/fs/cgroup cgmfs 100 0 100 0% /run/cgmanager/fs + One of the misassigned false positive reports: Tue Jun 14 10:35:13 CEST 2016 - Filesystems NOT ok yellow / (94% used) has reached the WARNING level (90%) Filesystem 1024-blocks Used Available Capacity Mounted on udev 505668 0 505668 0% /dev tmpfs 102980 884 102096 1% /run /dev/sda1 155570664 137658232 10003236 94% / tmpfs 5120 4 5116 1% /run/lock none 1132776 516 1132260 1% /tmp cgroup 12 0 12 0% /sys/fs/cgroup + Excerpt from the "Client data" link from the status page of the item directly above this one: [df] Filesystem 1024-blocks Used Available Capacity Mounted on udev 2013012 0 2013012 0% /dev tmpfs 405736 46464 359272 12% /run /dev/sda1 75620456 2591688 69950340 4% / tmpfs 2028668 0 2028668 0% /dev/shm tmpfs 5120 4 5116 1% /run/lock tmpfs 2028668 0 2028668 0% /sys/fs/cgroup cgmfs 100 0 100 0% /run/cgmanager/fs tmpfs 405736 0 405736 0% /run/user/0 Please notice + the different set of file systems due to the reports are in reality coming from systems with different init systems. + the different overall size of /dev/sda1. The system hasn't rebooted since like 10 days and only has an 80 GB disk while the yellow report comes from a system with a 160 GB disk. * The "Client data" link always have the correct "df" and "netstat" output even in cases where the service status page shows the "df" or "netstat" output from the wrong host. So I assume it's a bug in report handling and neither on the client side nor on the network side of the of the daemon. * The misassigned reporting of the "raid" test started on 22nd of April 2016. That fits very well to the fact that I upgraded that server on 20th of April 2016 from 4.3.26 (plus CGI-regression fixes) to 4.3.27. The misassigned reporting of the "disk" test was first seen in our (IRC) logs on 15th of May 2016, i.e. weeks after the upgrade of Xymon to 4.3.27. But the misassigned "ports" test seems to have started already on 13th of April 2016 and that test was green for more than a year beforehands), i.e. while still 4.3.26 (plus CGI-regression fixes) was running. Can't say when the misassigned cpu tests started since I didn't find them via obvious false positives but because an actual report reported xen daemons on a webserver. * There are hosts whose reports gets assigned to other hosts while being assigned reports from yet another host, e.g. Host 1 (web server with nearly full disk)'s procs test is (occassionally) reported as being for host 2 (a Xen server, easily spottable by several daemons containing "xen" in their name), host 2's cpu test is (occassionally) reported for host 3 (another web server, but with very low disk usage) and host's 1 disk test is (occassionally) reported as belonging to host 3, too. So far we've noticed this only on one of our two xymon servers, let's call it "Server B". And even explicitly grepping in the log history of our Xymon IRC bot for potential cases (like ports on servers where they never were nor shouldn't be) on Server A didn't bring up anything similar. Both run on Debian 8 Jessie ("Stable") on the amd64 architecture with backported xymon packages from Debian Unstable/Testing (i.e. always the newest packages -- as I test the packages there under live conditions before uploading them to Debian). Both use Apache 2.4 as webserver. Server A: * real hardware * about 850 hosts * configured "critical view" setup * only reports to itself. * permanently monitored by our group, i.e. I expect that we would notice such misassignements rather quickly, especially if they happen in the "critical view". * Also runs some other monitoring-like tools like e.g. Smokeping. Server B: * virtual machine * about 35 hosts * Some hosts (actually network-enabled environment sensors) report once per minute. Those reports get passed through to a separate task ("CMD xymond_channel --channel=status ...") which logs _all_ their values and archives them month-wise for later reference. * no "critical view" * reports to itself and to Server A. * Also tries to report to Server C (which was a test installation of the Xymon 5 branch, now offline) * mostly monitors customer machines (call it "Xymon as a Service" ;-), i.e. not that much looked at by our group. * Writes "Flushed 1 stale messages for 0.0.0.0:0" every 5 minutes into /var/log/xymon/clientdata.log, probably related to Server C being offline. So far we only noticed that strange behaviour on Server B -- which is the less busier server (despite that bunch of minute-wise reporting hosts). Actually the first notion has been seen when the VM "Server B" reported to Server A that its (non-existing) "raid" test went purple. And then today one of our customers argued about his disk test going yellow several times a day despite the accused partition is at 4% most of the time according to Xymon -- except when it's at 94% for 5 minutes and then back at 4% again... (And I can't imagine an old 80 GB disk being stuffed with 75 GB of data within 5 minutes) One single _other_ host reporting to Server B is permanently on 94% disk usage and its (and only its) yellow thresholds is set to 95%, hence its reports are green for the same set of data. So does anyone have an idea what goes wrong on our Server B so that it mixes up reports from multiple hosts while Server A doesn't show these symptoms? Kind regards, Axel Beckert -- Axel Beckert <beckert at phys.ethz.ch> support: +41 44 633 26 68 IT Services Group, HPT H 6 voice: +41 44 633 41 89 Departement of Physics, ETH Zurich CH-8093 Zurich, Switzerland http://nic.phys.ethz.ch/

3 3

Graph has broken link
by Foster.Patch＠accuweather.com 16 Jun '16

16 Jun '16

Hello, I created a script that monitors connection traffic on a server, called "aspnethealth". Unfortunately, the graphs on the monitor are broken... I've attached a picture displaying this. My script is also attached, and also pasted below. (Also here http://pastebin.com/Pdnq6AnA) I've been looking at the rrd and also http://xymon.sourceforge.net/xymon/help/howtograph.html but am clueless as to what I need to do. I am a dummy when it comes to Linux. Any assistance with this would be much appreciated! Thanks, Foster Patch Web Server Technician AccuWeather $MAX_RUN_TIME = 300 $TIME_Exceeded = $false $STATUS = "green" $STATUSCODE = "&green Everything is looking just fine! &green" $TEST_FILE = "${env:ProgramFiles(x86)}\BBWin\tmp\aspnethealth" $CurrSTATUS = "&green" $ExeSTATUS = "&green" $WaitSTATUS = "&green" $QueueSTATUS = "&green" $TimeSTATUS = "&green" $Source = @" using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; using System.Diagnostics; namespace ASPNETPerformance { public static class Query { public static IntPtr userHandle = new IntPtr(0); public static string hostName = "."; public static string categoryName = "ASP.NET Applications"; public static string instanceName = "__Total__"; public static float CurrentConnections() { string counterName = "Current Connections"; PerformanceCounter counter = new PerformanceCounter("Web Service", counterName, "_Total", hostName); return counter.NextValue(); } public static float RequestsExecuting() { string counterName = "Requests Executing"; PerformanceCounter counter = new PerformanceCounter(categoryName, counterName, instanceName, hostName); return counter.NextValue(); } public static float RequestWaitTime() { string counterName = "Request Wait Time"; PerformanceCounter counter = new PerformanceCounter(categoryName, counterName, instanceName, hostName); return counter.NextValue(); } public static float RequestsInApplicationQueue() { string counterName = "Requests In Application Queue"; PerformanceCounter counter = new PerformanceCounter(categoryName, counterName, instanceName, hostName); return counter.NextValue(); } public static float RequestExecutionTime() { string counterName = "Request Execution Time"; PerformanceCounter counter = new PerformanceCounter(categoryName, counterName, instanceName, hostName); return counter.NextValue(); } } } "@ try{ Add-Type -TypeDefinition $Source -Language CSharp } catch { } $CurrentConnections = [ASPNETPerformance.Query]::CurrentConnections() $RequestsExecuting = [ASPNETPerformance.Query]::RequestsExecuting() $RequestWaitTime = [ASPNETPerformance.Query]::RequestWaitTime() $RequestsInApplicationQueue = [ASPNETPerformance.Query]::RequestsInApplicationQueue() $RequestExecutionTime = [ASPNETPerformance.Query]::RequestExecutionTime() if ($TIME_Exceeded) { $STATUS = "yellow" $STATUSCODE = "Something is wrong with this test reporting to Hobbit" } if(($CurrentConnections -gt 15000) -and (!($CurrentConnections -gt 16000))) { $CurrSTATUS = "&yellow" $STATUSCODE = "&yellow Current Connections exceed 15000 &yellow" } if($CurrentConnections -gt 16000) { $CurrSTATUS = "&red" $STATUSCODE = "&red Current Connections exceed 16000 &red" } if(($RequestsExecuting -gt 3000) -and (!($RequestsExecuting -gt 5000))) { $ExeSTATUS = "&yellow" $STATUSCODE = "&yellow Requests Executing exceeds 3000 &yellow" } if($RequestsExecuting -gt 5000) { $ExeSTATUS = "&red" $STATUSCODE = "&red Requests Executing exceeds 5000 &red" } if(($RequestWaitTime -gt 60000) -and (!($RequestWaitTime -gt 70000))) { $WaitSTATUS = "&yellow" $STATUSCODE = "&yellow Request Wait Time exceeds 60000 &yellow" } if($RequestWaitTime -gt 70000) { $WaitSTATUS = "&red" $STATUSCODE = "&red Requests Wait Time exceeds 70000 &red" } if(($RequestsInApplicationQueue -gt 20) -and (!($RequestsInApplicationQueue -gt 50))) { $QueueSTATUS = "&yellow" $STATUSCODE = "&yellow Requests in Queue exceed 20 &yellow" } if($RequestsInApplicationQueue -gt 50) { $QueueSTATUS = "&red" $STATUSCODE = "&red Requests in Queue exceed 50 &red" } if(($RequestExecutionTime -gt 60000) -and (!($RequestExecutionTime -gt 70000))) { $TimeSTATUS = "&yellow" $STATUSCODE = "&yellow Request Execution Time exceeds 60000 &yellow" } if($RequestExecutionTime -gt 70000) { $TimeSTATUS = "&red" $STATUSCODE = "&red Request Execution Time exceeds 70000 &red" } if(($CurrSTATUS -eq "&yellow") -or ($ExeSTATUS -eq "&yellow") -or ($WaitSTATUS -eq "&yellow") -or ($QueueSTATUS -eq "&yellow") -or ($TimeSTATUS -eq "&yellow")) { $STATUS = "yellow" } if(($CurrSTATUS -eq "&red") -or ($ExeSTATUS -eq "&red") -or ($WaitSTATUS -eq "&red") -or ($QueueSTATUS -eq "&red") -or ($TimeSTATUS -eq "&red")) { $STATUS = "red" } $end = Get-Date $STATUS = "$STATUS $end" $STATUS | out-file -Encoding ASCII $TEST_FILE -Force $Herp = "ASP.NET Application Information `n" | out-file -Encoding ASCII $TEST_FILE -Append $skeep = "$STATUSCODE`n`n" | out-file -Encoding ASCII $TEST_FILE -Append $Derp = $CurrSTATUS + " Current Connections: " + $CurrentConnections | out-file -Encoding ASCII $TEST_FILE -Append $Merp = $ExeSTATUS + " Requests Executing: " + $RequestsExecuting | out-file -Encoding ASCII $TEST_FILE -Append $Cherp = $WaitSTATUS + " Request Wait Time: " + $RequestWaitTime | out-file -Encoding ASCII $TEST_FILE -Append $Berp = $QueueSTATUS + " Requests In Queue: " + $RequestsInApplicationQueue | out-file -Encoding ASCII $TEST_FILE -Append $Flerp = $TimeSTATUS + " Request Exec Time: " + $RequestExecutionTime | out-file -Encoding ASCII $TEST_FILE -Append

2 1

XYmon & jmx
by linuxmasterjedi＠free.fr 16 Jun '16

16 Jun '16

Hi, What do you use to monitoring JMX interface ? I've found https://wiki.xymonton.org/doku.php/monitors:jmxstat Setup went fine but no data seems retrived (no graph). Are you using the same product ? Any advices regarding the installation steps ? Thanks

3 2

XymonPSclient EXT scripts
by davieb＠gmail.com 16 Jun '16

16 Jun '16

Zak, Should the ext script write to stdout or should it be directed to say a file in <xymon>\tmp to be sent to the Xymon server? I set it up on the server client-local.cfg: external:everyscan:sync:hotfix.ps1 As PSClient started the hotfix.ps1 (looking at the xymonclient.txt logfile), after 15m it still had not moved on... I was wondering if something was broke w/the hotfix.ps1. I ran it by hand and everything was output to the screen. Any help would be appreciated! Thanks, Dave

2 1

Is SNMP a solution
by josh＠imaginenetworksllc.com 15 Jun '16

15 Jun '16

DevMon is used for SNMP. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Jun 15, 2016 at 3:56 PM, usa ims via Xymon <xymon at xymon.com> wrote: > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon > > > ---------- Forwarded message ---------- > From: usa ims <usaims at yahoo.com> > To: Xymon Mailing List <xymon at xymon.com> > Cc: > Date: Wed, 15 Jun 2016 19:56:29 +0000 (UTC) > Subject: Is SNMP a solution > I have a xymon server located out in the Internet that uses custom scripts > for monitoring. > > I monitor this one server which is NATed behind a firewall. I use > 'xymon-rclient.sh' script to fetch the data. This server is only one that > is exposed on the internet, it has an IP of 192.169.100.50. > > But I want to monitor it's sister server '192.168.100.51' which is not > exposed on the Internet. > > Somebody said to me that SNMP might be a solution. > > If that is a solution, can somebody give me a high level explanation on > how this might work? > > Thanks in advance. > >

2 3

Is SNMP a solution
by novosirj＠rutgers.edu 15 Jun '16

15 Jun '16

http://devmon.sourceforge.net > On Jun 15, 2016, at 4:07 PM, usa ims via Xymon <xymon at xymon.com> wrote: > > > From: usa ims <usaims at yahoo.com> > Subject: Re: [Xymon] Is SNMP a solution > Date: June 15, 2016 at 4:07:42 PM EDT > To: Josh Luthman <josh at imaginenetworksllc.com> > Cc: Xymon Mailing List <xymon at xymon.com> > Reply-To: usa ims <usaims at yahoo.com> > > > Thanks but I need somebody to briefly explain this? > > > On Wednesday, June 15, 2016 4:03 PM, Josh Luthman <josh at imaginenetworksllc.com> wrote: > > > DevMon is used for SNMP. > > > Josh Luthman > Office: 937-552-2340 > Direct: 937-552-2343 > 1100 Wayne St > Suite 1337 > Troy, OH 45373 > > On Wed, Jun 15, 2016 at 3:56 PM, usa ims via Xymon <xymon at xymon.com> wrote: > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon > > > ---------- Forwarded message ---------- > From: usa ims <usaims at yahoo.com> > To: Xymon Mailing List <xymon at xymon.com> > Cc: > Date: Wed, 15 Jun 2016 19:56:29 +0000 (UTC) > Subject: Is SNMP a solution > I have a xymon server located out in the Internet that uses custom scripts for monitoring. > > I monitor this one server which is NATed behind a firewall. I use 'xymon-rclient.sh' script to fetch the data. This server is only one that is exposed on the internet, it has an IP of 192.169.100.50. > > But I want to monitor it's sister server '192.168.100.51' which is not exposed on the Internet. > > Somebody said to me that SNMP might be a solution. > > If that is a solution, can somebody give me a high level explanation on how this might work? > > Thanks in advance. > > > > > > > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `'

1 0

Is SNMP a solution
by usaims＠yahoo.com 15 Jun '16

15 Jun '16

I have a xymon server located out in the Internet that uses custom scripts for monitoring. I monitor this one server which is NATed behind a firewall. I use 'xymon-rclient.sh' script to fetch the data. This server is only one that is exposed on the internet, it has an IP of 192.169.100.50. But I want to monitor it's sister server '192.168.100.51' which is not exposed on the Internet. Somebody said to me that SNMP might be a solution. If that is a solution, can somebody give me a high level explanation on how this might work? Thanks in advance.

1 0

Monitoring logfiles with changing names on a linux client
by christian.becker＠rhein-zeitung.net 14 Jun '16

14 Jun '16

Hello out there, i did already search for this, but couldn't' find the right things; further my perl regexp knowledge is not the best. What I have: I have a linux client with xymon agents up and running, reporting his data to the Xymon server -> this is working fine. This linux client serves an application that creates 6 new different logfiles every day -> that's my problem. The names of the lofgiles are like this: Log.File_ABC#_YYYYMMDD_######.log The first # in the name could be numbers fom 1 to 6. YYYYMMDD is the actual day date, like 20160414. The ###### block is another numeric identifier, like 050601 or similar, which also changes every day, but not following any rule. What I want: My goal is to check those logiles for the keyword ERROR and let Xymon go red if this keyword occurs. What I'm looking for is the necessary config on both the client (in localclient.cfg) and the xymon server (in client-local.cfg and analysis.cfg) using regexp. I'm thinking about something like this to have in the linux client's localclient.cfg: LOG `ls -1 Log.File_ABC*_`date +%Y%m%d\`_*.log` But I'm not sure about the syntax here. Further I need to know if and how this could be implemented in client-local.cfg and analysis.cfg on the Xymon server. Anyone any idea? Regards Christian Christian Becker IT-Services Christian.Becker at rhein-zeitung.net<mailto:Christian.Becker at rhein-zeitung.net> _________________________________ Mittelrhein-Verlag GmbH August-Horch-Straße 28 D-56070 Koblenz Verleger und Geschäftsführer: Walterpeter Twer Reg.-Gericht Koblenz HRB 121 Finanzamt Koblenz Str.Nr. 22 65 10 285 2 www.rhein-zeitung.de<http://www.rhein-zeitung.de/>

3 4