November 2012 - Xymon - Xymon mailman web

ack's in xymon
by wrichardson＠llbean.com 12 Nov '12

12 Nov '12

My operations team would like to ack a yellow alert but would like that ack (check mark) to go away if it escalates to a red alert so it can be seen as a new event from "All non green view" As it is today if they ack a yellow alert for 24 hours that yellow ack'd alert could go back and forth from yellow to red and it would go unnoticed. So I edit the (OKCOLORS="green,blue,clear") in the xymonserver.cfg to look like OKCOLORS="green,blue,clear,red" I restart xymon but that didn't work same behavior as the default setting. Changing it to OKCOLORS="green,blue,clear,yellow" Now the above change will clear the ACK going from a red to a yellow but that's not what I was looking for. I need to clear the ACK going from a yellow to a red alert. Any thoughts on how to do this? Thanks I have looked but cant seem to find the answer to this one Bill Richardson Bill Richardson Systems Management Engineer wrichardson at llbean.com<mailto:rwrichardson at llbean.com> 207-552-2729

1 0

#comments not showing xymon 4.3.10
by Cedric.Steenbeke＠aznikolaas.be 12 Nov '12

12 Nov '12

Guys, Since we upgraded our Xymon server to 4.3 the #COMMENT tag is not working as it used to. For instance: 192.168.3.4 MYSERVER01 #COMMENT:"server comment" The comment does not show up in the Xymon overview. Any ideas? Cedric Steenbeke

2 1

Using "IGNORE" to exclude filesystems
by Mike.Burger＠FreedomMortgage.com 09 Nov '12

09 Nov '12

I have a host for which I'd like to ignore 27 filesystems...they're all at 97-100% full, and we expect them to be at that level...as such, there's no reason for Xymon to be alarming on them. Reading the analysis man page, I've tried a number of options to try to get this set of filesystems ignored, to no avail. The hostname of the system is "edmprod" and the filesystems are /da01 thru /da27. The stanza I've tried to use is: HOST=%edmprod.* DISK %^\/da* IGNORE I've also tried: HOST=edmprod DISK %^\/da* IGNORE And HOST=edmprod DISK /da01 IGNORE DISK /da02 IGNORE DISK /da03 IGNORE DISK /da04 IGNORE DISK /da05 IGNORE DISK /da06 IGNORE DISK /da07 IGNORE DISK /da08 IGNORE DISK /da09 IGNORE DISK /da10 IGNORE DISK /da11 IGNORE DISK /da12 IGNORE DISK /da13 IGNORE DISK /da14 IGNORE DISK /da15 IGNORE DISK /da16 IGNORE DISK /da17 IGNORE DISK /da18 IGNORE DISK /da19 IGNORE DISK /da20 IGNORE DISK /da21 IGNORE DISK /da22 IGNORE DISK /da23 IGNORE DISK /da24 IGNORE DISK /da25 IGNORE DISK /da26 IGNORE DISK /da27 IGNORE I'd love to know what I'm doing wrong, here, if anyone is so inclined to assist me. -- Mike Burger AIX Administrator <http://freedomhome.fhmc.local/intranet/main.jsp> Phone (317) 537-3680, Fax (317) 537-4680, Cell (317) 797-2040 E-mail: Mike.Burger at FreedomMortgage.com <mailto:Anthony.House at freedommortgage.com> "Once word leaks out that a pirate's gone soft, people begin to disobey you and then it's nothing but work, work, work, all the time." --Westley/The Dread Pirate Roberts CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.

8 16

Sending client data without client installed
by raylee88＠gmail.com 09 Nov '12

09 Nov '12

Greetings, I have a Cisco ACS appliance that runs a CentOS-based OS. I cannot install the Xymon client on it, but I can ssh to it from our Xymon server and run "show tech-support" to get the output of a long list of commands that include df -k, ps -aux, netstat -an, etc., and save the output to a file. The output has section headers, so it's easy to parse out each of the command outputs that I'm interested in with a Perl script. I was thinking to have the same Perl script look at each of the command outputs and apply some logic to them to determine what color their status should be and then do a ' bb localhost "status hostname.disk COLOR <additional text>" '. Same idea for hostname.procs, hostname.ports, etc. But this would essentially be like rewriting the Xymon client. Since I already have the outputs of df -k, ps -aux, netstat -an, etc., can I just send them to the Xymon server and have it magically know what to do with them to populate the disk, procs, and ports columns? I've already seen the tip at http://www.xymon.com/xymon/help/xymon-tips.html#noinstall , but that doesn't apply to my scenario. I also looked into using devmon, but the MIBs that Cisco ACS supports don't give me the data I'm looking for. Thanks, Ray

4 5

FW: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10
by Don.Kuhlman＠schawk.com 09 Nov '12

09 Nov '12

Thanks Larry. Looks like everything went purple again at 6:45 this morning. The logs still show 0 bytes. Any other suggestions for trying to figure this out? Regards, Don From: Larry Barber <lebarber at gmail.com<mailto:lebarber at gmail.com>> Date: Mon, 5 Nov 2012 17:19:53 -0600 To: Don Kuhlman <don.kuhlman at schawk.com<mailto:don.kuhlman at schawk.com>> Subject: Re: [Xymon] FW: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10 Xymonnet tends to be pretty quiet unless something goes wrong. You won't be able to tell for sure until you get one of your purple storms. Alerts are handled by a different module. Look in tasks.cfg to find it. Thanks, Larry Barber On Mon, Nov 5, 2012 at 3:53 PM, Don Kuhlman <Don.Kuhlman at schawk.com<mailto:Don.Kuhlman at schawk.com>> wrote: Hi Larry/all. I've noticed that the xymonnet.log and xymonnet-again.log files are staying at 0 bytes. Does that seem to be indicating a problem? (and Xymon hasn't gone purple all day, but I'm still not sending any email alerts to anyone). -rw-rw-rw- 1 xymon xymon 0 Nov 5 15:05 /var/log/xymon/xymonnet-again.log -rw-rw-rw- 1 xymon xymon 0 Nov 5 15:07 /var/log/xymon/xymonnet.log Thanks Don K From: Larry Barber <lebarber at gmail.com<mailto:lebarber at gmail.com>> Date: Mon, 5 Nov 2012 11:19:32 -0600 To: Don Kuhlman <don.kuhlman at schawk.com<mailto:don.kuhlman at schawk.com>> Cc: Xymon Email List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: Re: [Xymon] FW: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10 All the server side Xymon logs are in /var/log/xymon by default. Since you say that you are getting purple storms for conn and http tests, this suggests that the problem is likely with your xymonnet process. Check the xymonnet log, and when you see the purples check to see if there is a xymonnet instance running. If this instance has been running for more than a few minutes, kill it. If the xymonnet process is hanging, you might want to set the MAXTIME parameter on the xymonnet process in tasks.cfg. Doesn't really fix the problem, but it will at least stop things from going purple. Thanks, Larry Barber On Mon, Nov 5, 2012 at 10:01 AM, Don Kuhlman <Don.Kuhlman at schawk.com<mailto:Don.Kuhlman at schawk.com>> wrote: Update to this. While googling further, I saw a thread titled "[hobbit] stale alerts". This mentioned that there could be an external script that I created which may cause issues for xymon when it runs. I do have a diskstat.sh script that may be causing problems. For now, I'm setting it to DISABLED in the tasks.cfg file. Is there a way to see log information in xymon to try and verify something like this? Thanks Don K From: Don Kuhlman <don.kuhlman at schawk.com<mailto:don.kuhlman at schawk.com>> Date: Mon, 5 Nov 2012 08:34:29 -0600 To: Xymon Email List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10 Hi folks. We've been running xymon for about 10 months now. It's been fine all this time. However last week around Wednesday we started getting purple storms on the CONN and HTTP tests for all our hosts. I stop Xymon and restart it, or reboot the server (Linux 5.x) and then it comes back ok. This also happened Thursday, and then again Saturday around 2PM cst. Anyone have a link or source for which logs to look in on the server or xymon to see what may be causing the CONN and HTTP tests to randomly start failing like this or where to start troubleshooting? Can I use xymonlaunch —debug like this to see what is happening? /usr/lib64/xymon/server/bin/xymonlaunch --debug --config=/usr/lib64/xymon/server/etc/tasks.cfg --env=/usr/lib64/etc/xymonserver.cfg While searching the xymon forum and message boards, I saw some things that say it may be disk space or inodes, but it seems like we are ok there - df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 3899392 204731 3694661 6% / tmpfs 490139 6 490133 1% /dev/shm /dev/sda1 32768 51 32717 1% /boot df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda2 61312028 5748784 52448700 10% / tmpfs 1960556 188 1960368 1% /dev/shm /dev/sda1 516040 87716 402112 18% /boot DNS also seems fine. Thanks Don K _______________________________________________ Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon

2 5

RRD - Aberrant Behavior Detection with Holt-Winters Forecasting
by nicolas＠lienard.name 09 Nov '12

09 Nov '12

Hello When devices are too overloaded; we got lot of gap in the graph and it is not very nice. To avoid that, i would like to implement the abberrant behavior detection provided by RRDTOOL. Following http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html, i defined a new RRA in ~xymon/server/etc/rrddefinitions.cfg [test_rrd] RRA:AVERAGE:0.5:1:576 RRA:AVERAGE:0.5:6:576 RRA:AVERAGE:0.5:24:576 RRA:AVERAGE:0.5:288:576 # RRA:HWPREDICT:<array length>:<alpha>:<beta>:<period> # enable RRAs for aberrant detection, 1 day seasonal store for 5 day RRA:HWPREDICT:1440:0.1:0.0035:288 <------ new entry i got a simple test script running every 5 mn: sh-3.00$ cat test_rrd.sh #!/bin/sh HOSTNAME="test_device" value=42 $BB $BBDISP "status+200d $HOSTNAME.test_rrd green `date` test: $value" i got ncv conf in xymonserver.cfg, the graph is working fine with this basic definition: [test_rrd] TITLE test RRD YAXIS value DEF:a=test_rrd.rrd:test:AVERAGE LINE2:a#00CC00:test value GPRINT:a:LAST: \: %5.1lf (cur) GPRINT:a:MAX: \: %5.1lf (max) GPRINT:a:MIN: \: %3.1lf (min) GPRINT:a:AVERAGE: \: %5.1lf (avg)\n now, if i add the def: DEF:b=test_rrd.rrd:test:HWPREDICT the graph is not showing anymore [even if i didn t use the new DEF on a LINE] :( if i dump the rrd i can see the RRA : sh-3.00$ rrdtool dump /opt/xymon/data/rrd/test_device/test_rrd.rrd |grep HW <cf> HWPREDICT </cf> any idea is welcome. Thanks. regards Nico

2 1

xmond_channel --channel=data: There can be only one?
by David.Mills＠ngc.com 09 Nov '12

09 Nov '12

All - I know I did something bad, but don't know exactly what... I ran across this posting from Henrik (http://lists.xymon.com/archive/2011-August/032267.html) indicating one could see the data coming across to the server from all the clients using a command similar to this: xymoncmd xymond_channel --channel=data egrep "egrep '^\@\@'" (Henrik's example used "status" rather than "data") So, I kicked it off and walked away for about an hour. Fine. When I returned some colleagues came by with worried expressions asking me to look at our Non-Green report. Rather than the usual handful of non-green hosts, there were rows and rows of hosts showing 'purple' in various columns. I killed my manual "xymond_channel..." command and watched as the Non-Green report shrank quickly to a normal display. What happened? * It appeared the appearance of various "purple" hosts in the environment was random: random hosts with random columns (but I won't swear to that) * At least one monitored host's "proc"s ended up as "red", rather than "purple", but this transition was long after the initial onset of my command. The user believes there was no real "crash" of his procs on his host. * We have two servers monitoring the same hosts in our environment. On the host used to run my manual command, pretty much all graph data for the Xymon server itself shows a gap: no data available while my command ran. * From the brother server, looking at graphs for the deranged server, we see an odd *decrease* in CPU load for the duration of my extra command, and an *increase* in TCP traffic I'm guessing my manual command competed / randomly intercepted incoming data for this server, but does this fit all the oddities observed? In any case, I'm back at square one for my original goal of *safely* seeing what raw data is coming into the server from my clients. (FWIW - all hosts are Solaris 10 / Xymon 4.3.3) Help? Thanks! david ~~~~~~~~~~~~~~~~~~~ David Mills Systems Administrator Northrop Grumman 512-595-1238 david.mills at ngc.com

2 1

Different displays for "disabled" tests
by David.Mills＠ngc.com 09 Nov '12

09 Nov '12

All - I have a situation where I have two Xymon servers (both 4.3.3 / Solaris 10) working off the same NFS-shared data. The only differences between the configs that I know about are the lines in xymonserver.cfg detailing the host names, and IP addresses. However, for any given host with disabled tests, one server displays columns for that host which have been disabled as blue, but the other server completely omits the columns for that host which are disabled! In case I'm not being clear, if I have server "XYMON_1" showing columns "xymond" and "iostat2" for host "racecar" as disabled, my other server ("XYMON_2") will simply show all columns except "xymond" and "iostat2" for host "racecar". The column headers won't appear for that host at all. What causes this "dual" behavior? Thanks! david ~~~~~~~~~~~~~~~~~~~ David Mills Systems Administrator Northrop Grumman 512-595-1238 david.mills at ngc.com

2 1

Xymon Disk Space issue on 4.2.3
by novosirj＠umdnj.edu 08 Nov '12

08 Nov '12

Hi all, What can I make of this? red Fri Nov 2 14:16:37 EDT 2012 - Filesystems NOT ok red /project (2391754944 units free) has reached the PANIC level (26210000 units) /dev/mapper/VolGroup01-LogVol10 5986310368 3290516412 2391754944 58% /project The client is Linux, and the host is Solaris 9. Clearly some creative math going on here. Anything I can do? Thanks. -- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/EI-Academic Svcs. - ADMC 450, Newark

2 1

SVCS in analysis.cfg
by lebarber＠gmail.com 08 Nov '12

08 Nov '12

It seems that the man page for the SVCS entries in analysis.cfg is incomplete. In particular it doesn't show how to set the color, does anybody here know how to do that? Thanks, Larry Barber

1 0