November 2012 - Xymon - Xymon mailman web

ESXi Hardware Health Monitor
by sanderson＠CORNETSER.com 13 Nov '12

13 Nov '12

Fellow XYmoners, Here is my RC1 of a VMWare ESX vShpere Hardware health monitoring script for XYmon that can be used on any remote network to send data back to your XYmon display. I tie in to the CIM data produced by the hardware so you can get any notice of failures to the ESX hosts hardware. http://www.squidworks.net/2012/11/xymon-esx-hardware-health-monitor/ This is a bit different from the ESXi script that provides VM level health and stats. You get: FAN,CPU failure, MEMORY Module Errors, RAID, Drive and Controller errors, Power supply failures and lost redundancy and so on. Shannon Anderson, MCP, MCP+I, MCSE Support Centre Systems Engineer Corporate Network Services "Count on Us" 20010 Fisher Ave, Suite E Poolesville, MD 20837 direct: 240-425-4904 | main: 301.948.8077 | fax: 301.349.2518 http://www.cornetser.com<http://www.cornetser.com/> [Description: Description: Description: Description: Description: Facebook]Join CNS on Facebook<http://www.facebook.com/#!/CorporateNetworkServices?ref=ts> AWE Best Place to Work 2006-2012; Eco-Leader 2009-2012; Health and Wellness Trailblazer 2010-2012 ...ask me how you can feel happier about IT support with [Description: Description: Description: Description: untitled] P Please consider the environment before printing this email

3 4

ack's in xymon
by wrichardson＠llbean.com 12 Nov '12

12 Nov '12

My operations team would like to ack a yellow alert but would like that ack (check mark) to go away if it escalates to a red alert so it can be seen as a new event from "All non green view" As it is today if they ack a yellow alert for 24 hours that yellow ack'd alert could go back and forth from yellow to red and it would go unnoticed. So I edit the (OKCOLORS="green,blue,clear") in the xymonserver.cfg to look like OKCOLORS="green,blue,clear,red" I restart xymon but that didn't work same behavior as the default setting. Changing it to OKCOLORS="green,blue,clear,yellow" Now the above change will clear the ACK going from a red to a yellow but that's not what I was looking for. I need to clear the ACK going from a yellow to a red alert. Any thoughts on how to do this? Thanks I have looked but cant seem to find the answer to this one Bill Richardson Bill Richardson Systems Management Engineer wrichardson at llbean.com<mailto:rwrichardson at llbean.com> 207-552-2729

1 0

#comments not showing xymon 4.3.10
by Cedric.Steenbeke＠aznikolaas.be 12 Nov '12

12 Nov '12

Guys, Since we upgraded our Xymon server to 4.3 the #COMMENT tag is not working as it used to. For instance: 192.168.3.4 MYSERVER01 #COMMENT:"server comment" The comment does not show up in the Xymon overview. Any ideas? Cedric Steenbeke

2 1

Using "IGNORE" to exclude filesystems
by Mike.Burger＠FreedomMortgage.com 09 Nov '12

09 Nov '12

I have a host for which I'd like to ignore 27 filesystems...they're all at 97-100% full, and we expect them to be at that level...as such, there's no reason for Xymon to be alarming on them. Reading the analysis man page, I've tried a number of options to try to get this set of filesystems ignored, to no avail. The hostname of the system is "edmprod" and the filesystems are /da01 thru /da27. The stanza I've tried to use is: HOST=%edmprod.* DISK %^\/da* IGNORE I've also tried: HOST=edmprod DISK %^\/da* IGNORE And HOST=edmprod DISK /da01 IGNORE DISK /da02 IGNORE DISK /da03 IGNORE DISK /da04 IGNORE DISK /da05 IGNORE DISK /da06 IGNORE DISK /da07 IGNORE DISK /da08 IGNORE DISK /da09 IGNORE DISK /da10 IGNORE DISK /da11 IGNORE DISK /da12 IGNORE DISK /da13 IGNORE DISK /da14 IGNORE DISK /da15 IGNORE DISK /da16 IGNORE DISK /da17 IGNORE DISK /da18 IGNORE DISK /da19 IGNORE DISK /da20 IGNORE DISK /da21 IGNORE DISK /da22 IGNORE DISK /da23 IGNORE DISK /da24 IGNORE DISK /da25 IGNORE DISK /da26 IGNORE DISK /da27 IGNORE I'd love to know what I'm doing wrong, here, if anyone is so inclined to assist me. -- Mike Burger AIX Administrator <http://freedomhome.fhmc.local/intranet/main.jsp> Phone (317) 537-3680, Fax (317) 537-4680, Cell (317) 797-2040 E-mail: Mike.Burger at FreedomMortgage.com <mailto:Anthony.House at freedommortgage.com> "Once word leaks out that a pirate's gone soft, people begin to disobey you and then it's nothing but work, work, work, all the time." --Westley/The Dread Pirate Roberts CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.

8 16

Sending client data without client installed
by jlaidman＠rebel-it.com.au 09 Nov '12

09 Nov '12

On 1 November 2012 05:58, Andy Smith <abs at shadymint.com> wrote: > Raymond Lee wrote: > > Maybe too late now, but I have had good success with this for various > linux based appliances :- > > http://tools.rebel-it.com.au/**xymon-rclient/<http://tools.rebel-it.com.au/xymon-rclient/> (and now listed on Xymonton) Great to hear this is getting some use. In theory, this should work with a suitable client script in client/bin called something like xymonclient-cisco_acs.sh, probably based on the xymonclient-linux.sh. But if you only have one type of appliance, a custom script like Henrik suggested (and Raymond implemented) is a much simpler solution. Raymond, it would be good to see your final solution to this. J

2 1

FW: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10
by Don.Kuhlman＠schawk.com 09 Nov '12

09 Nov '12

Thanks Larry. Looks like everything went purple again at 6:45 this morning. The logs still show 0 bytes. Any other suggestions for trying to figure this out? Regards, Don From: Larry Barber <lebarber at gmail.com<mailto:lebarber at gmail.com>> Date: Mon, 5 Nov 2012 17:19:53 -0600 To: Don Kuhlman <don.kuhlman at schawk.com<mailto:don.kuhlman at schawk.com>> Subject: Re: [Xymon] FW: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10 Xymonnet tends to be pretty quiet unless something goes wrong. You won't be able to tell for sure until you get one of your purple storms. Alerts are handled by a different module. Look in tasks.cfg to find it. Thanks, Larry Barber On Mon, Nov 5, 2012 at 3:53 PM, Don Kuhlman <Don.Kuhlman at schawk.com<mailto:Don.Kuhlman at schawk.com>> wrote: Hi Larry/all. I've noticed that the xymonnet.log and xymonnet-again.log files are staying at 0 bytes. Does that seem to be indicating a problem? (and Xymon hasn't gone purple all day, but I'm still not sending any email alerts to anyone). -rw-rw-rw- 1 xymon xymon 0 Nov 5 15:05 /var/log/xymon/xymonnet-again.log -rw-rw-rw- 1 xymon xymon 0 Nov 5 15:07 /var/log/xymon/xymonnet.log Thanks Don K From: Larry Barber <lebarber at gmail.com<mailto:lebarber at gmail.com>> Date: Mon, 5 Nov 2012 11:19:32 -0600 To: Don Kuhlman <don.kuhlman at schawk.com<mailto:don.kuhlman at schawk.com>> Cc: Xymon Email List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: Re: [Xymon] FW: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10 All the server side Xymon logs are in /var/log/xymon by default. Since you say that you are getting purple storms for conn and http tests, this suggests that the problem is likely with your xymonnet process. Check the xymonnet log, and when you see the purples check to see if there is a xymonnet instance running. If this instance has been running for more than a few minutes, kill it. If the xymonnet process is hanging, you might want to set the MAXTIME parameter on the xymonnet process in tasks.cfg. Doesn't really fix the problem, but it will at least stop things from going purple. Thanks, Larry Barber On Mon, Nov 5, 2012 at 10:01 AM, Don Kuhlman <Don.Kuhlman at schawk.com<mailto:Don.Kuhlman at schawk.com>> wrote: Update to this. While googling further, I saw a thread titled "[hobbit] stale alerts". This mentioned that there could be an external script that I created which may cause issues for xymon when it runs. I do have a diskstat.sh script that may be causing problems. For now, I'm setting it to DISABLED in the tasks.cfg file. Is there a way to see log information in xymon to try and verify something like this? Thanks Don K From: Don Kuhlman <don.kuhlman at schawk.com<mailto:don.kuhlman at schawk.com>> Date: Mon, 5 Nov 2012 08:34:29 -0600 To: Xymon Email List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: Troubleshooting Purple CONN and HTTP Tests in Xymon 4.3.10 Hi folks. We've been running xymon for about 10 months now. It's been fine all this time. However last week around Wednesday we started getting purple storms on the CONN and HTTP tests for all our hosts. I stop Xymon and restart it, or reboot the server (Linux 5.x) and then it comes back ok. This also happened Thursday, and then again Saturday around 2PM cst. Anyone have a link or source for which logs to look in on the server or xymon to see what may be causing the CONN and HTTP tests to randomly start failing like this or where to start troubleshooting? Can I use xymonlaunch —debug like this to see what is happening? /usr/lib64/xymon/server/bin/xymonlaunch --debug --config=/usr/lib64/xymon/server/etc/tasks.cfg --env=/usr/lib64/etc/xymonserver.cfg While searching the xymon forum and message boards, I saw some things that say it may be disk space or inodes, but it seems like we are ok there - df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 3899392 204731 3694661 6% / tmpfs 490139 6 490133 1% /dev/shm /dev/sda1 32768 51 32717 1% /boot df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda2 61312028 5748784 52448700 10% / tmpfs 1960556 188 1960368 1% /dev/shm /dev/sda1 516040 87716 402112 18% /boot DNS also seems fine. Thanks Don K _______________________________________________ Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon

2 5

RRD - Aberrant Behavior Detection with Holt-Winters Forecasting
by jlaidman＠rebel-it.com.au 09 Nov '12

09 Nov '12

On 24 October 2012 06:19, Nico <nicolas at lienard.name> wrote: > now, if i add the def: > > DEF:b=test_rrd.rrd:test:HWPREDICT > > the graph is not showing anymore [even if i didn t use the new DEF on a > LINE] :( > In debug mode, showgraph.cgi will display the arguments sent to rrd_graph(). Try adding "--debug" to CGI_SHOWGRAPH_OPTS in cgioptions.cfg, and have a look at the parameters. Then try running "rrdtool graph ..." with the same parameters and see what it says. If you're able to grab the URL for the image, use wget to fetch the URL and show the contents. You should see the rrd_graph error message, like this: <html><head><title>Invalid request</title></head> <body>the RRD does not contain an RRA matching the chosen CF</body></html> This is what I get when I add a line with HWPREDICT, where I haven't defined it in my RRD file (yet). You might get a different message that indicates where the problem is. J

1 0

xmond_channel --channel=data: There can be only one?
by David.Mills＠ngc.com 09 Nov '12

09 Nov '12

All - I know I did something bad, but don't know exactly what... I ran across this posting from Henrik (http://lists.xymon.com/archive/2011-August/032267.html) indicating one could see the data coming across to the server from all the clients using a command similar to this: xymoncmd xymond_channel --channel=data egrep "egrep '^\@\@'" (Henrik's example used "status" rather than "data") So, I kicked it off and walked away for about an hour. Fine. When I returned some colleagues came by with worried expressions asking me to look at our Non-Green report. Rather than the usual handful of non-green hosts, there were rows and rows of hosts showing 'purple' in various columns. I killed my manual "xymond_channel..." command and watched as the Non-Green report shrank quickly to a normal display. What happened? * It appeared the appearance of various "purple" hosts in the environment was random: random hosts with random columns (but I won't swear to that) * At least one monitored host's "proc"s ended up as "red", rather than "purple", but this transition was long after the initial onset of my command. The user believes there was no real "crash" of his procs on his host. * We have two servers monitoring the same hosts in our environment. On the host used to run my manual command, pretty much all graph data for the Xymon server itself shows a gap: no data available while my command ran. * From the brother server, looking at graphs for the deranged server, we see an odd *decrease* in CPU load for the duration of my extra command, and an *increase* in TCP traffic I'm guessing my manual command competed / randomly intercepted incoming data for this server, but does this fit all the oddities observed? In any case, I'm back at square one for my original goal of *safely* seeing what raw data is coming into the server from my clients. (FWIW - all hosts are Solaris 10 / Xymon 4.3.3) Help? Thanks! david ~~~~~~~~~~~~~~~~~~~ David Mills Systems Administrator Northrop Grumman 512-595-1238 david.mills at ngc.com

2 1

Different displays for "disabled" tests
by jlaidman＠rebel-it.com.au 09 Nov '12

09 Nov '12

On 31 October 2012 09:30, Mills, David (IS) <David.Mills at ngc.com> wrote: > What causes this “dual” behavior? > Is this what would happen if xymond was run with "--no-purple" and those tests had gone purple? Can you confirm that the status is actually blue for these tests? J

1 0

Xymon Disk Space issue on 4.2.3
by novosirj＠umdnj.edu 08 Nov '12

08 Nov '12

Hi all, What can I make of this? red Fri Nov 2 14:16:37 EDT 2012 - Filesystems NOT ok red /project (2391754944 units free) has reached the PANIC level (26210000 units) /dev/mapper/VolGroup01-LogVol10 5986310368 3290516412 2391754944 58% /project The client is Linux, and the host is Solaris 9. Clearly some creative math going on here. Anything I can do? Thanks. -- ---- _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Sr. Systems Programmer |$&| |__| | | |__/ | \| _| |novosirj at umdnj.edu - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.|IST/EI-Academic Svcs. - ADMC 450, Newark

2 1