October 2012 - Xymon - Xymon mailman web

Proving that tasks.d/ directory is working
by Martin.Ward＠colt.net 20 Oct '12

20 Oct '12

Hi all, I have a few tasks that I have designed to run on the Xymon server. I have created files in the tasks.d/ subdirectory based on the configs in the tasks.cfg file but they don't seem to be running. One example entry from a file in tasks.d is: # "replicate_ldap" runs the in-house LDAP replication checks. [ldap_replication] ENVFILE /opt/xymon/server/etc/xymonserver.cfg CMD /opt/xymon/server/ext/ldap_check_replication LOGFILE $BBSERVERLOGS/ldap_replication.log INTERVAL 5m It's a simple configuration and the files exist but the log file has not been updated in weeks. I cannot find any log information about the tasks.cfg file or its inclusions, it doesn't seem to be mentioned anywhere in the Xymon log files, can someone give me a clue as to how I can debug this? |\/| [Description: Colt @ 20 email sig]<http://youtu.be/jmmcILTf88I> Martin Ward Manager, Technical Services Service Operations DDI:+44 (0) 20 7863 5218 / Fax: +44 (0)20 7863 9999 / www.colt.net<http://www.colt.net/> Colt Technology Services, Unit 12, Powergate Business Park, Volt Avenue, Park Royal, London, NW10 6PW, UK. Help reduce your carbon footprint | Think before you print. Registered in England and Wales, registered number 02452736, VAT number GB 645 4205 50 [Colt Disclaimer] This email is from an entity of the Colt group of companies. Colt Group S.A., K2 Building, Forte 1, 2a rue Albert Borschette, L-1246 Luxembourg, R.C.S. B115679. Corporate and contact information for our entities can be found at http://colt.net/uk/en/Colt-Group-of-Companies/index.htm. Internet communications are not secure and Colt does not accept responsibility for the accurate transmission of this message. Content of this email or its attachments is not legally or contractually binding unless expressly previously agreed in writing by Colt

2 2

Process report tweak? (a request from my boss's boss...)
by betsy.schwartz＠gmail.com 19 Oct '12

19 Oct '12

The process report shows a list of monitored processes at the top, with the missing process having a yellow dot. Is it possible in another release to list the bad process FIRST and SEPARATELY from the others? In email it gets sort of buried when there's a long list of monitored processes: red Fri Oct 12 15:50:56 EDT 2012 - Processes NOT ok &green /usr/sbin/cron (found 1, req. 1 or more) &red httpd (found 0, req. 1 or more) &green nscd (found 1, req. 1 or more) &green java (found 1, req. 1 or more) &red coldfusion8 (found 0, req. 1 or more) &green cron (found 1, req. 1 or more).... If people think this is a good idea, that is thanks Betsy

3 5

Brainstorming - open xymon tickets on *acknowledgement*? (and Xymon talking to Jira and ticket systems) (and a bit of Nagios)
by betsy.schwartz＠gmail.com 19 Oct '12

19 Oct '12

We're using Jira for ticketing, and Xymon for monitoring. Jira allows opening tickets via email, so we have a one-way communication. I've been asked (by someone coming from the world of Nagios) to investigate opening tickets on *ACKNOWLEDGE*. His reasoning is that if a ticket is opened on ACK, it is clear that every acked alert has a ticket and it is clear what ticket goes with the ack, and all tickets will have owners. If we open a ticket via email it might or might not have an owner right away, and there's no way to match the ticket with the dot on the screen. Also if we open tickets via email, we get multiple tickets when a service *flaps*, such as a disk that is cyclically going above the limits. Has anyone else worked with these issues? Thoughts I have so far: 1) process. Make sure that everyone who takes a ticket also acks the alert manually with the ticket number. This is clunky. 2) hook into the email ack (not currently enabled). When procmail parses the ack, trigger the ticket open email. Difficulty: alerts.cfg has logic to control which group gets which tickets; with procmail I am starting from scratch. 3) switch to Nagios (Not a short term project but it's on the table) 4) use a custom alert script that both sends the ticket email and sends an email ack. (difficulty: multiple tickets for a flapping service would do ... what?) (I think this is the most promising idea so far...) Interested in any ideas, thoughts, and ramblings on the subject(s)

2 1

Customising message subject in email alerts
by abs＠shadymint.com 19 Oct '12

19 Oct '12

Hi, This has been talked about a few times before and I am aware that we have the SCRIPT mechanism available for this purpose. However, what I really need to be able to do is annotate the subject line with the page where the object is located. The environment variables do not currently provide this information and I have been playing a little with findhost.cgi to see if it can be used within a script to extract the page. This has not been successful and anyway is flawed because the object may appear on multiple pages. I have even experimented with fetching hosts.cfg and parsing the pages which kindof works, but there must surely be a better way? Anyone had similar problems or got any other suggestions? Thanks -- Andy

2 1

Quick urgent question - *IS* there an issue with combo tests in 4.3.10?
by betsy.schwartz＠gmail.com 19 Oct '12

19 Oct '12

I'm scheduled to upgrade to 4.3.10 this weekend so if there's an issue with combo tests I need to verify that it won't affect us What are people seeing? Working ok for you? Not OK? sign/symptom/test case? thanks so much!!

2 1

Xymon Wiki Pages
by josh＠imaginenetworksllc.com 19 Oct '12

19 Oct '12

Who is responsible for this? http://en.wikibooks.org/wiki/System_Monitoring_with_Xymon I've been trying to make changes to the compiling guide for CentOS and since January I've not gotten my changes reviewed. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

2 3

diskstat.sh/RRD oddity
by sholmes42＠mac.com 18 Oct '12

18 Oct '12

This is just a comment on an oddity with respect to diskstat.sh and RRD. We make pretty heavy use of the diskstat.sh script, which I believe I downloaded from xymonton. When I installed it I used the standard clientlaunch.cfg stanza for the configuration and everything worked great. I was called to task today because we have been having some disk io issues on the RHEL VMs and someone was looking at the trend graphs for some servers to see if there was anything they could learn and they noticed that beginning at about 4pm local time on Monday the graphs for the number of sectors written per second on a couple of file systems on several VMs jumped from the 10 to 20 range to the 300 to 340 range and stayed there. The graph for number of disk writes per second had a corresponding jump up to about 40 or 50 from close to zero. In analyzing the data I discovered that the file system that was displaying this behavior is the same file system to which the diskstat.sh script is writing its temp files. It appears that for some reason, starting at 4pm on Monday the 5 minute test interval and the 5 minute average for RRD got in sync and all it was seeing was the data point that corresponded to its own writing activity and RRD was using it for the entire 5 minute average (of course, that's what RRD does). I 'fixed' it by changing the test interval to something less than 5 minutes. I tried 2, 3, and 4 minutes and they all had the effect of reducing the data in the plot back to the expected level, i.e. to the level it was before 4pm on Monday. The mystery remains why it suddenly started seeing and using its own disk activity at the same time on several different servers. Steve Holmes ITaP/Purdue University -- If they give you ruled paper, write the other way. -Juan Ramon Jimenez, poet, Nobel Prize in literature (1881-1958) I prayed for freedom for twenty years, but received no answer until I prayed with my legs. -Frederick Douglass, Former slave, abolitionist, editor, and orator (1817-1895)

5 10

Metrics Reports
by rblumenthal＠telmar.com 17 Oct '12

17 Oct '12

When trying to run a metrics report with multiple servers it cannot display the graph, however it works when choosing 1 server. Any insight on where I should check? Thanks

1 0

xymon failure this am
by betsy.schwartz＠gmail.com 17 Oct '12

17 Oct '12

Still on 4.3.7, on RHEL5. Had a custom test fail and hang last night, a perl script called from tasks.cfg . Not sure if this is related. But six hours later xymon started failing wtih 2012-10-17 06:51:00 Whoops ! Failed to send message (Connection failed) 2012-10-17 06:51:00 -> Could not connect to Xymon daemon at 10.100.5.42:1984 (Connection refused) 2012-10-17 06:51:00 -> Recipient '10.100.5.42', timeout 15 drilling into detail pages was also failing. Resetting fixed it and nobody thought to run the self test. Betsy PS the self test I created was a file monitor that looks for /home/xymon/TESTOK . If that file exists it sends email to whatever addresses are in /home/xymon/testalert.txt . All of our support team has privilege to touch those two files, and the rule is the last one in alerts.cfg . Created this after munging up alerts.cfg so some pages were not going out.

1 0

Plagued by purple
by David.Mills＠ngc.com 16 Oct '12

16 Oct '12

> Josh Luthman josh at imaginenetworksllc.com <mailto:xymon%40xymon.com?Subject=Re%3A%20%5BXymon%5D%20Plagued%20by%20purple&In-Reply-To=%3CCAN9qwJ-iF_r5AsQx5HFnpgK%3DPDnr0qtt1m%3DxnGjKdnVAFGwzSA%40mail.gmail.com%3E> > >Either the server isn't gathering it or the clients aren't sending the data. > >The raid script that I have is sent by the client. Thanks, Josh, but can you point out where I can check to see which of those options is my likely culprit? Also, don't quite follow your comment about your "raid script". What raid script? Sorry! David >On Oct 16, 2012 2:08 AM, "Mills, David (IS)" <David.Mills at ngc.com> wrote: > >> All . >> >> Am fairly new to Xymon and apologize if this is documented somewhere and I >> missed it. >> >> I have inherited a Xymon server (4.3.3 on Solaris 10) which shows itself >> as having about half its tests as .purple., the other half as green. As far >> as I can tell, everything.s running just fine. It seems to be a simple data >> refresh problem of some sort, but haven.t been able prove my hunch. >> >> The problem started after a reboot of the server. Everything came up OK, >> except for the following list of .missing. tests: >> >> Purple Tests: >> iostat iostat2 mem2 meta nfs prtdiag raid sar tm zones >> >> Green Tests: >> bbd conn cpu disk http info memory msgs >> ntp ports procs ssh trends >> >> I.ve restarted the server: (.$xymon_home/xymon.sh restart.) and tried to >> find something in the logs, but am not making ground on this fast enough. >> >> Help? >> >> Thanks! >> >> david ~~~~~~~~~~~~~~~~~~~ David Mills Systems Administrator Northrop Grumman 512-595-1238 david.mills at ngc.com

5 5