October 2014 - Xymon - Xymon mailman web

Sudden rash of "logfetch" crashes in libc on RHEL 6 Linux?!?
by earle＠isolar.DynDNS.ORG 21 Oct '14

21 Oct '14

Starting about 45 minutes ago as I type this, all of a sudden, 5 of my machines running Xymon 4.3.17 have gotten afflicted with a mysterious 'disease' - "logfetch" is endlessly crashing on all of them. The core files being created are getting corrupted for some reason, and they are magically getting deleted as a result, so I can't debug the problem. One machine is running 64-bit RHEL 6.5; the rest 64-bit RHEL 6.3. Client version is as I mentioned 4.3.17: xymon-client-4.3.17-1.el6.x86_64 This cycle is getting repeated every 1 minute and 40 seconds - help! Oct 20 15:42:42 linux1 kernel: logfetch[2334]: segfault at ef ip 000000355a4c835f sp 00007fff0ac56a80 error 6 in libc-2.12.so[355a400000+189000] Oct 20 15:42:42 linux1 abrt[2335]: //core.2334 fd(-1) is not a regular file with link count 1: Permission denied Oct 20 15:42:43 linux1 abrt[2335]: Saved core dump of pid 2334 (/usr/bin/logfetch) to /var/spool/abrt/ccpp-2014-10-20-15:42:42-2334 (335872 bytes) Oct 20 15:42:43 linux1 abrtd: Directory 'ccpp-2014-10-20-15:42:42-2334' creation detected Oct 20 15:42:43 linux1 abrtd: Package 'xymon-client' isn't signed with proper key Oct 20 15:42:43 linux1 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-10-20-15:42:42-2334' exited with 1 Oct 20 15:42:43 linux1 abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2014-10-20-15:42:42-2334, deleting I doubt that "libc" got simultaneously corrupted on all 5 systems so there must be something wrong in "logfetch". The segfault ip is always "000000355a4c835f" on the 4 RHEL 6.3 systems while the stack pointer varies. - Greg P.S. Why am I getting these "isn't signed with proper key" messages as well?

2 1

Recover messages
by colin.coe＠gmail.com 19 Oct '14

19 Oct '14

Hi all Is there any way to only send recover messages between, say 6AM-10PM? Some of those that I work with are getting grumpy about recover messages while they're sleeping. Thanks CC

2 1

batch configuration of critical systems?
by rlhamil2＠gmail.com 19 Oct '14

19 Oct '14

Is there a recommended/supported mechanism to configure (and as needed, update or replace the configuration) of critical systems from the command line? Assume an external database of service levels or priorities, from which the importance of monitored systems can be determined; and probably some heuristics to expand that into which tests for which hosts to treat as critical. I want to write a script to fetch that and create/update/replace the critical systems configuration on a xymon server; not live from the database, but periodically (probably run by cron as needed, e.g. 4x daily weekdays or something like that).

4 4

Modify email notification template?
by robr7535＠gmail.com 17 Oct '14

17 Oct '14

Hi, is there a way to modify the default email notification template on xymon 4.3.17 so that multiple items within the same monitored category are listed one per line? For example, when monitoring multiple processes, the email alert will list them in one long string, as opposed to breaking them apart on separate lines. If you have a lot of processes it's hard to tell which one cause the alert at first look. I've searched the documentation and the web, but found no obvious way to fix this besides using a custom external script to rewrite the message. Which is doable, but sounds like we just need a line break between each item so it's easier to read. What we have now in the default email alert: host,domain,net.procs red Fri Oct 17 13:34:50 EDT 2014 - Processes NOT ok &green sshd (found 1, req. between 1 and 1) &red Proc1 (found 0, req. between 1 and 1) &red Proc2 (found 0, req. between 1 and 1) &green Proc3 (found 2, req. 2 or more) &green Proc4 (found 1, req. between 1 and 1) &green Proc5 (found 1, req. between 1 and 1) &green Proc6 (found 2, req. between 2 and 2) &green Proc7 (found 4, req. 4 or more) &green Proc8 (found 1, req. between 1 and 1) &green Proc9 (found 1, req. between 1 and 1) What we would like to have: host,domain,net.procs red Fri Oct 17 13:34:50 EDT 2014 - Processes NOT ok &green sshd (found 1, req. between 1 and 1) &red Proc1 (found 0, req. between 1 and 1) &red Proc2 (found 0, req. between 1 and 1) &green Proc3 (found 2, req. 2 or more) &green Proc4 (found 1, req. between 1 and 1) &green Proc5 (found 1, req. between 1 and 1) &green Proc6 (found 2, req. between 2 and 2) &green Proc7 (found 4, req. 4 or more) &green Proc8 (found 1, req. between 1 and 1) &green Proc9 (found 1, req. between 1 and 1) Any feedback would be appreciated.

3 3

Multiple Issues with 4.3.17 install
by jlaidman＠rebel-it.com.au 17 Oct '14

17 Oct '14

Neal I have to say, apart from the mention of xymon_rrd crashing, much of what I see appears normal. The "network listener on 0.0.0.0:1984" indicates that xymond is listening to connections any IP addresses. xymond will go yellow when when it's restarted. The messages under "Latest error messages" all appear to be normal start-up messages for xymond. Cheers Jeremy On 17 October 2014 09:40, Neal, Jonathan W via Xymon <xymon at xymon.com> wrote: > _______________________________________________ > Xymon mailing list > Xymon at xymon.com > http://lists.xymon.com/mailman/listinfo/xymon > > > ---------- Forwarded message ---------- > From: "Neal, Jonathan W" <wes.neal at verizon.com> > To: "xymon at xymon.com" <xymon at xymon.com> > Cc: > Date: Thu, 16 Oct 2014 18:40:46 -0400 > Subject: Multiple Issues with 4.3.17 install > > I am coming from an early 4.2 install. I merged my bb-hosts, > hobbit-alerts.cfg and hobbit-clients.cfg files into the proper files in the > new 4.3.17 install. I also copied over the entire histlogs directory from > data. Currently xymon_rrd keeps dying and going purple with a Fatal > signal error. > > > > rrd-status log has this in it going back most of the day: > > > > 2014-10-16 19:23:00 Peer at 0.0.0.0:0 failed: Broken pipe > > 2014-10-16 19:23:00 Peer not up, flushing message queue > > 2014-10-16 19:24:43 Shutting down, flushing cached updates to disk > > 2014-10-16 19:28:39 Peer not up, flushing message queue > > 2014-10-16 20:00:35 Shutting down, flushing cached updates to disk > > 2014-10-16 20:00:36 Cache flush completed > > 2014-10-16 21:58:19 Peer not up, flushing message queue > > 2014-10-16 22:30:58 Shutting down, flushing cached updates to disk > > 2014-10-16 22:30:59 Cache flush completed > > 2014-10-16 22:31:14 Peer not up, flushing message queue > > > > Xymond is also constantly going yellow and I again see that 0.0.0.0:1984 > that is mentioned above: > > > > Statistics for Xymon daemon > > Version: 4.3.17 > > Up since 16-Oct-2014 22:31:09 (0 days, 00:04:59) > > > > Incoming messages : 937 > > - status : 885 > > - combo : 1 > > - extcombo : 22 > > - page : 0 > > - summary : 0 > > - data : 6 > > - client : 2 > > - notes : 0 > > - enable : 0 > > - disable : 0 > > - ack : 0 > > - config : 4 > > - query : 0 > > - xymondboard : 6 > > - xymondlog : 5 > > - drop : 0 > > - rename : 0 > > - dummy : 1 > > - ping : 0 > > - notify : 0 > > - schedule : 0 > > - download : 0 > > - Bogus/Timeouts : 5 > > Incoming messages/sec : 3 (average last 300 seconds) > > > > status channel messages: 885 (1 readers) > > stachg channel messages: 877 (1 readers) > > page channel messages: 37 (1 readers) > > data channel messages: 6 (1 readers) > > notes channel messages: 0 (0 readers) > > enadis channel messages: 0 (0 readers) > > client channel messages: 2 (1 readers) > > clichg channel messages: 0 (1 readers) > > user channel messages: 0 (0 readers) > > backfeed messages : 0 > > > > > > Latest error messages: > > Loading hostnames > > Loading saved state > > Setting up network listener on 0.0.0.0:1984 > > Setting up signal handlers > > Setting up xymond channels > > Setting up logfiles > > Setup complete > > > > Can anyone tell me what might be going on? > > Thanks in advance! > > > >

2 1

function run_command in ../lib/run.c assumes sh but uses any $SHELL
by gendalia＠iastate.edu 17 Oct '14

17 Oct '14

If I restart xymon while using tcsh, for example, suddenly my ntp test breaks, because: === red Thu Oct 16 22:39:47 2014 ntp NOT ok Service ntp on <elided> is not OK : Service unavailable Command: ntpdate -u -q -p 2 <elided> 2>&1 1: Permission denied. === /var/log/xymon/1, of course, is confused about the value of 2, because it can write there sometimes, which is also interesting. For the moment, I've hard coded 'export SHELL=/bin/sh' in the startup script for xymon, but the assumption shouldn't be made that $SHELL is a /bin/sh derivative. -Tracy

3 4

SSLv3 vs TLS, not working for !https
by gjs＠sonnit.dk 17 Oct '14

17 Oct '14

Hi all So looking at the recent POODLE SSL v3 attack, I started reconfiguring servers and hosts. I same across a problem, not with https sites - as Xymon supports TLS on those, but pop3, imaps and probably more. Services are reported down, but they are up, just not supporting SSLv3 Error output: Unspecified SSL error in SSL_connect to pop3s (58115/tcp) on host 192.168.201.xx: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure Unspecified SSL error in SSL_connect to imaps (57603/tcp) on host 192.168.201.xx: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure I'm not sure what to patch here, to make xymonnet use TLS instead of SSL on all protocols. Any suggestions would be most appreciated, as many of our hosts are now reported as down. -- Best regards Gorm J. Siiger

1 0

Multiple Issues with 4.3.17 install additional
by wes.neal＠verizon.com 16 Oct '14

16 Oct '14

The other things that seems to be happening is that all alerts are going to all possible recipients. The info panel for most all of the systems I checked look like this: conn wes.neal at one.verizon.com (R) - - 1h - purple,red page-ems-team1 at one.verizon.com 10m 1s - 2d - purple,red wes.neal at one.verizon.com(R) - - 1h - purple,red page-ems-team1 at one.verizon.com 10m 1s - 2d - purple,red page_ems_team3 at lists.verizonbusiness.com 10m 1s - 2d - purple,red page_ems_team3 at lists.verizonbusiness.com 10m 1s - 2d - purple,red page_ems_team3 at lists.verizonbusiness.com 10m 1s - 2d - purple,red page_ems_team3 at lists.verizonbusiness.com The rules which I copied over from my old system look like this though for this system in particular. HOST=$aspmdpbulk MAIL $ems1 REPEAT=60m RECOVERED MAIL page-ems-team1 at one.verizon.com SERVICE=conn DURATION>10 REPEAT=2d All of my rules in alerts.cfg pretty much look like the above. An email for everything and a page for connectivity. I have checked the file for white space, typos etc. I do not see any. Is there script or something out there that will parse the *.cfg files for errors?

1 0

Multiple Issues with 4.3.17 install
by wes.neal＠verizon.com 16 Oct '14

16 Oct '14

I am coming from an early 4.2 install. I merged my bb-hosts, hobbit-alerts.cfg and hobbit-clients.cfg files into the proper files in the new 4.3.17 install. I also copied over the entire histlogs directory from data. Currently xymon_rrd keeps dying and going purple with a Fatal signal error. rrd-status log has this in it going back most of the day: 2014-10-16 19:23:00 Peer at 0.0.0.0:0 failed: Broken pipe 2014-10-16 19:23:00 Peer not up, flushing message queue 2014-10-16 19:24:43 Shutting down, flushing cached updates to disk 2014-10-16 19:28:39 Peer not up, flushing message queue 2014-10-16 20:00:35 Shutting down, flushing cached updates to disk 2014-10-16 20:00:36 Cache flush completed 2014-10-16 21:58:19 Peer not up, flushing message queue 2014-10-16 22:30:58 Shutting down, flushing cached updates to disk 2014-10-16 22:30:59 Cache flush completed 2014-10-16 22:31:14 Peer not up, flushing message queue Xymond is also constantly going yellow and I again see that 0.0.0.0:1984 that is mentioned above: Statistics for Xymon daemon Version: 4.3.17 Up since 16-Oct-2014 22:31:09 (0 days, 00:04:59) Incoming messages : 937 - status : 885 - combo : 1 - extcombo : 22 - page : 0 - summary : 0 - data : 6 - client : 2 - notes : 0 - enable : 0 - disable : 0 - ack : 0 - config : 4 - query : 0 - xymondboard : 6 - xymondlog : 5 - drop : 0 - rename : 0 - dummy : 1 - ping : 0 - notify : 0 - schedule : 0 - download : 0 - Bogus/Timeouts : 5 Incoming messages/sec : 3 (average last 300 seconds) status channel messages: 885 (1 readers) stachg channel messages: 877 (1 readers) page channel messages: 37 (1 readers) data channel messages: 6 (1 readers) notes channel messages: 0 (0 readers) enadis channel messages: 0 (0 readers) client channel messages: 2 (1 readers) clichg channel messages: 0 (1 readers) user channel messages: 0 (0 readers) backfeed messages : 0 Latest error messages: Loading hostnames Loading saved state Setting up network listener on 0.0.0.0:1984 Setting up signal handlers Setting up xymond channels Setting up logfiles Setup complete Can anyone tell me what might be going on? Thanks in advance!

1 0

PROC change needed
by dewelker＠gmail.com 16 Oct '14

16 Oct '14

I have this java command-line that shows up in a PROC status (ps listing) on multiple lines. I'd like to be able to have this text replaced with some one-liner. I thought the way to do that was using a PROC line in the analysis.cfg file with the TEXT option, but that doesn't do anything to either the client data OR the PROC status page. Can this somehow be done in a xymond_worker? Can it ONLY be done in a xymond_worker? Sample analysis.cfg file: HOST=* PROC="java" 1 100 green TEXT="java - with a lot of parameters" Any ideas or thoughts would be appreciated. Thanks, David

3 3