December 2014 - Xymon - Xymon mailman web

[PATCH] Fix OpenBSD client support
by feld＠feld.me 30 Dec '14

30 Dec '14

OpenBSD client is broken and I'm not sure how long this has been the case. Attached patch against 4.3.18 branch fixes it. However, I'm noticing another problem I cannot explain: in the cpu output the "top" data has this error in it: warning: process count should be a non-negative number -- using default Reading through the Client Data top seems to be running with the wrong arguments: top -d -20 After further testing I can only get this message to appear in my own invocations of "top" if I give a negative number with the -d argument. Unfortunately while grepping through the code I cannot find where this is being called from -- it's certainly not in the xymonclient-openbsd.sh file. A note about the patch: The change in lib/sendmsg.c is due to OpenBSD not having msglen_t in <sys/msg.h>. It is unsigned long on FreeBSD and I assume it's the same on other platforms as well or there would be issues. The sendmsg.c probably won't apply cleanly to trunk because it appears much has changed in sendmsg.c and this code is about 200 lines earlier.

1 0

monitoring multiple log files on a single syslog server
by usaims＠yahoo.com 29 Dec '14

29 Dec '14

Greetings, I'm monitoring multiple log files on a single syslog server. I have a syslog server that stores the apache error logs in a server named ‘syslogserver’ looking for the word ‘CurlTransfer’. When xymon finds the word ‘CurlTransfer’ it sends an alert but it alerts that it is coming from ‘syslogserver’ and the guys are complaining that ‘nongreen.html’ is reporting the servername rather than the actual name of the server where the error originated from. Is there a way if xymon finds the word ‘CurlTransfer’ in ‘/opt/syslogs/Server09/apache2.error.log’, to get it to report in the ‘msgs’ column that it came from ‘Server09’? Here is my current config for this. [syslogserver] log:/opt/syslogs/Server09/apache2.error.log:4096 log:/opt/syslogs/Server10/apache2.error.log:4096 log:/opt/syslogs/Server11/apache2.error.log:4096 log:/opt/syslogs/Server12/apache2.error.log:4096 log:/opt/syslogs/Server13/apache2.error.log:4096 I have configured my analysis.cfg with: HOST= syslogserver LOG /opt/syslogs/Server09/apache2.error.log CurlTransfer COLOR=red LOG /opt/syslogs/Server10/apache2.error.log CurlTransfer COLOR=red LOG /opt/syslogs/Server11/apache2.error.log CurlTransfer COLOR=red LOG /opt/syslogs/Server12/apache2.error.log CurlTransfer COLOR=red LOG /opt/syslogs/Server13/apache2.error.log CurlTransfer COLOR=red Thanks in advance. usaims

1 0

Getting data from a "pull only" site
by john.thurston＠alaska.gov 26 Dec '14

26 Dec '14

I wanted to be able to see how my network looked from the "outside". Xymonnet running http tests was going to be perfect. A major sticking point was our local network access policies which made it very difficult for a host on the internet to access an "inside" server to deliver xymon messages. The solution I am currently testing uses: Outside: A Xymon server enabling only xymonnet and msgcache in tasks.cfg Inside: A line in my hosts.cfg for the "outside" server with a "pulldata" tag Enabling xymonfetch in tasks.cfg xymonnet on the "outside" Xymon server performs whatever tests are defined in its hosts.cfg. It reports the results to msgcache running locally. xymonfetch on the "inside" Xymon server finds the "pulldata" flag, contacts the "outside" server, pulls all new messages from it, and passes them to xymond running locally. xymond handles the messages using its regular let of rules. This lash-up lets me perform network tests from outside my network, but display the test results alongside all my other tests on the main Xymon web server. This is done even though the outside server is not allowed to reach the inside server. Limitations: The PULLDATA tag is supposed to support [=ip[:port]] modifiers. These modifiers do not work. xymonfetch always queries port 1984 of the specified host's IP address. This is a defect. The data is passed un-encrypted across the network. In my case, all of the data is a result of testing publicly accessible web pages. Consider what your messages contain before trying this. msgcache has an option to limit transfers to requests from only certain IP addresses. Since my "outside" network is NAT'd, I chose to implement my access restrictions at the firewall layers. One PULLDATA tag is all that is required to retrieve all new messages from the distant msgcache. If the host/client names on the messages match up with entries on the main Xymon server, the messages will be incorporated. All messages are accepted from the distant site, including "drop" and "rename" commands. This lash-up requires that you trust that no one will send malicious messages to the msgcache on the distant site. I only see one way around this; create a "pulling" server, running xymonfetch and xymonproxy, on a new IP address and use the --admin-senders option on xymond of the main Xyon server. I don't know how many (or how quickly) messages can be cached by msgcache. It was designed to accept messages from an isolated client. It is possible that it will not be able to keep up with an influx of messages from xymonnet. -- Do things because you should, not just because you can. John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska

1 0

Freebsd port may interest Henrik
by jungleboogie0＠gmail.com 22 Dec '14

22 Dec '14

Hello All, Interesting problem here with xymon-server and sslv2 and sslv3 on freeBSD that I'm told Henrik may like looking into... Failure log: http://package18.nyi.freebsd.org/data/101amd64-default-PR195796/2014-12-12_… Bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195796 Do you know how this could be corrected in the freebsd ports? The maintainer of the freeBSD port updated it just yesterday (19 December) bumping it to 4.3.18: https://www.freshports.org/net-mgmt/xymon-server -- inum: 883510009027723 sip: jungleboogie at sip2sip.info xmpp: jungle-boogie at jit.si

2 1

xymon on a large architecture
by sean.clark＠twcable.com 22 Dec '14

22 Dec '14

So I have a setup where, in total, I monitor about 300,000 hosts. It's regionalized though, with the largest site having roughly 34,000 hosts The xymon daemon runs on a VM, with 16 GB of Ram, and 4x2.5 Ghz xeon processors. The virtual NIC is a 1000Mb It currently handles 481 msgs/sec (the hosts.cfg is 1.2 Mb) Here are some ideas: (1) Obviously split pollers out into separate machines (2) Change the history daemon (Henrick as some sample daemons in the source code) - I changed it to write stachg and clichg and such are written out to mysql instead of files, and the textblog is compressed. This helped with managing history (3) Use JC Cleaver's patches http://terabithia.org/rpms/xymon/ -- he has several memory leak fixes, tuning optimizations. I would just say that if you use clientupdate, you will have to patch his things a little bit (4) An interesting thing you can do, if you have the means via a load balancer is --> (a) I have 10 pollers for this instance. All of them run xymonproxy on localhost (stuff they poll reports to localhost and gets forwarded up) (b) Using a load balancer for they xymon IP address, it inspects the first 12 characters of a xymon message – if it's something that can be combo'd in proxy it passes to a poller in the pool, otherwise, it sends it to the real xymon IP (5) JC has some nanomsg options for channels, and myself, I use AMQP for channels to communicate data outbound. Both of these help if you use xymondata outside of the stock app For your setup, rather than running more xymond's on different ports, setup xymonproxy's on different machines, and point your clients to them. Then you won't have to migrate your history files or anything else. But I would look at changing your history setup, as several million files gets very unwieldy to manage, but millions of rows in databases are much easer to do. From: "fmaillard.ext at orange.com<mailto:fmaillard.ext at orange.com>" <fmaillard.ext at orange.com<mailto:fmaillard.ext at orange.com>> Date: Wednesday, December 17, 2014 at 11:04 AM To: "xymon at xymon.com<mailto:xymon at xymon.com>" <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: [Xymon] xymon on a large architecture Hello, We’re running a quite large xymon setup, and have been dealing with performance issue for quite a while. Here are some stats to give an idea about the setup: - We have 2 xymon servers per datacenter, on 3 datacenter (all messages are sent to both servers for a given site) - Each xymon server receives on average between 200msg/s and 250msg/s. We’re getting peaks at 400msg/sec. - Each site hosts about 3000 hosts / 30 000 services We’ve been suspecting for a long time that we might be losing messages… and I think I finally tracked it down to xymond not fetching the messages quickly enough so that the kernel’s buffer fill up and messages get discarded (by the kernel). On one of our servers, even though I have already increased net.ipv4.tcp_rmem and net.ipv4.tcp_wmem I got the following output from “netstat -s”: 148909 packets pruned from receive queue because of socket buffer overrun 4453143 packets collapsed in receive queue due to low socket buffer And here I come to the question I’m having: 1/ Is 250msg/s too much for a single xymond instance? Is anyone running instances with a lot more traffic than that? 2/ I’m starting to look into running several instances of xymond on the same machine, by binding them to different ports. Another option is to set up new machines, but that would mean migrating history files (several million files), sorting out the firewalling issues (our xymon interfaces are deeply connected to our information system) so I’d rather like avoiding this option. Are there any guidelines on how to do this? 3/ Are there any settings and best practice that could improve performance? For instance, should we move to a massive use of combo statuses in order to lessen the number of messages received? Best regards, Francois Maillard Pilote des plateformes Supervision, DNS & FTP - Sysadmin Infrastructure Altran Méditerranée pour Orange/OF/DTSI/DSI/DFY/HBX Sophia Antipolis tél. 04 97 12 87 53 fmaillard.ext at orange.com<mailto:fmaillard.ext at orange.com> _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. ________________________________ This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

3 3

Hobbit agent
by Srikrishna.Karatalapu＠vudu.com 20 Dec '14

20 Dec '14

Hi; I had few questions regarding the hobbit client ? * Can the hobbit agent report to the xymon server? * Can we have both hobbit and xymon client running on a host ? Thanks Krishna

2 1

xymon custom graphs
by Sheel.Shah＠rd.com 19 Dec '14

19 Dec '14

Hello, I am trying to set up some custom graphs to monitor memcached. I am using the script Olivier Beau created a few years ago. The data is definitely displaying in Xymon itself, but I am unable to see the graphs for the same datasets. Could someone point me in the right direction of what I may be doing wrong? Thanks, Sheel I added the appropriate values to the xymonserver.cfg, adding "memcache-test=ncv" to the TEST2RRD line, and adding this line (memcache-test is the name of the column in Xymon). SPLITNCV_memcache-test="Hits:GAUGE,Misses:GAUGE,reqs:GAUGE,Gets:GAUGE,Sets:GAUGE,storage:GAUGE,items:GAUGE,hitratio:GAUGE,*:NONE" I verified that data appears to be collected in the ~/xymon/data/rrd/<hostname>/memcache-test,Gets.rrd, for example. Finally, here is my configuration in the graphs.cfg. [memcache-test] TITLE Memcache Utilization YAXIS DEF:reqs=memcache-test,reqs.rrd:Lambda:AVERAGE DEF:hitratio=memcache-test,hitratio.rrd:Lambda:AVERAGE DEF:gets=memcache-test,Gets.rrd:Lambda:AVERAGE DEF:sets=memcache-test,Sets.rrd:Lambda:AVERAGE DEF:items=memcache-test,items.rrd:Lambda:AVERAGE Def:storage=memcache-test,storage.rrd:Lambda:AVERAGE LINE1:reqs#@COLOR@:Requests GPRINT:reqs:LAST: \: %5.1lf%s (cur) GPRINT:reqs:MAX: \: %5.1lf%s (max) GPRINT:reqs:MIN: \: %5.1lf%s (min) GPRINT:reqs:AVERAGE: \: %5.1lf%s (avg)\n LINE1:hitratio#@COLOR@:CacheHitRatio GPRINT:hitratio:LAST: \: %5.1lf%s (cur) GPRINT:hitratio:MAX: \: %5.1lf%s (max) GPRINT:hitratio:MIN: \: %5.1lf%s (min) GPRINT:hitratio:AVERAGE: \: %5.1lf%s (avg)\n LINE1:gets#@COLOR@:Gets GPRINT:gets:LAST: \: %5.1lf%s (cur) GPRINT:gets:MAX: \: %5.1lf%s (max) GPRINT:gets:MIN: \: %5.1lf%s (min) GPRINT:gets:AVERAGE: \: %5.1lf%s (avg)\n LINE1:sets#@COLOR@:Sets GPRINT:sets:LAST: \: %5.1lf%s (cur) GPRINT:sets:MAX: \: %5.1lf%s (max) GPRINT:sets:MIN: \: %5.1lf%s (min) GPRINT:sets:AVERAGE: \: %5.1lf%s (avg)\n COMMENT:Items Cached GPRINT:items:LAST: \: %5.1lf%s (cur) GPRINT:items:MAX: \: %5.1lf%s (max) GPRINT:items:MIN: \: %5.1lf%s (min) GPRINT:items:AVERAGE: \: %5.1lf%s (avg)\n LINE1:storage#@COLOR@:% Used Storage GPRINT:storage:LAST: \: %5.1lf%s (cur) GPRINT:storage:MAX: \: %5.1lf%s (max) GPRINT:storage:MIN: \: %5.1lf%s (min) GPRINT:storage:AVERAGE: \: %5.1lf%s (avg)\n

1 0

Bug, Error or my mistake in Xymon 4.3.11, analysis.cfg, LOG Statement???
by christian.becker＠rhein-zeitung.net 18 Dec '14

18 Dec '14

Hello all, this seems like a big text, but it might be a short reading and understanding..... I'm wondering if the following is either a bug, a config error or my missunderstanding. We are running Xymon 4.3.11. We have a couple of Linux servers running several distributions of Linux OS, but all containing Xymon 4.3.11 and above; installed as client. On our Xymon server, we have the following DEFAULT section in analysis.cfg: DEFAULT # These are the built-in defaults. UP 3 CLOCK 60 LOAD 5.0 10.0 DISK * 90 95 MEMPHYS 100 101 MEMSWAP 80 90 MEMACT 90 97 FILE /var/log/ntp SIZE>0 FILE %/var/(adm|log)/messages LOG %/var/(adm|log)/messages WARNING IGNORE=%(smbd|STORVSC:*.WARNING\!|gdm-simple-greeter|GdmDisplay|GdmSession|GDM|packagekitd|parport) COLOR=yellow LOG %/var/(adm|log)/messages %(I/O|read).error IGNORE=%(fd0|smbd|read_fd_with_timeout|Connection.reset.by.peer|error\.txt) COLOR=red LOG %/var/(adm|log)/messages ERROR IGNORE=%(fd0|smbd|read_fd_with_timeout|Connection.reset.by.peer|error\.txt|gdm-simple-greeter|GdmDisplay|GdmSession|GDM|packagekitd|parport) COLOR=red LOG %/var/(adm|log)/messages FAIL IGNORE=%(smbd|Connection.reset.by.peer|gdm-simple-greeter|GdmDisplay|GdmSession|GDM|packagekitd|parport|NT_STATUS_LOGON_FAILURE|LOGIN) COLOR=red LOG %/var/(adm|log)/messages CRITICAL IGNORE=%(smbd|gdm-simple-greeter|GdmDisplay|GdmSession|GDM|packagekitd|parport) COLOR=red (I don't want to dicuss about the sense or the absurdity of the configured IGNORE statements here; let's simply say they are OK for us.... Shouldn't matter for my question here.....) As one can read in the man page of analysis.cfg (which is available at https://www.xymon.com/help/manpages/man5/analysis.cfg.5.html ) i've read the following: Note that Xymon defaults to case-insensitive pattern matching; if that is not what you want, put "(?-i)" between the "%" and the regular expression to turn this off. E.g. "%(?-i)WARNING" will match the word WARNING only when it is upper-case. We don't have (?-i) in front of our keywords; that should match our keywords regardless of their upper and lower case. So far, so good. That was my meaning..... A colleague called me saying that one of our servers had a problem with a openvpn connection. I logged in to the server and inspected /var/log/messages, and I found these lines (I've changed IP's and ports to #): Dec 1 12:06:18 open-vpn ovpn-server[39555]: ##.##.##.##:##### TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity) Dec 1 12:06:18 open-vpn ovpn-server[39555]: ##.##.##.##:##### TLS Error: TLS handshake failed Dec 1 12:06:18 open-vpn ovpn-server[39555]: ##.##.##.##:##### Fatal TLS error (check_tls_errors_co), restarting Dec 1 12:06:18 open-vpn ovpn-server[39555]: ##.##.##.##:##### SIGUSR1[soft,tls-error] received, client-instance restarting Dec 1 12:06:23 open-vpn ovpn-server[39555]: TCP connection established with [AF_INET]##.##.##.##:##### Dec 1 12:06:24 open-vpn ovpn-server[39555]: ##.##.##.##:##### TLS: Initial packet from [AF_INET]##.##.##.##:#####, sid=######## ######## Dec 1 12:07:06 open-vpn ovpn-server[39555]: ##.##.##.##:##### TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity) Dec 1 12:07:06 open-vpn ovpn-server[39555]: ##.##.##.##:##### TLS Error: TLS handshake failed Dec 1 12:07:06 open-vpn ovpn-server[39555]: ##.##.##.##:##### Fatal TLS error (check_tls_errors_co), restarting Dec 1 12:07:06 open-vpn ovpn-server[39555]: ##.##.##.##:##### SIGUSR1[soft,tls-error] received, client-instance restarting As you can see, there are the keywords "Error", "error" and "failed" in these lines. At this time the msgs test of that server was at state green. This made me think that the statement of the man page of analysis.cfg regarding case-insensitive might be wrong. I've tested this and inserted this line in the DEFAULT section of analysis.cfg: LOG %/var/(adm|log)/messages Error IGNORE=%(fd0|smbd|read_fd_with_timeout|Connection.reset.by.peer|error\.txt|gdm-simple-greeter|GdmDisplay|GdmSession|GDM|packagekitd|parport) COLOR=red After a couple of minutes, the msgs test for that server changed to red. Did i understand something wrong or miss something? Or is this really a bug? Do i need to configure my keywords in a different way? Best regards Christian Christian Becker IT-Services Christian.Becker at rhein-zeitung.net<mailto:Christian.Becker at rhein-zeitung.net> _________________________________ Mittelrhein-Verlag GmbH August-Horch-Straße 28 D-56070 Koblenz Verleger und Geschäftsführer: Walterpeter Twer Reg.-Gericht Koblenz HRB 121 Finanzamt Koblenz Str.Nr. 22 65 10 285 2 www.rhein-zeitung.de<http://www.rhein-zeitung.de/>

1 0

Upgrade hobbit 4.2.0 to Xymon
by kpeng＠partycity.com 18 Dec '14

18 Dec '14

I only have my fresh install note early this year. I installed Xymon server on Centos 6.1. You can use Notepad open the attached note. I had an old Hobbit server as well. I left that untouched, and after installed this Xymon, I copied over the configuration files from old Hobbit, and the server now is running production. Hopefully this helps. Good luck! -----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Srikrishna Karatalapu Sent: Wednesday, December 3, 2014 4:40 PM To: xymon at xymon.com Subject: [Xymon] Upgrade hobbit 4.2.0 to Xymon Hi; I am working on upgrading hobbit to xymon on cetos6.5 vm. Wanted to check if there was any documentation available that would help me. I was hoping to get the list rpms for xymon and any another dependencies I would require for the upgrade . Thanks Krishna _______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

2 1

html_mail.pl colour
by bdale＠kitchen-net.com.au 18 Dec '14

18 Dec '14

I am trying to setup HTML email alerts, I found this script on xymonton which appears to be exactly what I need https://wiki.xymonton.org/doku.php/alerts:html_mail I followed the instructions and added it to my alerts.cfg. It successfully sends HTML emails, however it doesn't seem to contain a colour to match the alert which I think it should? I assumed I would get a line down the left side similar to what you see on the xymon webpages. This is what I get emailed: [cid:image001.png at 01D019DE.64A62AB0] If I look at the HTML source for the email I think based on this I can assume the scripts logic is successfully matching the alert colour to the correct .gif it needs: <body background="cid:bkg-red.gif"> I checked on the xymon server and these files appear to exist where they need to: usr/lib/xymon/server/www/gifs# ls arrow.gif bkg-red.gif clear.gif favicon-purple.ico green-recent.gif red-ack.gif xymonbody.css bkg-blue.gif bkg-yellow.gif clear-recent.gif favicon-red.ico purple-ack.gif red.gif yellow-ack.gif bkg-clear.gif blue-ack.gif favicon-blue.ico favicon-unknown.ico purple.gif red-recent.gif yellow.gif bkg-green.gif blue.gif favicon-clear.ico favicon-yellow.ico purple-recent.gif unknown.gif yellow-recent.gif bkg-purple.gif blue-recent.gif favicon-green.ico green.gif README unknown-recent.gif zoom.gif Am I doing something obivously wrong? And how do I go about troubleshooting this Note: - This example is from a custom test, but I don't think that could be the cause of the problem as the script is determing the colour of the alert as expected. - Xymon 4.3.17 running on Debian 6 Brandon

3 3