November 2013 - Xymon - Xymon mailman web

FreeBSD Actual Memory Usage
by jlaidman＠rebel-it.com.au 22 Nov '13

22 Nov '13

Good Xymon Folks Xymon doesn't support reporting "actual" memory usage for FreeBSD systems - that is, available memory that may or may not be in use for buffers and cache. It only reports, graphs, and alerts on, swap and physical memory usage. Some OSes use some of the unused memory for filesystem caching and other purposes related to performance, and so the reported free memory count goes down over time even though the memory available to for use is not decreasing. On my systems, free memory is only a few percent. So this doesn't give any indication of the risk of memory "resource exhaustion". So what I really need is to report on "actual" memory usage. From what reading I've done, for FreeBSD this would be total memory subtract "free" + "inactive" memory, although that depends on who you ask. The "actual" memory reporting is not only a problem for FreeBSD, but for all supported OSes except for Linux, IRIX and Windows. I suspect many of these OSes report their free memory to include used-but-available memory also, and so the "real" available memory is a useful number. Even real memory usage reporting seems to have caused trouble in the past for FreeBSD, as Xymon has had to have its own client-side binary for getting the memory numbers, as is also the case for HPUX and the other *BSDs, NetBSD and OpenBSD. For all other OS types, standard OS tools (free, sar) are used to get memory usage numbers. Memory usage reporting in Xymon seems to be quite a mixed bag, in general. Some clients report usage to [memory], some to [freemem], some to [meminfo], some to [free]. The Irix client has no specific memory usage report at all. This is by no means a complaint - I'm sure Henrik would have been much happier if all OSes had a standard memory query interface, and I suspect a lot of these different reporting methods were for legacy support. So. I need to get actual memory reported for some FreeBSD systems, so I'm trying to work out the best way to do this. I'd like to fix it in a way that fits in with the "standard" model, but as I described above, there isn't really a "standard" model. Here are several ways I can think of to solve the problem: 1. FreeBSD reports [top] header output, if top is installed (as do many OSes). The server-side code could simply grab the numbers from there. This would work for other OSes that don't report "active" memory, and could be a common interface to memory usage metrics. Any new OS that doesn't have extensive client support could simply report a [top] section in client data, and Xymon would magically start reporting actual free memory. The down-side to this is that top isn't installed everywhere. Although on my systems it is, so that's OK. The other down-side is that it requires patches to the server code. 2. I could replace the "freebsd-meminfo" binary that comes with the Xymon client, so that the "free" figure has the "inactive" memory added (or whatever adjustments are appropriate). This doesn't solve the problem for any other OS. Perhaps that's OK - perhaps the problem is very much OS dependent because each OS has its own unique memory management. I think the binary can be replaced by a simple shell script that parses the "top" header or "sysctl vm.vmtotal" for the correct figures. (It seems that "top" gets all its numbers from sysctl anyway, so I'd do the latter, so as to avoid a dependency.) 3. I could report each of the different memory metrics separately to Xymon: active, inactive, wired, cache, buffers, free. Then I can graph them all, and look for various conditions on each of them separately, or in certain combinations that make sense. This is the most flexible option, and would provide the highest degree of insight to someone trying to troubleshoot a sluggish server, but it requires a lot more work on both client and server. It's also specific to *BSD systems. So, any other suggestions on the best way to achieve this? Which of the above is the best approach, do you think? The other issue I have is that nobody seems to agree on what's a useful measure to keep an eye on. The Xymon server-side code for Darwin reports used memory as the sum of active, inactive and wired. But other sources use the sum of active, wired, cache and buffers. Yet other sources say that buffers cannot be freed, and also that inactive pages are kind-of available if needed. My intention is to be able to predict when it's time to add RAM to avoid performance degradation, but it's not clear what numbers are going to give me that. Cheers Jeremy

2 4

Question about TIME keyword in alerts.cfg
by mburger＠bubbanfriends.org 20 Nov '13

20 Nov '13

> The "man page" for alerts.cfg says: > > ==================================== > An example: > > HOST=www.foo.com > MAIL webmaster at foo.com SERVICE=http REPEAT=1h > MAIL unixsupport at foo.com SERVICE=cpu,disk,memory > > The first line defines a rule for alerting when something breaks on the > host "www.foo.com". > There are two recipients: webmaster at foo.com is notified if it is the > "http" service that fails, and the notification is repeated once an hour > until the problem is resolved. > unixsupport at foo.com is notified if it is the "cpu", "disk" or "memory" > tests that report a failure. Since there is no "REPEAT" setting for this > recipient, the default is used which is to repeat the alert every 30 > minutes. > > OK, suppose now that the webmaster complains about getting e-mails at 4 AM > in the morning. The webserver is not supposed to be running between 9 PM > and 8 AM, so even though there is a problem, he doesn't want to hear about > it until 7:30 - that gives him just enough time to fix the problem. So you > must modify the rule so that it doesn't send out alerts until 7:30 AM: > > HOST=www.foo.com > MAIL webmaster at foo.com SERVICE=http REPEAT=1h TIME=*:0730:2100 > MAIL unixsupport at foo.com SERVICE=cpu,disk,memory > > Adding the TIME setting on the recipient causes the alerts for this > recipient to be suppressed, unless the time of day is within the interval. > So with this setup, the webmaster gets his sleep. > ==================================== > > > This makes sense, but I'm confused and obsessing about the word > "suppressed" in the second to last sentence. > > Suppose http went "red" at midnight. No alert was sent then, because of > the TIME keyword in the webmaster's line (unixsupport didn't get an alert > either, because it wasn't configured for it). > > So far, so good. Now suppose 7:30AM comes, and http is still red (no > status change). What happens? Does the webmaster receive an email? Even > though there is no new http status change to trigger it? > > > Thanks, > > glauber Hello, Glauber. Your understanding is correct...the email should be received once the Xymon server's time reaches the start time (in this case, 7:30AM) and the alarm is still active. Note, my testing also indicates that the test won't even change color at all until after the start time indicated in your TIME option. -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1

1 0

Question about TIME keyword in alerts.cfg
by glauber.ribeiro＠experian.com 20 Nov '13

20 Nov '13

The "man page" for alerts.cfg says: ==================================== An example: HOST=www.foo.com MAIL webmaster at foo.com SERVICE=http REPEAT=1h MAIL unixsupport at foo.com SERVICE=cpu,disk,memory The first line defines a rule for alerting when something breaks on the host "www.foo.com". There are two recipients: webmaster at foo.com is notified if it is the "http" service that fails, and the notification is repeated once an hour until the problem is resolved. unixsupport at foo.com is notified if it is the "cpu", "disk" or "memory" tests that report a failure. Since there is no "REPEAT" setting for this recipient, the default is used which is to repeat the alert every 30 minutes. OK, suppose now that the webmaster complains about getting e-mails at 4 AM in the morning. The webserver is not supposed to be running between 9 PM and 8 AM, so even though there is a problem, he doesn't want to hear about it until 7:30 - that gives him just enough time to fix the problem. So you must modify the rule so that it doesn't send out alerts until 7:30 AM: HOST=www.foo.com MAIL webmaster at foo.com SERVICE=http REPEAT=1h TIME=*:0730:2100 MAIL unixsupport at foo.com SERVICE=cpu,disk,memory Adding the TIME setting on the recipient causes the alerts for this recipient to be suppressed, unless the time of day is within the interval. So with this setup, the webmaster gets his sleep. ==================================== This makes sense, but I'm confused and obsessing about the word "suppressed" in the second to last sentence. Suppose http went "red" at midnight. No alert was sent then, because of the TIME keyword in the webmaster's line (unixsupport didn't get an alert either, because it wasn't configured for it). So far, so good. Now suppose 7:30AM comes, and http is still red (no status change). What happens? Does the webmaster receive an email? Even though there is no new http status change to trigger it? Thanks, glauber --- Glauber Ribeiro Experian Automotive, Schaumburg IL Infrastructure and Information Security e: glauber.ribeiro at experian.com p: 224-698-3108 f: 224-698-5032

1 0

Changing rrd definitions
by john.r.rothlisberger＠accenture.com 19 Nov '13

19 Nov '13

Hi all - I wanted to change the default rrd graphs and thought this may be relatively easy but then again.. Xymon 4.3.10 Ubuntu 12.10 Currently, all of the graphs are based on 576 data points which works great for graphs that are: 48 hours @ 5 minute average = 576 12 days @ 30 minute average = 576 48 days @ 2 hours average = 576 576 days @ 1 day average = 576 I want to increase some of the granularity and also reduce the final graph to 1 year. So, I decided on these settings: (based on 864 data points) 72 hours @ 5 minute average = 864 9 days @ 15 minute average = 864 27 days @ 45 minute average = 864 360 days @ 600 minute average = 864 (I was searching for a nice number that I could use for 1 year and this was as close as I could get) I changed rrdefinitions.cfg to match the above (I hope my numbers are right): RRA:AVERAGE:0.5:1:864 RRA:AVERAGE:0.5:3:864 RRA:AVERAGE:0.5:9:864 RRA:AVERAGE:0.5:120:864 I also changed "RRDWIDTH" in xymonserver.cfg from 576 to 864. I assumed that all of my old graphs would remain as they were and that new graphs would take on the new characteristics above. 2 Things occurred: * All the graphs grew to 864 data points in width (ok, that is fine) * All the graphs still retained the original titles "Last 48 Hours", "Last 12 Days", etc. I found those graph titles in showgraph.c: showgraph.c: glegend = "Last 12 Days"; showgraph.c: glegend = "Last 48 Days"; showgraph.c: glegend = "Last 576 Days"; That leads to these questions: * I can recompile showgraph.c - should I? Is there an easier way? * If new graphs take on the new characteristics, will I be able to change the old rrd files? I understand that I can't create data that is not already there but is there a way to transition over time? Is this a crazy idea? Thanks, John Upcoming PTO: None _____________________________________________________________________ John Rothlisberger IT Strategy, Infrastructure & Security - Technology Growth Platform TGP for Business Process Outsourcing Accenture 312.693.3136 office _____________________________________________________________________ ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. . ______________________________________________________________________________________ www.accenture.com

1 0

Disabled check emails - how?
by rectifier＠gmail.com 15 Nov '13

15 Nov '13

I was wondering if anyone had a similar situation to me: I have many admins that can disable a Xymon check in my company. I would like to receive an email when they do this. Hobbit-alerts can only use red-green-yellow and NOTICE cannot just send disabled checks, have been trying to get that to work. It is very important for me to be able to be alerted when someone disables a check so i can see who it is. thanks

2 1

BBWin won't run external scripts
by stella＠rider.edu 15 Nov '13

15 Nov '13

We have a new install of Windows 2008 r2. For some reason, BBWin will not run the couple of external scripts we have. If I run them by hand (cscript sricpt.vbs) from the command line, I have no problem. But if not it seems the scripts are being ignored. For example, <load value="c:\windows\system32\cscript.exe //B C:\BBWin\ext\SystemCheck.vbs" timer="5m" /> Event log shows the externals.dll is being loaded. But can't see what's wrong. Nothing in logs. (note we default the installation to C:\BBWin). Disk, CPU, msgs, processes correctly report. Just not these externals. I tried version 0.12 and 0.13 and still no luck. Any clues? -- °((( = (( ===°°° ((( ================================================

2 1

combine status and data in same column with graph
by vadud3＠gmail.com 14 Nov '13

14 Nov '13

Hi All, I sent some text to status channel and some NCVs to the data channel for same test/columnname. So on the same script I have the followings ... $XYMON $XYMSRV "status $MACHINE.$COLUMN $COLOR `date` ${MSG} " $XYMON $XYMSRV "data $MACHINE.$COLUMN foo: 1 bar: 2 " Then I placed the columnname=ncv in TEST2RRD and restarted xymon rrdtool dump $COLUMN.rrd shows all the text from status msgs in it, yikes! How do I tell xymon to look at only the data channel for the rrds and leave the status channel alone for this test/columnname? Thanks -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

3 9

Using rclient to connect to a Tripplite terminal server
by Paul.Root＠CenturyLink.com 14 Nov '13

14 Nov '13

I recently got some new terminal servers from Tripplite, and I want to monitor them. Since I can't really build the client on it, I thought I would use the xymon-rclient.sh script from Jeremy Laidman. It seems to work pretty well, more or less it ssh's to the client in question, transfers over the proper xymonclient script, and populates the results. The script runs from tasks.cfg. The tripplite is some form of minimal linux, so I copied xymonclient-linux.sh to xymonclient-tripplite.sh, and started editing. This didn't work. Rclient happily ran the script and gave good output I can see from debugging. It seems that xymon ignores it, I'm guessing because it doesn't recognize 'tripplite' as a valid os type. Is there a way to add 'tripplite' as an os type? Would that be client-local.cfg? So instead, I modified xymonclient-linux.sh and have it run. The tripplite doesn't have uptime, so I get that from /proc/uptime and /proc/loadaverage and do a bunch of manipulations with dc. Also, top won't run without a terminal, and you can't set a terminal, so the 'cpu' output is a bit sparse. Ps doesn't have any options, and says COMMAND instead of CMD, so I fix that. Also, df needed some frobbing. The last thing really, is vmstat, it doesn't have one. So I was thinking again to using /proc/vmstat, and manipulate that, but I'm not really sure how to start there, and there isn't a whole lot in /proc/vmstat. Any ideas there? nr_free_pages 2651 nr_inactive_anon 0 nr_active_anon 1573 nr_inactive_file 3249 nr_active_file 3114 nr_anon_pages 1535 nr_mapped 1051 nr_file_pages 6407 nr_dirty 0 nr_writeback 0 nr_slab_reclaimable 1334 nr_slab_unreclaimable 699 nr_page_table_pages 96 nr_unstable 0 nr_bounce 0 nr_vmscan_write 0 nr_writeback_temp 0 Does anybody use rclient.sh and/or have any advice? Thanks, Paul. Paul Root Senior Engineer CenturyLink Network Reliability Operations Center 600 Stinson Blvd, N.E. Flr 2N Minneapolis, MN 55413 Direct: (651)312-5207 Paul.Root at centurylink.com

2 2

authentification check tests for cookies acceptance
by schritte＠ub.uni-frankfurt.de 13 Nov '13

13 Nov '13

Hi all, I want to test login on an authentification form with http. The requested url checks for the acceptance of cookies. If cookies aren't accepted an error message is returned even if authentification is successful. There is no need to transfer cookies contents, only xymon has to to send the information "yes, cookies accepted" with the http-test to the url. Is that possible? kind regards Rolf -- Mit freundlichen Gruessen Rolf Schrittenlocher Lokales Bibliothekssystem Frankfurt Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 798 28817 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at ub.uni-frankfurt.de

2 4

WinPSClient - Updates?
by JasonRP＠techheads.com 13 Nov '13

13 Nov '13

How would I go about getting involved? I’d be willing to put some effort into this, but I’ve never contributed to a project in sourceforge, so I’m not sure if I have to be authorized of if anyone can do it. I don’t have the familiarity with BBWin to determine what other functionality needs to be added to WinPSClient, but if you can point me in the right direction I can check it out. I’ve been working with PowerShell for a few years, so I’m pretty up to speed in order to be useful. :) --- Jason Richards Pagh From: David Baldwin <david.baldwin at ausport.gov.au<mailto:david.baldwin at ausport.gov.au>> Date: Tuesday, November 12, 2013 at 8:55 PM To: Jason Richards Pagh <jasonrp at techheads.com<mailto:jasonrp at techheads.com>>, "xymon at xymon.com<mailto:xymon at xymon.com>" <xymon at xymon.com<mailto:xymon at xymon.com>> Cc: David Baldwin <David.Baldwin at ausport.gov.au<mailto:David.Baldwin at ausport.gov.au>> Subject: [Xymon] Re: WinPSClient - Updates? Jason, Thanks for the email. So it looks like the WinPSClient hasn’t been updated in a few months. I had seen a thread from quite a while ago about some issues David was running into, but can’t tell if he’s still around and willing/able to work on it. In principle I am interested, but don't have a lot of time to work on it at present. I’ve been able to get XymonPSClient.exe (aka SrvAny) to run powershell natively as a 64-bit executable. The only change necessary to enable this was changing the PowerShell executable to running as %windir%\sysnative\WindowsPowerShell\v1.0\powershell.exe. The sysnative portion prevents a 32-bit program (SrvAny) from being redirected to %windir%\SysWow64. (See http://msdn.microsoft.com/en-us/library/windows/desktop/aa384187(v=vs.85).a…<http://msdn.microsoft.com/en-us/library/windows/desktop/aa384187%28v=vs.85%…>). Thanks - I found the "sysNative" workaround a month or two ago also. Very useful for running PS scripts from BBWin since it is still 32-bit only. I also found a better service wrapper than SrvAny, namely nssm (http://nssm.cc/) This has the ability to detect if powershell fails and restart it, which is something SrvAny lacks. It also comes in both 32-bit and 64-bit versions, so when using the latter the sysnative piece isn’t necessary. I’ve only been working at this for a day or two, so it probably needs more testing, but it looks promising. That's an awesome improvement. Definitely a big advance :) Is there any work (design contemplation) about how xymonclient.ps1 could handle external scripts? I assume this was just a future phase and not so much a limitation or barrier that hadn’t yet been overcome. I'd played around with things like start-job in PS, but I hadn't worked through all of the legacy reporting mechanisms from BBNT/BBWin (e.g. writing text files into directory - that's an easy one in practice) but more to the point worked out throttling mechanisms for long-running scripts that don't exit, etc. Had also considered just running external scripts as scheduled tasks, but not had time to investigate. >From memory there were some other issues such as dealing with memory leaks (periodic service restart would solve that). David. -- David Baldwin - Senior Systems Administrator (Datacentres + Networks) Information and Communication Technology Services Australian Sports Commission http://ausport.gov.au Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616 david.baldwin at ausport.gov.au<mailto:david.baldwin at ausport.gov.au> Leverrier Street Bruce ACT 2617 ________________________________ Keep up to date with what's happening in Australian sport visit www.ausport.gov.au<http://www.ausport.gov.au> This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender. ________________________________

1 0