October 2005 - Xymon - Xymon mailman web

Client memory test problem
by rsmrcina＠wi.rr.com 28 Oct '05

28 Oct '05

I have a host where the SWAP test is coming up yellow at 21%. This host does have the following specified in hobbit-cleints.cfg: HOST=golxi01 MEMACT 95 98 If I comment out the MEMACT values, SWAP goes back to green. If I uncomment them, SWAP becomes yellow again. Quite strange... -- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006

2 2

EXHOST usage
by jonescr＠cisco.com 28 Oct '05

28 Oct '05

Perhaps it's because I'm working on this at 4am, but I'm having a problem with the EXHOST option, that according to hobbitd_alert --test isn't working, I also am not sure how to do a particular host/service exclusion. Heres basically what my below alert config is meant to accomplish. 1. For any alerts on any servers, send alerts to an alert email address. 2. For 2 particular web servers (web5.mydomain.com and web6.mydomain.com) send an alert to one person, but *not *the alert alias. 3. For a set of oracle servers, send an extra alert message to an alternate email address/cellphone. 4. After hours (from 5pm until 8am), only send alerts to an alternate email address (but still need the seperate alert for the web5 and web6 hosts described in #2). 5. After hours (from 5pm until 8am), send an alert to my cellphone for any hosts and services being red for more than 30 mins. 6. Do not alert for high load average on a particular server from 6-10am. My first problem is I am not sure how to implement #2 (exclude alerts for the msgs on web5 and web6 from being sent to the main alert email, and instead send them to the alternate address). I'm thinking one solution would be to define it as the very first rule and use the "STOP" option, like: HOST=$WEB_SERVERS SERVICE=msgs COLOR=red MAIL webdev at mydomain.com STOP Also, according to the tests I did with hobbit_alert --test, the last rule is not working...Is there a more logical way to simply specify that a single host/service combo be ignored during a certain timeframe? My hobbit-alerts.cfg is below, if you see anything wrong or have suggestions on a better way to accomplish my list above I would appreciate it. # hobbitd-alerts.cfg # oradb1-9.mydomain.com $ORACLE_SERVERS=%oradb(.).mydomain.com # web1-9.mydomain.com $WEB_SERVERS=%web(.).mydomain.com # All hosts $ALL_HOSTS=* # Send me an email alert if any service on any host goes red. # Note: This rule will probably be removed once the alert rules are deemed fully working. HOST=$ALL_HOSTS SERVICE=* COLOR=red MAIL alert at mydomain.com # Notify webdev about (only) web errors on web5 and web6 # FIXME: Need to make it so that alert at mydomain.com does NOT get these! HOST=$WEB_SERVERS SERVICE=msgs COLOR=red MAIL webdev at mydomain.com # Send an alert to dba phone on oracle-specific problems HOST=$ORACLE_SERVERS SERVICE=msgs,oradb,orasys COLOR=red FORMAT=sms MAIL dbacell at cellphone.com # Send me a page if any hosts go red for more than 30 minutes during offhours HOST=$ALL_HOSTS SERVICE=* COLOR=red DURATION>30 TIME=1700-0800 FORMAT=sms MAIL mycell at cellphone.com # Ignore high load average warnings for dataproc1 in the mornings EXHOST=dataproc1.mydomain.com SERVICE=cpu COLOR=red TIME=0600-1000 MAIL me at mydomain.com

2 6

local TCP test
by saam＠tomikoni.de 28 Oct '05

28 Oct '05

Hi, I have written a netstat tcp test, which i find particularly useful. "netstat" tcp connections sent by the hobbit client will be scanned on the hobbit server for sourceport, targetport and connection state. This helps you verify that a process is listening on port 80, that at least 5 connections are established to your database server, that the telnet port is really closed or that there are no more than 100 connections in TIME_WAIT state. The handling/display is quite similar to the "PROC" option in hobbit-clients.cfg. If there is interest, I would like to contribute it, because i think that hobbit is a great tool (I'm currently migrating from bb). How is the procedure? Regards, Mirko

3 3

Problem with NCV graphs
by gsteer＠firstwave.com.au 28 Oct '05

28 Oct '05

I'm having problems getting a NCV graph to work. The status script runs on the monitoring client without error and the status result appears on the Hobbit web page in the format expected. The rrd file gets created but the data doesn't get added consistently. The figures that do get graphed are large ones e.g a heartbeat value was 3 but a value of 32579345 gets put into the rrd Here's the script fragment that sends the status message. (I've tried replacing the = with a : but same results.) -- /usr/local/bb/bbc1.9e-btf/bin/bb 202.12.141.141 "status mailserver,domain,com.heartbeat $colour `date` HeartbeatTime=$elapsed " -- What I've added to hobbitserver.cfgr.cfg TEST2RRD="cpu=la,disk,inode,qtree,memory, $PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,vmstat,vmstat5,iostat,netstat, temperature,apache,bind,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,citrix,bbgen, bbtest,bbproxy,hobbitd,heartbeat,heartbeat=ncv" NCV_heartbeat="HeartbeatTime:COUNTER" The graph specification in hobbitgraph.cfg [heartbeat] FNPATTERN heartbeat.rrd TITLE Email Heartbeat YAXIS Transit Time DEF:HeartbeatTime at RRDIDX@=@RRDFN@:HeartbeatTime:AVERAGE LINE1:HeartbeatTime at RRDIDX@#@COLOR@:Heartbeat Period @RRDPARAM@ COMMENT:\n GPRINT:HeartbeatTime at RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:HeartbeatTime at RRDIDX@:MAX: \: %5.1lf (max) GPRINT:HeartbeatTime at RRDIDX@:MIN: \: %5.1lf (min) GPRINT:HeartbeatTime at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n COMMENT:\n I did write a do_heartbeat.c routine for an older version of Hobbit but the NCV option seemed more attractive that editing any new versions of hobbit I install. -- Geoff Steer <gsteer at firstwave.com.au> -------------------------------Safe Stamp----------------------------------- The sender's Anti-virus Service scanned this email. It is safe from known viruses.

1 0

Missed pages going from Yellow to Red
by hobbit＠pvaughan.us 27 Oct '05

27 Oct '05

This morning I had a filesystem start to fill up and go to a yellow state. Hobbit sent the Email as expected. But then the alarm went to a red state. Once it goes to a red state, I should get a page. At first I thought it might be something to do with the long REPEAT time on this rule, but it looks like there is only a problem when the state is yellow and then switches to red. Has anyone else noticed similar behavior?

1 1

[hobbit] short vs. canonical name
by adam.scheblein＠marquette.edu 27 Oct '05

27 Oct '05

I have run into this in my administration of HP-UX and Tru64 where just like you, my HP-UX servers use the short names, and the Tru64 are fully qualified. What I have done is in the bb-hosts file put the entry like this: IP server.fqdn.com # CLIENT:shortname [tests] Ex: 1.2.3.4 foo.bar.com # CLIENT:foo !ftp ssh !telnet Because then when the servers send their messages to hobbit, Hobbit is able to translate the messages no matter what the name is... Hope this helps, Adam ________________________________ From: Christian, Ron [mailto:rchristian at columbia.com] Sent: Wednesday, October 26, 2005 1:11 PM To: hobbit at hswn.dk Subject: [hobbit] short vs. canonical name We're bringing up 4.1.2 on a QA machine, and have noticed an anomaly. Our environment is mostly Solaris 9 with a few machines running Red Hat Enterprise AS. We have Hobbit 4.0.4 running on our production administrative server and Hobbit 4.1.2 server running on our QA box. The intention was to qualify 4.1.2 and push it into the production environment if it seemed reasonable to do so. I've run into a problem with the QA instance on Red Hat machines. I can see a couple ways to fix it but want to find out what is considered "best practices" The Solaris node names are the short (non-canonical) form, but the node names of RHEE tend to be fully qualified. It looks like we will have more Red Hat and fewer Solaris machines in the near future, so this issue will become significant. It looks like different parts of 4.1.2 arrive at the machine name in different ways. The name of the server is set explicitly as BBSERVERHOSTNAME in hobbitserver.cfg. It appears that this is used both for the built-in tests and server extensions. The client, on the other hand, appears to get it's hostname in a different fashion depending on whether it's a built-in test or an extension. With the client short name in bbhosts, but the client uname defaulting to the fully qualified name, the built-in tests work but extensions do not. It appears that the built-in tests somehow figure out the client's short name, and the server recognizes it and displays the data. However, when extensions are invoked, the client name is set to the fully qualified name, the name is munged to hostname,my,domain (note commas) and the server doesn't recognize the client. I see a few ways around this. We could set the uname on all the Linux boxes to the short name, and remember that we have to do this for any new installs. (I'm not sure if this will break anything else.) Or, we could set up a hobbit alias for each Linux machine (ick...). Or, we could change runclient.sh to use "hostname -s" instead of "uname -n" (But only on Linux boxes.) or change the "MACHINE="`echo $MACHINEDOTS | sed [...]`" expression to dump the domain name instead of changing the dots to commas. Have others run into this, and is there an accepted solution? Ron - Silence is golden. Duct tape is silver.

1 0

disabled hosts, re-enabled on hobbitd restart
by rdeal＠tigr.ORG 26 Oct '05

26 Oct '05

Should hobbitd maintain disabled hosts when restarted? The files are still in the data/disabled directory but the hosts are listed as enabled after a restart.

3 2

[hobbit] Missed pages going from Yellow to Red
by Jason.Chambers＠geosoft.com 26 Oct '05

26 Oct '05

The only problem that I have had with yellow to red is that, using the big brother client for windows, I changed the disk capacity warning and critical to something really small. Like 5% and 10%. Theres like 58% used, and I got a yellow alert instead of a red alert for some strange reason. Jason Chambers IT Helpdesk Support Geosoft Inc. 85 Richmond St. West - 8th Floor Toronto, Ontario, Canada M5H 2C9 Tel: 416-369-0111 x344 Fax: 416-369-9599 www.geosoft.com -----Original Message----- From: Pat Vaughan [mailto:hobbit at pvaughan.us] Sent: Wednesday, October 26, 2005 3:28 PM To: hobbit at hswn.dk Subject: [hobbit] Missed pages going from Yellow to Red This morning I had a filesystem start to fill up and go to a yellow state. Hobbit sent the Email as expected. But then the alarm went to a red state. Once it goes to a red state, I should get a page. At first I thought it might be something to do with the long REPEAT time on this rule, but it looks like there is only a problem when the state is yellow and then switches to red. Has anyone else noticed similar behavior? To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

short vs. canonical name
by rchristian＠columbia.com 26 Oct '05

26 Oct '05

We're bringing up 4.1.2 on a QA machine, and have noticed an anomaly. Our environment is mostly Solaris 9 with a few machines running Red Hat Enterprise AS. We have Hobbit 4.0.4 running on our production administrative server and Hobbit 4.1.2 server running on our QA box. The intention was to qualify 4.1.2 and push it into the production environment if it seemed reasonable to do so. I've run into a problem with the QA instance on Red Hat machines. I can see a couple ways to fix it but want to find out what is considered "best practices" The Solaris node names are the short (non-canonical) form, but the node names of RHEE tend to be fully qualified. It looks like we will have more Red Hat and fewer Solaris machines in the near future, so this issue will become significant. It looks like different parts of 4.1.2 arrive at the machine name in different ways. The name of the server is set explicitly as BBSERVERHOSTNAME in hobbitserver.cfg. It appears that this is used both for the built-in tests and server extensions. The client, on the other hand, appears to get it's hostname in a different fashion depending on whether it's a built-in test or an extension. With the client short name in bbhosts, but the client uname defaulting to the fully qualified name, the built-in tests work but extensions do not. It appears that the built-in tests somehow figure out the client's short name, and the server recognizes it and displays the data. However, when extensions are invoked, the client name is set to the fully qualified name, the name is munged to hostname,my,domain (note commas) and the server doesn't recognize the client. I see a few ways around this. We could set the uname on all the Linux boxes to the short name, and remember that we have to do this for any new installs. (I'm not sure if this will break anything else.) Or, we could set up a hobbit alias for each Linux machine (ick...). Or, we could change runclient.sh to use "hostname -s" instead of "uname -n" (But only on Linux boxes.) or change the "MACHINE="`echo $MACHINEDOTS | sed [...]`" expression to dump the domain name instead of changing the dots to commas. Have others run into this, and is there an accepted solution? Ron - Silence is golden. Duct tape is silver.

1 0

[hobbit] EXHOST usage
by adam.scheblein＠marquette.edu 26 Oct '05

26 Oct '05

Greetings, Is there a way to make the apache graphs show up in the http page? Thanks, Adam

2 1