November 2006 - Xymon - Xymon mailman web

Problem with TIME specification in hobbit-alerts.cfg?
by Eric.vdM＠Softlution.com 07 Nov '06

07 Nov '06

Hello, I'm still having problems with alert configuration in Hobbit. I've configured notifications as follows: > # On which colors should we alert? > $ALERT_COLORS=COLOR=purple,red > > # Mail rule > $MAIL_BEHEER=MAIL beheer at softlution.com FORMAT=TEXT REPEAT=10069920 RECOVERED NOTICE > > # SMS rule > $SMS_SUPPORT=SCRIPT /Hobbit/server/ext/pg/smsalert support FORMAT=SMS REPEAT=30 TIME=W:0800:0900,W:1700:2300,60:0800:2300 DURATION>5 RECOVERED NOTICE > > PAGE=<page> $ALERT_COLORS > $SMS_SUPPORT > > HOST=%.* $ALERT_COLORS > $MAIL_BEHEER hobbitd_alert --dumpconfig and hobbitd_alert --test seem to think this configuration is fine. smsalert is basically a wrapper around smstools' sendsms command, I've added phone number aliases so the actual phone numbers don't show up in the info column pages, and I've added basic logging with timestamps and sendsms exit codes. It has worked fine so far.) However, this morning at 8:00 AM no SMS was sent on a red condition which had already lasted for hours (it started at 00:44 AM) and is shown on the page <page>. A mail alert was sent as soon as the condition started, but an SMS is never sent and neither notifications.log nor my smsalert.log show any attempt to send an SMS. Am I doing something wrong here? Best regards, Eric.

1 0

ganglia-style graph aggregation with hobbit
by gn1＠sanger.ac.uk 07 Nov '06

07 Nov '06

Hello, I'd like to completely replace the ganglia system we have here with hobbit. Most of the needed features are there, except the ability to produce aggregated graphs with multiple hosts, as in http://monitor.millennium.berkeley.edu/?c=PSI%20Cluster&m=&r=hour&s=descend… I suppose the following things will need to be done: - find a neat way to group the hosts (all the hosts in a group or a page for instance) - collect all the values from the rrd for all the hosts in a group and add them (when different from 0/NaN) - graph the result What would be the best way to do it? I'd like not to reinvent the wheel, so if some bits are already existing, I'd better use them... Cheers, Gildas

7 14

Multi-host process graphs
by jonescr＠cisco.com 06 Nov '06

06 Nov '06

Has anyone had any luck in creating multi-hosts process graphs? Here are the steps I have done so far in trying to get it to work. If anyone sees what I am doing wrong, any advice would be appreciated. 1. Created a multi definition in hobbitgraphs.cfg [processes-multi] FNPATTERN processes.(.+).rrd TITLE Multi-host Process counts YAXIS # DEF:cnt at RRDIDX@=@RRDFN@:count:AVERAGE -l 0 -A LINE2:cnt at RRDIDX@#@COLOR@:@RRDPARAM@ GPRINT:cnt at RRDIDX@:LAST: \: %10.1lf (cur) GPRINT:cnt at RRDIDX@:MAX: \: %10.1lf (max) GPRINT:cnt at RRDIDX@:MIN: \: %10.1lf (min) GPRINT:cnt at RRDIDX@:AVERAGE: \: %10.1lf (avg)\n 2. Edited hobbit/server/web/hostgraphs_form and added the extra option for process counts: <option value="processes">Process counts</option> 3. Made sure that the process rrds are there (I know they are anyway because I see the graphs in trends): $ ls data/rrd/prod-web-1/processes.* data/rrd/prod-web-1/processes.Apache_portal.rrd data/rrd/prod-web-1/processes.Apache_tomcat.rrd data/rrd/prod-web-1/processes.Apache_research.rrd data/rrd/prod-web-1/processes.Apache_main.rrd data/rrd/prod-web-1/processes.Apache_proxy.rrd Now when I go to the Metrics Report, and select prod-web-2 and prod-web2, along with Process counts. I get a missing graph, so something isn't working right. The same hosts work fine with other multi-host graphs like la1, cpu, etc. any ideas? -Charles

1 0

Port Monitoring
by paul.moore＠verizonbusiness.com 06 Nov '06

06 Nov '06

Is there a way to setup hobbit's port monitoring to alert when a specific device has X number of established connections on particular port? IE alerting when one client has 20 sessions connected to port 80 signifying a DOS attack? Paul Moore V966-5159 MSO OSS Support -=-=-=-=-= Pinky, Are You Pondering What I'm Pondering? Well, I think so Brain but if Jimmy cracks corn and no one cares, why does he keep doing it?

2 2

[hobbit] Hosts going purple
by CHRIS.MORRIS＠RWEnpower.com 06 Nov '06

06 Nov '06

Check the size of the client message file compared to the MAXMSG_CLIENT, MAXMSG_STATUS and MAXMSG_STACHG settings in the hobbitserver.cfg file, if you have lots of ports open on that host it can create a large amount of output which is too big for Hobbit to process. > -----Original Message----- > From: Jeff Newman [SMTP:jeffnewman75 at gmail.com] > Sent: Thursday, November 02, 2006 4:12 PM > To: hobbit at hswn.dk > Subject: [hobbit] Hosts going purple > > Hi, > > I am running the latest hobbit server/client (with all the patches > applied) and I still have AIX hosts going purple due to the netstat > command (nearest I can tell) > > From ps: > # ps -ef | grep hob > hobbit 13306 16692 0 03:58:28 - 0:00 sh > /usr/local/hobbit/client/bin/hobbitclient.sh > hobbit 15572 13306 0 03:58:28 - 0:00 sh > /usr/local/hobbit/client/bin/hobbitclient-aix.sh > root 16262 12252 0 11:07:52 pts/0 0:00 grep hob > hobbit 16692 1 0 Oct 30 - 0:03 > /usr/local/hobbit/client/bin/hobbitlaunch > --config=/usr/local/hobbit/client/etc/clientlaunch.cfg > --log=/usr/local/hobbit/client/logs/clientlaunch.log > --pidfile=/usr/local/hobbit/client/logs/clientlaunch.devarb2.pid > hobbit 18104 15572 64 03:58:29 - 206:41 netstat -an > hobbit 20870 15572 0 03:58:29 - 2:55 grep ^tcp > > If I go on that host (which is purple right now) and run "netstat -an > | grep ^tcp" it runs just fine. > > This happens mostly on AIX 4.3.3 > > There is nothing in any log file on the client to indicate any problem. > > Can provide any needed info. > > Thanks, > Jeff > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > **************************************************************************** The information contained in this email is intended only for the use of the intended recipient at the email address to which it has been addressed. If the reader of this message is not an intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination or copying of the message or associated attachments is strictly prohibited. If you have received this email in error, please contact the sender by return email or call 01793 877777 and ask for the sender and then delete it immediately from your system.Please note that neither RWE npower nor the sender accepts any responsibility for viruses and it is your responsibility to scan attachments (if any). *****************************************************************************

2 3

Warnings and patch oddities
by rmunsch＠solutionsforprogress.com 06 Nov '06

06 Nov '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm seeing a slew of these on a fresh DL of hobbit 4.2.0 and the allinone.patch: currant:~# patch -p5 <allinone.patch missing header for unified diff at line 4 of patch patching file hobbit-4.2.0/bbdisplay/bbgen.h missing header for unified diff at line 39 of patch patching file hobbit-4.2.0/bbdisplay/loadbbhosts.c missing header for unified diff at line 132 of patch patching file hobbit-4.2.0/bbdisplay/loadbbhosts.h missing header for unified diff at line 145 of patch patching file hobbit-4.2.0/bbdisplay/loaddata.c missing header for unified diff at line 244 of patch patching file hobbit-4.2.0/bbdisplay/pagegen.c missing header for unified diff at line 376 of patch patching file hobbit-4.2.0/bbnet/bbtest-net.c i don't really know what they mean. I note also that the patching instructions are not correct as-is. They state: http://www.hswn.dk/hobbitsw/patches/ 'All patches should be applied using GNU patch. "cd" to the hobbit-4.2.0 directory, then run "patch -p0 <allinone.patch".' However, since the path specified in the patch file is /home/henrik/hobbit/release/hobbit-4.2.0 i do not think this will work unless you are Henrik :) - -- Rob Munsch Solutions For Progress IT www.solutionsforprogress.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFT4hQBvBcJFK6xYURAgmbAJ95lVRJNE+Jjp/9st7VnKtx08iqhACeIUv0 2aprSUKlWaDoTENC0Dk8mUE= =WBvU -----END PGP SIGNATURE-----

2 1

bbgen frequent yellow alerts - hobbitd problem?
by pope8086＠gmail.com 06 Nov '06

06 Nov '06

Hi, We are running a new installation of Hobbit 4.2 on Solaris 10 running in a non-global zone. Server is a v240 but I don't think that matters here. The problem here is that our bbgen status turns yellow with fairly high frequency, sometimes multiple times an hour, at (what seem like) random intervals. In the yellow alert bbgen reports: "hobbitd status-board not available" During this time the hobbitd daemon is still running and the next time that bbgen runs the alert (usually) turns green. I've tested this by running bbgen every second, every 15 seconds, and every minute. The same is also true if I run bbgen by hand. During the 'yellow alert' time window the bb2.html gets updated with "All Monitored Systems OK" When all monitored systems are NOT OK. When the status turns green again this page reflects the correct status for the non-green systems. Below are the output from some commands/logs. These logs don't really seem to help, so let me know if there is anything else that I can send along to debug this issue. Any help is appreciated - we're near the point of frustration to where we may have to pull the plug on Hobbit and go back to our old BB installation. Thanks in advance. -Jon (logs below) hobbitd log from --debug. Way less entries here than normal. 2006-11-03 10:54:00 -> do_message/1 (12 bytes): hobbitdboard 2006-11-03 10:54:00 -> update_statistics 2006-11-03 10:54:00 <- update_statistics 2006-11-03 10:54:00 -> oksender 2006-11-03 10:54:00 <- oksender(1-a) 2006-11-03 10:54:00 -> setup_filter: hobbitdboard 2006-11-03 10:54:00 <- setup_filter: hobbitdboard 2006-11-03 10:54:00 <- do_message/1 2006-11-03 10:54:01 -> do_message/1 (0 bytes): 2006-11-03 10:54:01 -> update_statistics 2006-11-03 10:54:01 <- update_statistics 2006-11-03 10:54:01 <- do_message/1 $BB --debug $BBDISP "hobbitdboard" (with no --debug on a 'failure' I get no output. I'm assuming this is the same cause of the bbgen yellow alert) 2006-11-03 10:54:01 Transport setup is: 2006-11-03 10:54:01 bbdportnumber = 1984 2006-11-03 10:54:01 bbdispproxyhost = NONE 2006-11-03 10:54:01 bbdispproxyport = 0 2006-11-03 10:54:01 Recipient listed as '10.xxx.xxx.xxx' 2006-11-03 10:54:01 Standard BB protocol on port 1984 2006-11-03 10:54:01 Will connect to address 10.xxx.xxx.xxx port 1984 2006-11-03 10:54:01 Connect status is 0 2006-11-03 10:54:01 Sent 12 bytes 2006-11-03 10:54:01 Closing connection bbgen --debug --report (this one turned bbgen yellow/unavailable. Note the quick disconnect.) 2006-11-03 09:51:03 load_state() 2006-11-03 09:51:03 Transport setup is: 2006-11-03 09:51:03 bbdportnumber = 1984 2006-11-03 09:51:03 bbdispproxyhost = NONE 2006-11-03 09:51:03 bbdispproxyport = 0 2006-11-03 09:51:03 Recipient listed as '10.xxx.xxx.xxx' 2006-11-03 09:51:03 Standard BB protocol on port 1984 2006-11-03 09:51:03 Will connect to address 10.xxx.xxx.xxx port 1984 2006-11-03 09:51:03 Connect status is 0 2006-11-03 09:51:03 Sent 126 bytes 2006-11-03 09:51:03 Closing connection bbgen --debug --report (this one worked fine) 2006-11-03 09:54:00 load_state() 2006-11-03 09:54:00 Transport setup is: 2006-11-03 09:54:00 bbdportnumber = 1984 2006-11-03 09:54:00 bbdispproxyhost = NONE 2006-11-03 09:54:00 bbdispproxyport = 0 2006-11-03 09:54:00 Recipient listed as '10.xxx.xxx.xxx' 2006-11-03 09:54:00 Standard BB protocol on port 1984 2006-11-03 09:54:00 Will connect to address 10.xxx.xxx.xxx port 1984 2006-11-03 09:54:00 Connect status is 0 2006-11-03 09:54:00 Sent 126 bytes 2006-11-03 09:54:00 Read 16384 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 1 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 24578 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 24578 bytes 2006-11-03 09:54:00 Read 16503 bytes 2006-11-03 09:54:00 Closing connection

6 9

Larrd data problem, Implementation Assistance - New User
by jkwade＠futurefrontiers.com 06 Nov '06

06 Nov '06

Almost forgot, I had another problem. My larrd data get's the following errors on the Hobbit Server: update using time 1162503307 when last update time is 1162503307 (minimum one second step) 2006-11-02 15:35:07 RRD error updating /usr/local/apache2/htdocs/hobbit/data/rrd/iesx004/netstat.rrd from 192.168.20.33: Template contains more DS definitions than RRD So, how do I fix this and still keep my old larrd data. I get the same message with my iostat and vmstat rrd coming in from the larrd BB clients. Thanks.James

1 1

[hobbit] Custom Graph Problem
by Al.Jeffcoat＠orhs.org 06 Nov '06

06 Nov '06

Thanks Dan, I'm assuming that the following lines under database means that it has not received data for that time entry.  <row><v> NaN </v></row> There is no graph being shown on the hobbit web page for this service test. I'm still curious why the graph type is DERIVE instead of GAUGE, which is defined in hobbitserver.cfg. I added spaces between the DS name and the value, removed the .rrd, and let it regenerate, however, it's still a type DERIVE rrd. Is the correct thing spaces or no spaces on the status page? The ones that are working have no spaces between the DS name and value pair. They weren't working until I removed the spaces. If I manually create the rrd with rrdtool create, I'm wondering if it will start working, it just seems that the creation of the rrd is broken. Al _____ From: Dan Vande More [mailto:bigdan at gmail.com] Sent: Sunday, November 05, 2006 10:49 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Custom Graph Problem You gave us the first part of rrd dump, but you should also check to see whether rrd has any data in it. Is the graph's png showing up, but showing NaN? Is the graph not showing up at all? It sounds silly, but try putting spaces in your output so it looks like this: order_gen : 19 It seems like I've had that issue before too. I have about 25 extra tests I'm doing, and I use spaces in all of them. On 11/5/06, Jeffcoat, Al <Al.Jeffcoat at orhs.org> wrote: Hello All, I'm trying to create a custom script/graph that graphs some data from a mssql server. I've got the service (order_gen) and page showing up in hobbit, with the following output: Sun Nov 5 22:18:50 EST 2006 There are currently 19 requests in the ORDER GENERATION queue order_gen:19 Status unchanged in 9 hours, 17 minutes Status message received from 172.31.22.50 hobbit graph ncv:order_gen <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&action=menu> Zoom graph <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&graph=hourly&action=selzoom> I've defined the following in hobbitserver.cfg: TEST2RRD="......,order_gen=ncv" . . . NCV_order_gen="order_gen:GAUGE" . . GRAPHS="......,order_gen" It's creating an RRD under the hostname called order_gen.rrd. While trying to figure out why there is no graph coming up for this, I ran "rrdtool dump order_gen.rrd". Here's what's coming out: <ds> <name> ordergen </name> <type> DERIVE </type> I've search high and low on the hobbit archives looking for a similar situation, and can't find it, even though I thought I had seen it while originally working on these scripts on Friday. Can someone help me with the next steps on this? I'm not sure where to look next. This is my first custom graph, and it was a fun adventure in diving into how some things work in hobbit. After some trial and error, I've got 4 graphs working that are doing the same thing, which is trending different information from our main clinical app into hobbit for management perusal. This is the only one that's "out of order". TIA, Al This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message. To reply to our email administrator directly, send an email to: postmaster at orlandoregional.org . If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare. This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message. To reply to our email administrator directly, send an email to: postmaster at orlandoregional.org . If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare.

1 0

Different Custom Graph Question
by Thomas.Kern＠hq.doe.gov 06 Nov '06

06 Nov '06

I have a custom client running on a z/VM system with the hobbit server being run by a different group in our organization. I would like to provide CPU data on a smaller interval, such as 1 minute. I have modified my client to do that. Now comes the question, can the rrd files for my host be modified to record and report on that finer interval without altering the setups for all the other hosts that are being monitored? /Thomas Kern /301-903-2211

2 1