- Xymon - Xymon mailman web

bbgen frequent yellow alerts - hobbitd problem?
by pope8086＠gmail.com 06 Nov '06

06 Nov '06

Hi, We are running a new installation of Hobbit 4.2 on Solaris 10 running in a non-global zone. Server is a v240 but I don't think that matters here. The problem here is that our bbgen status turns yellow with fairly high frequency, sometimes multiple times an hour, at (what seem like) random intervals. In the yellow alert bbgen reports: "hobbitd status-board not available" During this time the hobbitd daemon is still running and the next time that bbgen runs the alert (usually) turns green. I've tested this by running bbgen every second, every 15 seconds, and every minute. The same is also true if I run bbgen by hand. During the 'yellow alert' time window the bb2.html gets updated with "All Monitored Systems OK" When all monitored systems are NOT OK. When the status turns green again this page reflects the correct status for the non-green systems. Below are the output from some commands/logs. These logs don't really seem to help, so let me know if there is anything else that I can send along to debug this issue. Any help is appreciated - we're near the point of frustration to where we may have to pull the plug on Hobbit and go back to our old BB installation. Thanks in advance. -Jon (logs below) hobbitd log from --debug. Way less entries here than normal. 2006-11-03 10:54:00 -> do_message/1 (12 bytes): hobbitdboard 2006-11-03 10:54:00 -> update_statistics 2006-11-03 10:54:00 <- update_statistics 2006-11-03 10:54:00 -> oksender 2006-11-03 10:54:00 <- oksender(1-a) 2006-11-03 10:54:00 -> setup_filter: hobbitdboard 2006-11-03 10:54:00 <- setup_filter: hobbitdboard 2006-11-03 10:54:00 <- do_message/1 2006-11-03 10:54:01 -> do_message/1 (0 bytes): 2006-11-03 10:54:01 -> update_statistics 2006-11-03 10:54:01 <- update_statistics 2006-11-03 10:54:01 <- do_message/1 $BB --debug $BBDISP "hobbitdboard" (with no --debug on a 'failure' I get no output. I'm assuming this is the same cause of the bbgen yellow alert) 2006-11-03 10:54:01 Transport setup is: 2006-11-03 10:54:01 bbdportnumber = 1984 2006-11-03 10:54:01 bbdispproxyhost = NONE 2006-11-03 10:54:01 bbdispproxyport = 0 2006-11-03 10:54:01 Recipient listed as '10.xxx.xxx.xxx' 2006-11-03 10:54:01 Standard BB protocol on port 1984 2006-11-03 10:54:01 Will connect to address 10.xxx.xxx.xxx port 1984 2006-11-03 10:54:01 Connect status is 0 2006-11-03 10:54:01 Sent 12 bytes 2006-11-03 10:54:01 Closing connection bbgen --debug --report (this one turned bbgen yellow/unavailable. Note the quick disconnect.) 2006-11-03 09:51:03 load_state() 2006-11-03 09:51:03 Transport setup is: 2006-11-03 09:51:03 bbdportnumber = 1984 2006-11-03 09:51:03 bbdispproxyhost = NONE 2006-11-03 09:51:03 bbdispproxyport = 0 2006-11-03 09:51:03 Recipient listed as '10.xxx.xxx.xxx' 2006-11-03 09:51:03 Standard BB protocol on port 1984 2006-11-03 09:51:03 Will connect to address 10.xxx.xxx.xxx port 1984 2006-11-03 09:51:03 Connect status is 0 2006-11-03 09:51:03 Sent 126 bytes 2006-11-03 09:51:03 Closing connection bbgen --debug --report (this one worked fine) 2006-11-03 09:54:00 load_state() 2006-11-03 09:54:00 Transport setup is: 2006-11-03 09:54:00 bbdportnumber = 1984 2006-11-03 09:54:00 bbdispproxyhost = NONE 2006-11-03 09:54:00 bbdispproxyport = 0 2006-11-03 09:54:00 Recipient listed as '10.xxx.xxx.xxx' 2006-11-03 09:54:00 Standard BB protocol on port 1984 2006-11-03 09:54:00 Will connect to address 10.xxx.xxx.xxx port 1984 2006-11-03 09:54:00 Connect status is 0 2006-11-03 09:54:00 Sent 126 bytes 2006-11-03 09:54:00 Read 16384 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 1 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 24578 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 32767 bytes 2006-11-03 09:54:00 Read 24578 bytes 2006-11-03 09:54:00 Read 16503 bytes 2006-11-03 09:54:00 Closing connection

6 9

Larrd data problem, Implementation Assistance - New User
by jkwade＠futurefrontiers.com 06 Nov '06

06 Nov '06

Almost forgot, I had another problem. My larrd data get's the following errors on the Hobbit Server: update using time 1162503307 when last update time is 1162503307 (minimum one second step) 2006-11-02 15:35:07 RRD error updating /usr/local/apache2/htdocs/hobbit/data/rrd/iesx004/netstat.rrd from 192.168.20.33: Template contains more DS definitions than RRD So, how do I fix this and still keep my old larrd data. I get the same message with my iostat and vmstat rrd coming in from the larrd BB clients. Thanks.James

1 1

[hobbit] Custom Graph Problem
by Al.Jeffcoat＠orhs.org 06 Nov '06

06 Nov '06

Thanks Dan, I'm assuming that the following lines under database means that it has not received data for that time entry.  <row><v> NaN </v></row> There is no graph being shown on the hobbit web page for this service test. I'm still curious why the graph type is DERIVE instead of GAUGE, which is defined in hobbitserver.cfg. I added spaces between the DS name and the value, removed the .rrd, and let it regenerate, however, it's still a type DERIVE rrd. Is the correct thing spaces or no spaces on the status page? The ones that are working have no spaces between the DS name and value pair. They weren't working until I removed the spaces. If I manually create the rrd with rrdtool create, I'm wondering if it will start working, it just seems that the creation of the rrd is broken. Al _____ From: Dan Vande More [mailto:bigdan at gmail.com] Sent: Sunday, November 05, 2006 10:49 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Custom Graph Problem You gave us the first part of rrd dump, but you should also check to see whether rrd has any data in it. Is the graph's png showing up, but showing NaN? Is the graph not showing up at all? It sounds silly, but try putting spaces in your output so it looks like this: order_gen : 19 It seems like I've had that issue before too. I have about 25 extra tests I'm doing, and I use spaces in all of them. On 11/5/06, Jeffcoat, Al <Al.Jeffcoat at orhs.org> wrote: Hello All, I'm trying to create a custom script/graph that graphs some data from a mssql server. I've got the service (order_gen) and page showing up in hobbit, with the following output: Sun Nov 5 22:18:50 EST 2006 There are currently 19 requests in the ORDER GENERATION queue order_gen:19 Status unchanged in 9 hours, 17 minutes Status message received from 172.31.22.50 hobbit graph ncv:order_gen <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&action=menu> Zoom graph <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&graph=hourly&action=selzoom> I've defined the following in hobbitserver.cfg: TEST2RRD="......,order_gen=ncv" . . . NCV_order_gen="order_gen:GAUGE" . . GRAPHS="......,order_gen" It's creating an RRD under the hostname called order_gen.rrd. While trying to figure out why there is no graph coming up for this, I ran "rrdtool dump order_gen.rrd". Here's what's coming out: <ds> <name> ordergen </name> <type> DERIVE </type> I've search high and low on the hobbit archives looking for a similar situation, and can't find it, even though I thought I had seen it while originally working on these scripts on Friday. Can someone help me with the next steps on this? I'm not sure where to look next. This is my first custom graph, and it was a fun adventure in diving into how some things work in hobbit. After some trial and error, I've got 4 graphs working that are doing the same thing, which is trending different information from our main clinical app into hobbit for management perusal. This is the only one that's "out of order". TIA, Al This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message. To reply to our email administrator directly, send an email to: postmaster at orlandoregional.org . If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare. This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message. To reply to our email administrator directly, send an email to: postmaster at orlandoregional.org . If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare.

1 0

Different Custom Graph Question
by Thomas.Kern＠hq.doe.gov 06 Nov '06

06 Nov '06

I have a custom client running on a z/VM system with the hobbit server being run by a different group in our organization. I would like to provide CPU data on a smaller interval, such as 1 minute. I have modified my client to do that. Now comes the question, can the rrd files for my host be modified to record and report on that finer interval without altering the setups for all the other hosts that are being monitored? /Thomas Kern /301-903-2211

2 1

[hobbit] Custom Graph Problem
by Al.Jeffcoat＠orhs.org 06 Nov '06

06 Nov '06

I'll give that a shot. I had spaces in the output originally, and it wasn't working, so I took them out, and the other three graphs started working. One didn't work, and the DS name was missing an underscore, but it was a GAUGE type rrd, all I needed to do to fix it was to remove the underscore from the hobbitgraph.cfg file. Al Al Jeffcoat eServer Certified Specialist - pSeries Administration and Support for AIX 5L v5.2 Enterprise SAN and Storage Administrator System Programmer III (321)843-1051 Al.Jeffcoat at orhs.org (UPDATED) _____ From: Dan Vande More [mailto:bigdan at gmail.com] Sent: Sunday, November 05, 2006 10:49 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Custom Graph Problem You gave us the first part of rrd dump, but you should also check to see whether rrd has any data in it. Is the graph's png showing up, but showing NaN? Is the graph not showing up at all? It sounds silly, but try putting spaces in your output so it looks like this: order_gen : 19 It seems like I've had that issue before too. I have about 25 extra tests I'm doing, and I use spaces in all of them. On 11/5/06, Jeffcoat, Al <Al.Jeffcoat at orhs.org> wrote: Hello All, I'm trying to create a custom script/graph that graphs some data from a mssql server. I've got the service (order_gen) and page showing up in hobbit, with the following output: Sun Nov 5 22:18:50 EST 2006 There are currently 19 requests in the ORDER GENERATION queue order_gen:19 Status unchanged in 9 hours, 17 minutes Status message received from 172.31.22.50 hobbit graph ncv:order_gen <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&action=menu> Zoom graph <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&graph=hourly&action=selzoom> I've defined the following in hobbitserver.cfg: TEST2RRD="......,order_gen=ncv" . . . NCV_order_gen="order_gen:GAUGE" . . GRAPHS="......,order_gen" It's creating an RRD under the hostname called order_gen.rrd. While trying to figure out why there is no graph coming up for this, I ran "rrdtool dump order_gen.rrd". Here's what's coming out: <ds> <name> ordergen </name> <type> DERIVE </type> I've search high and low on the hobbit archives looking for a similar situation, and can't find it, even though I thought I had seen it while originally working on these scripts on Friday. Can someone help me with the next steps on this? I'm not sure where to look next. This is my first custom graph, and it was a fun adventure in diving into how some things work in hobbit. After some trial and error, I've got 4 graphs working that are doing the same thing, which is trending different information from our main clinical app into hobbit for management perusal. This is the only one that's "out of order". TIA, Al This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message. To reply to our email administrator directly, send an email to: postmaster at orlandoregional.org . If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare. This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message. To reply to our email administrator directly, send an email to: postmaster at orlandoregional.org . If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare.

1 0

help with LOG check
by Schrittenlocher＠rz.uni-frankfurt.de 06 Nov '06

06 Nov '06

Hi, it seems I haven't understood the log checks right :-( The goal I want to achieve is getting a log always presented in hobbit "msgs" "Full log" in its total, wether there are changes in it or not. My settings on the server: hobbit-clients.cfg: LOG /pica/prod/log/pica.log STAT IGNORE=%20032 HOST=lbsprod client-local.cfg: [lbsprod] log:`/pica/ffm/com/sys/hobbit/client/ext/reduce_message.ksh`:20480 The script reduce_message.ksh creates a modified copy of the log, adds `date +%H%M%S` at the beginning of each line (example: Stand 12:58:59 2006-11-06 12:39:52:94 UPD2000_60 STAT=00000001 CPL\FET \PPN 182300404 P Stand 12:58:59 2006-11-06 12:39:52:95 UPD2000_60 STAT=00000001 167: Beschreibung ist NICHT vorhanden. Stand 12:58:59 2006-11-06 12:40:52:00 UPD1000_60 STAT= LBS databse error. Foutcode uit database is 104400300 Stand 12:58:59 2006-11-06 12:40:52:79 UPD1000_60 STAT=00000001 Fehlercode nach Aufruf Server 18b, F ) and returns the log's name. So the log timestamp and content always differs from the old one and I hoped I could get hobbit to consider the whole log always as new and to send the total. Unfortunately it doesn't, it sends the last 8 lines (942 Byte) while the whole log has 11 kB: -rw-r--r-- 1 bb prod 11043 Nov 3 13:20 /pica/prod/log/pica.log Any ideas what I am doing wrong or how to achieve the goal? thanks for help Rolf -- Mit freundlichen Gruessen Rolf Schrittenlocher HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 - 798 28817 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at rz.uni-frankfurt.de

2 2

Custom Graph Problem
by Al.Jeffcoat＠orhs.org 06 Nov '06

06 Nov '06

Hello All, I'm trying to create a custom script/graph that graphs some data from a mssql server. I've got the service (order_gen) and page showing up in hobbit, with the following output: Sun Nov 5 22:18:50 EST 2006 There are currently 19 requests in the ORDER GENERATION queue order_gen:19 Status unchanged in 9 hours, 17 minutes Status message received from 172.31.22.50 <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&action=menu> <http://hobbit/hobbit-cgi/hobbitgraph.sh?host=srv607sql&service=ncv:orde r_gen&graph_width=576&graph_height=120&disp=srv607sql&nostale&color=gree n&graph=hourly&action=selzoom> I've defined the following in hobbitserver.cfg: TEST2RRD="......,order_gen=ncv" . . . NCV_order_gen="order_gen:GAUGE" . . GRAPHS="......,order_gen" It's creating an RRD under the hostname called order_gen.rrd. While trying to figure out why there is no graph coming up for this, I ran "rrdtool dump order_gen.rrd". Here's what's coming out: <ds> <name> ordergen </name> <type> DERIVE </type> I've search high and low on the hobbit archives looking for a similar situation, and can't find it, even though I thought I had seen it while originally working on these scripts on Friday. Can someone help me with the next steps on this? I'm not sure where to look next. This is my first custom graph, and it was a fun adventure in diving into how some things work in hobbit. After some trial and error, I've got 4 graphs working that are doing the same thing, which is trending different information from our main clinical app into hobbit for management perusal. This is the only one that's "out of order". TIA, Al This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message. To reply to our email administrator directly, send an email to: postmaster at orlandoregional.org . If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare.

4 3

Implementation Assistance - New User
by jkwade＠futurefrontiers.com 04 Nov '06

04 Nov '06

Hello All, I'm still having difficulties moving my existing Big Brother install over to Hobbit. I'm considering just continuing to use Big Brother. I'll do a summary to the group once I've gotten everything moved over and the problems that I've had. I need assistance with the clients. The server is working. 1) I did one client yesterday, and some of the scripts that I got from deadcat don't work. The reason is they source bbsource.local, so I'm working on that. What have other folks done with regard to deadcat scripts sourcing bbssys.local? 2) I had larrd installed on Big Brother, and assumed that the larrd scripts on the clients could be disabled because Hobbit did iostat and vmstat. However, they turned purple. So, I restarted the larrd scripts from within Hobbit. However, they still stay purple. They don't seem to update properly. Do you have to use larrd on the clients or does Hobbit do an iostat, vmstat test like larrd? 3) Does anyone use Hobbit for clients? I'm considering just living my clients using Big Brother because of less implementation hours. What's the advantage to moving to hobbit? 4) Are there any additional add-ins for hobbit, ie.. plugins Thanks..James

2 5

[hobbit] rrd-status.log error messages
by tj_yang＠hotmail.com 04 Nov '06

04 Nov '06

>illegal attempt to update using time 1162473047 when last update time is 1162473047 (minimum one second step)time on hobbit client was out of ntp sync.tjFrom: tj_yang at hotmail.comTo: hobbit at hswn.dkDate: Fri, 3 Nov 2006 19:06:18 -0600Subject: [hobbit] rrd-status.log error messages Hi, I have no clue what these error message mean ?Can I just delete the d241 memory.swap.rrd database to start from scratch ?Another one is bad data rrd channel, could this because of bb client rrd message ?<snip>2006-11-02 07:10:47 RRD error updating /var/opt/hobbit/data/rrd/d241.test.com/memory.swap.rrd from 11.61.2.141: illegal attempt to update using time 1162473047 when last update time is 1162473047 (minimum one second step)2006-11-02 08:37:08 Bad data in channel, skipping it2006-11-02 08:37:08 Dropping (more) garbled data2006-11-02 11:17:54 RRD error updating /var/opt/hobbit/data/rrd/ob02.test.com/tcp.conn.rrd from 11.61.2.141: illegal attempt to update using time 1162487874 when last update time is 1162487874 (minimum one second step)<snip>T.J. YangStay in touch with old friends and meet new ones with Windows Live Spaces Check it out! _________________________________________________________________ Stay in touch with old friends and meet new ones with Windows Live Spaces http://spaces.live.com/signup.aspx

1 0

one missing step in slab custom graph example
by tj_yang＠hotmail.com 04 Nov '06

04 Nov '06

I followed though the procedure in R1 and I am happy to see the custom graph got generated easily.One missing step I found is that there is no instruction on adding "slab" column definition in R1.R1: http://www.hswn.dk/hobbit/help/howtograph.htmlT.J. Yang _________________________________________________________________ Check the weather nationwide with MSN Search: Try it now! http://search.msn.com/results.aspx?q=weather&FORM=WLMTAG

1 0