March 2005 - Xymon - Xymon mailman web

[hobbit] BOARDBUSY messages in logs
by dstuffle＠deltafaucet.com 04 Mar '05

04 Mar '05

I see, thanks. My main goal is trying to get the mail acks to work. That's why I was digging through the log files. Anyway, my mail acks are only sometimes working. I haven't been able to find a pattern to it yet so I will need to investigate more. I do know though that adding MESSAGE=something to the subject line when replying to an email alert doesn't work. Should that work, or am I misunderstanding something? I get this in the acklog (all one line): 1109963670 126510 30 126510 np_filename_not_used zenmaster.memory red ~~~~~~~~~~~~~~\nAcked by: "Stuffle, David" dstuffle at deltafaucet.com Several times that I acked, I get nothing at all in the acklog. Thanks, David -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Friday, March 04, 2005 12:13 PM To: hobbit at hswn.dk Subject: Re: [hobbit] BOARDBUSY messages in logs On Fri, Mar 04, 2005 at 12:00:22PM -0500, Stuffle, David wrote: > I am still seeing BOARDBUSY messages in my logs like the following: > larrd-status.log: > 2005-03-04 11:36:11 Tried to down BOARDBUSY: Invalid argument This one is harmless, and happens when Hobbit shuts down. The master daemon (hobbitd) sends a shutdown-message to all of the hobbitd_channel worker modules, then marks the shared memory segments used for communication with the modules for deletion. Depending on how fast the workers respond, they may try to access the shared memory segment when it has been deleted, and you'll get this message. > My log files don't seem to be getting updated properly and I'm > wondering if this has anything to do with it. I'm on Fedora. Which log files, specifically ? The only files that Hobbit normally updates while running are the historical logs in the data/hist/ and data/histlogs/ directories - and they only get updated when there are status changes. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

3 2

[hobbit] bb-central
by trotunda＠opushealthcare.com 04 Mar '05

04 Mar '05

Dan, Thanks for the info. I just started my learning curve a few days ago and already have 32 servers monitored. I am under the gun to prove Hobbit a better tool than Nagios and it looks to me like incorporating bb-central is all the ammo I will need. Very powerful if I can get it to work well. Thanks again, Tim Austin, Texas. -----Original Message----- From: Daniel J McDonald [mailto:dan.mcdonald at austinenergy.com] Sent: Friday, March 04, 2005 12:33 PM To: Hobbit List Subject: Re: [hobbit] bb-central On Fri, 2005-03-04 at 11:49 -0600, Tim Rotunda wrote: > Can anyone tell me if bb-central will work with Hobbit if I do not > have bbclient loaded? It looks like it but install says to change > bbrun.sh script and I don’t have one of those. In hobbit, it would be hobbitlaunch.cfg that you would fiddle with. You just create a new entry like this one I made for bbmrtg.pl: [bbmrtg] ENVFILE /var/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD $BBHOME/ext/bbmrtg.pl LOGFILE $BBSERVERLOGS/bbmrtg.log INTERVAL 5m > Do I just need to see what the format of that is? Am I barking up > the wrong tree here? I have 100 plus servers I need to monitor and I > really don’t want to deploy anything across all of them. Ditto. I haven't played with bb-central under hobbit yet, but that's coming up soon on my agenda. Any pointers you come up with are most appreciated. -- Daniel J McDonald, CCIE # 2495, CNX Austin Energy dan.mcdonald at austinenergy.com To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

2 1

recovery emails on alerts which don't generate pages == undesired behaviour
by blysik＠shutterfly.com 04 Mar '05

04 Mar '05

Hi, Previously, we've only wanted to alert on red and purples, and then send recovery emails when it changes out of a red or purple state. This was easily accomplished by setting --alertcolors to red,purple. Recently however, I've gotten some requests from people who want to get alerted on a few monitors when they yellow. No problem, I thought. I'll just change --alertcolors back to default, and then add COLOR=red,purple to all the existing alert definitions to start. This caused the problem where a monitor would go into a yellow state (not causing a page because COLOR=red,purple) and then go back to green (which would then send a recovery page). This current behaviour doesn't make sense. Why would I want to be alerted on a recovery of something that never generated an alert in the first place? It would make more sense if the recovery email condition was tied to the COLOR definition. So an alert with COLOR=red,yellow,purple (the default) would send a recovery message on leaving the red, yellow, or purple state to green, blue, or clear. And an alert with COLOR=red,purple would only send a recovery message if it was leaving a red or purple state. Or maybe I'm braindead and can't see how this can be accomplished currently. Comments? -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer

3 3

[hobbit] Iostat graphs
by Kevin.Hanrahan＠novainfo.com 04 Mar '05

04 Mar '05

You might be right...I honestly don't remember which one is on that particular server. I have had problems getting a single I/O script to work consistently across all our *NIX platforms (solaris, linux, and aix....maybe SCO someday) so I am really just looking for a good I/O tool that works cross-platform...and the graphs would be a major bonus too. Suggestions? Kevin -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Friday, March 04, 2005 1:32 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Iostat graphs On Fri, Mar 04, 2005 at 12:24:37PM -0500, Kevin.Hanrahan at novainfo.com wrote: > Here is the output of one of the systems: This doesn't look like the iostat output from the LARRD bf-iostat.sh script. More like something from the vmio script, but not quite. So it's quite understandable that you don't get any graphs, because this is an input format that is simply not recognized. If you can point me to the script you're using and some of the data you want to graph, I'll see if I can come up with something. But much of the data should already be available if you have the vmstat graphs (particularly the vmstat "wa" column for how much cpu time is spent in i/o wait). Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk Note: The information contained in this email and in any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. The recipient should check this email and any attachments for the presence of viruses. Sender accepts no liability for any damages caused by any virus transmitted by this email. If you have received this email in error, please notify us immediately by replying to the message and delete the email from your computer. This e-mail is and any response to it will be unencrypted and, therefore, potentially unsecure. Thank you. NOVA Information Systems, Inc.

1 0

bb-central
by trotunda＠opushealthcare.com 04 Mar '05

04 Mar '05

Can anyone tell me if bb-central will work with Hobbit if I do not have bbclient loaded? It looks like it but install says to change bbrun.sh script and I don't have one of those. Do I just need to see what the format of that is? Am I barking up the wrong tree here? I have 100 plus servers I need to monitor and I really don't want to deploy anything across all of them. Thanks for any info, Tim

2 1

[hobbit] Iostat graphs
by Kevin.Hanrahan＠novainfo.com 04 Mar '05

04 Mar '05

Here is the output of one of the systems: dwprod|iostat|green||1109955157|1109956353|1109958153|0|0|10.164.134.32|-1|| green Fri Mar 4 12:19:43 EST 2005 Virtual Memory Info (/usr/bin/vmstat 3 10) System Configuration: lcpu=4 mem=12288MB kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 1 2 1742256 221056 0 0 0 57 187 0 1118 11551 2259 13 5 65 17 2 4 1743281 220039 0 0 0 3638 35155 0 1383 21640 1561 28 14 2 56 3 4 1743464 219850 0 0 0 5302 59687 0 1380 32418 2129 30 22 11 36 3 5 1744448 218898 0 0 0 8531 68216 0 2418 9874 2417 14 15 12 59 2 4 1746221 217134 0 0 0 5888 53164 0 1180 56781 1929 28 14 19 39 2 3 1746231 217070 0 0 0 4739 29844 0 894 18563 2034 35 15 14 35 3 2 1746961 216412 0 0 0 6969 42484 0 986 14507 2087 51 26 7 17 2 2 1744307 219070 0 0 0 6787 47050 0 913 16042 1918 49 16 12 24 2 2 1744324 219032 0 0 0 5609 35614 0 797 15220 1527 34 12 24 30 4 2 1744324 219044 0 0 0 5046 28956 0 777 13252 1590 32 11 22 34 &green There's a avg of 2 processes in run queue. &green There's a avg of 3 blocked processes. &green There's a avg swap of 0 available. &green There's a avg of 0 k paged in. &green There's a avg of 0 k paged out. &green There's a avg scan rate of 44463 present. &green There's a avg sys cpu time of 0 percent. Disk I/O Statistics (/usr/bin/iostat -t 2 5) System configuration: lcpu=4 tty: tin tout avg-cpu: % user % sys % idle % iowait 0.3 25.6 12.7 5.5 64.8 17.0 0.0 0.0 32.2 12.1 21.2 34.4 0.0 0.0 22.6 17.2 6.6 53.6 0.0 0.0 34.8 13.8 3.8 47.8 0.0 0.0 37.8 18.8 4.5 38.9 &green There's an avg I/O wait time of 43. I do get an IOSTAT column with this data in it but the graph at the bottom has always been blank. -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Friday, March 04, 2005 11:49 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Iostat graphs On Fri, Mar 04, 2005 at 11:39:30AM -0500, Kevin Hanrahan wrote: > Has anyone ever gotten iostat to put any data into the larrd graphs? > I get plenty of data for iostat but the graphs are always blank. > Anybody got any advice? Lets see what data you're sending into the iostat handler. Could you do a ~hobbit/server/bin/bb 127.0.0.1 "hobbitdlog HOSTNAME.iostat" and show us the outpu ? If the iostat data is sent as "data" messages (i.e. you dont have an iostat column), then you'll need to snoop the messages going out on the hobbit data-channel: Login as the hobbit user, then run server/bin/bbcmd sh hobbitd_channel --channel=data cat |tee log |grep iostat Wait for a while until an iostat message shows up, then pick it out of the logfile. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk Note: The information contained in this email and in any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. The recipient should check this email and any attachments for the presence of viruses. Sender accepts no liability for any damages caused by any virus transmitted by this email. If you have received this email in error, please notify us immediately by replying to the message and delete the email from your computer. This e-mail is and any response to it will be unencrypted and, therefore, potentially unsecure. Thank you. NOVA Information Systems, Inc.

2 1

[hobbit] recovery emails on alerts which don't generate pages == undesired behaviour
by Kevin.Hanrahan＠novainfo.com 04 Mar '05

04 Mar '05

The patch for the page-a-minute prob did fix it. I havent tried RC5 yet but if it includes the patch, it should fix it as well KEvin -----Original Message----- From: Bruce Lysik [mailto:blysik at shutterfly.com] Sent: Friday, March 04, 2005 1:02 PM To: hobbit at hswn.dk Subject: RE: [hobbit] recovery emails on alerts which don't generate pages == undesired behaviour > The current code - i.e. RC4 plus the post-RC4 patch, plus the fix I > sent out yesterday to stop alerts from going off every minute - should > behave the way you want. As you say, it doesn't make sense to get a > recovery message when you didn't get the alert. I just tested it to be > absolutely certain it behaves, and it does. > > If you're confused about what version you've got (quite > understandable): Unpack the hobbit-4.0-RC4 archive; grab the latest > post-RC4 patch (I updated it yesterday) from http://www.hswn.dk/beta/ > and apply it with "cd hobbit-4.0-RC4; patch -p0 </tmp/post-RC4.patch" > copy over your Makefile from the old setup and run make, make install. Sweet! Yeah, I rolled back the RC4 patch when it started paging every minute, and didn't get a chance to try with the fix. I'll download the latest post-RC4 patch and give it a shot. -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk Note: The information contained in this email and in any attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. The recipient should check this email and any attachments for the presence of viruses. Sender accepts no liability for any damages caused by any virus transmitted by this email. If you have received this email in error, please notify us immediately by replying to the message and delete the email from your computer. This e-mail is and any response to it will be unencrypted and, therefore, potentially unsecure. Thank you. NOVA Information Systems, Inc.

1 0

[hobbit] recovery emails on alerts which don't generate pages == undesired behaviour
by blysik＠shutterfly.com 04 Mar '05

04 Mar '05

> The current code - i.e. RC4 plus the post-RC4 patch, plus the fix I > sent out yesterday to stop alerts from going off every minute - > should behave the way you want. As you say, it doesn't make sense to > get a recovery message when you didn't get the alert. I just tested it > to be absolutely certain it behaves, and it does. > > If you're confused about what version you've got (quite > understandable): Unpack the hobbit-4.0-RC4 archive; grab the latest > post-RC4 patch (I updated it yesterday) from http://www.hswn.dk/beta/ > and apply it with "cd hobbit-4.0-RC4; patch -p0 </tmp/post-RC4.patch" > copy over your Makefile from the old setup and run make, make install. Sweet! Yeah, I rolled back the RC4 patch when it started paging every minute, and didn't get a chance to try with the fix. I'll download the latest post-RC4 patch and give it a shot. -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer

1 0

BOARDBUSY messages in logs
by dstuffle＠deltafaucet.com 04 Mar '05

04 Mar '05

I am still seeing BOARDBUSY messages in my logs like the following: larrd-status.log: 2005-03-04 11:36:11 Tried to down BOARDBUSY: Invalid argument I've narrowed it down to the following code in hobbitd_channel.c, but I don't know how to fix it. My log files don't seem to be getting updated properly and I'm wondering if this has anything to do with it. I'm on Fedora. /* * Let master know we got it by downing BOARDBUSY. * This should not block, since BOARDBUSY is upped * by the master just before he ups GOCLIENT. */ do { s.sem_num = BOARDBUSY; s.sem_op = -1; s.sem_flg = IPC_NOWAIT; n = semop(channel->semid, &s, 1); } while ((n == -1) && (errno == EINTR)); if (n == -1) { errprintf("Tried to down BOARDBUSY: %s\n", strerror(errno)); } Can anyone provide any assistance? Thanks David Stuffle This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

2 1

Iostat graphs
by Kevin＠ewormhole.com 04 Mar '05

04 Mar '05

Hi all, Has anyone ever gotten iostat to put any data into the larrd graphs? I get plenty of data for iostat but the graphs are always blank. Anybody got any advice? Thanks KEvin

2 1