January 2005 - Xymon - Xymon mailman web

[hobbit] Valid alerts decleration?
by blysik＠shutterfly.com 27 Jan '05

27 Jan '05

> > > $HG-TEST=host01|host02 > > > Whoops! Drop the $ from the start of this line too! I thought that might be a possibility too, but no go. -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer

1 0

[hobbit] Valid alerts decleration?
by blysik＠shutterfly.com 27 Jan '05

27 Jan '05

> Looking back at Henrik's earlier email, I think this should be... > > HOST=%$HG-TEST Nope, that doesn't work either: -bash-2.05b$ ./bin/bbcmd --env=etc/hobbitserver.cfg hobbitd_alert --dump-config HOST=% SCRIPT /opt/bb/server/ext/email bruce_mail FORMAT=SCRIPT REPEAT=30 MAIL blysik at shutterfly.com REPEAT=30 Nor does defining the definition like this work: $HG-TEST=%host01|host02 or $HG-TEST=%(host01|host02) Hmm. Wonder what I'm missing. -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer

1 0

[hobbit] Valid alerts decleration?
by blysik＠shutterfly.com 27 Jan '05

27 Jan '05

> If in doubt, you can make hobbitd_alert print out how it interpreted > the config file: > > cd ~hobbit/server > ./bin/bbcmd --env=etc/hobbitserver.cfg hobbitd_alert --dump-config Thanks! This is useful. A related question: if hobbit-alerts.cfg is modified, does hobbit automatically pick up the new version, or is a restart required? -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer

2 1

easy way to set larrd tags for a group of systems in bb-hosts?
by tgeorgoulias＠nandomedia.com 27 Jan '05

27 Jan '05

I've been populating my bb-hosts files and adding tags to use many of the tests offered by hobbit that weren't in my BB setup. In doing so, I couldn't help but notice that for some sets of systems, they all use the same tags. For example, say my bb-hosts looks something like this, where different groups of servers have different services, but each server in that group has the same thing: IP web1.foo.com # "LARRD:*,vmstat:vmstat1|vmstat4" http://www.foo.com/ IP web2.foo.com # "LARRD:*,vmstat:vmstat1|vmstat4" http://www.foo.com/ IP web3.foo.com # "LARRD:*,vmstat:vmstat1|vmstat4" http://www.foo.com/ IP db1.foo.com # "LARRD:*,vmstat:vmstat|vmstat4" IP db2.foo.com # "LARRD:*,vmstat:vmstat|vmstat4" IP db3.foo.com # "LARRD:*,vmstat:vmstat|vmstat4" IP build1.foo.com # "LARRD:*,vmstat:vmstat2|vmstat3" ftp IP build2.foo.com # "LARRD:*,vmstat:vmstat2|vmstat3" ftp IP build3.foo.com # "LARRD:*,vmstat:vmstat2|vmstat3" ftp Is there an easy way that I can define a macro or meta tag that would capture this data, so I could do something like this (fabricated syntax to follow) where I group into a single variable and just reference that? web_monitors = ( "LARRD:*,vmstat:vmstat1|vmstat4" http://www.foo.com/ ) web1.foo.com # $web_monitors I checked the bb-host manpage and found a reference to using an include file to keep bb-hosts short, but I think I would still have to have individual lines in each include file. Just thought I'd ask. Tom

2 1

alerts and stop
by ctbeers＠syr.edu 27 Jan '05

27 Jan '05

Is there a way to stop rules from processing after issuing an alert (maybe it already does it). What I am looking for is specific alerts in the beginning of the file that would alert people to problems. Once the alert was issued it would stop after processing that definition (with a keyword STOP or something). This would allow me to create specific rules in the beginning of the file (for specific hosts, services, etc) and then a catchall at the bottom. If the specific rules were triggered, the generic catch all would not be. Chris

2 1

RECOVERED keyword in alerts
by blysik＠shutterfly.com 27 Jan '05

27 Jan '05

Hi, I'm trying to understand the RECOVERED keyword. If I add it to a rule, does it alert on both error and recovered states? Or do I have to define an extra line for every rule if I also want a recovery email for all alerts. -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer

2 1

Valid alerts decleration?
by blysik＠shutterfly.com 27 Jan '05

27 Jan '05

Hi, I just wanted to verify that his alerts snippet is valid (and does what I think it should do). It appears so from the documentation, but I wanted to run it by the group: HOST=$HG-ADMIN SCRIPT=/blah/blah/paging onduty SERVICE=procs SCRIPT=/blah/blah/paging ops SERVICE=procs SCRIPT=/blah/blah/paging onduty SERVICE=http,ping DURATION=6 SCRIPT=/blah/blah/paging ops SERVICE=http,ping DURATION=6 SCRIPT=/blah/blah/paging ops SERVICE=* EXSERVICE=procs,http,ping -- Bruce Z. Lysik <blysik at shutterfly.com> Operations Engineer

2 1

[hobbit] dual hobbit system config question
by KauffmanT＠nibco.com 27 Jan '05

27 Jan '05

Yes - my config has bbnet/bbpager/bbdisplay all on the same system, and the D/R system is bbdisplay with bbnet/bbpager failover running. I would like to just have one BBNET test system running at a time. I was planning on detecting the failure of the primary BBNET by having a script on the D/R system look for the ftp or sshd tests -- if I don't see these: Jan 27 12:25:06 elkhart sshd[19099]: Did not receive identification string from ::ffff:10.8.254.11 Jan 27 12:25:06 elkhart pure-ftpd: (?(a)whq-bbd.nibco.com) [INFO] New connection from whq-bbd.nibco.com Jan 27 12:25:06 elkhart pure-ftpd: (?(a)whq-bbd.nibco.com) [INFO] Logout. Then the network tests aren't running (elkhart is the failover, whq-bbd is the primary). The nice part about pulling these from the log is that they are time-stamped for my convenience. -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Thursday, January 27, 2005 11:42 AM To: hobbit at hswn.dk Subject: Re: [hobbit] dual hobbit system config question On Thu, Jan 27, 2005 at 10:21:50AM -0500, Kauffman, Tom wrote: > I currently run two BBDisplay systems, with the second serving as a > BBNET and BBPAGER failover system. It also serves as a document server > and our computer room X server. We recover this system at our D/R > hotsite, and part of the recovery scripting in place doctors the > bb-hosts file on this and all other recovered systems to set this up as > the D/R BB system. > > I see failover (as such) isn't in the initial cut of hobbit. What > problems am I setting myself up for if I allow the network tests to run > on both systems? I can handle the redundant paging by kludging my own > 'fallover' script to rename my paging script on the fallover, but it > looks like network testing would be harder to fake out. (I've only got > about 200 devices I'm testing) > > And I want both systems to have matching LARRD graphs :-) which is the > reason I set things up as is. As I understand your description, the two BBDISPLAY servers are normally running in parallel, each with their own set of data for webpages, RRD files, history logs etc - correct ? Also, is the primary BBNET server running on the same box as the primary BBDISPLAY server, or a separate system ? If BBDISPLAY and BBNET functions are combined on the same server both at the main site and the D/R site, then my suggestion is pretty simple: Just run the two sites completely in parallel, with each of the BBNET servers reporting to "their own" BBDISPLAY server. The only downside of this is that the measurements on the two sites might disagree! So there is no guarantee that you'll have identical displays at the two sites. On the D/R server you just disable the "[bbpage]" task in hobbitlaunch.cfg - that way, no alarms get sent. You can sync the bb-hosts and hobbit-alerts.cfg files between the two servers without any problems; Hobbit doesn't use the "BBDISPLAY", "BBNET" or "BBPAGER" tags in the bb-hosts file at all. Handling failover in this situation means somehow getting the D/R server to detect that Hobbit is down on the primary server, and then starting up the hobbitd_alert task to process alerts. You'll probably need to do some script that periodically either checks the primary server itself, or runs bb 127.0.0.1 "query primaryserver.bbd" and takes action when that status changes. Another option is to setup a hobbitd worker module that picks up events from the "stachg" (status change) channel, and reacts to changes in the state of the primary server. That might be a more elegant solution - see the "hobbitd_sample.c" file in the Hobbit sources for an example of how to write a worker module. If you only want the BBNET function running on one server at a time (e.g. because you must have identical displays at the two sites), or you have the BBNET server running on a different system at the primary site, then the situation becomes a bit more complex. The main site where BBNET normally runs would of course be configured to send the results to both of the BBDISPLAY servers. As described above, you can handle failover of the alert function by having the hobbitd_alert module turned off normally, and enabling it if the primary BBPAGER goes down. The problem as I see it would be how the D/R server detects that the primary BBNET server has failed. One possibility would be to enable the bbtest-net "--report" option; this makes bbtest-net send in a report about itself as the last status report from one cycle of the network tests. Something on the D/R server could monitor when this status was last received, and if it becomes too old it would then fire up the network testing task on the D/R server. If the status-report lifetime that bbtest-net generates was set to e.g. 10 minutes, then this could trigger when the status-report turned purple - so you could handle it with the same module that keeps an eye on changes in the primary BBDISPLAY server status. I haven't finally decided how to handle failover in Hobbit. I imagine that it means implementing a special "heartbeat" message that the servers send to each other - the BBDISPLAY servers exchange one, and the BBNET servers send one to the BBDISPLAY servers to inform them that they are alive. One of the BBDISPLAY servers then acts as an arbitrator, and decides which tasks must run where, and announces this to all of the servers, who then start or stop tasks as needed. If the arbitrator crashes, another BBDISPLAY server will have to take over. > (We monitor connectivity to RF antennas used for bar-code scanning > applications and our motley assortment of oracle and SAP instances on a > mob of AIX systems). Sounds like a lot of custom scripts are in use :-) Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

dual hobbit system config question
by KauffmanT＠nibco.com 27 Jan '05

27 Jan '05

I currently run two BBDisplay systems, with the second serving as a BBNET and BBPAGER failover system. It also serves as a document server and our computer room X server. We recover this system at our D/R hotsite, and part of the recovery scripting in place doctors the bb-hosts file on this and all other recovered systems to set this up as the D/R BB system. I see failover (as such) isn't in the initial cut of hobbit. What problems am I setting myself up for if I allow the network tests to run on both systems? I can handle the redundant paging by kludging my own 'fallover' script to rename my paging script on the fallover, but it looks like network testing would be harder to fake out. (I've only got about 200 devices I'm testing) And I want both systems to have matching LARRD graphs :-) which is the reason I set things up as is. (We monitor connectivity to RF antennas used for bar-code scanning applications and our motley assortment of oracle and SAP instances on a mob of AIX systems). TIA Tom Kauffman

2 1

[hobbit] vmstat graphing with CPU io wait
by CHRIS.MORRIS＠RWEnpower.com 27 Jan '05

27 Jan '05

Henrik, AIX also reports i/o wait in its vmstat output in column 16 under wa of cpu which it would be nice to have in the graphs. kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 0 2 238300 239351 0 0 0 60 108 0 110 79 83 8 30 30 32 0 2 238803 238827 0 0 0 0 0 0 498 3066 334 1 2 97 1 Chris > -----Original Message----- > From: Henrik Stoerner [SMTP:henrik at hswn.dk] > Sent: Tuesday, January 25, 2005 5:04 PM > To: hobbit at hswn.dk > Subject: Re: [hobbit] vmstat graphing with CPU io wait > > On Tue, Jan 25, 2005 at 08:27:37AM -0500, Tom Georgoulias wrote: > > Henrik Storner wrote: > > > > >Where do you get the I/O wait information from ? > > > > On RHEL3 (procps-2.0.17-10), there is a value for it in column 14 of > > vmstat's output, labeled "wa" under "cpu" > > Aha! So that's it - I had been wondering a bit why my load graphs > didn't always add up to 100% ! > > This is quite interesting, and definitely something that should be > tracked. So I hope you don't mind that I've tried adding it myself ... > > One annoying bit with the RRD files is that changing the dataset > (e.g. adding an extra variable) is not possible. So adding the > cpu_wait data will break any existing vmstat data that has been > collected. So if we're gonna break the vmstat RRD layout for Linux > clients, we might as well do it now before the official release. And > that should also include getting the very old layout (the one from > Linux 2.2 kernels, with the "r b w" proces-counts) aligned with the > new layout - effectively creating a single vmstat RRD format > regardless of what Linux version you are running. > > So: I've modified the Linux vmstat RRD layout to always include the > "cpu_w" (from the very old vmstat version) and "cpu_wait" columns > (from the latest vmstat versions). If the client doesn't report a > value for these, they are set to the special RRD-value "undefined". So > when someone upgrades a system from Linux 2.2. to 2.4, or from 2.4 to > 2.6, the vmstat data will still work. > > I've also defined a "vmstat1" graph similar to the normal "vmstat" > graph, but with the cpu_wait data added (it stacks on top of the > "system" time, below "user" time). > > Some sample graphs (they don't have any data yet, so you're probably > better off waiting a couple of hours before you view them): > > > Linux 2.6 host: > http://www.hswn.dk/hobbit-cgi/hobbitgraph.sh?host=voodoo.hswn.dk&service=v > mstat1&graph=hourly > > Linux 2.4 host: > http://www.hswn.dk/hobbit-cgi/hobbitgraph.sh?host=tyge.sslug.dk&service=vm > stat1&graph=hourly > > Linux 2.2 host (actually 2.4, but an old vmstat version): > http://www.hswn.dk/hobbit-cgi/hobbitgraph.sh?host=fenris.hswn.dk&service=v > mstat1&graph=hourly > > > Henrik > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > **************************************************************************** The information contained in this email is intended only for the use of the intended recipient at the email address to which it has been addressed. If the reader of this message is not an intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination or copying of the message or associated attachments is strictly prohibited. If you have received this email in error, please contact the sender by return email or call 01793 877777 and ask for the sender and then delete it immediately from your system.Please note that neither RWE npower nor the sender accepts any responsibility for viruses and it is your responsibility to scan attachments (if any). *****************************************************************************

3 7