Well, we watch for the presence of processes today. It would be nice to be able to track cpu and size of "important" processes over time.

Another problem is detecting CPU hogs (sometimes things run away), another problem is detecting processes with memory leaks -- they just grow and grow and grow. How can Hobbit help?

GLH

-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Tuesday, July 24, 2007 3:31 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Sat, Jul 21, 2007 at 07:16:12PM -0400, Asif Iqbal wrote:

• Display column only when it is red (http://www.*hobbit*mon.com/*hobbit*on/2006/08/msg00920.html)

I'll leave that for later. There will probably be an entire new version with just display things.

• SNMP trap by default
• SNMP probe option builtin

Too much for now. I need to dig into the Net-SNMP library API to do that.

• Process specific alert (http://www.hswn.dk/hobbiton/2005/11/msg00159.html)

Already in 4.2.0 via the GROUP definition in hobbit-clients.cfg and the corresponding rule in hobbit-alerts.cfg

• Comment TAG for DOWNTIME (http://www.hobbitmon.com/hobbiton/2007/04/msg00141.html)

Has been implemented for 4.3.0

• Add functionalities in `delay' (http://www.hswn.dk/hobbiton/2005/06/msg00272.html)

Haven't looked at that.

• CPU/Memory Usage per process (http://osdir.com/ml/monitoring.hobbit/2007-01/msg00429.html)

Probably impossible. Most "ps" implementations can report the current amount of cpu/memory a process uses, but that's a snapshot (ever noticed how "top" always has itself in the top list of cpu-using processes?). What's interesting is not how much cpu/memory a process uses exactly when the Hobbit client runs the "ps" command, but how much it has used on average since the last client run - similar to what "vmstat" reports for the system as a whole. I don't know of any way to get this data.

Another problem with this is identifying what a process is. A long-running daemon often forks child-processes that are short-lived; should we add their cpu-utilisation to that of the long-running process? If yes, then we have to monitor all processes that are started (so running once every N seconds is not sufficient); if no, then you won't spot the cpu hog because it was spawned as a child process.

• Text based alert for `msgs'. Currently it shows as html in my email (http://osdir.com/ml/monitoring.hobbit/2007-01/msg00203.html)

Easily done with an alert script.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

Thanks!

-----Original Message----- From: shea_greg at emc.com [mailto:shea_greg at emc.com] Sent: Tuesday, July 24, 2007 3:56 PM To: hobbit at hswn.dk Cc: shea_greg at emc.com Subject: RE: [hobbit] Highlights of the 4.3.0 version

Hi Greg,

I needed to do this originally with BB to track a memory leak with HP OpenView's pmd process, when we used to use it.

#!/bin/sh

SCRIPTS IN THE BBHOME/ext DIRECTORY ARE ONLY RUN IF # THEY ARE DEFINED

IN THE ENTRY FOR THE CURRENT HOST # LISTED IN THE ext/bb-bbexttab FILE.

BBPROG SHOULD JUST CONTAIN THE NAME OF THIS FILE # USEFUL WHEN YOU GET

ENVIRONMENT DUMPS TO LOCATE # THE OFFENDING SCRIPT...

BBPROG=bb-pmd.sh; export BBPROG

TEST NAME: THIS WILL BECOME A COLUMN ON THE DISPLAY # IT SHOULD BE AS

SHORT AS POSSIBLE TO SAVE SPACE...

NOTE YOU CAN ALSO CREATE A HELP FILE FOR YOUR TEST # WHICH SHOULD BE

PUT IN www/help/$TEST.html. IT WILL # BE LINKED INTO THE DISPLAY AUTOMATICALLY.

TEST="pmd"

BBHOME CAN BE SET MANUALLY WHEN TESTING.

OTHERWISE IT SHOULD BE SET FROM THE BB ENVIRONMENT

#BBHOME=/opt/BB/bb19c ; export BBHOME # FOR TESTING

if test "$BBHOME" = "" then echo "BBHOME is not set... exiting" exit 1 fi

if test ! "$BBTMP" # GET DEFINITIONS IF NEEDED then # echo "*** LOADING BBDEF ***" . $BBHOME/etc/bbdef.sh # INCLUDE STANDARD DEFINITIONS fi

PMDMEM=/bin/ps -e -o vsz -o comm | grep " pmd" | awk '{printf "%d", $1/1024}' if test "$PMDMEM" = "" then COLOR="clear" else COLOR="green" fi

AT THIS POINT WE HAVE OUR RESULTS. NOW WE HAVE TO SEND IT TO # THE

BBDISPLAY TO BE DISPLAYED...

MACHINE NAME MUST EITHER BE A REAL MACHINE NAME, OR # LOOK LIKE A REAL

MACHINE (in the case of arbitrary measurements # like temperature). IF THE NAME YOU ARE USING DOESN'T EXIST # IN THE DNS THEN IT SHOULD BE LISTED IN THE bb-hosts FILE WITH noping, # PREFERABLY IN IT'S OWN GROUP...

NOTE THE COMMAS HERE - YOU NEED THEM!

MACHINE=echo $MACHINE | $SED 's/\./,/g' # HAS TO BE IN A,B,C FORM

THE FIRST LINE IS STATUS INFORMATION... STRUCTURE IMPORANT!

THE REST IS FREE-FORM - WHATEVER YOU'D LIKE TO SEND...

LINE="PMD Statistics. " SUMMARY=" PMD memory usage is $PMDMEM"

NOW USE THE BB COMMAND TO SEND THE DATA ACROSS # SEND IT TO BBDISPLAY

$BB $BBDISP "status $MACHINE.$TEST $COLOR date $LINE $SUMMARY MB"

-----Original Message----- From: Hubbard, Greg L [mailto:greg.hubbard at eds.com] Sent: Tuesday, July 24, 2007 4:44 PM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

Well, we watch for the presence of processes today. It would be nice to be able to track cpu and size of "important" processes over time.

Another problem is detecting CPU hogs (sometimes things run away), another problem is detecting processes with memory leaks -- they just grow and grow and grow. How can Hobbit help?

GLH

-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Tuesday, July 24, 2007 3:31 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Sat, Jul 21, 2007 at 07:16:12PM -0400, Asif Iqbal wrote:

• Display column only when it is red (http://www.*hobbit*mon.com/*hobbit*on/2006/08/msg00920.html)

I'll leave that for later. There will probably be an entire new version with just display things.

• SNMP trap by default
• SNMP probe option builtin

Too much for now. I need to dig into the Net-SNMP library API to do that.

• Process specific alert (http://www.hswn.dk/hobbiton/2005/11/msg00159.html)

Already in 4.2.0 via the GROUP definition in hobbit-clients.cfg and the corresponding rule in hobbit-alerts.cfg

• Comment TAG for DOWNTIME (http://www.hobbitmon.com/hobbiton/2007/04/msg00141.html)

Has been implemented for 4.3.0

• Add functionalities in `delay' (http://www.hswn.dk/hobbiton/2005/06/msg00272.html)

Haven't looked at that.

• CPU/Memory Usage per process (http://osdir.com/ml/monitoring.hobbit/2007-01/msg00429.html)

Probably impossible. Most "ps" implementations can report the current amount of cpu/memory a process uses, but that's a snapshot (ever noticed how "top" always has itself in the top list of cpu-using processes?). What's interesting is not how much cpu/memory a process uses exactly when the Hobbit client runs the "ps" command, but how much it has used on average since the last client run - similar to what "vmstat" reports for the system as a whole. I don't know of any way to get this data.

Another problem with this is identifying what a process is. A long-running daemon often forks child-processes that are short-lived; should we add their cpu-utilisation to that of the long-running process? If yes, then we have to monitor all processes that are started (so running once every N seconds is not sufficient); if no, then you won't spot the cpu hog because it was spawned as a child process.

• Text based alert for `msgs'. Currently it shows as html in my email (http://osdir.com/ml/monitoring.hobbit/2007-01/msg00203.html)

Easily done with an alert script.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

On Tue, Jul 24, 2007 at 11:21:15PM +0200, Francesco Duranti wrote:

I've a problem with bbhostgrep of the latest snapshots ... It seems that he cannot get the data from the bb-hosts file...

The current snapshot has this fixed.

Regards, Henrik

On 7/24/07, Henrik Stoerner <henrik at hswn.dk> wrote:

On Sat, Jul 21, 2007 at 07:16:12PM -0400, Asif Iqbal wrote:

...

• Display column only when it is red (http://www.*hobbit*mon.com/*hobbit*on/2006/08/msg00920.html)

I'll leave that for later. There will probably be an entire new version with just display things.

...

• SNMP trap by default
• SNMP probe option builtin

Too much for now. I need to dig into the Net-SNMP library API to do that.

...

• Process specific alert (http://www.hswn.dk/hobbiton/2005/11/msg00159.html)

Already in 4.2.0 via the GROUP definition in hobbit-clients.cfg and the corresponding rule in hobbit-alerts.cfg

...

• Comment TAG for DOWNTIME (http://www.hobbitmon.com/hobbiton/2007/04/msg00141.html)

Has been implemented for 4.3.0

...

• Add functionalities in `delay' (http://www.hswn.dk/hobbiton/2005/06/msg00272.html)

Haven't looked at that.

...

• CPU/Memory Usage per process (http://osdir.com/ml/monitoring.hobbit/2007-01/msg00429.html)

Probably impossible. Most "ps" implementations can report the current amount of cpu/memory a process uses, but that's a snapshot (ever noticed how "top" always has itself in the top list of cpu-using processes?). What's interesting is not how much cpu/memory a process uses exactly when the Hobbit client runs the "ps" command, but how much it has used on average since the last client run - similar to what "vmstat" reports for the system as a whole. I don't know of any way to get this data.

Another problem with this is identifying what a process is. A long-running daemon often forks child-processes that are short-lived; should we add their cpu-utilisation to that of the long-running process? If yes, then we have to monitor all processes that are started (so running once every N seconds is not sufficient); if no, then you won't spot the cpu hog because it was spawned as a child process.

...

• Text based alert for `msgs'. Currently it shows as html in my email (http://osdir.com/ml/monitoring.hobbit/2007-01/msg00203.html)

Easily done with an alert script.

Well all my messages show up in html format. Wouldn't it be nice to generate the email, or have a choice to generate email, as text type instead of html type.

Also this email may suggest that text based email alert is possible.

http://www.hobbitmon.com/hobbiton/2005/10/msg00382.html

However, I might be misreading that email.

Again, please understand this is still just a low priority feature request.

Until then I will just explore the script idea that you suggested.

Appreciate all your work really!

Regards,

Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu

On Sat, Jul 21, 2007 at 09:34:11PM -0400, Scott Walters wrote:

Great to see the summary, these features look great. I'd like to request more RRDs and reports about the monitoring system and the servers/services monitored. For example:

I think the following could be "gauge" metrics:

Number of devices monitored Number of services monitored Number of host.service in green state Number of host.service in yellow state Number of host.service in red state Number of host.service in XXX state

You mean like this:

Statistics:
Hosts                      :  4321
Pages                      :   286
Status messages            : 22331
- Red                      :   907 ( 4.06 %)
- Red (non-propagating)    :   809 ( 3.62 %)
- Yellow                   :   353 ( 1.58 %)
- Yellow (non-propagating) :   210 ( 0.94 %)
- Clear                    :  1970 ( 8.82 %)
- Green                    : 17052 (76.36 %)
- Purple                   :   452 ( 2.02 %)
- Blue                     :   578 ( 2.59 %)

The first three are from the current "bbgen --report" status message; I've added the breakdown of the colors now. Will put these into an RRD for tracking trends.

I am thinking these could be done by creating counters within hobbit (since boot):

Number of state changes Number of state changes per server Number of state changes per service Number of notifications sent

The state changes can be calculated from the history logs. This is preferable, I think, because that way it won't get reset if the Hobbit server is restarted.

Notifications - it would make sense to have the alert module provide some statistics that we could put into a trend graph.

If you like, I could draft up some graphs and reports I'd like to see. My above description might be hard to visualize. I definitely think hobbit could benefit from internal counters, similarly to how on OS keeps tracks of context switches and the like.

Please do. The graphs I've created about the Hobbit "internals" have been mostly for my own use as debugging / performance evaluation data. If we can provide some data that is interesting to management, that would be a good thing.

Regards, Henrik

On Mon, Jul 23, 2007 at 08:59:54AM +0200, Charles Goyard wrote:

what a nice hook to tell about a bug in nopropred/nopropyellow: I _often_ (but not always) get a red status with nopropred on the bb2 page. Full report here:

[Jun, 21th] (http://www.hswn.dk/hobbiton/2007/06/msg00311.html)

noprop's don't affect the bb2 page - they only control if a status affects the color of the "main page" that the status is on. If you want to remove these from the BB2 page, run bbgen with "--bb2-ignorecolumns=procs_master,is_master".

Regards, Henrik

...

If you like, I could draft up some graphs and reports I'd like to see. My above description might be hard to visualize.

Henrik, you're right about using the histories for reports. That data keeps its integrity unlike the RRD averages, much better for reports.

For a given input period (Last 7 days, June 2007, etc.)

• Servers with the most state changes, sorted by highest to lowest (Maybe just Top 10). Clicking on server would generate list of state changes. "Look Bob, your server is not stable you need to get your developers under control!"

• Services with the most state changes, sorted by highest to lowest (Maybe just Top 10). Clicking on service would generate list of the state changes for that period. "PHB, the web group is performing way too many undocumented code changes."

• Red events with longest durations (for events still open, use start time to NOW as duration)

• Yellow events with longest durations (for events still open, use start time to NOW as duration)

• All ping/fping/conn events.

You could piece-meal some of those from the eventlog report, but I'd prefer a single page that showed them all. For weekly, quarterly meetings, turnover, etc.

Scott

On Tue, Jul 24, 2007 at 09:18:49AM -0500, Ralph Mitchell wrote:

On 7/23/07, Scott Walters <scott at packetpushers.com> wrote:

...

• Services with the most state changes, sorted by highest to lowest (Maybe just Top 10). Clicking on service would generate list of the state changes for that period. "PHB, the web group is performing way too many undocumented code changes."

Heh, that would be useful. I've got a perl script using SOAP to get BigIP pool status and some joker has transferred some machines between BigIPs without removing the old definitions. So, there's a bunch of systems/ports that flip/flop between enable & disable. Whether they're red or green depends on which report comes in last.

That should actually be caught by another 4.3.0 feature: Flap detection. If a status changes more than 10 times in 10 minutes, Hobbit deems it "flapping" and stops logging status changes - instead, it fixes the status at the most critical level reported.

Any hosts flapping are reported on the "hobbitd" status display.

Regards, Henrik

On Mon, Jul 23, 2007 at 09:44:11PM -0400, Scott Walters wrote:

For a given input period (Last 7 days, June 2007, etc.)

• Servers with the most state changes, sorted by highest to lowest (Maybe just Top 10). Clicking on server would generate list of state changes. "Look Bob, your server is not stable you need to get your developers under control!"

• Services with the most state changes, sorted by highest to lowest (Maybe just Top 10). Clicking on service would generate list of the state changes for that period. "PHB, the web group is performing way too many undocumented code changes."

I've whipped up a very rough implementation as part of the eventlog report on the Hobbit demo site. Could you try generating a report at http://www.hswn.dk/hobbit-cgi/bb-eventlog.sh and let me know if the data at the top is something in the right direction?

The nice thing about making it an add-on or variant of the eventlog report is that there's already all of the nice filtering for hosts, pages, time-periods etc in place, plus the "allevents" logfile parsing is also done.

Regards, Henrik

Hi Henrik,

New Feature Request.-

I would like to add some network news on the pages. Suppose we found that some server has bad disk and some one will fix it later so here i want add the info that this issue has taken care.

On 7/25/07, Scott Walters <scott at packetpushers.com> wrote:

On 7/24/07, Henrik Stoerner <henrik at hswn.dk> wrote:

...

I've whipped up a very rough implementation as part of the eventlog report on the Hobbit demo site. Could you try generating a report at http://www.hswn.dk/hobbit-cgi/bb-eventlog.sh and let me know if the data at the top is something in the right direction?

Bingo. But since that was so easy, here are a few more:

• So voodoo.hswn.dk has had 12 state changes . . . what were they? It would be nice If the server name and service name could be HTML links which would generate a report of the state changes for the specified server/service over the given period.

• Also, please show the total. If there are more then 10 hosts/services use an "Other" at the end of the list. I love seeing single hosts on 100+ node installs with 25% of activity. You know where to focus.

• And I would imagine the "Top X" where X is configurable will be requested.

• And print the report period on the page so you know what you are looking at.

...

The nice thing about making it an add-on or variant of the eventlog report is that there's already all of the nice filtering for hosts, pages, time-periods etc in place, plus the "allevents" logfile parsing is also done.

We'll keep requesting features until it gets hard ;)

Scott Walters -PacketPusher

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

-- Thanks Sabeer MZ

Henrik Stoerner wrote :

On Wed, Jul 25, 2007 at 11:26:33AM +0530, Sabeer MZ wrote:

...

New Feature Request.-

I would like to add some network news on the pages. Suppose we found that some server has bad disk and some one will fix it later so here i want add the info that this issue has taken care.

Several possibilities already:

1. Ack the red/yellow statuses you have, and put this information in the acknowledgement text.
2. Disable the server and provide the information in the disable text.
3. Create a host "notes" file with the information.

Wasn't there a bb_bulletin feature too ?

-- Charles Goyard - charles.goyard at orange-ftgroup.com - (+33) 1 45 38 01 31 Orange Business Services - online multimedia // ingénierie

Many thanks. I ll check it out...

On 7/25/07, Henrik Stoerner <henrik at hswn.dk> wrote:

On Wed, Jul 25, 2007 at 11:26:33AM +0530, Sabeer MZ wrote:

...

New Feature Request.-

I would like to add some network news on the pages. Suppose we found that some server has bad disk and some one will fix it later so here i want add the info that this issue has taken care.

Several possibilities already:

1. Ack the red/yellow statuses you have, and put this information in the acknowledgement text.
2. Disable the server and provide the information in the disable text.
3. Create a host "notes" file with the information.

I'd use 1) or 2). I don't see the need for a fourth way of doing this.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

-- Thanks Sabeer MZ

On 7/31/07, Jones, Jason (Altrincham) <JasonAS_Jones at mentor.com> wrote:

I've been looking through the archives at the sample server side module Henrick showed us (http://www.hswn.dk/hobbiton/2007/01/msg00487.html) I'm just curious if anyone knows how to send data from the client side to the server the same way the standard hobbit client tests do, looking at bb I'm guessing either bb data or bb client, but when I run bb <hobbitIP> "client <hostname>.<os>" nothing happens and the man pages don't mention how to actually send the data etc.

I have a Hobbit client install running a BigIP check. In client/etc/clientlaunch.cfg:

  [bigip-v4]
      ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
      CMD $HOBBITCLIENTHOME/ext/bigip/bigip3.sh
      LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log
      INTERVAL 5m

After doing what it needs to do to get the status, the script sends off a status message to the server like this:

  MACHINE=`echo $NAME | sed -e 's/\./,/g'`
  MESSAGE="status $MACHINE.$TEST $COLOR `date`<P><font size=+2>The

$BIGIP BigIP says: $NAME $TEST is $STATE</font>" $BB $BBDISP $MESSAGE

Is that what you're looking for??

Ralph Mitchell

On Tue, Jul 24, 2007 at 10:15:02PM -0400, Scott Walters wrote:

On 7/24/07, Henrik Stoerner <henrik at hswn.dk> wrote:

...

I've whipped up a very rough implementation as part of the eventlog report on the Hobbit demo site. Could you try generating a report at http://www.hswn.dk/hobbit-cgi/bb-eventlog.sh and let me know if the data at the top is something in the right direction?

Bingo. But since that was so easy, here are a few more: [snip] We'll keep requesting features until it gets hard ;)

Reports are usually rather boring things to do, but this one was fun. Have a look at the current state of this report at the Hobbit demo site http://www.hswn.dk/hobbit-cgi/hobbit-topchanges.sh (You can also just go to the demo site, and pick the "Reports" -> "Top Changes" report).

Should cover everything you've asked for - at least until now, that is.

Regards, Henrik

Reports are usually rather boring things to do, but this one was fun. Have a look at the current state of this report at the Hobbit demo site http://www.hswn.dk/hobbit-cgi/hobbit-topchanges.sh (You can also just go to the demo site, and pick the "Reports" -> "Top Changes" report).

Should cover everything you've asked for - at least until now, that is.

Looks really great! Can we have next to the numbers also the percentage related to all event changes in the defined timeframe?

Johann

On Wed, Jul 25, 2007 at 04:12:06PM +0200, Johann Eggers wrote:

...

Reports are usually rather boring things to do, but this one was fun. Have a look at the current state of this report at the Hobbit demo site http://www.hswn.dk/hobbit-cgi/hobbit-topchanges.sh

Looks really great! Can we have next to the numbers also the percentage related to all event changes in the defined timeframe?

Sure, already done.

I also added something I felt was missing: When you have the top-10 list showing that host "foo" has the most status changes, then when you click on that host I wanted an overview of what services put it in the top-10. So I added a summary by service when you click on a host in the top-10 display.

And likewise when you click on a service in the top-10 list, it gives you a list of the hosts that were counted for that service.

Regards, Henrik

On Wed, Jul 25, 2007 at 01:11:47PM -0400, Scott Walters wrote:

On 7/25/07, Henrik Stoerner <henrik at hswn.dk> wrote:

...

I also added something I felt was missing: When you have the top-10 list showing that host "foo" has the most status changes, then when you click on that host I wanted an overview of what services put it in the top-10. So I added a summary by service when you click on a host in the top-10 display.

Wow. That is awesome. Great idea. Someone needs to talk to the mail admin of SMTP for www.sslug.dk!

Could you also add the report period to the server and services "sub-reports"?

Done.

Could you add state changes by day of week and hour of day? And since this all been so easy, how about trending reports based on an interval?

Let's leave those for now - these will be more difficult to implement. The only other addition I'd like to make for this report now is to have it count the event durations instead of the number of changes, so you can have a top-10 report of the hosts (or services) that have the longest outages. Could be useful when playing the "blame game". "Look - the DB people are always soooo slow when it comes to cleaning up the filled tables".

Regards, Henrik

On Thu, Jul 26, 2007 at 02:35:26PM -0500, Ralph Mitchell wrote:

Just a couple of minor observations on the top-10 list:

1. the right hand box, "Top 10 Services" has a "Host" column. Probably should be "Service"??

Of course - fixed.

2. I was lazy and just put in the date, with no time of day, and scored an "Internal Server Error". Could it default to "from 00:00:00" & "to 23:59:59", or maybe have a "last XX minutes OR from/to", same as the Notification Report??

I'll have to do some extra checking on that input. I've also added some buttons so you can easily select the last/current year/month/week.

Regards, Henrik

One thing that might be useful would be to alter the importance flag so if a critical system goes down it comes in with an exclamation mark etc. so it stands out from the other hobbit alerts....just a thought. Jason.

-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: 25 July 2007 15:05 To: hobbit at hswn.dk Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Tue, Jul 24, 2007 at 10:15:02PM -0400, Scott Walters wrote:

On 7/24/07, Henrik Stoerner <henrik at hswn.dk> wrote:

...

I've whipped up a very rough implementation as part of the eventlog report on the Hobbit demo site. Could you try generating a report at http://www.hswn.dk/hobbit-cgi/bb-eventlog.sh and let me know if the data at the top is something in the right direction?

Bingo. But since that was so easy, here are a few more: [snip] We'll keep requesting features until it gets hard ;)

Reports are usually rather boring things to do, but this one was fun. Have a look at the current state of this report at the Hobbit demo site http://www.hswn.dk/hobbit-cgi/hobbit-topchanges.sh (You can also just go to the demo site, and pick the "Reports" -> "Top Changes" report).

Should cover everything you've asked for - at least until now, that is.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

On 7/25/07, Scott Walters <scott at packetpushers.com> wrote:

Perfect. You rock Henrik.

Not quite, Could you add the total at the bottom of the "Top X" list.

Scott Walters -PacketPusher

Wonder if there is any way to tell a client what it's status is so it can be autonomous? What I mean is this: suppose there was a way for the Hobbit client to tell the server that service X was now in state Y, and a client-side module could then activate response Z on its own?

I know the Hobbit model is to have the server own the configurations, but how do we solve the "trust" problem?

GLH

-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Tuesday, July 24, 2007 3:41 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Sun, Jul 22, 2007 at 08:01:12PM -0400, Asif Iqbal wrote:

Here is another feature I like to see.

A way for the hobbit server to request hobbit clent to run a command locally based on an alert. [snip] So whenever there is a msgs alert run that command locally on the client

Run this as a client extension:

#!/bin/sh

Get the current status of the "msgs" column

MSGSSTATUS=$BB $BBDISP "query $MACHINE.msgs" | awk '{ print $1 }

Get the command we must run from the client config

CMD=grep "^msgsrecovercmd:" $BBTMP/logfetch.$MACHINEDOTS.cfg | sed -e 's!^msgsrecovercmd:!!'

If "msgs" is red and there is a command, run it

if test "$MSGSSTATUS" = "red" -a "$CMD" != "" then $CMD fi

exit 0

Before doing this, consider the security implications of having your servers run commands that they fetch from a remote host without authentication.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

On Tuesday 24 July 2007 22:55:02 Hubbard, Greg L wrote:

Wonder if there is any way to tell a client what it's status is so it can be autonomous? What I mean is this: suppose there was a way for the Hobbit client to tell the server that service X was now in state Y, and a client-side module could then activate response Z on its own?

I don't like band-aids like this.

"restart because it's down" prevents the real impact of problems being seen, and provides less motivation for fixing things properly. Instead, you sit with frequent short outages (which may avoid the attention of managers, production managers) which have end-user impact.

I like even less using a monitoring system to do this ...

Regards, Buchan

DOn't forget...this is the model that Tivoli and HP Openview, and many other commercial monitoring solutions provide and sell as a feature.

From my experience as a sys admin, I've alwys found that automatically restarting a service if it goes down to be "a bad thing"(TM).

In many solutions, logs get overwritten upon a restart that would be integral to the real resolution and prevention.

=G=

-----Original Message----- From: Tod Hansmann [mailto:thansmann at directpointe.com] Sent: Friday, August 03, 2007 10:40 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

In my experience, I have to agree. Hobbit is for monitoring so the information that x is down gets to people who can properly diagnose what is going on, not take generic actions. If generic actions were something that were required for X to function properly, it should be a feature of that software.

Hobbit CAN do some scripting based on alerts, but even that might be a bit more than a systems administrator wants to hinder himself with.

Tod Hansmann Network Engineer

-----Original Message----- From: Buchan Milne [mailto:bgmilne at staff.telkomsa.net] Sent: Friday, August 03, 2007 12:31 AM To: hobbit at hswn.dk Cc: Hubbard, Greg L Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Tuesday 24 July 2007 22:55:02 Hubbard, Greg L wrote:

Wonder if there is any way to tell a client what it's status is so it can be autonomous? What I mean is this: suppose there was a way for the Hobbit client to tell the server that service X was now in state Y, and a client-side module could then activate response Z on its own?

I don't like band-aids like this.

"restart because it's down" prevents the real impact of problems being seen, and provides less motivation for fixing things properly. Instead, you sit with frequent short outages (which may avoid the attention of managers, production managers) which have end-user impact.

I like even less using a monitoring system to do this ...

Regards, Buchan

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

Well, I use Netcool which has the opposite philosophy -- there is a "process automation" system that watches processes and restarts them if they fail, while also logging restarts. You can configure a "restart" parameter to be anything from 0 (forever) to any number of times. I like to set a reasonable number so persistent errors eventually kill the process, but occasional errors do not. Log files are not overwritten, but are appended and rotated.

But whatever. My view seems to be in the minority -- guess the rest of you don't mind 24x7x365 babysitting.

GLH

-----Original Message----- From: Galen Johnson [mailto:Galen.Johnson at sas.com] Sent: Friday, August 03, 2007 10:18 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

DOn't forget...this is the model that Tivoli and HP Openview, and many other commercial monitoring solutions provide and sell as a feature.

From my experience as a sys admin, I've alwys found that automatically restarting a service if it goes down to be "a bad thing"(TM).

In many solutions, logs get overwritten upon a restart that would be integral to the real resolution and prevention.

=G=

-----Original Message----- From: Tod Hansmann [mailto:thansmann at directpointe.com] Sent: Friday, August 03, 2007 10:40 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

In my experience, I have to agree. Hobbit is for monitoring so the information that x is down gets to people who can properly diagnose what is going on, not take generic actions. If generic actions were something that were required for X to function properly, it should be a feature of that software.

Hobbit CAN do some scripting based on alerts, but even that might be a bit more than a systems administrator wants to hinder himself with.

Tod Hansmann Network Engineer

-----Original Message----- From: Buchan Milne [mailto:bgmilne at staff.telkomsa.net] Sent: Friday, August 03, 2007 12:31 AM To: hobbit at hswn.dk Cc: Hubbard, Greg L Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Tuesday 24 July 2007 22:55:02 Hubbard, Greg L wrote:

Wonder if there is any way to tell a client what it's status is so it can be autonomous? What I mean is this: suppose there was a way for the Hobbit client to tell the server that service X was now in state Y, and a client-side module could then activate response Z on its own?

I don't like band-aids like this.

"restart because it's down" prevents the real impact of problems being seen, and provides less motivation for fixing things properly. Instead, you sit with frequent short outages (which may avoid the attention of managers, production managers) which have end-user impact.

I like even less using a monitoring system to do this ...

Regards, Buchan

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

I am definitely in the "monitor only" camp. As appealing as "self-healing" may seem, I've seen attempts go horrible wrong too many times. For example, shutting down Oracle for upgrades and then being restarted in the middle of the upgrade. Not good.

I also agree that "self-healing" lends itself to band-aids that avoid root-cause determination. I don't think this requires "baby-sitting," but a commitment to fixing things once. I have also had the displeasure of making permanent band-aids, but I cannot condone it.

All of those "operational" aspects aside, I've convinced myself from a security point of view, corrective action from monitoring is bad-- a clear violation of the separation of duties. You don't want your auditors "cleaning up" the numbers as they go over your books.

You know what's better than your webserver being automatically restarted when it crashes? Your webserver not crashing.

I completely support the absence of corrective actions from monitor triggers. The question I have yet to answer satisfactorily is,"Should the monitoring system perform additional data collection after specific errors?" For example, running a particular "find" command when disk usage increases to try and identify which files are causing the partition to fill.

Scott Walters -PacketPusher

On 8/3/07, Hubbard, Greg L <greg.hubbard at eds.com> wrote:

Well, I use Netcool which has the opposite philosophy -- there is a "process automation" system that watches processes and restarts them if they fail, while also logging restarts. You can configure a "restart" parameter to be anything from 0 (forever) to any number of times. I like to set a reasonable number so persistent errors eventually kill the process, but occasional errors do not. Log files are not overwritten, but are appended and rotated.

But whatever. My view seems to be in the minority -- guess the rest of you don't mind 24x7x365 babysitting.

GLH

-----Original Message----- From: Galen Johnson [mailto:Galen.Johnson at sas.com] Sent: Friday, August 03, 2007 10:18 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

DOn't forget...this is the model that Tivoli and HP Openview, and many other commercial monitoring solutions provide and sell as a feature. From my experience as a sys admin, I've alwys found that automatically restarting a service if it goes down to be "a bad thing"(TM).

In many solutions, logs get overwritten upon a restart that would be integral to the real resolution and prevention.

=G=

-----Original Message----- From: Tod Hansmann [mailto:thansmann at directpointe.com] Sent: Friday, August 03, 2007 10:40 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

In my experience, I have to agree. Hobbit is for monitoring so the information that x is down gets to people who can properly diagnose what is going on, not take generic actions. If generic actions were something that were required for X to function properly, it should be a feature of that software.

Hobbit CAN do some scripting based on alerts, but even that might be a bit more than a systems administrator wants to hinder himself with.

Tod Hansmann Network Engineer

-----Original Message----- From: Buchan Milne [mailto:bgmilne at staff.telkomsa.net] Sent: Friday, August 03, 2007 12:31 AM To: hobbit at hswn.dk Cc: Hubbard, Greg L Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Tuesday 24 July 2007 22:55:02 Hubbard, Greg L wrote:

...

Wonder if there is any way to tell a client what it's status is so it can be autonomous? What I mean is this: suppose there was a way for the Hobbit client to tell the server that service X was now in state Y, and a client-side module could then activate response Z on its own?

I don't like band-aids like this.

"restart because it's down" prevents the real impact of problems being seen, and provides less motivation for fixing things properly. Instead, you sit with frequent short outages (which may avoid the attention of managers, production managers) which have end-user impact.

I like even less using a monitoring system to do this ...

Regards, Buchan

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

Sometimes the real world runs interference for Utopia. While in Utopia you want to analyse, find the root cause, and fix everything before proceding, you can't always do that. When an outage of one hour costs your company tens of thousands of dollars, you can't justify withholding a simple bandaid (so long as you don't then ignore the long term fix).

Most everything I do in Hobbit is a custom script. Restarting crashed processes is one of the least of my worries. Although in some rare cases I do just that (short term), with appropriate logging and email to the app developement team. The corporate expense of having the app down is too great to let Utopian ideas prevail.

Most of the automated Hobbit stuff I do is not restarting dead apps (luckily, that is very infrequent around here). It's more mundane. One example is disk space. A full filesystem would shut many things down. Apps should not fill a filesystem, but sometimes they do. So my custom Hobbit scripts first scream and scream about low disk space, even analysing things down to specific subdirectories and fast growing files and doing trend analysis. But if their call is not answered, they start freeing up space from a "private reserve" I have set aside to deal with emergencies. So if we experience a sudden unexpected blowup in a filesystem at 3am, Hobbit keeps things running in production until the appropriate people can look into and diagnose the problem. This may not be Utopian behavior, but it sure is practical at 3am in the morning!

But my vote would be for Hobbit out-of-the-box to NOT attempt automated repair actions. That should be left to the Hobbit administrator. We can write custom monitor scripts or custom alert scripts to add this functionality if it's appropriate for our environments. It's trivial to integrate your own scripting into Hobbit.

I sure wish I worked in Utopia though. The job would be a helluva lot less stressful! :-)

-----Original Message----- From: scottrwalters at gmail.com [mailto:scottrwalters at gmail.com] On Behalf Of Scott Walters Sent: Friday, August 03, 2007 11:15 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Highlights of the 4.3.0 version

I am definitely in the "monitor only" camp. As appealing as "self-healing" may seem, I've seen attempts go horrible wrong too many times. For example, shutting down Oracle for upgrades and then being restarted in the middle of the upgrade. Not good.

I also agree that "self-healing" lends itself to band-aids that avoid root-cause determination. I don't think this requires "baby-sitting," but a commitment to fixing things once. I have also had the displeasure of making permanent band-aids, but I cannot condone it.

All of those "operational" aspects aside, I've convinced myself from a security point of view, corrective action from monitoring is bad-- a clear violation of the separation of duties. You don't want your auditors "cleaning up" the numbers as they go over your books.

You know what's better than your webserver being automatically restarted when it crashes? Your webserver not crashing.

I completely support the absence of corrective actions from monitor triggers. The question I have yet to answer satisfactorily is,"Should the monitoring system perform additional data collection after specific errors?" For example, running a particular "find" command when disk usage increases to try and identify which files are causing the partition to fill.

Scott Walters -PacketPusher

On 8/3/07, Hubbard, Greg L <greg.hubbard at eds.com> wrote:

Well, I use Netcool which has the opposite philosophy -- there is a "process automation" system that watches processes and restarts them if they fail, while also logging restarts. You can configure a "restart" parameter to be anything from 0 (forever) to any number of times. I like to set a reasonable number so persistent errors eventually kill the process, but occasional errors do not. Log files are not overwritten, but are appended and rotated.

But whatever. My view seems to be in the minority -- guess the rest of you don't mind 24x7x365 babysitting.

GLH

-----Original Message----- From: Galen Johnson [mailto:Galen.Johnson at sas.com] Sent: Friday, August 03, 2007 10:18 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

DOn't forget...this is the model that Tivoli and HP Openview, and many

other commercial monitoring solutions provide and sell as a feature. From my experience as a sys admin, I've alwys found that automatically

restarting a service if it goes down to be "a bad thing"(TM).

In many solutions, logs get overwritten upon a restart that would be integral to the real resolution and prevention.

=G=

-----Original Message----- From: Tod Hansmann [mailto:thansmann at directpointe.com] Sent: Friday, August 03, 2007 10:40 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Highlights of the 4.3.0 version

In my experience, I have to agree. Hobbit is for monitoring so the information that x is down gets to people who can properly diagnose what is going on, not take generic actions. If generic actions were something that were required for X to function properly, it should be a feature of that software.

Hobbit CAN do some scripting based on alerts, but even that might be a

bit more than a systems administrator wants to hinder himself with.

Tod Hansmann Network Engineer

-----Original Message----- From: Buchan Milne [mailto:bgmilne at staff.telkomsa.net] Sent: Friday, August 03, 2007 12:31 AM To: hobbit at hswn.dk Cc: Hubbard, Greg L Subject: Re: [hobbit] Highlights of the 4.3.0 version

On Tuesday 24 July 2007 22:55:02 Hubbard, Greg L wrote:

...

Wonder if there is any way to tell a client what it's status is so it can be autonomous? What I mean is this: suppose there was a way

...

for the Hobbit client to tell the server that service X was now in state Y, and a client-side module could then activate response Z on its own?

I don't like band-aids like this.

"restart because it's down" prevents the real impact of problems being

seen, and provides less motivation for fixing things properly. Instead, you sit with frequent short outages (which may avoid the attention of managers, production managers) which have end-user impact.

I like even less using a monitoring system to do this ...

Regards, Buchan

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

On 8/3/07, Haertig, David F (Dave) <haertig at avaya.com> wrote:

Most everything I do in Hobbit is a custom script. Restarting crashed processes is one of the least of my worries. Although in some rare cases I do just that (short term), with appropriate logging and email to the app developement team. The corporate expense of having the app down is too great to let Utopian ideas prevail.

Agreed, though sometimes it's worth the effort for an extra few minutes of downtime to do *some* analysis.

Most of the automated Hobbit stuff I do is not restarting dead apps

(luckily, that is very infrequent around here). It's more mundane. One example is disk space. A full filesystem would shut many things down. Apps should not fill a filesystem, but sometimes they do. So my custom Hobbit scripts first scream and scream about low disk space, even analysing things down to specific subdirectories and fast growing files and doing trend analysis. But if their call is not answered, they start freeing up space from a "private reserve" I have set aside to deal with emergencies. So if we experience a sudden unexpected blowup in a filesystem at 3am, Hobbit keeps things running in production until the appropriate people can look into and diagnose the problem. This may not be Utopian behavior, but it sure is practical at 3am in the morning!

What sort of trend analysis do your scripts perform? We have a few boxes that are notorious for filling up their disk space, and I haven't yet come up with an idea of how to neatly track exactly what it is that keeps filling up the disk.

But my vote would be for Hobbit out-of-the-box to NOT attempt automated

repair actions. That should be left to the Hobbit administrator. We can write custom monitor scripts or custom alert scripts to add this functionality if it's appropriate for our environments. It's trivial to integrate your own scripting into Hobbit.

Due to the demands of some of the other admins, I have implemented a script that does some rudimentary restarting, and even looks at the status of the specific Hobbit alert in question, so that it doesn't try to restart something, if the alert has been disabled (such as for a planned downtime).

It wasn't all that hard to write, and I also would prefer Hobbit NOT have auto-restart logic out of the box.

I sure wish I worked in Utopia though. The job would be a helluva lot

less stressful! :-)

Working in the real world isn't as bad, compared to working the real world where management _thinks_ you actually work in Utopia, and yet still can't spare an extra second of downtime for real-time root cause analysis. ;-)

On Monday 06 August 2007 21:25:46 Haertig, David F (Dave) wrote:

I try to identify filesystem "space hogs" via custom scripts I wrote a long time ago when using BB. 99% of my custom stuff is done in PERL.

I use 'du -k' to get the size of all directories in the filesystem. I then cut those results down to only the first and second level directories (but you could go as deep as you want). I store the size of each subdirectory in a small "database". I did this ages ago and my code uses PERL's "Storable" module to store the accumulated date into a file (called my "database"). These days I'd just use Hobbit's easily accessed RRD files. I then use PERL's Statistics::Descriptive::least_squares_fit() to calculate the slope and linear correlation coefficient of the "best fit line".

This would be really useful to do on directories monitored with the dir option in client-local.cfg plus DIR option in hobbit-clients, e.g. to be able to specify alerts at specified "time before disk is full".

This allows me to see how fast each subdirectory is growing/shrinking, and how linear that growth/reduction is. I trigger yellow/red conditions based on rate of growth and predicted fill time at current growth rate, in addition to the standard "95% full = red" test.

The above makes it fairly easy to identify which subdirectory is your problem, which is often times good enough to identify the file/process that is killing you. When that's not, I have a seperate test that tries to identify problem files a different way. BB/Hobbit uses 'top' to identify cpu-hogging processes. Many times you see files hogging space are directly tied to processes hogging cpu (runaway process = runaway file in many cases). 'top' identifies the process(es), then "lsof -p <pid>" is used to identify the files that the suspect process has open. Finding a cpu-hogger that has a filespace-hogger open is usually the holy grail you seek.

The "CPU usage by process" graph is the utopian one ...

As a "repair" action for Hobbit, I squirreled away 2Gb of diskspace in 100Mb chunks for critical filesystems. "dd if=/dev/zero of=/filesystem/DiskSpaceReserve/reserve01 bs=1024 count=102400", then "cp reserve01 reserve02", etc. to build up the reserve.

lvextend may be another useful command here ...

Regards, Buchan

Hi list, I am trying to compile hobbit on Solaris 9, make is failing:

bash-2.05# pwd /apps/hobbit/bbgen-3.5/build bash-2.05# cd .. bash-2.05# make CC="gcc" CFLAGS="-g -O2 -Wall -Wno-unused -D_REENTRANT -DHAVE_RPCENT -DMAXMSG=8192 -DBBDPORTNUMBER=1984 -I. -Ipwd/include" SSLFLAGS="-DBBGEN_SSL" SSLINCDIR="-I/usr/local/ssl/include" SSLLIBS="-L/usr/local/ssl/lib -lcrypto -lssl" NETLIBS="-lresolv -lsocket -lnsl" make -C lib all Usage : make [ -f makefile ][ -K statefile ]... [ -d ][ -dd ][ -D ][ -DD ] [ -e ][ -i ][ -k ][ -n ][ -p ][ -P ][ -q ][ -r ][ -s ][ -S ][ -t ] [ -u ][ -w ][ -V ][ target... ][ macro=value... ][ "macro +=value"... ] make: Fatal error: Unknown option -C' *** Error code 1 make: Fatal error: Command failed for target lib-build'

it is using -C option which is not a right option, which option I can use and chaning in /bbgen-3.5/build/Makefile.rules is sufficient? Please let me know.

Thanks in advance

Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.

Robert <rgoud at yahoo.com> wrote: Hi list, I am trying to compile hobbit on Solaris 9, make is failing:

bash-2.05# pwd /apps/hobbit/bbgen-3.5/build bash-2.05# cd .. bash-2.05# make CC="gcc" CFLAGS="-g -O2 -Wall -Wno-unused -D_REENTRANT -DHAVE_RPCENT -DMAXMSG=8192 -DBBDPORTNUMBER=1984 -I. -Ipwd/include" SSLFLAGS="-DBBGEN_SSL" SSLINCDIR="-I/usr/local/ssl/include" SSLLIBS="-L/usr/local/ssl/lib -lcrypto -lssl" NETLIBS="-lresolv -lsocket -lnsl" make -C lib all Usage : make [ -f makefile ][ -K statefile ]... [ -d ][ -dd ][ -D ][ -DD ] [ -e ][ -i ][ -k ][ -n ][ -p ][ -P ][ -q ][ -r ][ -s ][ -S ][ -t ] [ -u ][ -w ][ -V ][ target... ][ macro=value... ][ "macro +=value"... ] make: Fatal error: Unknown option -C' *** Error code 1 make: Fatal error: Command failed for target lib-build'

it is using -C option which is not a right option, which option I can use and chaning in /bbgen-3.5/build/Makefile.rules is sufficient? Please let me know.

Thanks in advance

Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.

Pinpoint customers who are looking for what you sell.

Robert wrote:

Hi list, I am trying to compile hobbit on Solaris 9, make is failing:

bash-2.05# pwd /apps/hobbit/bbgen-3.5/build bash-2.05# cd .. bash-2.05# make CC="gcc" CFLAGS="-g -O2 -Wall -Wno-unused -D_REENTRANT -DHAVE_RPCENT -DMAXMSG=8192 -DBBDPORTNUMBER=1984 -I. -Ipwd/include" SSLFLAGS="-DBBGEN_SSL" SSLINCDIR="-I/usr/local/ssl/include" SSLLIBS="-L/usr/local/ssl/lib -lcrypto -lssl" NETLIBS="-lresolv -lsocket -lnsl" make -C lib all Usage : make [ -f makefile ][ -K statefile ]... [ -d ][ -dd ][ -D ][ -DD ] [ -e ][ -i ][ -k ][ -n ][ -p ][ -P ][ -q ][ -r ][ -s ][ -S ][ -t ] [ -u ][ -w ][ -V ][ target... ][ macro=value... ][ "macro +=value"... ] make: Fatal error: Unknown option -C' *** Error code 1 make: Fatal error: Command failed for target lib-build'

it is using -C option which is not a right option, which option I can use and chaning in /bbgen-3.5/build/Makefile.rules is sufficient? Please let me know.

You need to use gmake and not Solaris make.

Or you can always use the packages from http://www.blastwave.org/ .

-- -m

Try sunfreeware.com...

From: Robert [mailto:rgoud at yahoo.com] Sent: Friday, August 10, 2007 12:25 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Hobbit instlalation on Solaris 9

Mike, I spent lot of time on that site, I am trying to download CSWhobbit but when I click on it to download it is showing bunch of dependencies and I can't download any of those. I am not sure what I am doing wrong, could you please let me know how to download from there. Thanks in advance

Mike Arnold <hobbit at razorsedge.org> wrote:

Robert wrote:

Hi list, I am trying to compile hobbit on Solaris 9, make is failing:

bash-2.05# pwd /apps/hobbit/bbgen-3.5/build bash-2.05# cd .. bash-2.05# make CC="gcc" CFLAGS="-g -O2 -Wall -Wno-unused -D_REENTRANT -DHAVE_RPCENT -DMAXMSG=8192 -DBBDPORTNUMBER=1984 -I. -Ipwd/include" SSLFLAGS="-DBBGEN_SSL" SSLINCDIR="-I/usr/local/ssl/include" SSLLIBS="-L/usr/local/ssl/lib -lcrypto -lssl" NETLIBS="-lresolv

-lsocket

-lnsl" make -C lib all Usage : make [ -f makefile ][ -K statefile ]... [ -d ][ -dd ][ -D ][ -DD ] [ -e ][ -i ][ -k ][ -n ][ -p ][ -P ][ -q ][ -r ][ -s ][ -S ][ -t ] [ -u ][ -w ][ -V ][ target... ][ macro=value... ][ "macro +=value"... ] make: Fatal error: Unknown option -C' *** Error code 1 make: Fatal error: Command failed for target lib-build'

it is using -C option which is not a right option, which option I can use and chaning in /bbgen-3.5/build/Makefile.rules is sufficient? Please let me know.

You need to use gmake and not Solaris make.

Or you can always use the packages from http://www.blastwave.org/ .

-- -m

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online. <http://us.rd.yahoo.com/evt=48251/*http:/smallbusiness.yahoo.com/webhost ing/?p=PASSPORTPLUS>

Robert wrote:

Mike, I spent lot of time on that site, I am trying to download CSWhobbit but when I click on it to download it is showing bunch of dependencies and I can't download any of those. I am not sure what I am doing wrong, could you please let me know how to download from there. Thanks in advance

Blastwave's web pages are just informative. To use blastwave you must have pkg-get installed.

HOWTO Use Blastwave http://www.blastwave.org/howto_S8.html

Once you have pkg-get installed, you can install Hobbit like this: pkg-get -i hobbit hobbit_client

Hobbit then lives in /opt/csw/libexec/hobbit .

-- -mike

Robert a écrit :

Hi list, hi Robert, I am trying to compile hobbit on Solaris 9, make is failing:

bash-2.05# pwd /apps/hobbit/bbgen-3.5/build bash-2.05# cd .. bash-2.05# make CC="gcc" CFLAGS="-g -O2 -Wall -Wno-unused -D_REENTRANT -DHAVE_RPCENT -DMAXMSG=8192 -DBBDPORTNUMBER=1984 -I. -Ipwd/include" SSLFLAGS="-DBBGEN_SSL" SSLINCDIR="-I/usr/local/ssl/include" SSLLIBS="-L/usr/local/ssl/lib -lcrypto -lssl" NETLIBS="-lresolv -lsocket -lnsl" make -C lib all Usage : make [ -f makefile ][ -K statefile ]... [ -d ][ -dd ][ -D ][ -DD ] [ -e ][ -i ][ -k ][ -n ][ -p ][ -P ][ -q ][ -r ][ -s ][ -S ][ -t ] [ -u ][ -w ][ -V ][ target... ][ macro=value... ][ "macro +=value"... ] make: Fatal error: Unknown option -C' *** Error code 1 make: Fatal error: Command failed for target lib-build'

are you using gmake or the standard solaris make ?

it is using -C option which is not a right option, which option I can use and chaning in /bbgen-3.5/build/Makefile.rules is sufficient? Please let me know.

Thanks in advance

I thought it'd gotten a little off topic too :)

Original question I asked:

Hi All,

Is there a way to filter out hosts/sites based on pagename or just a regular expression? Trying to do an availability report for all sites except one and wondering if there is a way.

Thanks, Jason.

-----Original Message----- From: Trent Melcher [mailto:trent.melcher at sitel.com] Sent: 09 August 2007 16:56 To: hobbit at hswn.dk Subject: Re: [hobbit] Filter reports?

Talk about hijacking a thread, and its not even close to anything about Filter reports. Folks, please create a new email with a new subject for your "New" posts. This will make following threads stay on subject.

Thanks Trent

On Thu, 2007-08-09 at 05:59 -0700, Robert wrote:

Hi list, I am trying to compile hobbit on Solaris 9, make is failing:

bash-2.05# pwd /apps/hobbit/bbgen-3.5/build bash-2.05# cd .. bash-2.05# make CC="gcc" CFLAGS="-g -O2 -Wall -Wno-unused -D_REENTRANT -DHAVE_RPCENT -DMAXMSG=8192 -DBBDPORTNUMBER=1984 -I. -Ipwd/include" SSLFLAGS="-DBBGEN_SSL" SSLINCDIR="-I/usr/local/ssl/include" SSLLIBS="-L/usr/local/ssl/lib -lcrypto -lssl" NETLIBS="-lresolv -lsocket -lnsl" make -C lib all Usage : make [ -f makefile ][ -K statefile ]... [ -d ][ -dd ][ -D ][ -DD ]

[ -e ][ -i ][ -k ][ -n ][ -p ][ -P ][ -q ][ -r ][ -s ][ -S ][ -t ] [ -u ][ -w ][ -V ][ target... ][ macro=value... ][ "macro +=value"... ] make: Fatal error: Unknown option -C' *** Error code 1 make: Fatal error: Command failed for target lib-build'

it is using -C option which is not a right option, which option I can use and chaning in /bbgen-3.5/build/Makefile.rules is sufficient? Please let me know.

Thanks in advance

---

Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

On Friday 03 August 2007 19:15:27 Scott Walters wrote:

I am definitely in the "monitor only" camp. As appealing as "self-healing" may seem, I've seen attempts go horrible wrong too many times. For example, shutting down Oracle for upgrades and then being restarted in the middle of the upgrade. Not good.

How about the easy example of a web server not responding. Do you restart it ? In the case I am thinking of, no. Since, the reason it is not responding is that the database server it (and another 4 webservers) is waiting for is having problems. Restarting the web server would drop the >1000 existing (working) sessions, causing a full-blown outage, and migrate the problem to the other 4 web servers that sit behind the same load balancer.

I also agree that "self-healing" lends itself to band-aids that avoid root-cause determination.

Or *prevent* the root-cause determination. For example, I had a problem on an LDAP server that appeared once in 2 or 3 weeks. I start it under a debugger, and when next experienced the problem, some online debugging (after taking it out of the pool) with a developer found and fixed the bug within one hour (and allowed me to understand the cause so I could work around it). A restart here would have meant waiting some more and another few outages.

I don't think this requires "baby-sitting," but a commitment to fixing things once. I have also had the displeasure of making permanent band-aids, but I cannot condone it.

We do have some applications that require supervision ... but for them we use daemon-tools or supervise-scripts (a re-implementation of daemon-tools), as these are *much* better at supervision than a monitoring system. If you really need a baby-sitter, the monitoring system isn't the best one ...

All of those "operational" aspects aside, I've convinced myself from a security point of view, corrective action from monitoring is bad-- a clear violation of the separation of duties. You don't want your auditors "cleaning up" the numbers as they go over your books.

You know what's better than your webserver being automatically restarted when it crashes? Your webserver not crashing.

I completely support the absence of corrective actions from monitor triggers. The question I have yet to answer satisfactorily is,"Should the monitoring system perform additional data collection after specific errors?" For example, running a particular "find" command when disk usage increases to try and identify which files are causing the partition to fill.

Or attach a debugger to the hung process and get a backtrace ?

Regards, Buchan

On Mon, Jul 23, 2007 at 06:14:14AM -0500, Daniel J McDonald wrote:

On Sun, 2007-07-22 at 00:08 +0200, Henrik Stoerner wrote:

...

This doesn't mean that I won't consider adding new stuff before the 4.3.0 release, but right now the plan is to get 4.3.0 shipped with the current set of features. But if I've missed someone's favourite patch or feature request, do let me know.

Get hobbitfetch to not crash, hang, or spin the cpu at 100%.

I know, this one is definitely a "must-fix-before-4.3.0" bug.

Henrik

All the new features sound great. It also sounds like nearly everyone has additional features they would like to see...do you use any sort of tool for tracking feature requests?

P.S. I might as well throw in my own feature request ;-)

• Content check should correctly follow 302 (redirects). I currently have to use a custom-made script that uses curl in order to do content checks. In fact, I will include it in case anyone wants to use it:

#!/bin/bash

contchk.sh written by Charles Jones (blazer0x at gmail.com) 6/6/2007

This script is designed to perform a content check on a URL and report the

status to a Hobbit server.

This script was created because Hobbits built-in content check

functionality

does not follow 302 redirects.

The script parses out a "contchk" tag in the bb-hosts file. The proper

syntax is: contchk;URL;REFERRER;CHECKSTRING

Note that CHECKSTRING cannot contain spaces so you must use regular

expression metacharacters, so use something like string.with.spaces

BBHTAG=contchk # Name of the tag in bb-hosts COLUMN=cont # Column display name in Hobbit CURL=/usr/bin/curl # Location of curl binary CURLOPTS="--connect-timeout 30 -m 30 -s -L -b cookiejar" # Curl options

Note: using grep because bbhostgrep fails on long lines

grep $BBHTAG $BBHOME/etc/bb-hosts | while read L do set $L # To get one line of output from bbhostgrep HOSTIP="$1" MACHINEDOTS="$2" MACHINE=echo $2 | $SED -e's/\./,/g' CHECKURL=echo $4 | awk -F";" '{print $2}' # Parse out the check URL REFERRER=echo $4 | awk -F";" '{print $3}' # Parse out the referrer string if [ "" != "$REFERRER" ]; then REFERRER="-e $REFERRER" fi CHECKSTRING=echo $4 | awk -F";" '{print $4}' # Parse out the check string $CURL $CURLOPTS $REFERRER $CHECKURL |grep -q "$CHECKSTRING" status=$? # Save greps return status if [ 0 -eq $status ]; then # grep returns 0 if it found something COLOR=green MSG="String <b>\"$CHECKSTRING\"</b> was found in <a href=$CHECKURL>$CHECKURL</a>" $BB $BBDISP "status $MACHINE.$COLUMN $COLOR date Content Check OK ${MSG} " else # grep didn't find anything COLOR=red MSG="String <b>\"$CHECKSTRING\"</b> was NOT FOUND in <a href=$CHECKURL>$CHECKURL</a>" $BB $BBDISP "status $MACHINE.$COLUMN $COLOR date Content Check FAILED ${MSG} " fi done exit 0

On Mon, Jul 23, 2007 at 12:16:53PM -0400, s_aiello at comcast.net wrote:

Just checking if the iconnames.patch will be included in 4.3.0. That is that patch that allowed &color-acked keyword to display the acknowledged icons in tests ?

Those patches that have been posted over the past year have all been merged into the code, so yes - it's included.

Regards, Henrik

On Mon, Jul 23, 2007 at 01:51:55PM -0700, Mike Arnold wrote:

Henrik Stoerner wrote:

...

This doesn't mean that I won't consider adding new stuff before the 4.3.0 release, but right now the plan is to get 4.3.0 shipped with the current set of features. But if I've missed someone's favourite patch or feature request, do let me know.

I'd like to see POWER5 CPU stats: http://www.docum.org/twiki/bin/view/Hobbit/AixPower5

No problem, except I don't understand why the Wiki claims that it is necessary to remove the '.' from the numbers. It seems this is to convert the data to percentages, but that can be done in the graph definition:

[vmstat-pc]
TITLE Used Physical CPU
YAXIS pc (100 = 1 CPU)
DEF:pc=vmstat.rrd:cpu_pc:AVERAGE
CDEF:pcpercent=pc,100,*
LINE2:pcpercent#00CC00

And the "-b 1024" in the Wiki graph definition looks bogus.

I'm cc'ing Stef Coene who wrote the Wiki entry to see if he can shed some light on this.

Regards, Henrik

On Tue, Jul 24, 2007 at 08:38:37AM +0200, Stef Coene wrote:

On Monday 23 July 2007, you wrote:

...

On Mon, Jul 23, 2007 at 01:51:55PM -0700, Mike Arnold wrote:

...

I'd like to see POWER5 CPU stats: http://www.docum.org/twiki/bin/view/Hobbit/AixPower5

No problem, except I don't understand why the Wiki claims that it is necessary to remove the '.' from the numbers. It seems this is to convert the data to percentages, but that can be done in the graph definition: The cpu_pc and cpu_ec has always a "." in it: kthr memory page faults cpu

---

r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec 2 2 1341331 15949 0 8 7 85 49 0 583 34667 1878 38 5 55 2 0.46 46.0

pc has always 2 numbers after the "." and ec 1 (I hope this stays the same for next AIX releases). Rrd wants integers (I think), so you have to strip the "." from the numbers.

No, RRD uses floating-point numbers everywhere. So I'll keep the numbers unmodified - then we won't have any problems if IBM does change the number of decimals they report.

Regards, Henrik

On 7/25/07, John Glowacki <johng at idttechnology.com> wrote:

Peter Welter wrote:

...

One of the things I'd like to see in 4.3.0 is what is already partly available on Sun-boxes in 4.2.0, the [iostatdisk] part. The stats are gathered but not plotted. I hope this monitor will make it, on all platforms, because you can pinpoint any disk performance problems much easier.

Thanks, Peter

That would be good to have. I have been asked if hobbit does this from other groups in the company.

Tracking disk IO gets complicated pretty quickly for a few reasons:

• OS's don't have common commands for measuring disk performance
• Do you watch IO by filesystem or spindle? If you have RAID, grabbing the data can become even more difficult.
• People can disagree on what good disk IO means, and even fewer understand disk IO workloads.
• I have no idea if Windows and the WMI has this kind of info.

For *ix, the "blocked processes" of vmstat is an excellent way to see if the server overall is IO bound. I would definitely like to see that a "stock" displayed metric in 4.3.0. Most *ix vmstat provides that number. Similar to the iostat for Solaris, the info is collected, just not displayed.

Scott Walters -PacketPusher