Thorsten, Because the MACHINE selection is hard coded, that doesn't scale very well (think 100's of hosts/tests). I tend to "read in" data definition files that define host:test then the code can be quite simple... this is what I'm currently playing, I don't claim it to be neat or anything:

================================================================================

#!/bin/sh

SNMP_LIST=$BBHOME/etc/snmp-list

BBHTAG=snmp-loop # What we put in bb-hosts to trigger this test COLUMN=snmp-probe # Name of the column, often same as tag in bb-hosts

$BBHOME/bin/bbhostgrep $BBHTAG | while read L do set $L # To get one line of output from bbhostgrep

  HOSTIP="$1"
  MACHINEDOTS="$2"
  MACHINE=`echo $2 | $SED -e's/\./,/g'`

  COLOR=green
  MSG="$COLUMN status for host $MACHINEDOTS

" HOST=grep $MACHINEDOTS $SNMP_LIST|cut -d":" -f1 COMMUNITY=grep $MACHINEDOTS $SNMP_LIST|cut -d":" -f2 MIB_OBJECT=grep $MACHINEDOTS $SNMP_LIST|cut -d":" -f3

  STATUS=`snmpwalk -v2c -c $COMMUNITY $HOST $MIB_OBJECT`
  RESULT=`echo $?`
  if [ $RESULT != 0 ]
  then
    COLOR=red
  fi
  $BB $BBDISP "status $MACHINE.$COLUMN $COLOR `date`

  ${MSG}
  ${STATUS}
  "
done

exit 0

From: thorsten.erdmann at daimler.com [mailto:thorsten.erdmann at daimler.com] Sent: 07 December 2009 16:18 To: hobbit at hswn.dk Subject: [hobbit] How to create multiple custum test under one column

Hi,

I want to do some application tests on several servers. All these test should be under one column named "app". Every test is unique for one server. So test "app" on "server1" is different from "app" on server2. All these test run remotely on the hobbit server, not on the destination servers.

For now I wrote a test-script for every test and define a unique testname for the bbhost file for every test-script. In the script I reported to the "app" column. Works fine, but I have to include every test-script in the hobbitlaunch file and so all these tests run simultaneously, resulting in a big number of processes on the hobbit server.

So I think of writing _one_ testscript for the "app" test and let this script decide which subscript should be run for each host. So all the different tests would be run under one master test script and so I hopefully get only one process.

BBHTAG=app # What we put in bb-hosts to trigger this test COLUMN=app # Name of the column, often same as tag in bb-hosts

TEMPFILE_OUTPUT=$BBTMP/$BBHTAG.output.tmp TEMPFILE=$BBTMP/$BBHTAG.tmp $BBHOME/bin/bbhostgrep $BBHTAG | while read L do echo "found hosts: $L" rm $TEMPFILEOUTPUT >/dev/null 2>&1 set $L # To get one line of output from bbhostgrep HOSTIP="$1" MACHINEDOTS="$2" MACHINE=echo $2 | $SED -e's/\./,/g'

  COLOR=green
  echo "Application test on $MACHINEDOTS" > $TEMPFILE

  if "$MACHINE" == "server1"
     . apptest_server1.sh
  fi
  if "$MACHINE" == "server2"
     . apptest_server2.sh
  fi
  MSG=`cat $TEMPFILE_OUTPUT | fold -s -w 100`

  if [ $DEBUG = TRUE ]
  then
    echo "status $MACHINE.$COLUMN $COLOR `date` $MSG"
  else
    $BB $BBDISP "status $MACHINE.$COLUMN $COLOR `date` $MSG"
  fi
  rm $TEMPFILE_OUTPUT >/dev/null 2>&1
done

exit 0

What do you think of this method? Maybe you have a better idea?

Thank you Thorsten Erdmann

If you are not the intended addressee, please inform us immediately that you have received this e-mail in error, and delete it. We thank you for your cooperation.

On Mon, Dec 7, 2009 at 11:17 AM, <thorsten.erdmann at daimler.com> wrote:

Hi,

I want to do some application tests on several servers. All these test should be under one column named "app". Every test is unique for one server. So test "app" on "server1" is different from "app" on server2. All these test run remotely on the hobbit server, not on the destination servers.

For now I wrote a test-script for every test and define a unique testname for the bbhost file for every test-script. In the script I reported to the "app" column. Works fine, but I have to include every test-script in the hobbitlaunch file and so all these tests run simultaneously, resulting in a big number of processes on the hobbit server.

So I think of writing _one_ testscript for the "app" test and let this script decide which subscript should be run for each host. So all the different tests would be run under one master test script and so I hopefully get only one process.

BBHTAG=app # What we put in bb-hosts to trigger this test COLUMN=app # Name of the column, often same as tag in bb-hosts

TEMPFILE_OUTPUT=$BBTMP/$BBHTAG.output.tmp TEMPFILE=$BBTMP/$BBHTAG.tmp $BBHOME/bin/bbhostgrep $BBHTAG | while read L do echo "found hosts: $L" rm $TEMPFILEOUTPUT >/dev/null 2>&1 set $L # To get one line of output from bbhostgrep HOSTIP="$1" MACHINEDOTS="$2" MACHINE=echo $2 | $SED -e's/\./,/g'

  COLOR=green
  echo "Application test on $MACHINEDOTS" > $TEMPFILE

  if "$MACHINE" == "server1"
     . apptest_server1.sh
  fi
  if "$MACHINE" == "server2"
     . apptest_server2.sh
  fi
  MSG=`cat $TEMPFILE_OUTPUT | fold -s -w 100`

  if [ $DEBUG = TRUE ]
  then
    echo "status $MACHINE.$COLUMN $COLOR `date` $MSG"
  else
    $BB $BBDISP "status $MACHINE.$COLUMN $COLOR `date` $MSG"
  fi

...

Thank you Thorsten Erdmann

If you are not the intended addressee, please inform us immediately that you have received this e-mail in error, and delete it. We thank you for your cooperation.

  rm $TEMPFILE_OUTPUT >/dev/null 2>&1
done

exit 0

What do you think of this method? Maybe you have a better idea?

I've used cron quite effectively. Most of the scripts were repeating every ten minutes, so I could schedule where in the ten minute block the tests would start. I also had some tests running every 5 minutes, 3 minutes, 1 minute and 30 seconds, so occasionally there would be a "planetary alignment" when a large bunch of tests ran together, causing a cpu spike on the server.

In my case, I was running a bunch of checks on web servers, so each "parent" script started by cron would run through a loop running a "child" script against each of a list of web servers. If the script finishes quickly, a bunch can be backgrounded to run near simultaneously, something like this:

 for x in  1 2 3 4 5 6 7 8
 do
   $SCRIPTDIR/script server$x   &
 done
 wait

Ralph Mitchell