OK... I enabled the modules one at a time until I reached 'larrdstatus'. When I enabled that module, hobbitd began crashing again. Here is the contents of the hobbitlaunch.cfg.

The hobbittasks.cfg file is loaded by "hobbitlaunch".

It controls which of the Hobbit modules to run, how often, and

with which parameters, options and environment variables. #

This is the main Hobbit daemon. You cannot live without this one. [hobbitd]

HEARTBEAT
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
CMD hobbitd --debug --restart=$BBTMP/hobbitd.chk

--checkpoint-file=$BBTMP/hobbitd.chk --checkpoint-interval=600 --log=$BBSERVERLOGS/hobbitd.log --admin-senders=127.0.0.1,$BBSERVERIP

"bbstatus" saves status-logs in text- and html-format, like the old

Big Brother

daemon does. Unless you are using add-ons that directly access the

log-files, you

will not need to run this module, and it is recommended that you

keep it disabled # since storing the raw logs on disk can cause a significant load on your server.

[bbstatus] DISABLED ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=status --log=$BBSERVERLOGS/status.log hobbitd_filestore --status --html

"bbhistory" keeps track of the status changes that happen, in a

manner that is # compatible with the Big Brother history logs. You probably do want to run this.

[bbhistory] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=stachg --log=$BBSERVERLOGS/history.log hobbitd_history

"bbdata" saves information sent using the BB "data" protocol, like

the old Big Brother

daemon does. Unless you are using add-ons that directly access the

data-files, you

will not need to run this module, and it is recommended that you

keep it disabled # since storing the raw data on disk can cause a significant load on your BB server. # LARRD uses data-files, but since LARRD is handled by the hobbitd_larrd module, you do # not need to run the "bbdata" module to get LARRD graphs.

[bbdata] DISABLED ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=data --log=$BBSERVERLOGS/data.log hobbitd_filestore --data

"bbnotes" saves web note-files, that are sent using the BB "notes"

protocol. This is # disabled by default; if you use the BB "notes" protocol, then you should enable this.

[bbnotes] DISABLED ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=notes --log=$BBSERVERLOGS/notes.log hobbitd_filestore --notes

"bbenadis" updates the files used to indicate that a host or test

has been enabled or disabled. # These files are used by bbgen and the "maint.pl" script to determine what is currently enabled # and disabled, so you probably want to run this module.

[bbenadis] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=enadis --log=$BBSERVERLOGS/enadis.log hobbitd_filestore --enadis

"bbpage" sends out alerts. Note that this module is NOT compatible

with the old Big Brother # system - it uses a different configuration file to determine how alerts get sent. If you want # alerts to go out via pager, e-mail or some other means, then you must run this module.

[bbpage] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=page --log=$BBSERVERLOGS/page.log hobbitd_alert

"larrdstatus" updates RRD files with information that arrives as

"status" messages. # If you want RRD graphs of your monitoring data, then you want to run this.

[larrdstatus] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=status --log=$BBSERVERLOGS/larrd-status.log hobbitd_larrd --rrddir=$BBVAR/rrd

"larrddata" updates RRD files with information that arrives as

"data" messages. # If you want RRD graphs of your monitoring BB data, then you want to run this.

[larrddata] DISABLED ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=data
--log=$BBSERVERLOGS/larrd-data.log hobbitd_larrd --rrddir=$BBVAR/rrd

"bbdisplay" runs the bbgen tool to generate the Hobbit webpages from

the status information that # has been received. Big Brother updated the webpages once every 5 minutes. The default here is to # run it every minute for faster updates, but you can change it if you have a highly loaded server # and dont need updates that often.

[bbdisplay] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd GROUP generators CMD bb-display.sh LOGFILE $BBSERVERLOGS/bb-display.log INTERVAL 1m

"larrdcolumn" is responsible for updating the contents of the LARRD

overview page, found on the # "trends" column for each host. Since the set of graphs does not change very often, we run this # less frequently than the normal webpage updates. We also make sure (with the "GROUP" setting) # that they don't run simultaneously with the infocolumnn and bbdisplay tasks. [larrdcolumn] DISABLED ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd GROUP generators CMD bb-larrdcolumn --hobbitd --rrddir=$BBVAR/rrd --column=trends LOGFILE $BBSERVERLOGS/bb-display.log INTERVAL 15m

"infocolumn" is responsible for updating the contents of the INFO

pages, found on the # "info" column for each host. Since the content does not change unless there is a

configuration change, we update these less frequently than the

normal webpage updates. # We also make sure (with the "GROUP" setting) that they don't run simultaneously with

the larrdcolumnn and bbdisplay tasks.

[infocolumn] DISABLED ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd GROUP generators CMD bb-infocolumn --hobbitd --column=info LOGFILE $BBSERVERLOGS/bb-display.log INTERVAL 15m

"bbnet" runs the bbtest-net tool to perform the network based tests

• i.e. http, smtp, ssh, dns and # all of the various network protocols we need to test.

[bbnet] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD bbtest-net --report --ping --checkresponse LOGFILE $BBSERVERLOGS/bb-network.log INTERVAL 5m

"bbretest" picks up the tests that the normal network test consider

"failed", and re-does those # tests more often. This enables Big Brother to pick up a recovered network service faster than # if it were tested only by the "bbnet" task (which only runs every 5 minutes). So if you have # servers with very high availability guarantees, running this task will make your availability # reports look much better.

[bbretest] ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg NEEDS hobbitd CMD $BBHOME/ext/bbretest-net.sh LOGFILE $BBSERVERLOGS/bb-retest.log INTERVAL 1m

On Tue, 1 Mar 2005 10:10:12 -0800, Brian Lynch <brianlynch at gmail.com> wrote:

Henrik,

Tried the patch and hobbitd still crashed.

Tried the disable module test. Disabled all the modules you mentioned above and hobbitd has not crashed since 8:30 PST this morning. I'll try adding back in modules until it crashes.

• Brian

On Tue, 1 Mar 2005 09:33:51 +0100, Henrik Stoerner <henrik at hswn.dk> wrote:

...

I've found something that might explain it, but I am certainly not sure if it's the cause of your problems. This bug would only trigger if some client of yours was sending in very large "data" messages, like 100 KB or more.

Could you try the attached patch and let me know if it solves your problem?

If not, I would like you to try and disable all of the hobbitd worker modules in hobbitlaunch.cfg - i.e. bbstatus, bbhistory, bbdata, bbnotes, bbenadies, bbpage, larrdstatus and larrddata. Restart hobbit and see if it still crashes. If it doesn't then we've narrowed down the problem - if it does, then the problem is somewhere else than where I've been looking so far.

Thanks, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk