Rich,

Here's what we're looking at :

Hosts: 10.1.1.101 innatrack # DOWNTIME=*:0300:0430 telnet ftp 172.16.1.24 CMD_Bagger_PLC #DOWNTIME=*:0230:0630 http://172.16.1.24

History.log. it gives the "cannot open" errors when it's down and the "will not update" after restarting the service:

Checking I just noticed this error at the same time as my last failures, however it is not with the previous set. 2007-04-06 05:26:09 Tried to down BOARDBUSY: Invalid argument . . . 2007-04-06 05:26:56 Cannot create histlog file '/home/hobbit/data/histlogs/MCC_Flex_I/O_Rack_2/http/Fri_Apr_6_05:26:56_ 2007' : No such fi le or directory 2007-04-06 05:26:56 Cannot open host logfile '/home/hobbit/data/hist/MCC_Flex_I/O_Rack_2' : No such file or directory 2007-04-06 05:26:56 Cannot open status historyfile '/home/hobbit/data/hist/Preblend_Flex_I/O.http' : No such file or directory 2007-04-06 05:26:56 Cannot create histlog file '/home/hobbit/data/histlogs/Preblend_Flex_I/O/http/Fri_Apr_6_05:26:56_20 07' : No such file or directory 2007-04-06 05:26:56 Cannot open host logfile '/home/hobbit/data/hist/Preblend_Flex_I/O' : No such file or directory 2007-04-06 05:26:56 Cannot open status historyfile '/home/hobbit/data/hist/Preblend_Point_I/O.http' : No such file or directory 2007-04-06 05:26:56 Cannot create histlog file '/home/hobbit/data/histlogs/Preblend_Point_I/O/http/Fri_Apr_6_05:26:56_2 007' : No such fil e or directory 2007-04-06 05:26:56 Cannot open host logfile '/home/hobbit/data/hist/Preblend_Point_I/O' : No such file or directory 2007-04-06 05:26:56 Cannot open status historyfile '/home/hobbit/data/hist/Relco_I/O_Rack.http' : No such file or directory 2007-04-06 05:26:56 Cannot create histlog file '/home/hobbit/data/histlogs/Relco_I/O_Rack/http/Fri_Apr_6_05:26:56_2007' : No such file or directory 2007-04-06 05:26:56 Cannot open host logfile '/home/hobbit/data/hist/Relco_I/O_Rack' : No such file or directory 2007-04-06 05:31:42 Will not update /home/hobbit/data/hist/voicemail.memory - color unchanged (green) 2007-04-06 05:31:46 Will not update /home/hobbit/data/hist/vscognos.uptime - color unchanged (green) 2007-04-06 05:32:04 Will not update /home/hobbit/data/hist/abra.uptime - color unchanged (green) 2007-04-06 05:32:45 Will not update /home/hobbit/data/hist/voicemail.procs - color unchanged (green) 2007-04-06 05:32:45 Will not update /home/hobbit/data/hist/voicemail.disk - color unchanged (green) 2007-04-06 05:33:06 Will not update /home/hobbit/data/hist/voicemail.msgs - color unchanged (green) 2007-04-06 05:33:10 Will not update /home/hobbit/data/hist/voicemail.netstat - color unchanged (green) 2007-04-06 05:33:12 Will not update /home/hobbit/data/hist/svetovit.uptime - color unchanged (green) 2007-04-06 05:33:40 Will not update /home/hobbit/data/hist/jdedeploy.uptime - color unchanged (green) 2007-04-06 05:33:57 Will not update /home/hobbit/data/hist/ems_burke.uptime - color unchanged (green) 2007-04-06 05:33:58 Will not update /home/hobbit/data/hist/kronos.uptime

• color unchanged (green) 2007-04-06 05:34:06 Will not update /home/hobbit/data/hist/burke-utility.uptime - color unchanged (green) 2007-04-06 05:35:36 Will not update /home/hobbit/data/hist/vs4100.uptime
• color unchanged (green) 2007-04-06 05:39:20 Will not update /home/hobbit/data/hist/voicemail.uptime - color unchanged (green)

Hobbitd.log only updates after restarting it and says simply setup complete

Page.log shows this : 2007-03-30 00:45:20 Tried to down BOARDBUSY: Invalid argument 2007-04-06 04:36:10 Tried to down BOARDBUSY: Invalid argument 2007-04-06 04:36:10 Worker process died with exit code 0, terminating 2007-04-06 05:26:09 Tried to down BOARDBUSY: Invalid argument

Rrd-data.log :

2007-04-06 03:12:37 Internal error: Duplicate match ignored 2007-04-06 04:36:10 Tried to down BOARDBUSY: Invalid argument 2007-04-06 05:26:09 Tried to down BOARDBUSY: Invalid argument

I did an ls -l before doing the restart and it was the same set of files with similar mod times as what's shown before the restart.

Bill Hart Burke Corporation -----Original Message----- From: Rich Smrcina [mailto:rsmrcina at wi.rr.com] Sent: Friday, April 06, 2007 9:12 AM To: hobbit at hswn.dk Subject: Re: [hobbit] More Downtime issues

Bill,

Please send your bb-hosts entries for those that have the downtime specified. Also, at the time this incident occurs (or when you discover

it) and BEFORE you restart Hobbit, send the last several lines of the log file where you see the messages and a 'ls -l' of /var/log/hobbit (or

whereever your log files live).

Send the recent messages from any log file that was updated on the current day (or the day the incident occurred).

Bill Hart wrote:

This is getting really annoying.

I added another DOWNTIME setting for a different machine.

This morning I came in early to find the system had died again.

The only entries I'm finding in the logs are the history.log that shows unable to access the historyfile for any of the systems. Restarting hobbit does the trick, and it always happens right around when an enable or disable is scheduled via downtime.

The error makes it seem like the files are just flat locked, I don't know if there's something in hobbit locking them all open or what.

Is there anything I can do to help troubleshoot this further ? I mean

I've checked all the logs, half of them haven't even had any logs added in weeks (no errors), others have some benign warnings (such as not able to find the disabled directory from bb-xsnmp), but nothing showing issues except the history side.

Thanks,

Bill Hart

Computer Support Supervisor

Burke Corporation

-- Rich Smrcina VM Assist, Inc. Phone: 414-491-6001 Ans Service: 360-715-2467 rich.smrcina at vmassist.com

Catch the WAVV! http://www.wavv.org WAVV 2007 - Green Bay, WI - May 18-22, 2007

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk