Henrik,

Last Sunday, July 23rd, at approximately 6:00 AM, almost all of my hosts' histfiles were cleared, and lost the 'color' status:

Sun Jul 23 06:06:25 2006 1153649185 888

This has caused the History page to fail.

I am running 4.2-beta-20060605.

I did try the latest version, but that did not make any difference.

All other functionality is present, the history files are just corrupt.

Nothing appears in any log file during this timeframe.

My Hobbit server is Linux, and until now I have been able to correct any issues that arose.

Thanks,

Michael Frey

This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

Output Requested:

[root at nbtbigbro1 lib]# ./availability ~hobbit/data/hist/nbtsanecc2p,nro,glic,com.conn 0 date +%s 2006-07-31 10:50:24 Got entry starting 2869 lasting 1154354555 2006-07-31 10:50:24 Reporting starts with this entry: Sun Jul 23 05:38:50 2006 1153647530 2869

2006-07-31 10:50:24 Duration for color 0: 0 2006-07-31 10:50:24 Duration for color 1: 0 2006-07-31 10:50:24 Duration for color 2: 0 2006-07-31 10:50:24 Duration for color 3: 0 2006-07-31 10:50:24 Duration for color 4: 0 2006-07-31 10:50:24 Duration for color 5: 0 2006-07-31 10:50:24 Color 0: Count=0, pct=0.00 2006-07-31 10:50:24 Color 1: Count=0, pct=0.00 2006-07-31 10:50:24 Color 2: Count=0, pct=0.00 2006-07-31 10:50:24 Color 3: Count=0, pct=0.00 2006-07-31 10:50:24 Color 4: Count=0, pct=0.00 2006-07-31 10:50:24 Color 5: Count=0, pct=0.00 2006-07-31 10:50:24 Availability: 100.00, color =0 2006-07-31 10:50:24 History file status: OK You have new mail in /var/spool/mail/root [root at nbtbigbro1 lib]#

Contents of hist file: (color status is missing)

Sun Jul 23 05:38:50 2006 1153647530 2869 Sun Jul 23 06:26:39 2006 green 1153650399 128621 Mon Jul 24 18:10:20 2006 red 1153779020 1138 Mon Jul 24 18:29:18 2006 green 1153780158 60483 Tue Jul 25 11:17:21 2006 red 1153840641 3398 Tue Jul 25 12:13:59 2006 green 1153844039 86990 Wed Jul 26 12:23:49 2006 clear 1153931029 587 Wed Jul 26 12:33:36 2006 green 1153931616 178738 Fri Jul 28 14:12:34 2006 red 1154110354 301 Fri Jul 28 14:17:35 2006 green 1154110655 9162 Fri Jul 28 16:50:17 2006 red 1154119817 254 Fri Jul 28 16:54:31 2006 green 1154120071

File requested:

I have not modified anything, so I am not sure how an unknown color text would be passed.

It also all happened at the same time, and not for all hosts or tests, although conn seems to be the hardest hit. Every host I have looked at, had at least one Hostory graph failure; except for any new host eneterd after July 23rd.

Michael Frey Intel Senior Systems Engineer The Guardian Life Insurance Company of America 3900 Burgess Place, 2-West Bethlehem, PA 18017 E-Mail: Michael_frey at glic.com Phone: 610-807-7889 Fax: 610-807-6003

henrik at hswn.dk (Henrik Stoerner) 07/31/2006 10:40 AM Please respond to hobbit at hswn.dk

To hobbit at hswn.dk cc

Subject Re: [hobbit] hist files

Hi Michael,

On Mon, Jul 31, 2006 at 09:24:49AM -0400, Michael Frey wrote:

Henrik,

Last Sunday, July 23rd, at approximately 6:00 AM, almost all of my hosts' histfiles were cleared, and lost the 'color' status:

Sun Jul 23 06:06:25 2006 1153649185 888

This has caused the History page to fail.

Could you send one of the corrupted files from the data/hist/ directory to henrik at hswn.dk ? I'd like to see just what this corruption looks like.

Also, there's a debugging tool included in Hobbit - the lib/availability utility which is built together with Hobbit, but not installed. Could you try running this with

cd hobbit-4.2-beta*/lib ./availability ~hobbit/data/hist/myhost.conn 0 date +%s

The snippet you've copied above looks like the color text is missing altogether. That could happen if there's some way that an unknown color text is passed to the history module ... I'll make sure it doesn't trip up the history log reporting.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.

The following error occured while attempting to apply the patch:

[hobbit at nbtbigbro1 hobbit-4.2-beta-20060605]$ patch -p0 <history-missing.patch (Stripping trailing CRs from patch.) patching file lib/availability.c Hunk #4 FAILED at 304. Hunk #5 FAILED at 321. 2 out of 8 hunks FAILED -- saving rejects to file lib/availability.c.rej (Stripping trailing CRs from patch.) patching file hobbitd/hobbitd_history.c

cannot find file 'lib/availability.c.rej'

I can find no corelation of events to anything near the timeframe this happened. Our environment was patched at 3 AM, and this all occured at 5:30AM.

The Hobbit server is Linux, so it did not receive any patches.

Others said the Hobbit page was 'going crazy', and a supervisor reset the Hobbit server. (the Hobbit server is a VMware virtual server, and resetting was the easy solution)

I have no details on what 'going crazy' means, no one is helpful.

Michael Frey

henrik at hswn.dk (Henrik Stoerner) 07/31/2006 11:52 AM Please respond to hobbit at hswn.dk

To hobbit at hswn.dk cc

Subject Re: [hobbit] hist files

On Mon, Jul 31, 2006 at 10:55:50AM -0400, Michael Frey wrote:

[snip]

ok, the attached patch should let you read the history file again. It cannot bring back the events that were lost, of course - but at least you should be able to view the history.

It also all happened at the same time, and not for all hosts or tests, although conn seems to be the hardest hit. Every host I have looked at,

had at least one Hostory graph failure; except for any new host eneterd after July 23rd.

I cannot see how that could happen, but just in case the patch modifies the history module to not log anything that doesn't have a valid color code.

The time when this happened, would it happen to concide with any cron-jobs, disk intensive activity, or anything else that might cause file operations to fail or go slow ?

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

This message, and any attachments to it, may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are notified that any use, dissemination, distribution, copying, or communication of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately by return e-mail and delete the message and any attachments. Thank you.