It's definitely some sort of "data in memory" corruption that occurs that is repeatable - I've noticed that when I restart when the problem first occurs, loading the chk file that is saved, it gets this message:
2011-04-05 08:55:32 Too few fields in record - found 6, expected 17 2011-04-05 08:55:32 Too few fields in record - found 6, expected 17 2011-04-05 08:55:32 Too few fields in record - found 6, expected 17 2011-04-05 08:55:32 Too few fields in record - found 6, expected 17 2011-04-05 08:55:32 Too few fields in record - found 6, expected 17 2011-04-05 08:55:32 Too few fields in record - found 6, expected 17 2011-04-05 08:55:32 Too few fields in record - found 6, expected 17 2011-04-05 08:55:32 Too few fields in record - found 6, expected 17
This matches up with the number of Cookies it couldn't find - I am guessing it's missing the cookies in those records
And more and more of those messages depending on how long I wait to restart (I.e. As the acknowledge problem gets wose and worse)
If I restart when I am not showing signs of it not finding cookies, I do not get that message in the xymonlaunch.log - it just works fine and exactly as I expect
Is there some sort of memory limit or that I am hitting? My xymond process takes up 524 MB of memory right now.
Just looking for any steps to take next
On 4/5/11 9:00 AM, "Clark, Sean" <sean.clark at twcable.com> wrote:
I have xymon 4.3.2 installed now
Every 4 days, almost exactly, I start losing the ability to acknowledge some alerts. As time progresses, it gets worse and worse at first it's random, some can be acknowledged, some can't Then, more and more can not be acknowledged
New alerts, Existing alerts that were already acknowledged, it doesn't matter
This is a fairly impacting issue, and others on the list have said they have this same problem
All I have is that find_cookie in lib/rbt.c is not finding the cookie, despite it being visible in the hobbitdboard
2011-04-05 05:23:09 Cookie 115771 not found, dropping ack 2011-04-05 05:23:09 Cookie 54483 not found, dropping ack 2011-04-05 05:23:09 Cookie 47469 not found, dropping ack 2011-04-05 06:38:55 Cookie 86204 not found, dropping ack 2011-04-05 06:41:37 Cookie 86204 not found, dropping ack
This is what my logs start filling up with.
Can anyone on this list point me to at least some starting point to try and solve this? It's seriously impacting my xymon implementation
--
This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.