acknowledgements does not survive xymon restart
What config changes are you making that requires such frequent restarts?
Changes to hosts.cfg, alerts.cfg, analysis.cfg, client-local.cfg; these are files that get changed most frequently in my environment. None of which require a restart to pick up the changes. I think the only one that *does* need a restart, would be xymonserver.cfg, and the only component that would need to be restarted would be xymond, not the full stack.
Erik D. Schminke | Associate Systems Programmer Hormel Foods Corporation | One Hormel Place | Austin, MN 55912 Phone: (507) 434-6817 edschminke at hormel.com | www.hormelfoods.com
Don't get me wrong: i don't do frequent restarts usually, but from time to time i need a restart, or the whole server must be restarted bc. of patches. And as i constantly have to add new checks with new NCV definitions the TEST2RRD and SPLITNCV settings in xymonserver.cfg changes, this needs also a restart.
Because we have such a huge numer of servers a lot of departments use Xymon regularly (luckily), and use the ack mechanism to organize their work (add ticket number to an alert, do some evaluation reports and so on). To have >100 alerts ack'ed is normal situation. And it creates a lot of extra work to restore this. In old BB times the acks always survived downtimes and restarts, but now there are no more ack files stored in the acks dir, so i thought it would be restored from the info in the alert.chk file, what is not the case.
Norbert
From: EDSchminke at Hormel.com To: xymon at xymon.com Cc: norbert.kriegenburg at de.ibm.com Date: 11/01/2018 12:58 PM Subject: Re: [Xymon] acknowledgements does not survive xymon restart
What config changes are you making that requires such frequent restarts?
Changes to hosts.cfg, alerts.cfg, analysis.cfg, client-local.cfg; these are files that get changed most frequently in my environment. None of which require a restart to pick up the changes. I think the only one that *does* need a restart, would be xymonserver.cfg, and the only component that would need to be restarted would be xymond, not the full stack.
Erik D. Schminke | Associate Systems Programmer Hormel Foods Corporation | One Hormel Place | Austin, MN 55912 Phone: (507) 434-6817 edschminke at hormel.com | www.hormelfoods.com
Hi Norbert,
as I can not remember that I encountered this I just tested this (in a mini setup, xymon 4.3.28 on Debian9) and my 2 acks survived the xymon-restart. As your environment is fairly large this could be size/load problem.
Random ideas:
- is the restart “clean” or does something crash during the stop-phase (logfiles)?
- you could try is to manually force writing a checkpoint-file by sending SIGUSR1 to xymond before restart.
- if you have a redundant/multi-server setup: Is there any chance that an other xymon-server is propagating incomplete state (
xymond_distributenot enabled)?
All the best Thomas
On 01 Nov 2018, at 13:16, Norbert Kriegenburg <norbert.kriegenburg at de.ibm.com> wrote:
Don't get me wrong: i don't do frequent restarts usually, but from time to time i need a restart, or the whole server must be restarted bc. of patches. And as i constantly have to add new checks with new NCV definitions the TEST2RRD and SPLITNCV settings in xymonserver.cfg changes, this needs also a restart.
Because we have such a huge numer of servers a lot of departments use Xymon regularly (luckily), and use the ack mechanism to organize their work (add ticket number to an alert, do some evaluation reports and so on). To have >100 alerts ack'ed is normal situation. And it creates a lot of extra work to restore this. In old BB times the acks always survived downtimes and restarts, but now there are no more ack files stored in the acks dir, so i thought it would be restored from the info in the alert.chk file, what is not the case.
Norbert
<graycol.gif>EDSchminke---11/01/2018 12:58:35 PM---What config changes are you making that requires such frequent restarts? Changes to hosts.cfg, alert
From: EDSchminke at Hormel.com To: xymon at xymon.com Cc: norbert.kriegenburg at de.ibm.com Date: 11/01/2018 12:58 PM Subject: Re: [Xymon] acknowledgements does not survive xymon restart
What config changes are you making that requires such frequent restarts?
Changes to hosts.cfg, alerts.cfg, analysis.cfg, client-local.cfg; these are files that get changed most frequently in my environment. None of which require a restart to pick up the changes. I think the only one that *does* need a restart, would be xymonserver.cfg, and the only component that would need to be restarted would be xymond, not the full stack.
Erik D. Schminke | Associate Systems Programmer Hormel Foods Corporation | One Hormel Place | Austin, MN 55912 Phone: (507) 434-6817 edschminke at hormel.com | www.hormelfoods.com
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Hi Thomas,
thx for your suggestions, but unfortunately this did not catch the issue. The restart runs without messages, nothing suspicious in the logs. But after restart and after some minutes all acks are gone. Also the ack table on bottom of the nongreen page is empty.
My alert.chk file is always up-to-date, a SIGUSR1 does not change anything. But it is quite large (5,1MB) currently due to the lot of alerts (my access to one of the DMZ is blocked creating a lot of conn/ssh/rdp alerts).
I wrote a script to mass-ack such alerts, otherwise the noise would be unmanageable. This acks all the red conns/ssh/rdp for this DMZ, and i can see the acks in the nongreen page. Until next restart...
Btw: In difference what Erik wrote: at least new CLASS settings in analysis.cfg need a xymon.sh restart to take effect (just checked).
Norbert
From: Thomas Eckert <thomas.eckert at it-eckert.de> To: xymon <xymon at xymon.com> Cc: Norbert Kriegenburg <norbert.kriegenburg at de.ibm.com> Date: 11/02/2018 10:45 AM Subject: Re: [Xymon] acknowledgements does not survive xymon restart
Hi Norbert,
as I can not remember that I encountered this I just tested this (in a mini setup, xymon 4.3.28 on Debian9) and my 2 acks survived the xymon-restart. As your environment is fairly large this could be size/load problem.
Random ideas:
- is the restart “clean” or does something crash during the stop-phase (logfiles)?
- you could try is to manually force writing a checkpoint-file by sending SIGUSR1 to xymond before restart.
- if you have a redundant/multi-server setup: Is there any chance that an
other xymon-server is propagating incomplete state (
xymond_distributenot enabled)?
All the best Thomas On 01 Nov 2018, at 13:16, Norbert Kriegenburg < norbert.kriegenburg at de.ibm.com> wrote:
Don't get me wrong: i don't do frequent restarts usually, but from
time to time i need a restart, or the whole server must be restarted
bc. of patches.
And as i constantly have to add new checks with new NCV definitions
the TEST2RRD and SPLITNCV settings in xymonserver.cfg changes, this
needs also a restart.
Because we have such a huge numer of servers a lot of departments use
Xymon regularly (luckily), and use the ack mechanism to organize
their work (add ticket number to an alert, do some evaluation reports
and so on).
To have >100 alerts ack'ed is normal situation.
And it creates a lot of extra work to restore this.
In old BB times the acks always survived downtimes and restarts, but
now there are no more ack files stored in the acks dir, so i thought
it would be restored from the info in the alert.chk file, what is not
the case.
Norbert
<graycol.gif>EDSchminke---11/01/2018 12:58:35 PM---What config
changes are you making that requires such frequent restarts? Changes
to hosts.cfg, alert
From: EDSchminke at Hormel.com
To: xymon at xymon.com
Cc: norbert.kriegenburg at de.ibm.com
Date: 11/01/2018 12:58 PM
Subject: Re: [Xymon] acknowledgements does not survive xymon restart
What config changes are you making that requires such frequent
restarts?
Changes to hosts.cfg, alerts.cfg, analysis.cfg, client-local.cfg;
these are
files that get changed most frequently in my environment. None of
which
require a restart to pick up the changes. I think the only one that
*does*
need a restart, would be xymonserver.cfg, and the only component that
would
need to be restarted would be xymxm-multiack.sh -t conn -c clear -a
rdp -d 144000 -r "FW blocked" -i de152911ond, not the full stack.
Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Place | Austin, MN 55912
Phone: (507) 434-6817
edschminke at hormel.com | www.hormelfoods.com
_______________________________________________
Xymon mailing list
Xymon at xymon.com
http://lists.xymon.com/mailman/listinfo/xymon
participants (3)
-
EDSchminke＠Hormel.com
-
norbert.kriegenburg＠de.ibm.com
-
thomas.eckert＠it-eckert.de