On Oct 24, 2014, at 14:50, J.C. Cleaver <cleaver at terabithia.org> wrote:

...

On Fri, October 24, 2014 8:13 am, Nicole Beck wrote: Hello, I think this has been asked before, but it was a long time ago and I wondered if something changed since then.

How long will the status of MSGS stay red? We just setup monitoring recently and it seems like it stays red for about 30 minutes. Is that normal? Is this configurable?

We are running Xymon server 4.2.3.

Nicole,

The duration of the 'msgs' test is actually a function of how many cycles back logfetch will scan for content to include in the log data going forward (actual calculation of the color is via the regex's performed by xymond_client).

logfetch will look back 6 runtime-positions which, combined with the default xymonclient run interval of 5m, ends up causing the 30m figure.

The former value is compiled in, however the run frequency is configurable. (We run our clients on 100s cycles, which means our msgs tests last for 10-12m.)

I'm not sure how easy the 6x positions would be to be made dynamic or a runtime option, but that would be nice.

Could have sworn the number of lines to look at was configurable too. Maybe I'm thinking of BB?

On 27 October 2014 01:26, Bill Arlofski <waa-hobbitml at revpol.com> wrote:

I have not tested this, but it would seem to indicate that it would cause the client to send the Xymon server all the lines that match the trigger pattern (regardless of how far back in time they go in the logfile) which should cause the test to stay non-green until the logfile is rotated and no more lines with the trigger pattern exist.

I haven't verified this, but my understanding of how the "logfetch" process works is that it keeps state of where it got up to in each logfile, and for the next (5 minute) round, it starts looking for matches only from that point onwards. This means, if there's a trigger match in the log file, the client will send it to the server in that round only.

J

On 28 October 2014 09:58, Bill Arlofski <waa-hobbitml at revpol.com> wrote:

Other ideas? Can I somehow hammer this square peg into a round hole?

You can create a dynamic file based on the logfile, and alert on that. For example, in client-local.cfg, something like this:

log:LOG=/tmp/zlic.status; M=$(date +%M); [ $(expr $M % 10) -ge 5 ] && rm -f $LOG; grep "ArchivingAccountsLimit exceeded" /var/log/messages >> $LOG; [ -s $LOG ] && echo "$LOG":4096

I'm assuming that /var/log/messages is rotated daily. What happens here is that zlic.status will get the log entries from your current messages file (updated every 5 minutes) appended to it. If there are no log entries, then the filename is not echoed and Xymon will ignore it (and no alerts possible).

The trick here is that the zlic.status file is emptied only every second run (every 10 minutes) prior to appending the log entries. By shrinking the file size, logfetch thinks the file has been rotated, zeroes its status, and starts looking at the file from the beginning.

Note that if you get a log entry in your messages file just prior to rotation, then you'll only get an alert between the time the message is detected and the messages file is rotated, which could be only a few minutes, or even not at all if the timing isn't favourable. So in other words, this will generate an alert that persists until the next rotation of messages, or messages in the last 0-24 hours. If you want to go for longer than that, you could perhaps grep from the current and previous messages file, so you're alerting on any messages in the last 24-48 hours.

Another way to do this is to use a "file:" definition, similarly creating a status file and then alarming on the file's size (non-zero indicating an alertable log entry). For example:

file:LOG=/tmp/zlic.status; grep "ArchivingAccountsLimit exceeded" /var/log/messages >> $LOG; echo $LOG

Then in analysis.cfg, create a matching entry and alert on size>0. A down-side to this approach is that you get a particularly unhelpful message along the lines of "FILE /tmp/zlic.status red size >0".

A third and similar way to do this is to create a file that exists only if the licencing log is not detected. Like so:

file:LOG=/tmp/zlic.OK; grep "ArchivingAccountsLimit exceeded" >/dev/null && rm -f $LOG || touch $LOG; echo $LOG

Then in analysis.cfg, create a matching entry and alert on "noexist".

Yet another way to do this is to use a pseudo-file to generate a status message. For example:

file:COL=green; MSG="licencing OK"; LOGS=$(grep "ArchivingAccountsLimit exceeded" /var/log/messages); [ "$LOGS" ] && { COL=red; MSG="licencing error"; }; echo "status ${MACHINE}.zlic $COL $(date) $MSG" | $XYMON $XYMSRV @

There is no output from this pseudo-file, so Xymon will not take any "file" connotations from it and will simply ignore it, except for the side-effects from the $XYMON command that's also run here. This is tantamount to having a client-side ext script, and you may simply prefer to do that. But this can be deployed centrally.

A few notes:

1. None of these specific examples have been tested, and may contain syntax errors, but scriptlets like these have been used on production systems.
2. I deliberately avoided using colons and backticks, because they are interpreted by the logfetch binary, and break the scriptlets.
3. These scriptlets take up to 15 minutes to start reporting after being added to client-local.cfg. When I'm testing these sort of things, I like to bring up a xymoncmd shell, and paste in the bits between the backticks, and look for errors or unexpected output.

J

Nicole

On 29 October 2014 05:16, Nicole Beck <nskyrca at syr.edu> wrote:

What I’m seeing is that I get an alert for my trigger string (which has a timestamp on it), and then I keep getting alerts for the same trigger string (with the same timestamp) for the next 30 minutes.

How often do you get the repeated alerts? Or how many in that 30 minutes?

I’m not sure if anything else was append to the log file in that 30 minutes. I stop getting the alerts after 30 minutes and don’t have to wait until the log is rotated for the alert to clear.

Do you have ALERTREPEAT defined in xymonserver.cfg? The default is 30 seconds, but you may have it less than that.

Similarly, do you have "REPEAT" defined in alerts.cfg for the rule matching these alerts? (The "REPEAT" value in alerts.cfg defaults to the setting of ALERTREPEAT.)

Is your message status (red?) staying non-green for the 30 minutes, or non-green for only a short time, or flapping like red/green/red/green?

The way messages get to Xymon are via the client data. So during an "event" you can click on the "Client data available" link at the bottom of your "msgs" page for the host, and it should show you all of the client data, and you can search for the logfilename to see what log lines the client sent to the server. Or you can click on the logfile name on the "msgs" page for a modified client data report showing just the log lines for that logfile.

What I'm trying to understand is whether you are getting the same messages sent multiple times from the client causing multiple events, or whether the one event is generating multiple alerts.

From what I can tell, a red "msgs" status will stay red for only one 5-minute client cycle. The next time the client sends its client data report, if the logfile in question has no new matching lines, it will actively generate a green status.

J