Force logfetch to only process complete lines?
RHEL 6.9 and RHEL 7.4 Xymon v4.3.28
This may be documented somewhere and I'm just not able to find it. But is there a way to force logfetch to only scan complete lines and discard any partials it might retrieve based on the MAXCHECK setting?
I've been getting quite a few alerts on highly active systems where the offending line would normally be excluded due to the first part of a search that is missing.
A simple example, I want to ignore the alert triggers for /var/log/messages where the system name is test-system and :\sheader\ssubject: is also in the line. Since test-system comes right after the date/time stamp, that causes the ignore check to not work if test-system is not retrieved by logfetch.
analysis.cfg
Red alert on CRITICAL or ERROR or SERIOUS (with exceptions)
LOG %.* %(?-i)CRITICAL|ERROR|SERIOUS COLOR=red IGNORE=%(?-i)test-system.*:\sheader\ssubject:
I've tried adjusting the MAXCHECK setting but it didn't make a difference one way or the other.
client-local.cfg
log:/var/log/messages:10240 # 10KB default log:/var/log/messages:1024000 # 1MB
Thanks.
Larry D. Bonham
Financial Network Inc. 10401-F Baur Olivette, MO 63132
(314) 400-9412 voice (314) 997-5647 fax
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
Second request. No one else having this particular problem? Any help would be appreciated. Modifying logfetch.c is pretty much beyond my limited C skills.
Thanks.
Larry B.
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Larry Bonham Sent: Monday, February 26, 2018 5:28 PM To: xymon at xymon.com Subject: [Xymon] Force logfetch to only process complete lines?
RHEL 6.9 and RHEL 7.4 Xymon v4.3.28
This may be documented somewhere and I'm just not able to find it. But is there a way to force logfetch to only scan complete lines and discard any partials it might retrieve based on the MAXCHECK setting?
I've been getting quite a few alerts on highly active systems where the offending line would normally be excluded due to the first part of a search that is missing.
A simple example, I want to ignore the alert triggers for /var/log/messages where the system name is test-system and :\sheader\ssubject: is also in the line. Since test-system comes right after the date/time stamp, that causes the ignore check to not work if test-system is not retrieved by logfetch.
analysis.cfg
Red alert on CRITICAL or ERROR or SERIOUS (with exceptions)
LOG %.* %(?-i)CRITICAL|ERROR|SERIOUS COLOR=red IGNORE=%(?-i)test-system.*:\sheader\ssubject:
I've tried adjusting the MAXCHECK setting but it didn't make a difference one way or the other.
client-local.cfg
log:/var/log/messages:10240 # 10KB default log:/var/log/messages:1024000 # 1MB
Thanks.
Larry D. Bonham
Financial Network Inc. 10401-F Baur Olivette, MO 63132
(314) 400-9412 voice (314) 997-5647 fax
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
Third request. I just can't believe that I'm the only one having this problem. It is a fairly frequent occurrence for me. Mainly with higher volume log files.
I simply want to drop any partial lines before they are compared with LOG alert definitions.
Based on the comments in logfetch.c (v4.3.28), the section between 509 and 562 would appear to handle this. But for whatever reason it is not consistently working for me. Maybe I'm overloading the MAXCHECK value and it is just truncating the output? Or I'm misunderstanding what the section is actually doing?
Once again, any help would be appreciated.
Larry B.
From: Larry Bonham Sent: Monday, March 5, 2018 10:05 AM To: xymon at xymon.com Subject: RE: Force logfetch to only process complete lines?
Second request. No one else having this particular problem? Any help would be appreciated. Modifying logfetch.c is pretty much beyond my limited C skills.
Thanks.
Larry B.
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Larry Bonham Sent: Monday, February 26, 2018 5:28 PM To: xymon at xymon.com<mailto:xymon at xymon.com> Subject: [Xymon] Force logfetch to only process complete lines?
RHEL 6.9 and RHEL 7.4 Xymon v4.3.28
This may be documented somewhere and I'm just not able to find it. But is there a way to force logfetch to only scan complete lines and discard any partials it might retrieve based on the MAXCHECK setting?
I've been getting quite a few alerts on highly active systems where the offending line would normally be excluded due to the first part of a search that is missing.
A simple example, I want to ignore the alert triggers for /var/log/messages where the system name is test-system and :\sheader\ssubject: is also in the line. Since test-system comes right after the date/time stamp, that causes the ignore check to not work if test-system is not retrieved by logfetch.
analysis.cfg
Red alert on CRITICAL or ERROR or SERIOUS (with exceptions)
LOG %.* %(?-i)CRITICAL|ERROR|SERIOUS COLOR=red IGNORE=%(?-i)test-system.*:\sheader\ssubject:
I've tried adjusting the MAXCHECK setting but it didn't make a difference one way or the other.
client-local.cfg
log:/var/log/messages:10240 # 10KB default log:/var/log/messages:1024000 # 1MB
Thanks.
Larry D. Bonham
Financial Network Inc. 10401-F Baur Olivette, MO 63132
(314) 400-9412 voice (314) 997-5647 fax
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
Have you looked at client-local.cfg on the xymon server? Or the man page for client-local.cfg? It's a bit finicky but may be able to do what you're asking without modifying any code.
=G=
On Thu, May 10, 2018 at 5:32 PM, Larry Bonham <larry at fni-stl.com> wrote:
Third request. I just can’t believe that I’m the only one having this problem. It is a fairly frequent occurrence for me. Mainly with higher volume log files.
I simply want to drop any partial lines before they are compared with LOG alert definitions.
Based on the comments in logfetch.c (v4.3.28), the section between 509 and 562 would appear to handle this. But for whatever reason it is not consistently working for me. Maybe I’m overloading the MAXCHECK value and it is just truncating the output? Or I’m misunderstanding what the section is actually doing?
Once again, any help would be appreciated.
Larry B.
*From:* Larry Bonham *Sent:* Monday, March 5, 2018 10:05 AM *To:* xymon at xymon.com *Subject:* RE: Force logfetch to only process complete lines?
Second request. No one else having this particular problem? Any help would be appreciated. Modifying logfetch.c is pretty much beyond my limited C skills.
Thanks.
Larry B.
*From:* Xymon [mailto:xymon-bounces at xymon.com <xymon-bounces at xymon.com>] *On Behalf Of *Larry Bonham *Sent:* Monday, February 26, 2018 5:28 PM *To:* xymon at xymon.com *Subject:* [Xymon] Force logfetch to only process complete lines?
RHEL 6.9 and RHEL 7.4
Xymon v4.3.28
This may be documented somewhere and I’m just not able to find it. But is there a way to force logfetch to only scan complete lines and discard any partials it might retrieve based on the MAXCHECK setting?
I’ve been getting quite a few alerts on highly active systems where the offending line would normally be excluded due to the first part of a search that is missing.
A simple example, I want to ignore the alert triggers for /var/log/messages where the system name is test-system and :\sheader\ssubject: is also in the line. Since test-system comes right after the date/time stamp, that causes the ignore check to not work if test-system is not retrieved by logfetch.
analysis.cfg
Red alert on CRITICAL or ERROR or SERIOUS (with exceptions)
LOG %.* %(?-i)CRITICAL|ERROR|SERIOUS COLOR=red IGNORE=%(?-i)test-system.*:\sheader\ssubject:
I’ve tried adjusting the MAXCHECK setting but it didn’t make a difference one way or the other.
client-local.cfg
log:/var/log/messages:10240 # 10KB default
log:/var/log/messages:1024000 # 1MB
Thanks.
=========================================================
Larry D. Bonham
Financial Network Inc. 10401-F Baur Olivette, MO 63132
(314) 400-9412 voice (314) 997-5647 fax
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
To be a bit more explicit...this section from the manpage:
*LOGFILE CONFIGURATION ENTRIES A logfile configuration entry looks like this: log:/var/log/messages:10240 ignore MARK trigger Oops The log:FILENAME:SIZE line defines the filename of the log, and the maximum amount of data (in bytes) to send to the Xymon server. FILENAME is usually an explicit full-path filename on the client. If it is enclosed in backticks, it is a command which the Xymon client runs and each line of output from this command is then used as a filename. This allows scripting which files to monitor, e.g. if you have logfiles that are named with some sort of timestamp. If FILENAME is enclosed in angle brackets it is treated as a glob and passed through the local glob(3) function first. The ignore PATTERN line (optional) defines lines in the logfile which are ignored entirely, i.e. they are stripped from the logfile data before sending it to the Xymon server. It is used to remove completely unwanted "noise" entries from the logdata processed by Xymon. "PATTERN" is a regular expression. The trigger PATTERN line (optional) is used only when there is more data in the log than the maximum size set in the "log:FILENAME:SIZE" line. The "trigger" pattern is then used to find particularly interesting lines in the logfile
- these will always be sent to the Xymon server. After picking out the "trigger" lines, any remaining space up to the maximum size is filled in with the most recent entries from the logfile. "PATTERN" is a regular expression.*
IIRC, you can even have multiple *ignore* entries. You should have messages in your xymon logs if the file is too big when it's fetched...also, I think you will also run up against the Xymon max data size in the server configs.
=G=
Thanks Galen. I really appreciate the response.
I have looked at client-local.cfg and experimented with different settings. Do you know if there is a practical limit to the size setting? I know 10240 is the default but I would like it as large as possible.
Most of my problems are related to LOG settings in analysis.cfg. I could try moving all that to client-local.cfg. I am correct in saying that client-local.cfg does require duplication of global settings if you have settings for a specific host (e.g. settings based on class RHEL7 will need to be duplicated for specific hosts)? We have over 250 so I’d like as many lumped together as possible.
I’ll review all this and reply tomorrow. Thanks.
Larry
From: Galen Johnson [mailto:solitaryr at gmail.com] Sent: Thursday, May 10, 2018 8:12 PM To: Larry Bonham Cc: xymon at xymon.com Subject: Re: [Xymon] Force logfetch to only process complete lines?
To be a bit more explicit...this section from the manpage:
LOGFILE CONFIGURATION ENTRIES A logfile configuration entry looks like this:
log:/var/log/messages:10240
ignore MARK
trigger Oops
The log:FILENAME:SIZE line defines the filename of the log, and the maximum amount of data (in bytes) to send to the Xymon server. FILENAME is usually an explicit full-path filename on the client. If it is enclosed in backticks, it is a command which the Xymon client runs and each line of output from this command is then used as a filename. This allows scripting which files to monitor, e.g. if you have logfiles that are named with some sort of timestamp. If FILENAME is enclosed in angle brackets it is treated as a glob and passed through the local glob(3) function first.
The ignore PATTERN line (optional) defines lines in the logfile which are ignored entirely, i.e. they are stripped from the logfile data before sending it to the Xymon server. It is used to remove completely unwanted "noise" entries from the logdata processed by Xymon. "PATTERN" is a regular expression.
The trigger PATTERN line (optional) is used only when there is more data in the log than the maximum size set in the "log:FILENAME:SIZE" line. The "trigger" pattern is then used to find particularly interesting lines in the logfile - these will always be sent to the Xymon server. After picking out the "trigger" lines, any remaining space up to the maximum size is filled in with the most recent entries from the logfile. "PATTERN" is a regular expression.IIRC, you can even have multiple ignore entries. You should have messages in your xymon logs if the file is too big when it's fetched...also, I think you will also run up against the Xymon max data size in the server configs.
=G=
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
unless I'm mistaken, you should be able to add the rules under os types. It should hopefully be obvious from looking in the existing client-local.cfg file. It's been a while since I messed in there but IIRC you can even do expressions to group them.
=G=
On Thu, May 10, 2018 at 9:53 PM, Larry Bonham <larry at fni-stl.com> wrote:
Thanks Galen. I really appreciate the response.
I have looked at client-local.cfg and experimented with different settings. Do you know if there is a practical limit to the size setting? I know 10240 is the default but I would like it as large as possible.
Most of my problems are related to LOG settings in analysis.cfg. I could try moving all that to client-local.cfg. I am correct in saying that client-local.cfg does require duplication of global settings if you have settings for a specific host (e.g. settings based on class RHEL7 will need to be duplicated for specific hosts)? We have over 250 so I’d like as many lumped together as possible.
I’ll review all this and reply tomorrow. Thanks.
Larry
*From:* Galen Johnson [mailto:solitaryr at gmail.com] *Sent:* Thursday, May 10, 2018 8:12 PM *To:* Larry Bonham *Cc:* xymon at xymon.com *Subject:* Re: [Xymon] Force logfetch to only process complete lines?
To be a bit more explicit...this section from the manpage:
*LOGFILE** CONFIGURATION ENTRIES *
A logfile configuration entry looks like this: log:/var/log/messages:10240 ignore MARK trigger Oops The log:FILENAME:SIZE line defines the
- these will always be sent to the Xymon server. After picking out the "trigger" lines, any remaining space up to the maximum size is filled in with the most recent entries from the logfile. "PATTERN" is a regular expression.*
IIRC, you can even have multiple *ignore* entries. You should have messages in your xymon logs if the file is too big when it's fetched...also, I think you will also run up against the Xymon max data size in the server configs.
=G=
CONFIDENTIALITY NOTICE: This electronic mail message is intended exclusively for recipient to which it is addressed. The contents of this message and any attachments may contain confidential and privileged information. Any unauthorized review, use, print, storage, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please advise the sender immediately by replying to the message's sender and delete all copies of this message and its attachments without disclosing the contents to anyone, or using the contents for any purpose.
Hi guys,
Does anyone know how I can exclude a directory from being checked by xymon-client for disk checks?
This has to be done client side because it appears to be breaking the client itself. Configuring the server to ignore the check won't prevent the client from trying to run checks on this directory.
Regards, JP Pitout
"This email is confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person as to do so could be a breach of confidentiality. Thank you for your co-operation. "
"The information contained in this message is intended solely for the individual to whom it is specifically and originally addressed. This message and its contents may contain confidential or privileged information from MTN Group. If you are not the intended recipient, you are hereby notified that any disclosure or distribution, is strictly prohibited. If you receive this email in error, please notify MTN Group immediately and delete it. MTN Group does not accept any liability or responsibility if action is taken in reliance on the contents of this information . Note that all personal emails are not authorized by MTN Group. "
What is the platform and why does checking it cause a problem?
If you are Linux or Unix it may help to add a switch to the df command.
Regards
Jeremt
On Mon, 14 May 2018 13:27 Jean-Pierre Pitout [ MTN South Africa ], < Jean-Pierre.Pitout at mtn.com> wrote:
Hi guys,
Does anyone know how I can exclude a directory from being checked by xymon-client for disk checks?
This has to be done client side because it appears to be breaking the client itself. Configuring the server to ignore the check won't prevent the client from trying to run checks on this directory.
Regards, JP Pitout
*"This email is confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person as to do so could be a breach of confidentiality. Thank you for your co-operation. "*
“The information contained in this message is intended solely for the individual to whom it is specifically and originally addressed. This message and its contents may contain confidential or privileged information from MTN Group. If you are not the intended recipient, you are hereby notified that any disclosure or distribution, is strictly prohibited. If you receive this email in error, please notify MTN Group immediately and delete it. MTN Group does not accept any liability or responsibility if action is taken in reliance on the contents of this information . Note that all personal emails are not authorized by MTN Group. ”
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Hi Jeremy, thank you for your response!
We're running RHEL and the xymon-client is using df for disk checks.
Recently Docker was installed and configured to use specific mount point for devicemapper (/app2/docker/devicemapper). Not sure if this is standard as it was configured by the developers.
Some issue is causing the client to hang when checking disk utilisation on this mount point. The issue could be with df itself but I thought excluding it from the check would be the simplest option.
Regards,
Jean-Pierre Pitout [ MTN South Africa ]
Senior Linux Engineer (RHCA)
IS Client Server Line-Of-Business Linux
ISClientServerLine-of-Business_Linux.ZA at mtn.comFrom: Jeremy Ruffer <jeremy.ruffer at gmail.com> Sent: 14 May 2018 14:34 To: Jean-Pierre Pitout [ MTN South Africa ] Cc: xymon Subject: Re: [Xymon] xymon-client: Exclude directory from being checked by client.
What is the platform and why does checking it cause a problem?
If you are Linux or Unix it may help to add a switch to the df command.
Regards
Jeremt
On Mon, 14 May 2018 13:27 Jean-Pierre Pitout [ MTN South Africa ], <Jean-Pierre.Pitout at mtn.com<mailto:Jean-Pierre.Pitout at mtn.com>> wrote:
Hi guys,
Does anyone know how I can exclude a directory from being checked by xymon-client for disk checks?
This has to be done client side because it appears to be breaking the client itself. Configuring the server to ignore the check won't prevent the client from trying to run checks on this directory.
Regards, JP Pitout
"This email is confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person as to do so could be a breach of confidentiality. Thank you for your co-operation. "
"The information contained in this message is intended solely for the individual to whom it is specifically and originally addressed. This message and its contents may contain confidential or privileged information from MTN Group. If you are not the intended recipient, you are hereby notified that any disclosure or distribution, is strictly prohibited. If you receive this email in error, please notify MTN Group immediately and delete it. MTN Group does not accept any liability or responsibility if action is taken in reliance on the contents of this information . Note that all personal emails are not authorized by MTN Group. "
Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon "The information contained in this message is intended solely for the individual to whom it is specifically and originally addressed. This message and its contents may contain confidential or privileged information from MTN Group. If you are not the intended recipient, you are hereby notified that any disclosure or distribution, is strictly prohibited. If you receive this email in error, please notify MTN Group immediately and delete it. MTN Group does not accept any liability or responsibility if action is taken in reliance on the contents of this information . Note that all personal emails are not authorized by MTN Group. "
As these df calls are located in the xymonclient-linux.sh (for your linux systems) on the client side, you have to add your FS to the EXCLUDES variable in this script, like EXCLUDES="$EXCLUDES -x additional_fs_type_to_ignore"
But these excludes only work for FS types, not for mount points. Of course you can throw away all the messages (simply add 2>/dev/null after the df calls), but if there is a stale mount or other reason that causes df to hang, this will not help.
Norbert
From: "Jean-Pierre Pitout [ MTN South Africa ]" <Jean-Pierre.Pitout at mtn.com> To: Jeremy Ruffer <jeremy.ruffer at gmail.com> Cc: xymon <xymon at xymon.com> Date: 14/05/2018 14:52 Subject: Re: [Xymon] xymon-client: Exclude directory from being checked by client. Sent by: "Xymon" <xymon-bounces at xymon.com>
Hi Jeremy, thank you for your response!
We're running RHEL and the xymon-client is using df for disk checks.
Recently Docker was installed and configured to use specific mount point for devicemapper (/app2/docker/devicemapper). Not sure if this is standard as it was configured by the developers.
Some issue is causing the client to hang when checking disk utilisation on this mount point. The issue could be with df itself but I thought excluding it from the check would be the simplest option.
Regards, ________________________________________________
Jean-Pierre Pitout [ MTN South Africa ]
Senior Linux Engineer (RHCA)
IS Client Server Line-Of-Business Linux
ISClientServerLine-of-Business_Linux.ZA at mtn.comFrom: Jeremy Ruffer <jeremy.ruffer at gmail.com> Sent: 14 May 2018 14:34 To: Jean-Pierre Pitout [ MTN South Africa ] Cc: xymon Subject: Re: [Xymon] xymon-client: Exclude directory from being checked by client.
What is the platform and why does checking it cause a problem?
If you are Linux or Unix it may help to add a switch to the df command.
Regards
Jeremt
On Mon, 14 May 2018 13:27 Jean-Pierre Pitout [ MTN South Africa ], < Jean-Pierre.Pitout at mtn.com> wrote: Hi guys,
Does anyone know how I can exclude a directory from being checked by xymon-client for disk checks?
This has to be done client side because it appears to be breaking the client itself. Configuring the server to ignore the check won't prevent the client from trying to run checks on this directory.
Regards, JP Pitout
"This email is confidential. If you have received it in error, you are on notice of its status. Please notify us immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person as to do so could be a breach of confidentiality. Thank you for your co-operation. "
“The information contained in this message is intended solely for the individual to whom it is specifically and originally addressed. This message and its contents may contain confidential or privileged information from MTN Group. If you are not the intended recipient, you are hereby notified that any disclosure or distribution, is strictly prohibited. If you receive this email in error, please notify MTN Group immediately and delete it. MTN Group does not accept any liability or responsibility if action is taken in reliance on the contents of this information . Note that all personal emails are not authorized by MTN Group. ”
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon “The information contained in this message is intended solely for the individual to whom it is specifically and originally addressed. This message and its contents may contain confidential or privileged information from MTN Group. If you are not the intended recipient, you are hereby notified that any disclosure or distribution, is strictly prohibited. If you receive this email in error, please notify MTN Group immediately and delete it. MTN Group does not accept any liability or responsibility if action is taken in reliance on the contents of this information . Note that all personal emails are not authorized by MTN Group. ”
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
participants (5)
-
Jean-Pierre.Pitout＠mtn.com
-
jeremy.ruffer＠gmail.com
-
larry＠fni-stl.com
-
norbert.kriegenburg＠de.ibm.com
-
solitaryr＠gmail.com