Can custom client-side tests leverage the server-side value in analysis.cfg?
Date: Thu, 12 Oct 2023 19:23:17 -0500 From: Grant Taylor <gtaylor at tnetconsulting.net> To: Xymon <xymon at xymon.com> Subject: Can custom client-side tests leverage the server-side value in analysis.cfg?
Hi,
I'm relatively new to Xymon -- working with it for three months at a new job -- and am wondering about how to have a custom client-side test leverage the value of the server-side analysis.cfg.
Yes, a simple way is to use the "config" command, this lets you download any .cfg (or arbitrary files, if ALLOWALLCONFIGFILES="TRUE" but don't do that :) files from the server's etc/ directory.
client/bin/xymon $XYMONSERVER "config analysis.cfg" > tmp/analysis.cfg
If you use "include" or "directory" directives in the .cfg files Xymon will send you the "flattened" expanded contents.
You will need to parse that file to do anything useful though.
Alternatively, fetching the hosts.cfg and using xymongrep client/bin/xymon $XYMONSERVER "config hosts.cfg" > tmp/hosts.cfg HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep mytag:* might be simpler for simple configuration data that you can put in a "tag:" for each host. xymongrep is normally used server-side to loop over multiple hosts, you will likely need to match just one client name from its output for a client script.
Alternatively, instead of using analysis.cfg the client-local.cfg file is logically cleaner (and automatically shipped to clients, see client/tmp/logfetch.HOSTNAME.cfg). This specifies the client-side part of log and file checks, although it's not documented, you can add arbitrary lines to this file, e.g. [myhostname] x-my-specialvarA one x-my-specialvarB two log:/var/log/message:16384 ignore MARK
Each client will pick this up and store its own "personal" stanza from the global client-local.cfg, this is automatically refreshed each client run cycle. Any non-documented parameters are explicitly ignored, but best placed above any of the documented ones to make sure.
Do I /have/ /to/ declare a color in a
statusordatamessage to Xymon?
status: yes (though "clear" might be an option), data: no. There's also a "modify" message you can use after the fact to change a status/colour (though take care not to trigger flapping). https://xymon.sourceforge.io/xymon/help/manpages/man1/xymon.1.html
Is there a way to have the color be determined by a value in the server-side analysis.cfg file?
E.g. I'd like to write a custom test to monitor /var/spool/mqueue on multiple MTAs and I'd like to have different values for the green / yellow / red levels on different servers.
Seems like fetching the hosts.cfg and using xymongrep should work for that without too much trouble, at least as an exercise, and is reasonably self-evident. hosts.cfg: 10.1.2.3 mailhost1 # ssh smtps MAILQ:qwarn=200,qcrit=500
Then in a bash script (light on error checking) for example: client/bin/xymon $XYMONSERVER "fetch hosts.cfg" > tmp/hosts.cfg HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep "MAILQ:*" | while read hostip hostname hostcfg; do if [[ "$hostname" == ${MACHINEDOTS} ]]; then mailcfg="" [[ "$hostcfg" =~ MAILQ:(.*) ]] && mailcfg="${BASH_REMATCH[1]}" for var in qwarn qcrit; do [[ "$mailcfg" =~ "($var)=([^,;]+) ]] && declare "cfg_${BASH_REMATCH[1]}"="${BASH_REMATCH[2]}" done # declare -p ${!cfg_*} ##DEBUG## # mailq logic using $cfg_qwarn $cfg_qcrit goes here # xymon status submit goes here fi done
You could dispense with the outer while loop by using read hostip hostname hostcfg < <(xymongrep MAILQ:* | grep "$MACHINEDOTS")
There is no xymongrep equivalent for client-local.cfg, you could adapt the above to parse the client cached copy (tmp/logfetch.HOSTNAME.cfg) - simple enough since the server does not provide the entire client-local config, just the client-specific matching stanza.
I know there were some old, old mailq client scripts, there's still some RRD related residue for parsing their output in Xymon. I've never used it, but likely this was one:
https://web.archive.org/web/20080718210020/http://www.deadcat.net/viewfile.p... The bf/nmailq-bf.sh script therein simply barfs out a bit of sendmail's "mailq" output. The point being if you have the data in an RRD then you can also use "DS" in analysis.cfg to alert based on a GAUGE type metric. There's more than one way to skin a (dead)cat :)
To scale up the configuration complexity or number of discrete service checks it would be better to dispense with xymongrep and distribute your own easy-to-parse configuration file(s), or use the client-local.cfg approach (my preference).
C.
Ideally I'd like the custom client-side test that I put in ${XYMONHOME}/ext/ to be identical across all clients and the only difference being the value each system has in the analysis.cfg file on the server.
Am I /required/ to send a color; green, yellow, red, and then have something override / update that color server side? Or can I omit the color and let something server side set it using values for the given client?
I'm thinking about things like DISK / LOAD / etc. where I can set warn and critical levels, or even ignore.
Or am I looking at a custom server-side xymond_channel receiver that will receive
data(?) from the client and alter things? -- Even thatdatais likely going to require a color.Would someone with more experience with Xymon than me give me a pointer in a direction that I can go research?
Please and thank you.
-- Grant. . . . unix || die
Subject: Digest Footer
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
End of Xymon Digest, Vol 151, Issue 10
Hi Conor,
Thank you for your reply.
The overall gist of what I've understood thus far is that it's /indirectly/ possible to have custom client-side scripts use server-side values by arranging for a copy of those values to be available (cached) on the client for local use. -- Is that an accurate summary?
On 10/13/23 1:20?PM, Conor McCarthy wrote:
Yes, a simple way is to use the "config" command, this lets you download any .cfg (or arbitrary files, if ALLOWALLCONFIGFILES="TRUE" but don't do that :) files from the server's etc/ directory.
client/bin/xymon $XYMONSERVER "config analysis.cfg" > tmp/analysis.cfg
If you use "include" or "directory" directives in the .cfg files Xymon will send you the "flattened" expanded contents.
I'm glad to know that the analysis.cfg will be flattened as we're making extensive use of includes.
You will need to parse that file to do anything?useful though.
Alternatively, fetching the hosts.cfg and using xymongrep ? client/bin/xymon $XYMONSERVER "config hosts.cfg" > tmp/hosts.cfg ? HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep mytag:* might be simpler for simple configuration data that you can put in a "tag:" for each host.
I'd rather have the values being checked against -- as opposed to what tests are run -- in the analysis.cfg file. It seems to me like having value entries in an additional file could potentially end up in some confusion. Which file do I check CPU load in? What about mail queue size? Type thing.
xymongrep is normally used server-side to loop over multiple hosts, you will likely need to match just one client name from its output for a client script.
That's the impression I had gotten.
I feel like the ability to access the config, and thus what other clients are being monitored, is a little bit of an information leak. Probably not one that matters. And it's not like I can do anything about that.
Alternatively, instead of using analysis.cfg the client-local.cfg file is logically cleaner (and automatically shipped to clients, see client/tmp/logfetch.HOSTNAME.cfg).
ACK
This specifies the client-side part of log and file checks, although it's not documented, you can add arbitrary lines to this file, e.g. ?[myhostname] ?x-my-specialvarA one ?x-my-specialvarB two ?log:/var/log/message:16384 ?ignore MARK
Good to know.
Each client will pick this up and store its own "personal" stanza from the global client-local.cfg, this is automatically refreshed each client run cycle.
Does the server send the personalized version? Or does the client receive the full version and only store it's own personal values?
Any non-documented parameters are explicitly ignored, but best placed above any of the documented ones to make sure.
Noted.
status: yes (though "clear" might be an option),
Interesting call on the "clear" color.
That has the added benefit of not declaring a green / yellow / read.
data: no.
Noted.
There's also a "modify" message you can use after the fact to change a status/colour (though take care not to trigger flapping).
Ya. I had seen modify. I figured that might be part of how the server side tweaking of client side colors might happen.
I believe there is some restriction on once a message has been modified, only the same thing can modify it again. -- I assume that restriction is only there until a new status message comes in from the client.
Seems like fetching the hosts.cfg and using xymongrep should work for that without too much trouble, at least as an exercise, and is reasonably self-evident.
ACK
hosts.cfg: ?10.1.2.3 mailhost1 #? ssh smtps MAILQ:qwarn=200,qcrit=500
Then in a bash script (light on error checking) for example: ? client/bin/xymon $XYMONSERVER "fetch hosts.cfg" > tmp/hosts.cfg ? HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep "MAILQ:*" | ? while read hostip hostname hostcfg; do ? ? if [[ "$hostname" == ${MACHINEDOTS} ]]; then ? ? ? mailcfg="" ? ? ? [[ "$hostcfg" =~ MAILQ:(.*) ]] && mailcfg="${BASH_REMATCH[1]}" ? ? ? for var in qwarn qcrit; do ? ? ? ? [[ "$mailcfg" =~ "($var)=([^,;]+) ]] && declare "cfg_${BASH_REMATCH[1]}"="${BASH_REMATCH[2]}" ? ? ? done ? ? ? # declare -p ${!cfg_*} ##DEBUG## ? ? ? # mailq logic using $cfg_qwarn $cfg_qcrit goes here ? ? ? # xymon status submit goes here ? ? fi ? done
This is some very tight and nice Bash. I like how you're using test ([[) and parameter expansion. -- Much to come back to and look at in more detail later.
Aside: Did you mean "fetch" or "config" when calling xymon?
I think the biggest gotcha that I have is that some of the systems in my config are using different host names and relying on CLIENT: to make things match. But I think that's a matter of finding and grinding out the CLIENT: from the ${hostcfg}.
You could dispense with the outer while loop by using ? read hostip hostname hostcfg < <(xymongrep?MAILQ:* | grep "$MACHINEDOTS")
ACK
There is no xymongrep?equivalent for client-local.cfg, you could adapt the above to parse the client cached copy (tmp/logfetch.HOSTNAME.cfg) - simple enough since the server does not provide the entire client-local config, just the client-specific matching stanza.
I like the idea that the server only provides the client specific stanza.
But I also like the idea that I can pull the analysis.cfg file using
${XYMON} ${XYMSRV} "config analysis.cfg".
Initial skim of the xymongrep man page make me think that it's intended for the hosts.cfg file and not going to work with other files. So I'd need to parse it.
I know there were some old, old mailq client scripts, there's still some RRD related residue for parsing their output in Xymon. I've never used it, but likely this was one: https://web.archive.org/web/20080718210020/http://www.deadcat.net/viewfile.p...
$RESEARCH++
The bf/nmailq-bf.sh script therein simply barfs out a bit of sendmail's "mailq" output. The point being if you have the data in an RRD then you can also use "DS" in analysis.cfg to alert based on a GAUGE type metric. There's more than one way to skin a (dead)cat :)
I have seen many references to "DS", but I'm still not quite sure what that means. I need to read the custom graphing again and work to understand it.
To scale up the configuration complexity or number of discrete service checks it would be better to dispense with xymongrep and distribute your own easy-to-parse configuration file(s), or use the client-local.cfg approach (my preference).
ACK
The ability to pull config files from the server makes this a bit easier. It means that the client can get access to the configuration data to be able to more accurately set clear / green / yellow / red colors in reports.
Thank you Conor. I'll re-read, research, and do some thinking.
-- Grant. . . . unix || die
participants (2)
-
conor＠conormccarthy.ie
-
gtaylor＠tnetconsulting.net