Can custom client-side tests leverage the server-side value in analysis.cfg?
Responding inline:
On Tue, 17 Oct 2023 at 02:47, Grant Taylor via Xymon <xymon at xymon.com> wrote:
The "DS" means "data source". It's a term used by the RRD (round-robin database) library (and tools) that Xymon uses for storing historical data.
Thank you for clarifying. I sort of deduced that as I got further into the custom graphs, which I've now got working. But it's one of those things that when you're newer to something "DS" is unfortunately vague and lost in a sea of new information.
Agreed. Xymon/Hobbit, and its spiritual predecessor BigBrother, come from a tradition of using a common collection of disparate tools for monitoring networked devices and hosts. One of these foundational tools is RRD, which is used by other network monitors (Cacti, MRTG, etc). The "culture" of Xymon can sometimes assume knowledge of how the other tools work. And while there's references to RRD in the Xymon doco, it couldn't hurt to add some extra definitions or explanations within the Xymon doco, to help those who don't have a history in this space that extends back to the last millenium (showing my age, aren't I).
As a general principle, Xymon clients operate in local mode or central
mode. In (the older) local mode, the clients do all of their own analysis and create their own data and status messages. In (the more modern) central mode, the clients do a lot less of that, and instead simply gather data for the server to parse and alert on.
Would it be fair to say that the older local mode uses status messages with color and newer central mode use data messages without color? -- As a rough generalization, not a hard and fast rule.
Yes and no. The central/local mode choice relates to the bulk of the core OS resource metric collection and processing. Choosing central mode doesn't prevent the use of client-side processing and status messages (with colour) being sent from the client. I know you didn't ask this, but I'm just making sure I'm not giving the wrong idea.
With local mode, the client looks for faults, and sends a status message with the colour in the header and a text report in the body.
With central mode, client data messages are sent to the server, into the xymond daemon. Then code on the server asks the xymond daemon for the client message of each host, parses them for possible problems, and constructs a status message with the colour in the header and a text report in the body.
So the last part of the process is the same. The Xymon server receives status messages from itself for clients in central mode, or from the clients when they're in local mode.
To bring it back to your phrasing, local mode uses status messages with colo(u)r and central mode also uses status messages with colo(ur) but those messages come from the Xymon server itself. In central mode, the colours are determined by the server (eg in analysis.cfg), whereas in local mode, the colours are determined by the client.
You could create a script in the "sections" subdirectory, and the
Xymon client will automatically run it and include the output in the [mailq] section of the client message.
This is very intriguing. I spent a moment looking for "section" (case insensitive) in the client directory tree (
fgrep -ir "section"type thing) and found reference to the client/local directory. Upon testing, I see that I end up with the output of my executable (currently a script) with a heading named[local:test_script.sh].
Yes, the "client/sections" directory is a relatively recent feature that was added as a "patch" to the original client-side script. My mistake for assuming that everyone had this. The patch is applied to the packages available from the Terabithia repository, but I don't believe it's included when Xymon is built from original source code. The "client/local" directory is "standard" Xymon (as long as you create the directory), but it has the side-effect of making section names with the "local" prefix. There are probably good reasons for this prefixing (I'm guessing to prevent collisions with standard section names), so if it works for you, that's great. My guess is that the next release of Xymon will include client/sections in the source, in addition to supporting client/local.
A snippet from the README in client/sections/ from the Terabithia package:
"This directory is intended for use with automatically deployed files. Local scripts for this host should be placed in the client/local/ directory instead."
It seems like the gist of the idea is to receive the client status
message, extract the section, and generate updates therefor.
Exactly. The client is as simple as it can be, so that it doesn't need constant tweaking. The server does all the smarts, including parsing, graphing, thresholding, so that the client host can spend more cycles doing its main function.
And optionally turn the mail queue size into a graphable data point
(either create a trends message with the number, or use a colon-separated value format for the Xymon parser to detect). The Xymon doco has a "how to setup custom graphs" page with a few different ways to do this type of thing.
Yep. I used that very page to get NCV based custom graphs working Friday evening.
I still need to explore the DS analysis / color declaration therefor.
You're well on your way, so well done! Feel free to ask more questions on this list. The more questions the better our shared knowledge! (Although it's easier to respond to if each question is in a separate thread.)
J
On 10/16/23 6:39?PM, Jeremy Laidman wrote:
Responding inline:
:-)
Agreed. Xymon/Hobbit, and its spiritual predecessor BigBrother, come from a tradition of using a common collection of disparate tools for monitoring networked devices and hosts. One of these foundational tools is RRD, which is used by other network monitors (Cacti, MRTG, etc).
Yep. I was exposed to RRD via MRTG, and subsequently Cacti, probably two decades ago. Both surprisingly and thankfully I've not needed to use RRD directly as things like MRTG, Cacti, and Xymon have abstracted them away for me. But I've long appreciated the suite of tools.
The "culture" of Xymon can sometimes assume knowledge of how the other tools work. And while there's references to RRD in the Xymon doco, it couldn't hurt to add some extra definitions or explanations within the Xymon doco, to help those who don't have a history in this space that extends back to the last millenium (showing my age, aren't I).
Waiter / waitress, the next cup of $COFFEE is on me.
Yes and no. The central/local mode choice relates to the bulk of the core OS resource metric collection and processing. Choosing central mode doesn't prevent the use of client-side processing and status messages (with colour) being sent from the client. I know you didn't ask this, but I'm just making sure I'm not giving the wrong idea.
ACK
Sounds like use what's best for the situation, even if there might be a slight affinity one way or the other.
With local mode, the client looks for faults, and sends a status message with the colour in the header and a text report in the body.
I've been making extensive use of this. I've been augmenting many cron jobs that historically have sent email to now also report to Xymon. I'm making extensive use of lifetime to make lack of reports stand out by going purple.
Honestly, this work effort has gotten me the closest "single pane of glass" visibility into environments. We're still a long way away from it. But we're getting closer every day.
With central mode, client data messages are sent to the server, into the xymond daemon. Then code on the server asks the xymond daemon for the client message of each host, parses them for possible problems, and constructs a status message with the colour in the header and a text report in the body.
My understanding is that's one way to do it.
My understanding that another way to do it is to hook in as a channel and receive a copy of all messages (of the proper type) and act appropriately.
So the last part of the process is the same. The Xymon server receives status messages from itself for clients in central mode, or from the clients when they're in local mode.
ACK
To bring it back to your phrasing, local mode uses status messages with colo(u)r and central mode also uses status messages with colo(ur) but those messages come from the Xymon server itself. In central mode, the colours are determined by the server (eg in analysis.cfg), whereas in local mode, the colours are determined by the client.
ACK
Yes, the "client/sections" directory is a relatively recent feature that was added as a "patch" to the original client-side script. My mistake for assuming that everyone had this.
After inheriting the -- let's go with -- existing legacy install at $DAY_JOB, I've since upgraded $DAY_JOB to 4.3.30, installed 4.3.30 at home, and installed 4.3.28 (what comes in distro files).
I've got a smattering of clients from 4.3.30 down to 4.2.3 (installed yesterday on an ancient FreeBSD 4.11). -- As in the green faces have just recently turned to green diamonds for that FreeBSD 4.11 system.
I'm trying to get my hands on a compilation of any version for AIX 5.3 (?). For now I'm running the hobbitclient-aix.sh via telnet w/ echo scheduled via cron. It's working and providing most of what I need and much of what I want.
I've sort of taken what previous colleagues poked once in a while and ran with it to make things better for $DAY_JOB, myself, and a friend that I help. I've also intrigued another friend enough to install and play with it.
I think I say that I'm liking Xymon. ;-)
The patch is applied to the packages available from the Terabithia repository, but I don't believe it's included when Xymon is built from original source code.
Please elaborate and / or point me where I can go do some more reading.
I've seen references to 4.4 being in alpha status.
The "client/local" directory is "standard" Xymon (as long as you create the directory), but it has the side-effect of making section names with the "local" prefix. There are probably good reasons for this prefixing (I'm guessing to prevent collisions with standard section names), so if it works for you, that's great.
Yep. Naming things and knowing where / how to start counting are three of the hardest things in computers.
My guess is that the next release of Xymon will include client/sections in the source, in addition to supporting client/local.
I'll keep that in mind.
A snippet from the README in client/sections/ from the Terabithia package:
"This directory is intended for use with automatically deployed files. Local scripts for this host should be placed in the client/local/ directory instead."
/me chuckles to himself muttering something about "/" vs "/usr" vs "/usr/local" vs "/opt".
Exactly. The client is as simple as it can be, so that it doesn't need constant tweaking. The server does all the smarts, including parsing, graphing, thresholding, so that the client host can spend more cycles doing its main function.
:-)
You're well on your way, so well done! Thank you.
Feel free to ask more questions on this list. The more questions the better our shared knowledge! (Although it's easier to respond to if each question is in a separate thread.)
ACK
I tend to try to keep any given thread relatively contained and start a new thread as needed.
-- Grant. . . . unix || die
participants (2)
-
gtaylor＠tnetconsulting.net
-
jeremy＠laidman.org