Hi all
I'm having some difficulty with a process counting/alerting problem for a
Linux client (Xymon client v4.3.27). I know the cause, but can't find a
good solution, short of hacking the code, or requesting a feature
enhancement.
The "ps" command produces output that shows child processes in a kind-of
tree structure ("forest" mode). So I sometimes see this (with some columns
removed for brevity):
3636 1 root 21720 xinetd -stayalive
29956 3636 nobody 21720 \_ xinetd -stayalive
For xinetd, usually there's only the "master" process. Occasionally there
is one extra like the above, and sometimes more than this. These are simply
child xinetd processes that haven't yet performed an exec of the real
worker process, so nothing unusual.
>From my perspective, the correct number of "master" xinetd processes is 1.
We don't want to have a second master instance of xinetd running. However,
if I add a PROC entry to analysis.cfg for xinetd, alerting when not 1, any
child processes are counted, and this causes an alert that I don't want.
This should be quite easy to solve. I just need to use a regular expression
that only matches when "xinetd" is at the start of the command string. So I
setup analysis.cfg as follows:
HOST=*
PROC "%^xinetd" 1 1 red
Awesomely, this worked!
But only when the Xymon server was v4.3.10. It seems that in v4.3.23, when
the "forest" option for "ps" was introduced in the client scripts, so was
some code to strip out the leading "\_" characters before matching. This
means that by the time the counting routine is reached, all of these lines
from "ps" look identical.
My work-around is to simply increase the "maximumcount" from 1 to 10 or
some arbitrary number (perhaps based on some historical data). But it's not
ideal. I'd like to be able to tell when a process is a child or a parent
(PPID=1) process and select for those when counting and thresholding. I
can't think of any other way to solve this without reverting xymonclient to
pre-v4.3.23.
I thought about this for a while, pondering the idea of adding a PPID=1
option, or some other way of specifying other fields from the "ps" output.
Then I realised that it might generally be very helpful to be able to
optionally match against the full line from the "ps" output, and not just
the "CMD" field or some other pre-defined field. For example, if I could
append a FIELD=all parameter to the PROC line in analysis.cfg, which allows
me to match anywhere along the "ps" output line, I could not only
match/exclude the "forest" characters, but also the username or the PPID
number or the time started or anything else that's interesting to me. The
fact that the counting routine add_count() is the same for proc, disk and
inode, means that I could do similar things there - such as using the
"IGNORE" keyword for any "df" output lines that started with
"/dev/myspecialfs". The same type of feature for PORT could allow me to
match UDP ports only, whereas I don't believe this is currently possible.
What are peoples' thoughts on this idea? Is there a more obvious solution I
haven't thought of yet?