Shawn On Wed, Mar 23, 2016 at 9:47 AM Shawn Heisey <hobbit at elyograg.org> wrote:
I see several graphs like this in xymon (4.3.23), where at least one of the port graphs (configured in analysis.cfg) has lots of data points missing:
https://www.dropbox.com/s/l483lilzh5n610k/flaky-xymon-port-graph.png?dl=0
Any idea how I can fix this? As you can see, the red line for http has data for the entire graph, while the blue line for ajp is very spotty. I just added the last two monitors less than an hour ago, so I don't have history on those yet.
This is not unique to this host. As far as I can tell, the port listings coming back from the client are always good, even when the graph has no line.
Thanks, Shawn
Firstly, I'm assuming you have setup port tracking with "PORT bla bla STATE=ESTABLISHED TRACK=ajp" or similar in analysis.cfg. It might be helpful to provide the relevant config section from there. I suppose there are three likely points of failure in the process of getting port data into the graphs. The first one is where the port data is received by Xymon and passed to the RRD parser xymond_rrd. The second is where the port lists are analysed to get the numbers. The third is where the numbers are added to the RRD file. If your "ports" tests show "red" or "clear" then that would suggest the first part is failing. If there are problems with the second or third parts, you may see useful messages logged in the rrd-status.log file. You could check to see if the port lists are being sent to the xymond_rrd process that handles status messages (and writes to the rrd-status.log file). If they are, then you'll know there's probably something amiss with the RRD analysis or update systems. Otherwise, if there are gaps in the messages being sent to the status channel, then it indicates that the RRD updating is probably fine. You can take a feed of the port status messages coming out of the status channel, so that you see everything that gets sent to xymond_rrd. First switch to the xymon user, and then run xymoncmd to setup your environment. Then run a command such as the following: $ xymond_channel --channel=status --filter='\|fremont\|ports\|' cat This will only display status channel messages for the host "fremont" and for the test name "ports". I'm using "cat" as the worker process, but you can substitute this with a grep, awk or sed command to reduce screen output as desired, or even write a script to parse the output. If you only want to know that a status message has been sent (and don't care about its contents), you could replace "cat" with "grep '^@@'", but for a first pass, "cat" works just fine. You can extend this idea to see what xymond_rrd is doing when it receives these messages, by running your own copy of xymond_rrd, with debugging enabled. Something like so: $ mkdir /tmp/myrrd $ xymond_channel --channel=status --filter='\|fremont\|ports\|' xymond_rrd --rrddir=/tmp/myrrd/ --debug This produces lots of output as xymond_rrd loads its configuration when it receives the first message, but then the output rate drops off to a few lines for each message. Due to buffering of the RRD file updates, it's tricky to debug whether xymond_rrd is sending updates to the RRD files. Cheers Jeremy