Hello,
I set up some custom graphs to monitor my Memcached instances. However, I appear to have run into an issue. I set up my xymonserver.cfg to collect all of my data into one NCV file, as evidenced by my TEST2RRD and my configuration below. I also believe I defined the datatypes appropriately. The issue that I am seeing is in the RRD file itself, when I use the RRDTOOL to dump the file. All of the values for the various datatypes are showing as NaN. Would someone be able to point me in the direction of what I could be doing wrong?
Thank you, Sheel
TEST2RRD="<other stuff>,memcache-test=ncv,MC-TFH-8K01-11221=ncv,MC-TFH-8K01-11222=ncv,MC-TFH-8K02-11221=ncv,MC-TFH-8K02-11222=ncv"
NCV_memcache-test="Hits:GAUGE,Misses:GAUGE,reqs:GAUGE,Gets:GAUGE,Sets:GAUGE,storage:GAUGE,items:GAUGE,hitratio:GAUGE,*:NONE" NCV_MC-TFH-8K01-11111="Hits:DERIVE,Misses:DERIVE,reqs:DERIVE,Gets:DERIVE,Sets:DERIVE,storage:DERIVE,items:DERIVE,hitratio:DERIVE,*:NONE"
On Tue, January 27, 2015 11:45 am, Sheel Shah wrote:
Hello,
I set up some custom graphs to monitor my Memcached instances. However, I appear to have run into an issue. I set up my xymonserver.cfg to collect all of my data into one NCV file, as evidenced by my TEST2RRD and my configuration below. I also believe I defined the datatypes appropriately. The issue that I am seeing is in the RRD file itself, when I use the RRDTOOL to dump the file. All of the values for the various datatypes are showing as NaN. Would someone be able to point me in the direction of what I could be doing wrong?
Thank you, Sheel
TEST2RRD="<other stuff>,memcache-test=ncv,MC-TFH-8K01-11221=ncv,MC-TFH-8K01-11222=ncv,MC-TFH-8K02-11221=ncv,MC-TFH-8K02-11222=ncv"
NCV_memcache-test="Hits:GAUGE,Misses:GAUGE,reqs:GAUGE,Gets:GAUGE,Sets:GAUGE,storage:GAUGE,items:GAUGE,hitratio:GAUGE,*:NONE" NCV_MC-TFH-8K01-11111="Hits:DERIVE,Misses:DERIVE,reqs:DERIVE,Gets:DERIVE,Sets:DERIVE,storage:DERIVE,items:DERIVE,hitratio:DERIVE,*:NONE"
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Sheel,
Many of our tests are using SPLITNCV, but that seems like it should be valid syntax.
Have you been doing development on this system? RRDtool will expect the template to match exactly and if the rrd file has a different set of DS's in it from its initial definition the values might not be being inserted properly.
You might want to try blowing away any existing RRD files for this type and letting xymond_rrd re-create them.
Also, can you try running xymond_rrd in --debug mode to see if any unusual errors are thrown on insertion?
HTH,
-jc
Hi J.C.
Thanks for your reply. Unfortunately, switching it to a SPLITNCV didn't have any effect. When I used the RRDTOOL to dump the contents of the RRD files, the data values are still showing as NaN. I confirmed the script I wrote is displaying the data in the new column on the Xymon homepage (column name is MC-TFH-8K01-11111). This is what is showing when I click the green icon to view the detail:
Hits : <number value> Misses : <number value> reqs : <number value> Gets : <number value> Gets per min : <number value> Sets per min : <number value> Sets : <number value> storage : <number value> items : <number value> hitratio : <percentage value> Used Connection : <percentage value>
Are the datatypes required to be uniquely named? In another column on the same host, I have similar data being displayed, with the same names. The graphing functions correctly in that column.
I apologize, but I am somewhat new to writing scripts in Xymon. I'm not familiar with the Xymon_rrd tool. Can you explain how to have it generate the rrd files and to run in debug mode?
Thanks, Sheel
-----Original Message----- From: J.C. Cleaver [mailto:cleaver at terabithia.org] Sent: Tuesday, January 27, 2015 9:19 PM To: Sheel Shah Cc: xymon at xymon.com Subject: Re: [Xymon] errors with custom graphs
On Tue, January 27, 2015 11:45 am, Sheel Shah wrote:
Hello,
I set up some custom graphs to monitor my Memcached instances. However, I appear to have run into an issue. I set up my xymonserver.cfg to collect all of my data into one NCV file, as evidenced by my TEST2RRD and my configuration below. I also believe I defined the datatypes appropriately. The issue that I am seeing is in the RRD file itself, when I use the RRDTOOL to dump the file. All of the values for the various datatypes are showing as NaN. Would someone be able to point me in the direction of what I could be doing wrong?
Thank you, Sheel
TEST2RRD="<other stuff>,memcache-test=ncv,MC-TFH-8K01-11221=ncv,MC-TFH-8K01-11222=ncv,MC-TFH-8K02-11221=ncv,MC-TFH-8K02-11222=ncv"
NCV_memcache-test="Hits:GAUGE,Misses:GAUGE,reqs:GAUGE,Gets:GAUGE,Sets:GAUGE,storage:GAUGE,items:GAUGE,hitratio:GAUGE,*:NONE" NCV_MC-TFH-8K01-11111="Hits:DERIVE,Misses:DERIVE,reqs:DERIVE,Gets:DERIVE,Sets:DERIVE,storage:DERIVE,items:DERIVE,hitratio:DERIVE,*:NONE"
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Sheel,
Many of our tests are using SPLITNCV, but that seems like it should be valid syntax.
Have you been doing development on this system? RRDtool will expect the template to match exactly and if the rrd file has a different set of DS's in it from its initial definition the values might not be being inserted properly.
You might want to try blowing away any existing RRD files for this type and letting xymond_rrd re-create them.
Also, can you try running xymond_rrd in --debug mode to see if any unusual errors are thrown on insertion?
HTH,
-jc
Sheel
Firstly let me suggest you delete the RRD file and confirm that it gets re-created, then wait 15 minutes for 3 samples to arrive, and dump it once more. If it's still not working, then more investigation is needed, but sometimes this fixes some problems, and helps to identify others.
Secondly, check the rrd-status.log file for indications of problems.
On 30 January 2015 at 08:34, Sheel Shah <Sheel.Shah at rd.com> wrote:
This is what is showing when I click the green icon to view the detail:
Hits : <number value>
Run the command "rrdtool info <filename.rrd> | grep last". This should give you some useful data.
Firstly, the timestamp of the last update should be shown. This is seconds-since-epoch so convert it to your current localtime with something like "perl -e 'print scalar localtime 1422583073'". Or run "date +%s" to show the current time, and calculate the difference. We'd expect the last_update time to update when you get more data.
Secondly, check that the "ds" names match what you're putting out, and that you have no extras or missing ones from your data. You should see, for example:
ds[Hits].last_ds = NaN
You might not get the NaN. Just make sure that everything else looks OK.
Are the datatypes required to be uniquely named?
Only per RRD file. If you have the same DS names for another test going to an RRD file with a different name, that's fine.
In another column on the same host, I have similar data being displayed, with the same names. The graphing functions correctly in that column.
Should be fine.
I apologize, but I am somewhat new to writing scripts in Xymon. I'm not familiar with the Xymon_rrd tool. Can you explain how to have it generate the rrd files and to run in debug mode?
No apology necessary. With the awesome flexibility of Xymon comes a degree of complexity, and using the extra features requires more than basic knowledge of how Xymon works.
The xymond_rrd process is run by the xymond_channel process, which in turn is run by xymonlaunch process - the master process that runs everything. You should know that there are two xymond_rrd processes, and two xymond_channel processes that run them, chained together. One chain is for data messages, and the other is for status messages. You are doing data collection from within test status messages (TEST2RRD), that that's the one relevant to your situation.
On the Xymon server, the configuration file that determines what runs and how it runs is tasks.cfg. Have a look in tasks.cfg and find the [rrdstatus] section. (There's a similar [rrddata] section, for the other channel, as discussed above.) This is what causes xymond_rrd to run for the status channel. You can edit tasks.cfg and add extra parameters to the command line, and what we want is to add the option to enable debugging. For example, you could update the [rrdstatus] section to look like this:
[rrdstatus]
ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=status
--log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd --debug
Notice how I added "--debug" to the end of the last line. After making the edit, simply kill xymond_rrd (eg "pkill xymond_rrd") and it will re-start with the new parameter. Use "pgrep -f -l ^xymond_rrd" to show the parameters being used by the two processes (one for data and one for status). If one now has "--debug" then you're OK.
The extra debug details will go into the log rrd-status.log. Have a look there for interesting details that might indicate a problem.
Cheers Jeremy
participants (3)
-
cleaver@terabithia.org
-
jlaidman@rebel-it.com.au
-
Sheel.Shah@rd.com