RRD Graph stopped graphing
I had two custom RRD graphs setup and running fine on client systems.
The script that collects the data was running every minute. After I
finished testing, I changed the frequency of the scripts on the clients
to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph
stopped displaying data. It ran like this for a couple days. I changed
clientlaunch.cfg back to 1 minute... and the graphs immediately started
graphing again. Is there a minimum time frame that is needed, or
something else that must be done? What am I missing?
-- Michael Beatty
When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points.
Thanks, Larry Barber
On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <Michael.Beatty at sherwin.com
wrote:
I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing?
-- Michael Beatty
______________________________**_________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/**mailman/listinfo/xymon<http://lists.xymon.com/mailman/listinfo/xymon>
How do I do that?
Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374
On 12/18/2012 02:13 PM, Larry Barber wrote:
When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points.
Thanks, Larry Barber
On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <Michael.Beatty at sherwin.com <mailto:Michael.Beatty at sherwin.com>> wrote:
I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty _______________________________________________ Xymon mailing list Xymon at xymon.com <mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon
man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval.
Thanks, Larry Barber
On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <Michael.Beatty at sherwin.com>wrote:
How do I do that?
Michael Beatty Sherwin-Williams IT Analyst/Developermichael.beatty at sherwin.com216-515-7374
On 12/18/2012 02:13 PM, Larry Barber wrote:
When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points.
Thanks, Larry Barber
On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty < Michael.Beatty at sherwin.com> wrote:
I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing?
-- Michael Beatty
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
It seems that any frequency above 10 minutes doesn't work. From what I understand, this has to do with the step and heartbeat of the rrd. It looks like by default the step is 300 seconds and the heartbeat is 600 seconds (10 minutes). So, as long as it doesn't go above 10 minutes it will accept data, anything past 10 minutes will be tossed which would result in me not getting graphs.
So, in reading the man pages, I figured out that I can override the 600 second default by appending it to my ENV tags in xymonserver.cfg NCV_MyAlert="MyAlert:GAUGE:900"
At first glance, this worked perfectly. I deleted the rrd file, restarted xymon, did a rrd dump and the XML was now showing a heartbeat of 900 (15 minutes). So, I set my script to run every 15 minutes expecting a giant rainbow to arc over my screen, but, it still doesn't work.
My XML looks like this: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd"> <!-- Round Robin Database Dump --><rrd> <version> 0003 </version> <step> 300 </step> <!-- Seconds --> <lastupdate> 1355940819 </lastupdate> <!-- 2012-12-19 13:13:39 EST -->
<ds>
<name> Shortterm </name>
<type> GAUGE </type>
<minimal_heartbeat> 900 </minimal_heartbeat>
<min> 6.0000000000e+02 </min>
<max> NaN </max>
<!-- PDP Status -->
<last_ds> 1.080055 </last_ds>
<value> NaN </value>
<unknown_sec> 219 </unknown_sec>
</ds>My rrd info:
Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374
On 12/18/2012 06:11 PM, Larry Barber wrote:
man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval.
Thanks, Larry Barber
On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <Michael.Beatty at sherwin.com <mailto:Michael.Beatty at sherwin.com>> wrote:
How do I do that? Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com <mailto:michael.beatty at sherwin.com> 216-515-7374 <tel:216-515-7374> On 12/18/2012 02:13 PM, Larry Barber wrote:When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <Michael.Beatty at sherwin.com <mailto:Michael.Beatty at sherwin.com>> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty _______________________________________________ Xymon mailing list Xymon at xymon.com <mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon
I haven't done this myself but my coworker has used it to fix graphing on one of my scripts that went from every 5 minutes to every 15 minutes:
rrdtool tune your.rrd --heartbeat lambda:1200 # I don't know why he used 1200 since it seems like you would use 900
Regardless try man rrdtool to see the details.
~David
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Wednesday, December 19, 2012 13:38 To: Larry Barber Cc: xymon at xymon.com Subject: Re: [Xymon] RRD Graph stopped graphing
It seems that any frequency above 10 minutes doesn't work. From what I understand, this has to do with the step and heartbeat of the rrd. It looks like by default the step is 300 seconds and the heartbeat is 600 seconds (10 minutes). So, as long as it doesn't go above 10 minutes it will accept data, anything past 10 minutes will be tossed which would result in me not getting graphs.
So, in reading the man pages, I figured out that I can override the 600 second default by appending it to my ENV tags in xymonserver.cfg NCV_MyAlert="MyAlert:GAUGE:900"
At first glance, this worked perfectly. I deleted the rrd file, restarted xymon, did a rrd dump and the XML was now showing a heartbeat of 900 (15 minutes). So, I set my script to run every 15 minutes expecting a giant rainbow to arc over my screen, but, it still doesn't work.
My XML looks like this: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd"> <!-- Round Robin Database Dump --><rrd> <version> 0003 </version> <step> 300 </step> <!-- Seconds --> <lastupdate> 1355940819 </lastupdate> <!-- 2012-12-19 13:13:39 EST -->
<ds> <name> Shortterm </name> <type> GAUGE </type> <minimal_heartbeat> 900 </minimal_heartbeat> <min> 6.0000000000e+02 </min> <max> NaN </max>
<!-- PDP Status --> <last_ds> 1.080055 </last_ds> <value> NaN </value> <unknown_sec> 219 </unknown_sec> </ds>
My rrd info:
Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374 On 12/18/2012 06:11 PM, Larry Barber wrote: man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval.
Thanks, Larry Barber On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <Michael.Beatty at sherwin.com> wrote: How do I do that?
Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374 On 12/18/2012 02:13 PM, Larry Barber wrote: When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points.
Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <Michael.Beatty at sherwin.com> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing?
-- Michael Beatty
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
I figured it out. And I believe I found a bug in the process.
Xymon uses a default step of 300 seconds with a heart beat of 600 seconds (10 mintues). If RRD doesn't get a status in under 10 minutes, it'll take it as an unknown. In order to override the defaults, you need set the values in xymonserver.cfg. This builds the the data set part of the rrd. From the documentation:
NCV_slab="inodecache:GAUGE,dentrycache:GAUGE"
I figured out early on that you can add in the heartbeat. ie. NCV_slab="inodecache:GAUGE:1800,dentrycache:GAUGE:1800"
This sets the heartbeat for those two data sets to be 1800 seconds (30 minutes). What I failed to figure out early on is that xymon doesn't like this. For some reason (I would consider this a bug) if you supply your own paramters to override the default, xymon still sticks its default value into the rrdcreate string which ultimately left it to be: NCV_slab="inodecache:GAUGE:1800:600"
So, when the create tool executed, it took the default heartbeat value (600) and added it to the end of the create string. This set the MINIMUM value, which you can clearly see in the XML I have posted below. I failed to see this. This is where my problem lied. My values were fractions of seconds... rrd needed a value greater than 600, so it was throwing out my data.
In order to fix the issue I had to explicitly provide all of the rrdcreate parameters: NCV_slab"iinodecache:GAUGE:1800:0:U"
Also, another thing to note to make it work, you also need to put an entry into rrddefinitions for these records because this is where the step parameter is set: [slab] -s 900 RRA:AVERAGE:0.5:1:576
If this isn't a bug, it should at least be documented in the "Custom Graphs" section.
Michael Beatty
On 12/19/2012 10:25 PM, Gore, David W (David) wrote:
I haven't done this myself but my coworker has used it to fix graphing on one of my scripts that went from every 5 minutes to every 15 minutes:
rrdtool tune your.rrd --heartbeat lambda:1200 # I don't know why he used 1200 since it seems like you would use 900
Regardless try man rrdtool to see the details.
~David
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Wednesday, December 19, 2012 13:38 To: Larry Barber Cc: xymon at xymon.com Subject: Re: [Xymon] RRD Graph stopped graphing
It seems that any frequency above 10 minutes doesn't work. From what I understand, this has to do with the step and heartbeat of the rrd. It looks like by default the step is 300 seconds and the heartbeat is 600 seconds (10 minutes). So, as long as it doesn't go above 10 minutes it will accept data, anything past 10 minutes will be tossed which would result in me not getting graphs.
So, in reading the man pages, I figured out that I can override the 600 second default by appending it to my ENV tags in xymonserver.cfg NCV_MyAlert="MyAlert:GAUGE:900"
At first glance, this worked perfectly. I deleted the rrd file, restarted xymon, did a rrd dump and the XML was now showing a heartbeat of 900 (15 minutes). So, I set my script to run every 15 minutes expecting a giant rainbow to arc over my screen, but, it still doesn't work.
My XML looks like this: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd"> <!-- Round Robin Database Dump --><rrd> <version> 0003 </version> <step> 300 </step> <!-- Seconds --> <lastupdate> 1355940819 </lastupdate> <!-- 2012-12-19 13:13:39 EST -->
<ds> <name> Shortterm </name> <type> GAUGE </type> <minimal_heartbeat> 900 </minimal_heartbeat> <min> 6.0000000000e+02 </min> <max> NaN </max> <!-- PDP Status --> <last_ds> 1.080055 </last_ds> <value> NaN </value> <unknown_sec> 219 </unknown_sec> </ds>My rrd info:
Michael Beatty
On 12/18/2012 06:11 PM, Larry Barber wrote: man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval.
Thanks, Larry Barber On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <Michael.Beatty at sherwin.com> wrote: How do I do that?
When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points.
Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <Michael.Beatty at sherwin.com> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing?
Thank you for your help, but it isn't working. I've been playing around with everything I can think of, but I cannot get the graphs to display if I collect data in frequencies greater than 10 minutes. I was briefly looking through the code and I found in do_rrd.c that the default interval for RRD is 300 seconds (5 minuets). I cannot figure out a way to change this. The closest I have come to getting this to work was to add a heartbeat value to the NCV_ setting in xymonserver.cfg file to: NCV_mytest=dataset:GAUGE:1800. Then I was able to dump the XML using rrdtools, manually update the STEP to 900, and reload the rrd. When I do so, I get a strange behavior that makes me think the 300 value that is hardcoded overrides the changes I made. When the first report comes in, the rrd file is created, the second report grabs the correct data, then the third report the last_ds value is populated with the previously correct data and the current value is set to "NaN" and that is the way it stays for all subsequent tests.
Ultimately my RRD looks like:
<?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd"> <!-- Round Robin Database Dump --><rrd> <version> 0003 </version> <step> 900 </step> <!-- Seconds --> <lastupdate> 1356032234 </lastupdate> <!-- 2012-12-20 14:37:14 EST -->
<ds>
<name> Shortterm </name>
<type> GAUGE </type>
<minimal_heartbeat> 1800 </minimal_heartbeat>
<min> 6.0000000000e+02 </min>
<max> NaN </max>
<!-- PDP Status -->
<last_ds> 1.080055 </last_ds>
<value> NaN </value>
<unknown_sec> 434 </unknown_sec>
</ds>
<ds>
<name> Longterm </name>
<type> GAUGE </type>
<minimal_heartbeat> 1800 </minimal_heartbeat>
<min> 6.0000000000e+02 </min>
<max> NaN </max>
<!-- PDP Status -->
<last_ds> 1.080055 </last_ds>
<value> NaN </value>
<unknown_sec> 434 </unknown_sec>
</ds>Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com 216-515-7374
On 12/18/2012 06:11 PM, Larry Barber wrote:
man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval.
Thanks, Larry Barber
On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <Michael.Beatty at sherwin.com <mailto:Michael.Beatty at sherwin.com>> wrote:
How do I do that? Michael Beatty Sherwin-Williams IT Analyst/Developer michael.beatty at sherwin.com <mailto:michael.beatty at sherwin.com> 216-515-7374 <tel:216-515-7374> On 12/18/2012 02:13 PM, Larry Barber wrote:When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <Michael.Beatty at sherwin.com <mailto:Michael.Beatty at sherwin.com>> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty _______________________________________________ Xymon mailing list Xymon at xymon.com <mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon
participants (3)
-
david.gore＠verizon.com
-
lebarber＠gmail.com
-
Michael.Beatty＠sherwin.com