You know the data exists because you used the rrd dump tool to display it? Um, no, sorry I didn't explain myself very well. I know the data is being passed back to the Hobbit server because it is displayed in the web page (see example in original email below). The issue it that the graph that is displayed at the bottom of the page is empty.
I know that data for the other identical servers is being passed for the same reason, but on those pages the graphed data IS visible.
I have compared the page sources and they look identical, which is why I think it's the data.
Looking at the data files I am getting somewhat confused (or am typing commands wrong), please confirm my thoughts on this:
- When numeric data is transferred from the Hobbit client to the server it is stored in an rrd file thanks to the TEST2RRD config option, mine looks like: TEST2RRD="cpu=la,disk,inode,qtree,memory,postfixqueue=ncv,postfixdeliver ies=ncv".
- The RRD file in 1 takes its name from the test that the client specifies when it transfers its data.
- The directory path for where the data is stored is specified in the hobbitd channel=data using the --rrddir= parameter (/opt/hobbit/data/rrd for me).
- The actual path of the RRD data file is the directory specified in 3, plus the fully qualified host name (mc25.lon.dcn.colt.net) plus the RRD filename specified in 2, giving me "/opt/hobbit/data/rrd/mc25.lon.dcn.colt.net/postfixqueue.rrd".
- When I run "rrdtool --dump" on this file I should get real data, and it is from this data that ALL the graphs are generated, i.e. this file can hold data for up to 576 days.
Now this is where things go wrong:
For both the machine that is failing and for any machine that is working I get nothing when I run: $ rrdtool --dump /opt/hobbit/data/rrd/mc25.lon.dcn.colt.net/postfixqueue.rrd or $ rrdtool --dump /opt/hobbit/data/rrd/mc25.lon.dcn.colt.net/memory.rrd
yet I can view graphs for the memory usage on this machine going back for as long as I have had this server up, so I now have no idea what these RRD files are for.
The files are being updated regularly and the have a quantity of data in the (18KB or so).
So, after all of this I am confused about how and where the data is stored although I can say that the trends show that data was successfully stored for about two months after the service was first started, but then it stopped. Being that that was back in February I have no idea what changes might have happened to the servers.
8-(
|\/|artin
-----Original Message-----
From: Hubbard, Greg L [mailto:greg.hubbard at eds.com]
Sent: 24 July 2008 14:34
To: hobbit at hswn.dk
Subject: RE: [hobbit] Graphs are missing data, but it's there!
You know the data exists because you used the rrd dump tool todisplay it? Is the graph simply not shown at all, or is there a "hole" in the Web page where it normally would go? ("show page source" might have a clue). Some ideas/shots in the dark: a) check the logs b) meticulously compare a "working" system to the non-working system, and make sure that they really are identical. c) look at the trends page for this host to see if the graph is okay there... Etc. I am sure you know the drill -- a big pain to look under every rock, but it has to be done... GLH
From: Ward, Martin [mailto:Martin.Ward at colt.net]
Sent: Thursday, July 24, 2008 8:21 AM
To: hobbit at hswn.dk
Subject: RE: [hobbit] Graphs are missing data, but it'sthere! Thanks for the suggestion but that didn't work (I guess you meant rrd). Any other ideas? |\/|
-----Original Message-----
From: Roberts, James[mailto:James.Roberts at hants.gov.uk] Sent: 24 July 2008 12:47 To: hobbit at hswn.dk Subject: RE: [hobbit] Graphs are missing data, but it's there! you need to touch all the rdd.
From: Ward, Martin [mailto:Martin.Ward at colt.net]
Sent: 24 July 2008 12:43
To: hobbit at hswn.dk
Subject: [hobbit] Graphs are missing data, butit's there!
All,
I have a problem with one machine where its datais not being shown in the graphs even though the data exists.
The machine in question's Hobbit client sendsfive pieces of numeric data (email queues) and these are displayed on the web page for this service:
====
Thursday July 24 11:29:11 UTC 2008
ActiveStatus: green<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif> ActiveQueue: 106 ActiveTrend: tendency rising with -60 mails. BounceStatus: green <http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif> BounceQueue: 58 BounceTrend: tendency falling with 3 mails. DeferStatus: red <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif> DeferQueue: 150464 DeferTrend: tendency falling with 95 mails. CorruptStatus: green <http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif> CorruptQueue: 0 CorruptTrend: amount equal to last measure. IncomingStatus: red <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif> IncomingQueue: 247049 IncomingTrend: amount equal to last measure. Deferred Queue is too high but is decreasing already.
====
These numbers change over time and the valuesare accurate.
However, the graph that is displayed below thisdata is blank. I have historic data, the files exist, and what is more I have other machines that are configured identically to this one where the data IS graphed correctly.
Hobbit graphs are a bit of a black hole to me,can anyone suggest where I might look?
|\/|artin
The message is intended for the named addresseeonly and may not be disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security at colt.net and delete the message and any attachments without retaining any copies. Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party. Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.
The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security at colt.net and delete the message and any attachments without retaining any copies.
Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.
No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.
Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.