Hobbit Average Data
Hello,
I've got some folks here that want to keep data all the time.
ie. They don't want to use a Round Robin Database that averages
the data over time. They want the 5 minute data forever.
I know we can get into a debate as to the reality of this, but
it's want the performance team here wants. If hobbit can't do it,
they want me to use sar and input to another database.
Is there a way to do this with Hobbit?
Anyone else had this come up?
Thanks..James
James Wade
James Wade wrote:
Hello,
I’ve got some folks here that want to keep data all the time.
ie… They don’t want to use a Round Robin Database that averages
the data over time. They want the 5 minute data forever.
I know we can get into a debate as to the reality of this, but
it’s want the performance team here wants. If hobbit can’t do it,
they want me to use sar and input to another database.
Is there a way to do this with Hobbit?
Anyone else had this come up?
I went through an exercise several years ago, where I made some tweaks to big brother to log alerts to a mysql database, in addition to the other alerting mechanisms. I declared it a success, and then forgot about it since management wasn't really interested in the solution, since they were not aware of any problem. At any rate, it was fairly quick and easy to whip up something with perl and DBI.
Joe
On Mon, Nov 26, 2007 at 02:49:27PM -0600, James Wade wrote:
I've got some folks here that want to keep data all the time. ie. They don't want to use a Round Robin Database that averages the data over time. They want the 5 minute data forever.
I know we can get into a debate as to the reality of this, but it's want the performance team here wants. If hobbit can't do it, they want me to use sar and input to another database.
Is there a way to do this with Hobbit?
Not out-of-the-box, but doing it wouldn't require a lot of work. I assume you want to store the data that Hobbit normally extracts and puts into the RRD files ? I.e. you want to save e.g. just the current utilisation of the /home filesystem (96%, usually), and not the full status message ?
In that case, you will want to make some changes to the hobbit-4.2.0/hobbitd/do_rrd.c file, specifically the "create_and_update_rrd()" routine in there. This function handles ALL of the RRD updating, so it is the place you want to put your code to grab the RRD data and save somewhere permanent. In that routine, it ends up calling the rrd_update() function in the RRD library: result = rrd_update(pcount, updparams); "updparams" is a table of "pcount" strings, which are really the same strings that you would feed into an "rrd update" command that you run from the commandline. See the rrdupdate(1) man-page - Hobbit always uses the "-t" option to specify which datasets are being updated and the sequence in which the datavalues is provided (for RRD files that contain more than one set of data).
The simplest solution is probably to setup a pipe to a child process, and then write a line with the RRD filename, the template and the data to that pipe - the child process can then take care of splitting the data into individual records, and store them in the database.
Henrik
On Mon, Nov 26, 2007 at 10:22:01PM +0100, Henrik Stoerner wrote:
The simplest solution is probably to setup a pipe to a child process, and then write a line with the RRD filename, the template and the data to that pipe - the child process can then take care of splitting the data into individual records, and store them in the database.
There's an implementation of this in the current snapshot, including a new hobbitd_rrd manpage that describes how to run it, and what the input to your custom script looks like:
--processor=COMMAND Feed the raw RRD data into COMMAND via standard input. COMMAND must be a simple command with no options; if necessary, COMMAND can be a script that runs the real command with the necessary options. The data sent to COMMAND consists of lines of text of the form dsname1[:dsname2] time:value1[:value2] rrdtype [key1] [key2] First the dataset names, corresponding to the DS definitions in the RRD files. Next the timestamp of the update, followed by the values in the same order as the dsname-list. Third item is the type of RRD file. After that there may be between 0-2 addi- tional keys: For "disk" this is the filesystem mountpoint, for "tcp" tests it is the service name, for "tcp" "http" tests there is also the URL checked. Note that filesystem names and URLs have forwardslash substituted by comma. See the hobbitd/rrd/*.c sourcecode for details, look for the "setupfn" function calls. An example:
la 1196115478:7 la
la 1196115478:169 procs
la 1196115478:1 users
la 1196115478:244 clock
pct:used 1196115478:63:62338144 disk ,root
realmempct 1196115478:76 memory real
realmempct 1196115478:0 memory swap
realmempct 1196115478:20 memory actual
sec 1196115478:0.03 tcp conn
sec 1196115478:1.38 tcp http osiris.hswn.dk,Regards, Henrik
I'll ask a question first, what is it that is wrong with the current implementation. If the "people" want the ability to zoom into a five minute period from a year back, then you need to modify the rrd files.
We had a sort of similar requirement for capacity planning in that for servers that peak during the day, and are completely idle at night. After a period of time, the average down to one sample per day would not give a good indication of how the server is performing over time. What we did was create an rrd database that stored MAX values, that way when the data is reduced to one sample per day, it was the max value for the day. We only did this for a 15 minute uptime value chart and from memory, we set up a script that created the rrd file to our spec the first time a sample flowed through based on current functionality within hobbit to add custom processing for different alerts. I don't have access to the system right now to check how it was done.
Cheers
Phil
On 27/11/2007, Henrik Stoerner <henrik at hswn.dk> wrote:
On Mon, Nov 26, 2007 at 10:22:01PM +0100, Henrik Stoerner wrote:
The simplest solution is probably to setup a pipe to a child process, and then write a line with the RRD filename, the template and the data to that pipe - the child process can then take care of splitting the data into individual records, and store them in the database.
There's an implementation of this in the current snapshot, including a new hobbitd_rrd manpage that describes how to run it, and what the input to your custom script looks like:
--processor=COMMAND Feed the raw RRD data into COMMAND via standard input. COMMAND must be a simple command with no options; if necessary, COMMAND can be a script that runs the real command with the necessary options. The data sent to COMMAND consists of lines of text of the form dsname1[:dsname2] time:value1[:value2] rrdtype [key1] [key2] First the dataset names, corresponding to the DS definitions in the RRD files. Next the timestamp of the update, followed by the values in the same order as the dsname-list. Third item is the type of RRD file. After that there may be between 0-2 addi- tional keys: For "disk" this is the filesystem mountpoint, for "tcp" tests it is the service name, for "tcp" "http" tests there is also the URL checked. Note that filesystem names and URLs have forwardslash substituted by comma. See the hobbitd/rrd/*.c sourcecode for details, look for the "setupfn" function calls. An example:
la 1196115478:7 la la 1196115478:169 procs la 1196115478:1 users la 1196115478:244 clock pct:used 1196115478:63:62338144 disk ,root realmempct 1196115478:76 memory real realmempct 1196115478:0 memory swap realmempct 1196115478:20 memory actual sec 1196115478:0.03 tcp conn sec 1196115478:1.38 tcp http osiris.hswn.dk,Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Tel: 0400 466 952 Fax: 0433 123 226 email: philwild at gmail.com
Hi Phil,
On Tue, Nov 27, 2007 at 03:08:22PM +0900, Phil Wild wrote:
I'll ask a question first, what is it that is wrong with the current implementation. If the "people" want the ability to zoom into a five minute period from a year back, then you need to modify the rrd files.
Good question.
There's really nothing wrong with the current implementation, but it does have some limitations. You *can* tune the RRD definitions to save more 5-minute measurements, or store MIN/MAX values (in fact, the snapshots have had the capability for some time to configure the RRD file layout so you don't have to dig into the C code).
The main limitation is that RRD will *always* normalize data. It is quite rare for updates to happen exactly at 300 second intervals, and when they are slightly off then RRD will not store the raw data; instead, it will compute what the value probably would have been, if the measurement had happened exactly at the 300-second mark. So you really cannot get the exact measurements (the observed data) from the RRD files.
In most cases this doesn't matter since the differences are small, and measurements do happen roughly at the "right" time. But it is something you should consider when trying to tap into the Hobbit measurements.
The main reason I added this was because it would allow users an easy hook into the raw data that Hobbit collects. The data that is fed into the RRD files is extremely standardized, as opposed to most of the other data available in Hobbit. So if you do want to add some custom processing of monitoring data, this is a good place to do it. And I could see some uses for it myself.
Hobbit is intended to provide a good basis for your monitoring, but it cannot solve everyones needs. If I can add a few lines of code that lets users connect their own custom tools so they are fed by the data that Hobbit collects, then there is a much better chance that they will stick with Hobbit instead of picking some other tool.
So I added it, because it makes Hobbit much more flexible and allows you to make more use of the data that Hobbit collects.
Regards, Henrik
On Monday 26 November 2007, James Wade wrote:
Hello,
I've got some folks here that want to keep data all the time. ie. They don't want to use a Round Robin Database that averages the data over time. They want the 5 minute data forever.
I know we can get into a debate as to the reality of this, but it's want the performance team here wants. If hobbit can't do it, they want me to use sar and input to another database.
Is there a way to do this with Hobbit?
Anyone else had this come up? I created a perl script that updates all rrd files. It adds MAX and MIN. It also adds more data point. It uses more space to store, but the MAX and MIN can be handy to find out peaks. The extra data points are handy to zoom in on older data.
Stef
participants (5)
-
henrik＠hswn.dk
-
jkwade＠futurefrontiers.com
-
joe＠tmsusa.com
-
philwild＠gmail.com
-
stef.coene＠docum.org