On Tue, Aug 28, 2007 at 09:26:51AM -0400, Sean R. Clark wrote:
I have 18,102 RRD's, 17,671 of which are controlled by the hobbitd_channel (the others are written/populated from other sources)
The slice I have the data on has a busy% between 16-88% depending on what's going on (so yes, high I/O as well)
OK, then I'd suggest that you pick up the current snapshot of Hobbit from http://www.hswn.dk/beta/ and build that. The only parts you need to replace in your current setup are these binaries:
- hobbitd/hobbitd_channel
- hobbitd/hobbitd_rrd
- web/hobbitgraph.cgi
After running "make", shutdown Hobbit and copy these files to your ~hobbit/server/bin/ directory (it's probably wise to save the original ones first). Then start Hobbit again, and everything should be working fine - with a lot less I/O load, and no memory leak in hobbitd_channel.
What's changed internally is that updates of the RRD files are now cached for up to 30 minutes before being written to disk; the RRDtool library can handle "batch" updates of the data, so instead of updating the RRD file with 1 dataset every 5 minutes, it now gets 6 datasets in one operation every 30 minutes.
This also means that when you shutdown Hobbit, you'll see that the hobbitd_rrd process takes quite a long time to finish - it is busy writing all of the cached updates to disk. On my work server, this takes about 5 minutes.
Regards, Henrik