On Monday, 26 October 2009 20:55:15 shea_greg at emc.com wrote: Hi all,
First off, sorry for the long post, I'm trying to supply as much data as possible for analysis.
I have a single Hobbit server with approximately 3500 hosts, a mixture of windows and unix, some DB tests,
some BEA tests and a few custom tests. I have over 70000 RRD files which seems to be causing Hobbit performance
problems, most specifcally clock offset. I have a cron job that restarts Hobbit every 30 minutes otherwise the offset
grows so large it eats all memory and OOM kill starts. NTP is fine, it seems to be the time it takes for Hobbit to process
the client data. OS resides on RAID1 146GB drives SAS 15K RPM, second drive for RRDs is a single 300GB SAS 15K RPM.
At the end is a graph showing the clock offset. What else can I try?
Add more spindles.
70 000 RRD files will result in a minimum of 233 IOPS (assuming they are all being updated at 5-minute intervals). The EMC people I've spoken to say a 15k FC disk shouldn't really be averaging much more than 180 IOPS, 15k SAS or 15k SCSI wouldn't be any better. The 311 you seem to be doing isn't significant overhead for the minumum of 233, so it is unlikely that any tuning will help.
If you can't add spindles, you could look at the 4.3 branch, which has some features that allow scaling out to more hosts, or streamlining RRD writes (which may allow you to lose the clock offset, but will likely not reduce the load average much).
Regards, Buchan
Hi Buchan,
Thanks for your response. I bounced around the idea of external storage, but even here at EMC there is a cost associated with external storage, that's why I tried the second drive. I've read about the enhancements in 4.3, but thought I should upgrade from RH 4.7 to RH 5.3 first (RH is the official supported Linux) as there were IO improvements in the kernel. I also tried a newer version of RRD 1.2.30 and 1.3.8. RRD 1.3.8 doesn't work Hobbit 4.2.
On to the storage requisition process....
Thanks -Grs- Gregory R Shea EMC Corporation