hi all,

do you have a lot of memory ? If yes you can create tmpfs for your rrd and sync the tmpfs every couple hours.

We do it for a little hobbit server with 3500+ devices. For rrd hist and www dir.

Regards

Olivier AUDRY

----- Mail Original ----- De: "shea greg" <shea_greg at emc.com> À: bgmilne at staff.telkomsa.net, hobbit at hswn.dk Cc: "shea greg" <shea_greg at emc.com> Envoyé: Mardi 27 Octobre 2009 14h24:47 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne Objet: RE: [hobbit] performance help needed

...

On Monday, 26 October 2009 20:55:15 shea_greg at emc.com wrote: Hi all,

First off, sorry for the long post, I'm trying to supply as much data as possible for analysis.

I have a single Hobbit server with approximately 3500 hosts, a mixture of windows and unix, some DB tests,

some BEA tests and a few custom tests. I have over 70000 RRD files which seems to be causing Hobbit performance

problems, most specifcally clock offset. I have a cron job that restarts Hobbit every 30 minutes otherwise the offset

grows so large it eats all memory and OOM kill starts. NTP is fine, it seems to be the time it takes for Hobbit to process

the client data. OS resides on RAID1 146GB drives SAS 15K RPM, second drive for RRDs is a single 300GB SAS 15K RPM.

At the end is a graph showing the clock offset. What else can I try?

Add more spindles.

70 000 RRD files will result in a minimum of 233 IOPS (assuming they are all being updated at 5-minute intervals). The EMC people I've spoken to say a 15k FC disk shouldn't really be averaging much more than 180 IOPS, 15k SAS or 15k SCSI wouldn't be any better. The 311 you seem to be doing isn't significant overhead for the minumum of 233, so it is unlikely that any tuning will help.

If you can't add spindles, you could look at the 4.3 branch, which has some features that allow scaling out to more hosts, or streamlining RRD writes (which may allow you to lose the clock offset, but will likely not reduce the load average much).

Regards, Buchan

Hi Buchan,

Thanks for your response. I bounced around the idea of external storage, but even here at EMC there is a cost associated with external storage, that's why I tried the second drive. I've read about the enhancements in 4.3, but thought I should upgrade from RH 4.7 to RH 5.3 first (RH is the official supported Linux) as there were IO improvements in the kernel. I also tried a newer version of RRD 1.2.30 and 1.3.8. RRD 1.3.8 doesn't work Hobbit 4.2.

On to the storage requisition process....

Thanks -Grs- Gregory R Shea EMC Corporation