performance issue with multiple pagesets
Hi Henrik,
i loaded hobbit 4.0.5RC1 on my standby server (which isn't as fast as my production bbgen3.5 server..)
i'm having lots of iowait, (it isn't swapping), i suspect it's bbgen i have 8 pagesets, and my hobbit/www directory is over 20M... and generating all that every minute (sshhh.. i know..)
i'm using most of my pagesets only for the bb2.html, and i am not using the bb.html page at all.
here's what is looks for my unix pageset :
[root at bigbrother-stdby www]# ls -lh unix/ total 2.0M -rw-r--r-- 1 bb-dev alert 94K Jul 20 16:22 bb2.html -rw-r--r-- 1 bb-dev alert 1.8M Jul 20 16:22 bb.html -rw-r--r-- 1 bb-dev alert 519 Jul 20 16:22 bbnk.html lrwxrwxrwx 1 bb-dev alert 7 Jul 20 16:07 index.html -> bb.html [root at bigbrother-stdby www]#
--> Is there a way to disable the generating of the bb.html page for pagesets ?
-- Olivier Beau
On Wed, Jul 20, 2005 at 04:54:56PM +0200, Olivier Beau wrote:
i loaded hobbit 4.0.5RC1 on my standby server (which isn't as fast as my production bbgen3.5 server..)
i'm having lots of iowait, (it isn't swapping), i suspect it's bbgen
It would be interesting to see the output from your "bbgen" column if you run bbgen with the "--report" option.
I haven't seen bbgen do very much I/O. Sure, you're generating 8x2 MB of data per minute, but most disk today will easily deliver 10 mbit/sec throughput. And it's not random access, just sequential writing of a single file.
I'd be more inclined to think it's your RRD file updates that are responsible for the disk I/O you see. Try disabling the [rrddata] and [rrdstatus] tasks in hobbitlaunch.cfg and see if the iowait drops.
--> Is there a way to disable the generating of the bb.html page for pagesets ?
No.
Regards, Henrik
i've disabled [rrdstatus] and [rrddata]. i still have lots of I/O, practicaly no change.
i'm moved my www directory on another disk, and now most of the iowait is on the other disk. i then moved www in ramdisk, and all the iowait disapeared !
here is a paste of bbgen report, and then a paste of the unix bbgen report (witch are similar whether www is on disk or ram):
bbgen for Hobbit version 4.0.5RC1
Statistics: Hosts : 1617 Status messages : 8359 Purple messages : 8 Pages : 390
TIME SPENT Event Starttime Duration Startup 1121874341.021044 - Load links done 1121874341.021332 0.000288 Load bbhosts done 1121874341.147357 0.126025 ACK removal done 1121874341.147481 0.000124 Load STATE done 1121874341.540020 0.392539 Color calculation done 1121874341.542579 0.002559 Hobbit pagegen start 1121874341.542610 0.000031 Hobbit pagegen done 1121874344.373265 2.830655 BB2 generation done 1121874344.397772 0.024507 BBNK generation done 1121874344.399964 0.002192 Summary transmission done 1121874344.399968 0.000004 Run completed 1121874344.399969 0.000001 TIME TOTAL 3.378925
bbgen for Hobbit version 4.0.5RC1
Statistics: Hosts : 589 Status messages : 8359 Purple messages : 0 Pages : 1
TIME SPENT Event Starttime Duration Startup 1121874421.091144 - Load links done 1121874421.091429 0.000285 Load bbhosts done 1121874421.192460 0.101031 ACK removal done 1121874421.192561 0.000101 Load STATE done 1121874421.497978 0.305417 Color calculation done 1121874421.498891 0.000913 Hobbit pagegen start 1121874421.498925 0.000034 Hobbit pagegen done 1121874421.586783 0.087858 BB2 generation done 1121874421.606831 0.020048 BBNK generation done 1121874421.607756 0.000925 Run completed 1121874421.607758 0.000002 TIME TOTAL 0.516614
Selon Henrik Stoerner <henrik at hswn.dk>:
On Wed, Jul 20, 2005 at 04:54:56PM +0200, Olivier Beau wrote:
i loaded hobbit 4.0.5RC1 on my standby server (which isn't as fast as my production bbgen3.5 server..)
i'm having lots of iowait, (it isn't swapping), i suspect it's bbgen
It would be interesting to see the output from your "bbgen" column if you run bbgen with the "--report" option.
I haven't seen bbgen do very much I/O. Sure, you're generating 8x2 MB of data per minute, but most disk today will easily deliver 10 mbit/sec throughput. And it's not random access, just sequential writing of a single file.
I'd be more inclined to think it's your RRD file updates that are responsible for the disk I/O you see. Try disabling the [rrddata] and [rrdstatus] tasks in hobbitlaunch.cfg and see if the iowait drops.
--> Is there a way to disable the generating of the bb.html page for pagesets ?
No.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Olivier Beau
On Wed, Jul 20, 2005 at 06:14:04PM +0200, Olivier Beau wrote:
i've disabled [rrdstatus] and [rrddata]. i still have lots of I/O, practicaly no change.
OK, so you just have a lot of disk I/O for those pages.
i'm moved my www directory on another disk, and now most of the iowait is on the other disk. i then moved www in ramdisk, and all the iowait disapeared !
Just checking - the [bbstatus] module in hobbitlaunch.cfg is disabled ? It should be, or you *will* have lots of disk I/O.
here is a paste of bbgen report, and then a paste of the unix bbgen report (witch are similar whether www is on disk or ram):
Hosts : 1617 Status messages : 8359 Pages : 390
OK, so your main view if split on quite a few pages...
Hobbit pagegen done 1121874344.373265 2.830655 TIME TOTAL 3.378925
and it takes 3.4 seconds to generate the main view with those 390 pages. Not a lot. The other pageset takes even less:
Hobbit pagegen done 1121874421.586783 0.087858 BB2 generation done 1121874421.606831 0.020048 BBNK generation done 1121874421.607756 0.000925 TIME TOTAL 0.516614
0.5 secs total, and about 0.1 seconds to generate that page.
So building all of your pagesets should not take more than 5-10 seconds max. Your system should be idle at least 80% of the time.
Your system is probably doing a great job of buffering the disk writes, but I really can't see what the big problem is.
What OS are you running ?
Regards, Henrik
Selon Henrik Stoerner <henrik at hswn.dk>:
i'm moved my www directory on another disk, and now most of the iowait is on the other disk. i then moved www in ramdisk, and all the iowait disapeared !
Just checking - the [bbstatus] module in hobbitlaunch.cfg is disabled ? It should be, or you *will* have lots of disk I/O.
Yes, bbstatus is disabled, the i/o is clearly on the www directory; i didn't touch the data directory during my tests...
Your system is probably doing a great job of buffering the disk writes, but I really can't see what the big problem is.
well, all that i/o load hangs the server for a couple second, (pretty anoying while doing vi..), and i say a couple of "Whoops ! bb failed to send message - timeout" during the I/O load
that's why i wondered about disabling the generation of bb.html on some pagesets..
for now i think i'll have to stick in a better raid card (the default raid card on those server are pretty crummy...)
would it be possible to have those bb*.html page generated dynamicly ?
What OS are you running ?
i'm running redhat 3.0 on a compaq server with 2 CPU (1,1Ghz)
Regards,
Olivier
On Wed, Jul 20, 2005 at 07:28:44PM +0200, Olivier Beau wrote:
Your system is probably doing a great job of buffering the disk writes, but I really can't see what the big problem is.
well, all that i/o load hangs the server for a couple second, (pretty anoying while doing vi..), and i say a couple of "Whoops ! bb failed to send message - timeout" during the I/O load
[snip]
for now i think i'll have to stick in a better raid card (the default raid card on those server are pretty crummy...)
What OS are you running ?
i'm running redhat 3.0 on a compaq server with 2 CPU (1,1Ghz)
Hmmm - it shouldn't load your system that bad, it sounds like there's some driver problem. Or a really crappy disk controller/driver. I don't know what Linux kernel you're running, but I know there's been a lot of work going on lately with different algorithms for I/O scheduling and ensuring a "fair" ressource sharing for processes.
If writing ~20 MB of data in one go can cause your system to stall, it does sound like a problem. If you can reproduce it with a simple "dd if=/dev/zero of=dummyfile size=4k count=5000" then I'd be inclined to ask your Red Hat support contact if they have any ideas what to do.
that's why i wondered about disabling the generation of bb.html on some pagesets..
OK, the attached patch will let you do that - add "--no-pages" to the bbgen command line.
would it be possible to have those bb*.html page generated dynamicly ?
It's something I've thought about, yes - but haven't gotten around to implement yet. The individual status pages are generated on-the-fly so it would be a nice "clean" implementation to have everything be dynamically generated. Plus you'd always have an up-to-date view.
I guess I won't run out of things to do for quite some time :-)
Regards, Henrik
Selon Henrik Stoerner <henrik at hswn.dk>:
Hmmm - it shouldn't load your system that bad, it sounds like there's some driver problem. Or a really crappy disk controller/driver.
i've did some more investigations... and yes, the disk controller is REALLLLY crummy on those servers (compaq dl 380 g2); i'm having better performance with my old desktop (1Ghz and a 40G IDE drive !) i really have to tick in a better scsi board...
Thank you for the patch henrik :))
would it be possible to have those bb*.html page generated dynamicly ?
It's something I've thought about, yes - but haven't gotten around to implement yet. The individual status pages are generated on-the-fly so it would be a nice "clean" implementation to have everything be dynamically generated. Plus you'd always have an up-to-date view.
Yes, that would be great : having an real-time view, and saving ressource for hobbit server for large sites :)
-- Olivier Beau
participants (2)
-
henrik＠hswn.dk
-
olivier＠qalpit.com