-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I was looking at the amount of data travelling to my hobbit server, and figured some compression would be useful...
Given we are effectively transferring pure text data from place to place, the CPU overhead of compressing/de-compressing the data should be minimal, and the bandwidth advantage (for non-lan) are significant... Taking the most recent file for each machine in my hostdata directory, and adding it up I get 1137kB or 116720MB / year or 9.5GB/month ...
Copying each of the most recent files to a temp folder, I get: Raw data per 5 mins: 1137 /tmp/hobbitdata gzip data per 5 mins: 389 /tmp/hobbitdata bz2 data per 5 mins: 385 /tmp/hobbitdata
So, from the above, I could change from using 9.5GB/mth to 9.5 * 385/1137 = 3.2 GB.... that would be a saving of around $19 per month for me.... Anyone else feel like calculating how much this is worth to them?
Try this:
mkdir /tmp/hobbitdata
cd /var/lib/hobbit/hostdata
for host in *
do
file=ls -lrt $host|tail -1|awk '{print $9}'
cp $host/$file /tmp/hobbitdata
done
echo -en "Raw data per 5 mins: "
du -sk /tmp/hobbitdata
gzip /tmp/hobbitdata/*
echo -en "gzip data per 5 mins: "
du -sk /tmp/hobbitdata
gunzip /tmp/hobbitdata/*
bzip2 /tmp/hobbitdata/*
echo -en "bz2 data per 5 mins: "
du -sk /tmp/hobbitdata
rm -rf /tmp/hobbitdata
Might take a few minutes depending on number of hosts/etc...
So, is compression likely to be implemented into the hobbit client/server any time soon?
Regards, Adam
Adam Goryachev Website Managers www.websitemanagers.com.au -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAklHRMcACgkQGyoxogrTyiUf6ACfcDwQbwVuISQ70A814Z+hujvm ezoAn1RrxkP0SMjr9ibV2iO56QI/iqh+ =fLdN -----END PGP SIGNATURE-----
In <494744C7.20101 at websitemanagers.com.au> Adam Goryachev <mailinglists at websitemanagers.com.au> writes:
So, is compression likely to be implemented into the hobbit client/server any time soon?
It's been in the main development trunk since March:
r5558 | henrik | 2008-03-02 13:44:57 +0100 (søn, 02 mar 2008) | 2 lines
Support on-the-fly compress/decompress of data
Regards, Henrik
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Henrik Størner wrote:
In <494744C7.20101 at websitemanagers.com.au> Adam Goryachev <mailinglists at websitemanagers.com.au> writes:
So, is compression likely to be implemented into the hobbit client/server any time soon?
It's been in the main development trunk since March:
r5558 | henrik | 2008-03-02 13:44:57 +0100 (søn, 02 mar 2008) | 2 lines
Support on-the-fly compress/decompress of data
Oops... perhaps this is a reason for me to upgrade to the dev version, though I'd of course prefer to wait for a proper release.... oh, the dilemma!
Thanks for your hard work, and a great system BTW! It's been a while since we had a flurry of thank you emails :)
Regards, Adam -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAklH1A8ACgkQGyoxogrTyiXdtQCgpmwd/3FX9VFs85wEq9UsCpde a7MAn2iy9kZmRW7aO4FWsVKIQQaqAC5D =wlzR -----END PGP SIGNATURE-----
I may have missed something here...but is this compression done already or is there a need to configure it?
Also, when was 4.2.0 release? I'm using the 4.2.0 release at the office currently and I'd like to know if the compression has been active or not (since Henrik has included it since March, presumably 2008).
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
On Tue, Dec 16, 2008 at 11:15 AM, Adam Goryachev < mailinglists at websitemanagers.com.au> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Henrik Størner wrote:
In <494744C7.20101 at websitemanagers.com.au> Adam Goryachev < mailinglists at websitemanagers.com.au> writes:
So, is compression likely to be implemented into the hobbit client/server any time soon?
It's been in the main development trunk since March:
r5558 | henrik | 2008-03-02 13:44:57 +0100 (søn, 02 mar 2008) | 2 lines
Support on-the-fly compress/decompress of data
Oops... perhaps this is a reason for me to upgrade to the dev version, though I'd of course prefer to wait for a proper release.... oh, the dilemma!
Thanks for your hard work, and a great system BTW! It's been a while since we had a flurry of thank you emails :)
Regards, Adam -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAklH1A8ACgkQGyoxogrTyiXdtQCgpmwd/3FX9VFs85wEq9UsCpde a7MAn2iy9kZmRW7aO4FWsVKIQQaqAC5D =wlzR -----END PGP SIGNATURE-----
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Josh Luthman wrote:
I may have missed something here...but is this compression done already or is there a need to configure it?
Not sure...
Also, when was 4.2.0 release? I'm using the 4.2.0 release at the office currently and I'd like to know if the compression has been active or not (since Henrik has included it since March, presumably 2008).
August 2006 says sourceforge :) http://sourceforge.net/project/showfiles.php?group_id=128058
I would say from Henrik's response that it is only in the cvs version right now, ie, what will become 4.3.0 when it is released. So you won't have it now, and even if you upgrade to 4.2.2 you won't get it. You would need to install the development version from cvs to get this feature, or wait for 4.3.0
Regards, Adam
On Tue, Dec 16, 2008 at 11:15 AM, Adam Goryachev <mailinglists at websitemanagers.com.au <mailto:mailinglists at websitemanagers.com.au>> wrote:
Henrik Størner wrote:
In <494744C7.20101 at websitemanagers.com.au <mailto:494744C7.20101 at websitemanagers.com.au>> Adam Goryachev <mailinglists at websitemanagers.com.au <mailto:mailinglists at websitemanagers.com.au>> writes:
So, is compression likely to be implemented into the hobbit client/server any time soon?
It's been in the main development trunk since March:
r5558 | henrik | 2008-03-02 13:44:57 +0100 (søn, 02 mar 2008) | 2 lines
Support on-the-fly compress/decompress of data
Oops... perhaps this is a reason for me to upgrade to the dev version, though I'd of course prefer to wait for a proper release.... oh, the dilemma!
Thanks for your hard work, and a great system BTW! It's been a while since we had a flurry of thank you emails :)
Adam Goryachev Website Managers Ph: +61 2 8304 0000 adam at websitemanagers.com.au Fax: +61 2 8304 0001 www.websitemanagers.com.au -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFJSCjFGyoxogrTyiURApT2AJ9HxvS6V+ccbuW7K3MoTAJolkKCfwCgt5xg v51UdvDpeOLZsGSk7wnMjVc= =crQN -----END PGP SIGNATURE-----
In <494828C5.4070104 at websitemanagers.com.au> Adam Goryachev <mailinglists at websitemanagers.com.au> writes:
Josh Luthman wrote:
I may have missed something here...but is this compression done already or is there a need to configure it?
Not sure...
It's done, but only in the development version - neither 4.2.0 nor 4.2.2 has it. It was rather non-trivial to implement and in a key-function in Hobbit; so adding it to the 4.2.x series was out of the question.
It requires a new hobbitd daemon on the server side, and a new "bb" utility at the client. Compression is not turned on by default - you'll have to invoke "bb" as "bbz", or pass it the "--compress" option.
Also, when was 4.2.0 release? I'm using the 4.2.0 release at the office currently and I'd like to know if the compression has been active or not (since Henrik has included it since March, presumably 2008).
August 2006 says sourceforge :) http://sourceforge.net/project/showfiles.php?group_id=128058
Aug 10, 2006 to be exact.
Regards, Henrik
Oh, didn't think about it requiring a new binary. Not interested it that for my purposes if that is the case.
Thanks a bunch for clearing that up!
On 12/16/08, Henrik Størner <henrik at hswn.dk> wrote:
In <494828C5.4070104 at websitemanagers.com.au> Adam Goryachev <mailinglists at websitemanagers.com.au> writes:
Josh Luthman wrote:
I may have missed something here...but is this compression done already or is there a need to configure it?
Not sure...
It's done, but only in the development version - neither 4.2.0 nor 4.2.2 has it. It was rather non-trivial to implement and in a key-function in Hobbit; so adding it to the 4.2.x series was out of the question.
It requires a new hobbitd daemon on the server side, and a new "bb" utility at the client. Compression is not turned on by default - you'll have to invoke "bb" as "bbz", or pass it the "--compress" option.
Also, when was 4.2.0 release? I'm using the 4.2.0 release at the office currently and I'd like to know if the compression has been active or not (since Henrik has included it since March, presumably 2008).
August 2006 says sourceforge :) http://sourceforge.net/project/showfiles.php?group_id=128058
Aug 10, 2006 to be exact.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Henrik Størner wrote:
It requires a new hobbitd daemon on the server side, and a new "bb" utility at the client. Compression is not turned on by default - you'll have to invoke "bb" as "bbz", or pass it the "--compress" option.
Sorry to continue this conversation, but I'd like to clarify some implementation details.
If I have two hobbit servers, and one bbproxy server which transmits the data to both hobbit servers, and of course multiple bb clients behind the bbproxy.
Do I need to update *both* hobbit servers, or could I only upgrade one hobbit server, and the bbproxy, and bbproxy would be capable of compressing received updates, and forwarding them to the compression capable server, and forwarding the uncompressed updates to the older server?
Also, how do old hobbit clients, or bb windows clients, or hobbit windows clients behave when behind the bbproxy? Would I need to upgrade each client for it's updates to be compressed, or will bbproxy handle that for me?
Thanks, Adam
Adam Goryachev Website Managers www.websitemanagers.com.au -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAklIekYACgkQGyoxogrTyiUI6gCfV3QJ0IhoA/ejrA6eBxkFGwy0 LfsAn2xIol3ngZh7IzrH7GtpDGblv9Nb =XXSK -----END PGP SIGNATURE-----
In <49487A46.9020406 at websitemanagers.com.au> Adam Goryachev <mailinglists at websitemanagers.com.au> writes:
Henrik St�rner wrote:
It requires a new hobbitd daemon on the server side, and a new "bb" utility at the client. Compression is not turned on by default - you'll have to invoke "bb" as "bbz", or pass it the "--compress" option.
Sorry to continue this conversation, but I'd like to clarify some implementation details.
If I have two hobbit servers, and one bbproxy server which transmits the data to both hobbit servers, and of course multiple bb clients behind the bbproxy.
Do I need to update *both* hobbit servers, or could I only upgrade one hobbit server, and the bbproxy, and bbproxy would be capable of compressing received updates, and forwarding them to the compression capable server, and forwarding the uncompressed updates to the older server?
You've caught one of the "missing items" in the development version - bbproxy hasn't been made compression-aware yet. In the end, there will be a way of specifying whether bbproxy target servers can handle compression or not - so yes, you could have updates sent in compressed form to one hobbitd, and uncompressed to another.
At least, that's the plan. I haven't looked too much at the implementation details yet, so it might turn out differently.
Also, how do old hobbit clients, or bb windows clients, or hobbit windows clients behave when behind the bbproxy? Would I need to upgrade each client for it's updates to be compressed, or will bbproxy handle that for me?
If a target server supports compression, then I think bbproxy will perform the compression on all of the "large" messages, i.e. status- and client-messages. Other messages are too small to be worth the trouble of compressing them. So that will be transparent to the client.
Regards, Henrik
participants (3)
-
henrik＠hswn.dk
-
josh＠imaginenetworksllc.com
-
mailinglists＠websitemanagers.com.au