On Thu, 2005-06-09 at 07:41 +0200, Henrik Stoerner wrote:

When I pick up speed again, I haven't yet decided if I should go for the alert/acknowledge improvements that I've talked about for some time, or if I should tackle the issue of a Hobbit client and get that into a reasonable shape for the more common platforms. I have some ideas for a rather different client architecture than the current one, and the client is also *the* missing piece of the whole Hobbit puzzle - so I could be tempted to get that done. I'd like some feedback on what you find most urgent.

Personally, I'd most like to see a 'free' client (ie, GPL, without the BB license issue), and I'd also like to see *much* better SNMP support. ie, point it at a router, and it will automatically (or some tool) setup the various values to monitor (interfaces, byte counter thresholds, cpu, temperature, etc) or a switch, or firewall, or UPS, or whatever thingamabob you have lying around.

Beyond that, I would then go for more of the admin tools, like the bb-hosts editor recently being developed/released, and some other auto-discovery type tools.... This makes things easier for initial deployment/users who don't understand unix so much... But overall, if you make the right API, then other people can step in and make their own GUI tools...

Finally, what about some sort of compression/encryption protocol, so that it is possible to do more frequent test/report without using so much bandwidth? I know very little about these things, but the overhead does concern me somewhat, especially if you wanted to test every 30 seconds or something...

Regards, Adam

Henrik,

This might not be quite the right thread for this, but anyway...

There appears to be a bug in bbproxy which should probably be fixed sometime. When a message is to be sent to several servers, and the transfer to one sender fails, then the transfer to the other servers isn't attempted. (The state machine jumps straight to the exit state rather than looping back to send the message to the next server.) I first noticed this in an older version of bbproxy, but a quick glance at the code suggests it might still be a problem.

Also in bbproxy, I'd like to see an option where the proxy reports can be sent to a different list of servers from where ordinary status reports are sent. In my case for example, I am using bbproxy mainly for load balancing on the server where bbd (not yet hobbitd) runs, so it only send messages to the local server. However I would like the reports sent to both our servers. I think I sent you some code back in January which attempted to do this, but it probably got lost in the ether somewhere. I could resurrect it if you like.

Regarding the hobbit client, one of my main complaints against various other monitoring/management tools as that they run large complex processes as root with links deep into the OS. I wouldn't like to see the hobbit client go down that path.

Kevin.

Henrik Stoerner schrieb:

When I pick up speed again, I haven't yet decided if I should go for the alert/acknowledge improvements that I've talked about for some time, or if I should tackle the issue of a Hobbit client and get that into a reasonable shape for the more common platforms. I have some ideas for a rather different client architecture than the current one, and the client is also *the* missing piece of the whole Hobbit puzzle - so I could be tempted to get that done. I'd like some feedback on what you find most urgent.

Performance is not an issue, here. So I really would like to see a real hobbit client, first.

Best wishes, Dirk

Henrik,

Whatever break you take over the summer is certainly well deserved. You've provided the world with a wonderful tool and this reader is very appreciative of all of your hard work.

I must concur that a Hobbit client should take precedence over other development activities (except perhaps bug fixes).

I tend to like the SNMP idea myself. Building a client around that arthitecture could solve a large number of problems, perhaps the biggest is support of a diverse number of operating systems.

Henrik Stoerner wrote:

It seems version 4.0 has reached some stability - there are still a few odd bug reports, but nothing that looks like major problems. So I thought it would be worthwhile to let you know what my plans are.

First, I'm currently pretty busy with other stuff so questions raised here are not being answered as quickly as I'd like to. Rest assured that they have not been forgotten, and I will look at all of the reports and questions - but probably not very much for the next week or so.

I'm currently working on a 4.0.5 release which will be a performance- improvement release. The current Hobbit code has a flaw in the alert- module, that makes it use much more CPU time than it should - especially if you have a large number of statuses that are in an alert state, but which do not have any recipients defined in hobbit-alerts.cfg. This will be released sometime in June.

But summer is upon us, and that means less development activity - so things will slow down as the temperature rises. So don't expect a lot of activity during the summer.

When I pick up speed again, I haven't yet decided if I should go for the alert/acknowledge improvements that I've talked about for some time, or if I should tackle the issue of a Hobbit client and get that into a reasonable shape for the more common platforms. I have some ideas for a rather different client architecture than the current one, and the client is also *the* missing piece of the whole Hobbit puzzle - so I could be tempted to get that done. I'd like some feedback on what you find most urgent.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

-- Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (414)491-6001 Ans Service: (360)715-2467 rich.smrcina at vmassist.com

Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006

On Thu, Jun 09, 2005 at 07:41:42AM +0200, Henrik Stoerner wrote:

It seems version 4.0 has reached some stability - there are still a few odd bug reports, but nothing that looks like major problems. So I thought it would be worthwhile to let you know what my plans are.

Well, lots of responses so I'll try to pick out some trends and respond to them in one place.

The Hobbit client

Adam Goryachev :

Personally, I'd most like to see a 'free' client (ie, GPL, without the BB license issue)

Right now the vote seems to be in favour of working on the client. It certainly will be GPL.

Craig Cook :

Instead of writing a hobbit client from scratch, it may be worth looking at modifying the Nagios client.

That might be an idea, yes. I haven't spent much time with Nagios, so I am not really familiar with their architecture. I had the impression it was more SNMP focused than BB/Hobbit.

SNMP - or "How to collect data"

Adam Goryachev :

I'd also like to see *much* better SNMP support.

Craig Cook :

Building SNMP support into core hobbit would be a good idea, it is also non trivial.

Daniel J McDonald :

I'd really like the bb-central approach. Most of the status information can be grabbed from non-privileged accounts on all unix-like platforms. I concede that a client is necessary in the windows world.

There's no doubt that some sort of support in Hobbit for collecting data via SNMP would be very useful. However, I believe that's better implemented as a stand-alone tool, somewhat like the network tester. It would obviously rely on some library like Net-SNMP 5 for the dirty stuff of talking SNMP (meaning it would support SNMP v1, v2c and v3 automatically - although I have at one point implemented SNMP daemons and MIB instrumentation from scratch, I'd rather not repeat that excercise :-))

However, I don't want to base a Hobbit client on SNMP or any other central polling-style method of data collection. There are at least two reasons for that:

1. It doesn't scale well. My main setup has over 2000 boxes to monitor. Doing that from one central server would mean polling 7 systems per second - that just won't work. There are always some servers that are down causing timeouts... whether it happens via SNMP, ssh or some other protocol really doesn't matter. It probably works fine for a setup with 50 or even 100 systems, but not for me.

2. The central server needs to know about all kinds of systems If my central polling server runs an "ssh hobbit at clientIP uptime;df;ps" - then the central server must know how to interpret the output. That's one of the major design problems in Hobbit currently - everytime Redhat^Wsomeone comes up with a new layout for the "vmstat" output, Hobbit needs to be modified to recognize these data.

So my idea currently is to design a new type of client. It won't generate "status" messages, it will just collect data. Imagine a client that just sends Hobbit a "client data" package, like

os: Linux osver: 2.6.11 osid: Debian/Sarge i386 uptime: 173201 seconds loadaverage: 0.4 filesystem /: 26102 MB, 71% used inodes /: 10291029 total, 21% used filesystem /var: 102400 MB, 50% used inodes /var: 40182910 total, 7% used

This is one well-defined format that Hobbit needs to recognize, and based on these data it can match e.g. filesystem utilisation against a configuration file on the Hobbit server and generate the necessary status messages - so the end result will look exactly like what you have today, but with much less complexity in how e.g. the RRD handlers need to know about the types of systems that report into Hobbit.

The only drawback is that the client becomes slightly more advanced but not much; it's really just formatting the information differently before sending it off to the Hobbit server.

Another very nice thing about this is that you can easily (well, relatively) write Hobbit modules that handle new kinds of information.

And it can be done without breaking compatibility with the existing clients, so you can run a mix of BB and Hobbit clients without any problems.

Encryption/authentication and compression of status messages

Adam Goryachev :

Finally, what about some sort of compression/encryption protocol, so that it is possible to do more frequent test/report without using so much bandwidth?

Daniel J McDonald :

If we are building an extended protocol, we should support authentication as well.

There's already some IP-based access controls built into Hobbit; see the hobbitd man-page for the --status-senders, --maint-senders, --www-senders, --admin-senders options. The first one should be sufficient to block most attempts at sending fake status messages - an attacker would need to break into your network test server and send the fake messages from there.

However, authentication could be nice. I am tempted to handle both of these problems with one solution - and just implement an SSL-encrypted protocol where you can then use client-side certificates for authentication. That will be significant overhead on the processing side, but the good thing is that you can offload SSL to hardware devices fairly easy (and OpenSSL does support that kind of hardware).

Compression ... is it necessary ? All of the status messages in my setup combined are about 6 MB for 2000 servers - ie. 3 KB/server which gets updated every 300 secs (on average). So that's 10 bytes/second per server. So a rough bandwidth estimate for Hobbit would be 100 bps per server monitored. For a LAN, that's peanuts.

Well, thanks for the feedback - it's really good to learn what ideas and problems are the important ones.

Regards, Henrik