[hobbit] Hobbit newbie from BB: differences and what may I lose from migrating?

This is an awesome write up. Maybe you should consider adding it to your webpage?

-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Tuesday, August 01, 2006 5:13 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Hobbit newbie from BB: differences and what may I lose from migrating?

Hi Jordan,

I'll try to answer your questions. Since I also develop Hobbit I am probably slightly biased when it comes to the "is-this-more-difficult- to-do-than-with-BB" type of questions, but I am sure others will voice their opinions on that.

On Tue, Aug 01, 2006 at 12:36:29PM -0700, Jordan Mendler wrote:

First, after reading through whatever I could find on the website I am still a little bit confused about configuration and setup. With BB,

you

install and configure each client and server on the local machine, except for the universal bb-hosts. Is this the same on Hobbit, or does Hobbit use a central configuration file that is modified only on the server to configure clients? I am trying to figure out the difference between installing, maintaining and configuring BB and Hobbit setups.

First, let me stress that Hobbit is fully compatible with your existing BB clients. You can keep your current client setup and just switch to Hobbit on the server side, and all of your clients will continue to work as they do with BB as the server. So you can migrate the server side first, and then migrate clients when you find that it is convenient to do so - or you want to take advantage of some of the new stuff that is in Hobbit.

The Hobbit client configuration is maintained on the Hobbit server. Clients in Hobbit are designed to be *really* dumb; they just collect data, and all of the configuration of what to monitor, what thresholds to use for e.g. disk utilisation and so on is configured only on the Hobbit server.

This is a major difference between Hobbit and BB. With BB you have delegated the client administration to whoever manages each server. Hobbit centralizes the monitoring configuration, so you will probably have a group of people who take more control of the monitoring setup.

Hobbit looks alot more complex to setup, but once I get my feet wet is it any harder than BB?

I think it is easier, once you get used to the Hobbit way of doing things. But as I said, I am biased.

Second is performance. I know this list may be biased toward Hobbit, but is it actually faster? We have about 50-100 clients on BB and I did not notice any performance issues.

With that number of systems monitored, you probably will not see a huge difference. BB works quite well for a small number of systems, but when you move beyond a couple of hundred boxes the overhead of generating webpages through shell scripts becomes very noticeable. On my setup, the servers were simply choking on the disk I/O caused by BB saving every status in a separate file, and from the huge number of small cut-grep-awk-sed etc. commands that ran to generate webpages.

Hobbit looks like it is very complex, so does this mean it uses a lot of resources on the client and server? What speed/ram server is usually the minimum recommended for a dedicated Hobbit server? Would something

like a dual Pentium II 266mhz have any performance issues as a server,

if it does nothing else? What about for clients? We have still have some testing, stating and production servers left that are singe chip Pentium III 700-850 mhz, and even a couple Pentium II's. Just need to make sure all the resources used for things like graphs are taken from

the server and not each client.

The Hobbit server uses fewer ressources than the BB server. The main ressource usage is memory; Hobbit keeps everything in memory except the history logs and the RRD files used for graphs. That doesn't mean a whole lot, though: Here's a ps listing of the Hobbit processes running

on my main monitoring system - it handles about 2500 hosts:

$ ps vax|cut -c1-100|egrep "PID|hobbit" PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 732 ? Ss 1:24 0 101 1802 696 0.0 hobbitlaunch 735 ? S 2434:37 1 162 31357 29784 2.8 hobbitd 1470 ? S 14:50 0 99 2332 1088 0.1 hobbitd_channel --channel=stachg 1471 ? S 25:18 0 108 2515 1048 0.1 hobbitd_history 1472 ? S 964:26 0 99 2332 1264 0.1 hobbitd_channel --channel=page 1473 ? S 1227:34 0 154 5661 3912 0.3 hobbitd_alert 1474 ? S 4090:05 0 99 2332 1264 0.1 hobbitd_channel --channel=status 1475 ? D 2962:15 0 178 7381 4392 0.4 hobbitd_rrd 1476 ? S 259:55 0 99 2332 1208 0.1 hobbitd_channel --channel=data 1477 ? S 494:13 0 178 5141 2128 0.2 hobbitd_rrd 1478 ? S 126:20 0 99 2844 1832 0.1 hobbitd_channel --channel=client 1480 ? S 291:20 0 146 4485 2792 0.2 hobbitd_client 5552 ? S 0:00 0 669 2002 1352 0.1 sh -c vmstat 300 2 1>/usr/lib/hobbit/client/

As you can see, the biggest chunk of memory goes to the "hobbitd" process which is the one that keeps all state information. It's currently using some 31 MB of memory. (This box has 1 GB RAM).

A rough estimate of how much memory Hobbit needs would be the size of your bbvar/logs/ directory, plus 30 MB.

As for CPU usage, your PII/266 should be adequate for 50-100 servers. The box I'm running on is an old (7-8 years) Solaris server with a 900 MHz UltraSparc II processor. That's roughly comparable to a PII running at 1.2 GHz. And it handles 25 times as many hosts as you are aiming for.

Third is plugins. Are BB plugins compatible with Hobbit?

Yes.

Also how hard are plugins to write for Hobbit?

Plugins that run on the monitored client systems are as easy to write as for BB, since it is basically the same thing.

Hobbit also allows you to write plugins for the Hobbit server, which receive events from the Hobbit server daemon. This is used by the core Hobbit tools - e.g. the hobbitd_rrd processes you see in the ps-listing above are a plugin that handle updating of the RRD files from the status- and data-messages that are sent to Hobbit. There aren't any third-party plugins that use this yet (at least, I don't know of any), but writing them is fairly simple since it basically involves reading data from a pipe and processing it in whatever way you want.

I don't know if these even exist for bb, but I ultimately would like to integrate plugins that 1) monitor legato tape backup,

Dont know about this.

2. run nmap to see what ports are open/can be seen from an external machine,

The Hobbit client in version 4.2 (about to be released soon) reports details about the network services running on a host. So you can check for which ports are open/listening for connections, and trigger alerts if any unwanted ports show up.

3. run 'lshw -html' to show a list of all the hardware on the system,

This would typically be a client-side test.

4. monitor uptime,

This is standard.

5. monitor OS and kernel versions (uname -a and head -n 1 /etc/issue),

This data is collected by the Hobbit client.

6. maybe some more router/network monitoring stuff and

Hobbit comes with built-in network service monitoring. There is also an SNMP add-on which can be used for monitoring devices such as routers.

Fourth is relay. By this I mean monitoring systems on a private subnetwork that are only accessible to the Hobbit server by going through an intermediate server. Is this possible with Hobbit and is it any more difficult to do than on BB?

Two ways of doing that. First, there is a proxy utility which is used to forward Hobbit messages from one network to another. This is used if your client systems on the private subnet are allowed to make outgoing connections to the proxy, and the proxy can connect to the real Hobbit server.

Second, Hobbit 4.2 includes a set of tools where it's the server that contacts clients to pick up the data they have collected (i.e. the traffic is initiated by the server, where the normal BB setup is for the client to initiate the connection). Useful for DMZ style setups where clients are not allowed to generate outbound connections.

Fifth is portability. BB is very portable, I can make a 'model' client for say Red Hat and tar it and distribute it very easily to every server I have using only a few commands. Is Hobbit the same, or are there client dependencies or other things that may make this more difficult.

The Hobbit client uses only the system libraries and standard utilities found on your client systems. You will need at least one system where you can compile the client binaries (that's similar to the BB requirements), since a few of the client-side tools are written in C.

Once you have a client compiled for an OS, it is as portable as any binary that is dynamically linked on your platform. I.e. you can just copy it over as long as the same run-time libraries are available.

So far, we haven't managed to find any unix-like system that couldn't run the Hobbit client. Including some rather odd ones. The current list of client-side data collectors are

hobbitclient-aix.sh hobbitclient-darwin.sh hobbitclient-freebsd.sh hobbitclient-hp-ux.sh hobbitclient-irix.sh hobbitclient-linux.sh hobbitclient-netbsd.sh hobbitclient-openbsd.sh hobbitclient-osf1.sh hobbitclient-sunos.sh

Sixth is development. How active is the development of Hobbit, how big is the community, etc? How many people can attest to having fully functional hobbit setups, how long has it been around and how often are new releases usually made?

Hobbit started back in late 2002 when it was called the "bbgen toolkit". It was renamed to Hobbit in March 2005 when it had developed into a complete replacement for BB. More details in the hobbit(7) man-page available online at http://www.hswn.dk/hobbit/help/manpages/

It is actively being developed by me, but people on this list have made contributions of code. Some have picked up special projects like the Windows client and run that completely on their own. I'd say Hobbit currently has a very active user community, and the development community is slowly growing beyond just myself.

There are currently 433 subscribers to the Hobbit mailing list. According to the Sourceforge download statistics, it is downloaded about 1000 times per month. http://sourceforge.net/project/stats/?group_id=128058&ugn=hobbitmon&type =&mode=year

There was a thread on the mailing list back in May about who uses Hobbit. The results were summarized here: http://en.wikibooks.org/wiki/System_Monitoring_with_Hobbit/User_Guide#Wh o_use_Hobbit_.3F

New releases have usually happened frequently - 2-4 times a year. The current interval between the 4.1.2 release and version 4.2 is unusually long - a whole year. I don't expect that to happen again.

Also I saw something this morning about a Windows client -- how stable is that?

From what I hear it should be usable. But you can stick with the current BBNT client until it reaches version 1.0.

How stable is the Solaris version?

Rock-solid.

Is there a client for Mac OSX?

Yes. It will run the Hobbit server also, if you want to.

Is Hobbit like BB in the sense that you can change paths to system binaries like grep and sed to allow easy use on other UNIXes like OSX?

Adding a client for a new OS will require implementing both a client-side script to collect whatever data is interesting for this system, and implementing the data parsing on the Hobbit server-side. So it is somewhat more challenging. But since Hobbit already supports all of the common Unix systems, I doubt that you will need to worry about that. If you do have a system which is not on the list, I will help you with adding support for it.

When will 4.2 be officially released as a production version?

Probably by the end of this week.

Since we have a working BB setup for now, I need to decide if I should try to start migrating now or if I should wait some time for Hobbit to develop more before I migrate from BB.

I don't think you have to wait. But it's for You to decide.

Regards, Henrik

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk