Hi List,

I've just set up Xymon monitoring of a pfSense system and I thought I'd share a few details for others.

pfSense is a firewall system based on FreeBSD using the 'pf' packet filter system. The FreeBSD system is trimmed down to a smaller operating system than the normal distribution and has a very small number of available packages available in the default repository, all related to it's firewall functionality. I've seen reference to pulling in packages from the standard FreeBSD pkg system but have chosen not to go down that path.

My starting point is Jeremy Laidman's xymon-rclient.sh script from xymonton. This is a quite well thought out shell script (in contrast to many of the add-ons, including mine), that connects to the target system using ssh (or others that don't matter) and fires off a combination of the standard xymon client scripts and a few extra shell commands. The resulting output looks like the normal client log which is then handed back to xymon. The idea is that no special software is required on the target system.

After installing the script in the server/ext directory, you add a tasks entry (I use a tasks.d directory file) to run the script every 5 minutes. All pretty standard stuff so far. The recommended task entry is:

[xymon-rclient] ??????? ENVFILE $XYMONHOME/etc/xymonserver.cfg ??????? CMD /$XYMONHOME/ext/xymon-rclient.sh ??????? LOGFILE $XYMONSERVERLOGS/xymon-rclient.log ??????? INTERVAL 5m

This looks for RCLIENT tags on entries in your hosts.cfg file. My tag looks like this:

??? "RCLIENT:cmd(ssh -T -o BatchMode=yes admin at 10.0.1.54 /bin/sh),ostype(freebsd)"

Key points include the ostype(freebsd) part which identifies the target system type. This identifies the client script to send to the target system, in this case client/bin/xymonclient-freebsd.sh.

The /bin/sh at the end of the command solves a particular problem. It tells the ssh daemon to run this program rather than the normal login shell. This is important because if you connect to a pfSense system by ssh (BTW this must be enabled first in the pfSense web interface), you are greeted by a menu rather than a command prompt. It would be possible to patch the rclient.sh script to send an '8' (the option for a shell) as the first thing it does, but that would mess up using the script for any other target systems. Including the /bin/sh command bypasses the menu and sets up the system in a shell to begin work. Note that this is a more or less standard bourne shell. Bash would only be available as an unsupported package but is also completely unnecessary.

SSH Setup

To use ssh you will need to generate some key pairs and transport the public key parts to the target system.

On the xymon server machine I did this (as root), assuming the xymon server runs as user called 'xymon':

??? cd ~xymon

??? mkdir .ssh

??? chown xymon .ssh

??? cd .ssh

??? ssh-keygen -t rsa -f id_rsa -P ''

??? ssh-keygen -t dsa -f id_dsa -P ''

??? ssh-keygen -t ecdsa -f id_ecdsa -P ''

??? ssh-keygen -t ed25519 -f id_ed25519 -P ''

??? chown xymon *

Your ssh may have less or additional key types than mine, check your manpage for ssh-keygen. Also, that's two consecutive single quotes at the end of the line.

You then need to transfer your public keys over to the target system. To get your public keys, type:

??? cat ~xymon/.ssh/*.pub

And copy the resulting 4 lines of output (some of which may be wrapped over a few lines).

We need to exchange public keys with the target server. Log on to the pfSense system as the local xymon user. This is important because we need this user to capture the remote pfSense host's public key in xymon's .ssh/known_hosts file.

??? su - xymon -c "ssh admin at 10.0.1.54"

If you are a 'sudo' fan, adjust as necessary. Accept the remote system's key and then give the remote password. The pfSense menu should appear, select 8 to get a shell.

Create a .ssh directory:

??? mkdir .ssh

And add our public keys to the system.

??? cat >>.ssh/authorized_keys

Paste in our public keys from earlier and press control+D on a new line to end input. Type control+D again to exit the shell, then select option '0' to logout from the menu.

You should now be able to ssh to the pfSense system (as the xymon user) without accepting a new key or supplying a password. Test and verify.

??? su - xymon -c "ssh admin at 10.0.1.54"

Fixes to xymon-rclient.sh

I've sent a diff file to Jeremy Laidman a couple of days ago. He may or may not change the code on xymonton. That's how these things work.

Essentially there are two essential changes to reflect the target system not being Linux. One is a problem with the sent command stream over-running a fresh shell invocation intended to set up proper environment variables. The other change is to some sed GNU extensions used. There shouldn't be any impact on other systems.

There are a couple of other minor fixes that I'd recommend but are not essential.

I'm including the patch in this email (not sure if the mail list server approves) incase you have an unpatched copy.

Fixes for Xymon

The client script for a FreeBSD system includes a small C program to extract memory data. This is a problem since we don't want to install a program on the target machine. We can avoid including that section in the client data if the program isn't available but then we don't get memory tracking by xymon.

The reason for the program is that the installed memory on the system is available as a number of bytes. With recent systems this often excedes a 32 bit number. The requirement is a memory size in MB and shell arithmetic is limited to 32 bit integers. I've implemented two workarounds for this:

1. A massive hack - drop the last three digits of the size and divide by 1048 to get the size in MB. This should be surprisingly good up to memory sizes in TB. I've only tested to 32GB though.

2. Send the size in bytes back to xymon where it can properly process the numbers as unsigned longs.

Xymon already uses "memory free" from a later section (vmtotal) so we don't need to fake that up, although it also quite doable but commented out in my fix.

The other diff file includes patches for both the FreeBSD client script, and the xymond reader for freebsd client messages.

You have three choices depending on circumstances and how much you care:

1. Do nothing. You won't have a memory column. Remember to include a NOCOLUMNS:memory tag on your pfSense machine in this case.

2. Just patch the client script. This should be good to just short of 2TB pfSense memory. pfSense doesn't need anything like that much memory.

3. Patch the script and the xymond code. Should be good up to a full 64 bit used address space. No nasty hacks involved.

[Side note: netbsd and openbsd also have client programs for the same purpose. I don't have ready access to either of these systems to develop/test equivalent patches.]

Other systems

I've also tested this on a nanobsd system (the ssh stuff is a little harder because of read-only file systems) and a dell server running my hypervisor on a recent version of FreeBSD. Also found that msdosfs file system on FreeBSD doesn't have inodes (I've normally handled this with exclusions) so added it to the inodes exclusion list.

I hope this helps someone in the future,

Brian

P.S. If this email doesn't have the diff files attached, check for another email following close behind with them included as the message body.