April 2006 - Xymon - Xymon mailman web

OpenLDAP performance and replication monitoring with Hobbit
by bgmilne＠staff.telkomsa.net 09 May '06

09 May '06

We run a number of OpenLDAP servers, currently we have about 11 in production (depending how you count clusters) and a few in DR sites etc. We have been doing performance monitoring in the past (ie running scripts via snmp, monitoring with mrtg), but it seemed logical to move it to Hobbit and do everything in one place (useful for the syncrepl monitoring below). We have two separate instances that we manage, only one currently uses Hobbit for production monitoring (the other installation has hobbit installed, which is publicly accessible at present, but isn't used for production monitoring). The instance with the publicly accessible installation has OpenLDAP 2.1 and 2.2 servers, the other one has OpenLDAP 2.3 servers (using sync replication). So, I have written an extension script using perl-ldap (and Date::Manip) which we are now using via Hobbit's ncv module for performance monitoring. Replication status of sync-repl slaves is also checked (green for in sync, yellow for out of sync, clear if the sync-repl master could not be contacted to check the contextCSN, green if it is not a sync-repl slave). The script is available here: http://www.zarb.org/~bgmilne/bb-openldap.pl An extract from hobbitgraph.cfg is available here: http://www.zarb.org/~bgmilne/hobbitgraph-ol.cfg Some examples: http://196.25.211.20/hobbit/ldap/ldap.html Slave running OpenLDAP 2.2.x: http://196.25.211.20/hobbit-cgi/bb-hostsvc.sh?HOSTSVC=io.ol Master cluster running OpenLDAP 2.1.x: http://196.25.211.20/hobbit-cgi/bb-hostsvc.sh?HOSTSVC=metis.ol The only problem is with the parsing in ncv when a slave is out of sync. For slaves using sync-repl (no examples publicly available for now), the "&green Not a syncrepl slave" can be something like: &yellow dc=mydomain,dc=com out of sync 4 weeks 4 days 15:5:11 behind or &green dc=mydomain,dc=com in sync (with one such line for each syncrepl database, we have two or more on our production OL 2.3 boxen) Now, I could s/:/;/g or similar, but it would be nice if there were a way to prevent ncv from parsing a line (since I would also like to be able to graph the CIM extension results with ncv, but it has output and &green/&yellow/&red's all over). Regards, Buchan -- Buchan Milne ISP Systems Specialist B.Eng,RHCE(803004789010797),LPIC-2(LPI000074592)

2 1

hobbitclient on solaris: using iostat for cpu stat
by grossi＠acantho.net 09 May '06

09 May '06

Hi Henrik, I have seen that hobbit clients collects cpu stats via vmstat, now i think that the cpu information provided by "iostat -c" is more accurate, because it has the "%waiting for i/o" columns, wich is also in the linux version of vmstat. do you think you can add this information? -- Gianluca

2 1

Disk RRD problem with hobbit-client and OSF1 (Tru64)
by astavitsky＠yahoo.com 04 May '06

04 May '06

Hi! I'm using hobbit 4.1.2 server and clients. Some of my OSF1 (Tru64 4.0&5.1) clients' filesystems are being captured in RRD by mountpoint name and others by the device/domain. These are RRD files for a particular host:: ls -l HOSTNAME/disk* -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,arc.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,nfs,BOSS.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,nfs.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-26 20:13 HOSTNAME/disk,report.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,root_domain#root.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act,CWNY.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act,LongIsland.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act,NewJersey.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt,MakPrd,progesaPrd,Act.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,rpt.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,tmp_domain#tmp.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,usr_domain#usr.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,var,ase,mnt,asearc,arc.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,var,ase,mnt,asenfs,nfs.rrd -rw-r--r-- 1 hobbit hobbit 38532 2005-10-30 13:04 HOSTNAME/disk,var_domain#var.rrd And here is the hobbitlog for this: HOSTNAME|disk|green||1130372298|1130621525|1130623325|0|0|172.16.52.21|-1|||Y green Sat Oct 29 17:31:58 EDT 2005 - Filesystems OK Filesystem 512-blocks Used Available Capacity Mounted on root_domain#root 409600 252902 128032 67% / usr_domain#usr 2516576 1825636 638672 75% /usr var_domain#var 9437184 4114720 5284080 44% /var tmp_domain#tmp 409600 2682 385376 1% /tmp /nfs at asenfs 88821680 37803820 50792192 43% /nfs /arc at asearc 106643104 69304154 37257648 66% /arc nfs_dmn#nfs 88821680 37803820 50792192 43% /var/ase/mnt/asenfs/nfs arc_dmn#arc 106643104 69304154 37257648 66% /var/ase/mnt/asearc/arc /NFS at CEBOSS 377479160 298515328 78963832 80% /nfs/BOSS rpt_dmn#progesa 106643104 61776456 44815856 58% /rpt rptact_dmn#act 106643104 75460796 31101280 71% /rpt/MakPrd/progesaPrd/Act rptcwny_dmn#cwny 106643104 28278562 78320496 27% /rpt/MakPrd/progesaPrd/Act/CWNY rptli_dmn#li 71095392 19095496 51949920 27% /rpt/MakPrd/progesaPrd/Act/LongIsland rptnj_dmn#nj 71095392 3627144 67417760 6% /rpt/MakPrd/progesaPrd/Act/NewJersey I read earlier messages about hobbit_larrd misdetecting NT/UNIX disk output, but in this case it uses different columns for different filesystems on the SAME host. I'm at a loss why this is happening and how to fix it. Thanks,

4 4

Changing the frequency of a specific test
by ian.marsh＠hants.gov.uk 03 May '06

03 May '06

We're currently experiencing a problem with httpd on one of our servers that isn't being caught by hobbit. Is it possible to specify a custom test interval for just one test on one server? The hope is that it'll stand a better chance of catching the problem if it runs, say, every minute rather than every 5 minutes. Thank you, Ian Marsh IT Services, Network Services Hampshire County Council Telephone: 01962 845235 HPSN: 200 5235 email: ian.marsh at hants.gov.uk

6 12

hobbit gentoo ebuild?
by kolbjorn.barmen＠uninett.no 02 May '06

02 May '06

Before I make them myself, are there ebuilds for hobbit server and hobbit client out there already? -- Kolbjørn Barmen UNINETT Driftsenter

2 3

Bb-hostedit
by Stewart_Larsen＠doh.state.fl.us 02 May '06

02 May '06

I am unable to fin dbb-hostedit on deadcat and googling finds some discussion about it, but I can't find where to download it. Is it still available? -----Original Message----- file is maintained mostly by the (non-unix-savvy) staff here, using the bb-hostedit CGI script, and I'd rather not have them have to keep track of which host needed which NET tag, etc.

3 2

I'm starting to look at setting up 'propr' monitoring for my production SAP environment, and wonder if anyone else is playing with HACMP clusters
by KauffmanT＠nibco.com 02 May '06

02 May '06

I need to split the monitoring for SAP and Oracle from the monitoring of the host it runs on -- because, in an IBM HACMP environment, the workload can move. So the oracle tests, *some* of the disk space tests, and *some* of the proc tests move from one box to another. So does the primary IP address. I need to be able to track 'colorado' the box (the hostname does NOT move) as well as 'colorado' the IP address. 'Colorado' the address could actually be pointing to the 'thames' or 'yukon' boxes. At which time, 'colorado' the box will be responding to the 'colorado-bt' adapter address (if 'colorado' the box is up at all). I can get some of what I want by running two clients, one for 'colorado' the box, and one for 'SAP' or some other such synthetic name. The biggest two challenges I see are splitting the OS filesystems from the application filesystems for freespace reporting, and the shifting IP address. I may have an 'out' on the IP address in the future, if I can migrate to what IBM calls 'aliasing' -- where they add the running address as a second address on the adapter and respond to both. And to make this a bit more fun, I have THREE of these silly environments to work with. Henrik, is there any way to hang two different IP addresses on the same bb-host entry such that the network connectivity test will be happy if either one responds? They are always on the same IP network, just different host addresses (our convention is to add 100 to the last octet to get the boot-time address for the matching run-time address). Is anyone else doing something silly like this? TIA Tom Kauffman NIBCO, Inc CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.

6 11

[hobbit] fping tuning
by EES2Y＠hscmail.mcc.virginia.edu 02 May '06

02 May '06

> I've had a look at the fping sources. > > There aren't any really obvious reasons why it should take so long. > If you run it with "time", it also claims that the user- and > system-time > are really low (I tried with ~1500 hosts), but the wall-clock time is > like 90 seconds (default options). Which kind of points at the code > not really doing parallel pings. > > > I think I'll try some modifications to it over the week-end. If any of > it seems to improve it, I'll let you know. > I gave the source a quick peruse before sending my initial email, and my gut was telling me likewise. Thanks for looking in to it :) -Eric

3 3

fping tuning
by EES2Y＠hscmail.mcc.virginia.edu 30 Apr '06

30 Apr '06

We're monitoring 1420 IPs in hobbit, and it takes fping ~40 seconds to go through them all: <snip> [root at hobbit fping]# fping -i5 -b12 -f ips -r1 -t250 -B2 -q -s 1430 targets 1419 alive 11 unreachable 0 unknown addresses 55 timeouts (waiting for response) 1474 ICMP Echos sent 1420 ICMP Echo Replies received 0 other ICMP received 0.05 ms (min round trip time) 5.83 ms (avg round trip time) 281 ms (max round trip time) 40.704 sec (elapsed real time) </snip> Now, this seems a bit lengthy to me. I mean, if the avg round trip time is 5.83 ms, and there are 1430 hosts, should the total time in transit for all hosts should be 8336ms, or 8 seconds... right? Even when I remove the hosts that aren't responding, the results on are par with those above. Our polling interval is once every 60 seconds (which we want to maintain, because we like to know ASAP when something drops even one ping), so it's not a problem yet. We add hosts on a daily basis, however, so it will be a problem some time in the future and I'd like to fix it before it becomes a problem. This machine is a dual 3GHz Xeon /w 6GB of memory, running Fedora Core 5. I've flipped every bit in the kernel parameters that I dare via sysctl, with little to no effect on the poll time. Does anybody out there have any recommendations on a way to speed this up? Regards, -Eric Schwimmer Network Engineer UVA HSCS Network Engineering

5 7

Strange disk partitions being monitored
by gmbfly98＠gmail.com 28 Apr '06

28 Apr '06

Every now and then, Hobbit 4.1.2p1 reports strange directory names, and attempts to graph them in the Disk Utilization graphs. These are bogus directories and I'd like to remove them from the RRD history graphs. An example of these directories: /-/+ /--pidfile=/var/log/hobbit/hobbitlaunch.pid /5 /status How to I remove these from the RRD graphs without removing the history of all the other valid directories?

1 1