September 2007 - Xymon - Xymon mailman web

[hobbit] Disk Usage problems
by Robert-Manocchia＠IDEXX.com 04 Sep '07

04 Sep '07

We use an open source tool called hobbit to monitor file system disk usage among other things. Hobbit takes about 15 minutes to get up and running. The hobbit server is a virtual server located in the Westbrook data center. Please let me know how we can proceed. Robert Manocchia UNIX System Administrator IDEXX Laboratories 207 556-6860 EMail Robert-Manocchia at idexx.com -----Original Message----- From: Joshua Krause [mailto:joshuakrause at triad.rr.com] Sent: Tuesday, September 04, 2007 10:31 AM To: hobbit at hswn.dk Subject: [hobbit] Disk Usage problems I have a some servers grouped together with six other servers. Say I have "Server 1" and "Server 2" and Server 1 has the disks in a storage array mounted via /dev like so: Filesystem Size Used Avail Use% Mounted on /dev/sda2 7.9G 4.8G 2.7G 64% / /dev/sda1 99M 65M 30M 69% /boot /dev/sda3 24G 4.0G 19G 18% /export/home /dev/tr/17s 20G 202M 19G 2% /sys6 /dev/tr/8s 39G 749M 38G 2% /sys3 /dev/tr/11s 39G 878M 38G 3% /sys4 /dev/tr/14s 39G 886M 38G 3% /sys5 /dev/tr/5s 39G 875M 38G 3% /sys2 /dev/tr/2s 39G 837M 38G 3% /sys1 /dev/tr/23s 79G 5.0G 74G 7% /sys8 /dev/tr/20s 79G 4.7G 74G 6% /sys7 /dev/tr/17r 938G 36G 903G 4% /r6 /dev/tr/8r 1.9T 201G 1.7T 11% /r3 /dev/tr/11r 1.9T 225G 1.7T 12% /r4 /dev/tr/14r 1.9T 223G 1.7T 12% /r5 /dev/tr/2r 1.9T 199G 1.7T 11% /r1 /dev/tr/5r 1.9T 217G 1.7T 12% /r2 /dev/tr/20r 3.8T 1.4T 2.5T 36% /r7 /dev/tr/23r 3.8T 1.4T 2.5T 36% /r8 And Server 2 has what are called "vodfs" mounts which it mounts the drive via Server 1's ip like so: Filesystem Size Used Avail Use% Mounted on /dev/sda2 7.9G 4.6G 3.0G 61% / /dev/sda1 99M 34M 60M 37% /boot /dev/sda3 24G 2.8G 20G 13% /export/home 172.31.129.166:/dev/tr/5r 1.9T 217G 1.7T 12% /r2 172.31.129.166:/dev/tr/5s 39G 876M 38G 3% /sys2 172.31.129.166:/dev/tr/8s 39G 750M 38G 2% /sys3 172.31.129.166:/dev/tr/11s 39G 881M 38G 3% /sys4 172.31.129.166:/dev/tr/14r 1.9T 223G 1.7T 12% /r5 172.31.129.166:/dev/tr/14s 39G 887M 38G 3% /sys5 172.31.129.166:/dev/tr/11r 1.9T 225G 1.7T 12% /r4 172.31.129.166:/dev/tr/8r 1.9T 201G 1.7T 11% /r3 172.31.129.166:/dev/tr/2r 1.9T 199G 1.7T 11% /r1 172.31.129.166:/dev/tr/2s 39G 839M 38G 3% /sys1 172.31.129.166:/dev/tr/23r 3.8T 1.4T 2.5T 36% /r8 172.31.129.166:/dev/tr/17s 20G 202M 19G 2% /sys6 172.31.129.166:/dev/tr/23s 79G 5.0G 74G 7% /sys8 172.31.129.166:/dev/tr/17r 938G 36G 903G 4% /r6 172.31.129.166:/dev/tr/20s 79G 4.7G 74G 6% /sys7 172.31.129.166:/dev/tr/20r 3.8T 1.4T 2.5T 36% /r7 Now i need to have both servers monitoring the disk space because Server 2 is a backup to Server 1 and vice versa. So if I loose Server 1 then Server 2 automatically mounts the arrays as a /dev and not with the ip. My problem is that Server two is showing this: red /r2 (172% used) has reached the PANIC level (95%) red /sys2 (172% used) has reached the PANIC level (95%) red /sys3 (172% used) has reached the PANIC level (95%) red /sys4 (172% used) has reached the PANIC level (95%) red /r5 (172% used) has reached the PANIC level (95%) red /sys5 (172% used) has reached the PANIC level (95%) red /r4 (172% used) has reached the PANIC level (95%) red /r3 (172% used) has reached the PANIC level (95%) red /r1 (172% used) has reached the PANIC level (95%) red /sys1 (172% used) has reached the PANIC level (95%) red /r8 (172% used) has reached the PANIC level (95%) red /sys6 (172% used) has reached the PANIC level (95%) red /sys8 (172% used) has reached the PANIC level (95%) red /r6 (172% used) has reached the PANIC level (95%) red /sys7 (172% used) has reached the PANIC level (95%) red /r7 (172% used) has reached the PANIC level (95%) But as you can see from what the df -h command reads out they are no where near 100% let alone 172%. Does anyone have any suggestions? Thanks in advance To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

Disk Usage problems
by joshuakrause＠triad.rr.com 04 Sep '07

04 Sep '07

I have a some servers grouped together with six other servers. Say I have "Server 1" and "Server 2" and Server 1 has the disks in a storage array mounted via /dev like so: Filesystem Size Used Avail Use% Mounted on /dev/sda2 7.9G 4.8G 2.7G 64% / /dev/sda1 99M 65M 30M 69% /boot /dev/sda3 24G 4.0G 19G 18% /export/home /dev/tr/17s 20G 202M 19G 2% /sys6 /dev/tr/8s 39G 749M 38G 2% /sys3 /dev/tr/11s 39G 878M 38G 3% /sys4 /dev/tr/14s 39G 886M 38G 3% /sys5 /dev/tr/5s 39G 875M 38G 3% /sys2 /dev/tr/2s 39G 837M 38G 3% /sys1 /dev/tr/23s 79G 5.0G 74G 7% /sys8 /dev/tr/20s 79G 4.7G 74G 6% /sys7 /dev/tr/17r 938G 36G 903G 4% /r6 /dev/tr/8r 1.9T 201G 1.7T 11% /r3 /dev/tr/11r 1.9T 225G 1.7T 12% /r4 /dev/tr/14r 1.9T 223G 1.7T 12% /r5 /dev/tr/2r 1.9T 199G 1.7T 11% /r1 /dev/tr/5r 1.9T 217G 1.7T 12% /r2 /dev/tr/20r 3.8T 1.4T 2.5T 36% /r7 /dev/tr/23r 3.8T 1.4T 2.5T 36% /r8 And Server 2 has what are called "vodfs" mounts which it mounts the drive via Server 1's ip like so: Filesystem Size Used Avail Use% Mounted on /dev/sda2 7.9G 4.6G 3.0G 61% / /dev/sda1 99M 34M 60M 37% /boot /dev/sda3 24G 2.8G 20G 13% /export/home 172.31.129.166:/dev/tr/5r 1.9T 217G 1.7T 12% /r2 172.31.129.166:/dev/tr/5s 39G 876M 38G 3% /sys2 172.31.129.166:/dev/tr/8s 39G 750M 38G 2% /sys3 172.31.129.166:/dev/tr/11s 39G 881M 38G 3% /sys4 172.31.129.166:/dev/tr/14r 1.9T 223G 1.7T 12% /r5 172.31.129.166:/dev/tr/14s 39G 887M 38G 3% /sys5 172.31.129.166:/dev/tr/11r 1.9T 225G 1.7T 12% /r4 172.31.129.166:/dev/tr/8r 1.9T 201G 1.7T 11% /r3 172.31.129.166:/dev/tr/2r 1.9T 199G 1.7T 11% /r1 172.31.129.166:/dev/tr/2s 39G 839M 38G 3% /sys1 172.31.129.166:/dev/tr/23r 3.8T 1.4T 2.5T 36% /r8 172.31.129.166:/dev/tr/17s 20G 202M 19G 2% /sys6 172.31.129.166:/dev/tr/23s 79G 5.0G 74G 7% /sys8 172.31.129.166:/dev/tr/17r 938G 36G 903G 4% /r6 172.31.129.166:/dev/tr/20s 79G 4.7G 74G 6% /sys7 172.31.129.166:/dev/tr/20r 3.8T 1.4T 2.5T 36% /r7 Now i need to have both servers monitoring the disk space because Server 2 is a backup to Server 1 and vice versa. So if I loose Server 1 then Server 2 automatically mounts the arrays as a /dev and not with the ip. My problem is that Server two is showing this: red /r2 (172% used) has reached the PANIC level (95%) red /sys2 (172% used) has reached the PANIC level (95%) red /sys3 (172% used) has reached the PANIC level (95%) red /sys4 (172% used) has reached the PANIC level (95%) red /r5 (172% used) has reached the PANIC level (95%) red /sys5 (172% used) has reached the PANIC level (95%) red /r4 (172% used) has reached the PANIC level (95%) red /r3 (172% used) has reached the PANIC level (95%) red /r1 (172% used) has reached the PANIC level (95%) red /sys1 (172% used) has reached the PANIC level (95%) red /r8 (172% used) has reached the PANIC level (95%) red /sys6 (172% used) has reached the PANIC level (95%) red /sys8 (172% used) has reached the PANIC level (95%) red /r6 (172% used) has reached the PANIC level (95%) red /sys7 (172% used) has reached the PANIC level (95%) red /r7 (172% used) has reached the PANIC level (95%) But as you can see from what the df -h command reads out they are no where near 100% let alone 172%. Does anyone have any suggestions? Thanks in advance

1 0

monitoring xen machine status from Dom0
by tomg＠mcclatchyinteractive.com 04 Sep '07

04 Sep '07

Has anyone started thinking about or working on script(s) to monitor xen machines from the Dom0? It would be nice to have some of the data that xentop and other tools display in Hobbit and I would rather work with others on this list to develop the scripts rather than do everything on my own, if possible. I'm really new to Xen, so I still have a lot to learn. Tom -- Tom Georgoulias Sr. Systems Engineer McClatchy Interactive tomg at mcclatchyinteractive.com

2 3

patch to support new "st" vmstat field in RHEL5
by tomg＠mcclatchyinteractive.com 04 Sep '07

04 Sep '07

Red Hat Enterprise Linux 5 adds support for a 17th field in vmstat: "st" or %CPU stolen from a virtual machine. I'm using procps-3.2.7-8.1.el5, the latest production procps from Red Hat Network. I made a simple one line patch that adds this field to the default Linux OS layout, rather than create OS classes for the two linux distros that I know are supporting it (RHEL5 and Fedora 7). I'm guessing that this patch will collect and display a lot of NaN data for some distros, but I thought I'd pass it along. If there is a better way to handle this, let me know. Tom -- Tom Georgoulias Sr. Systems Engineer McClatchy Interactive tomg at mcclatchyinteractive.com

1 0

Problem since upgrading to 4.2.0 from 4.1 - "data flooding from"
by rob.macgregor＠gmail.com 04 Sep '07

04 Sep '07

I've just upgraded my Hobbit server from 4.1 to 4.2 (on a Gentoo 2006.1 host), having had a need for "Disabled until OK" :) The trouble that I now have is that reports from the client part of the server install are never making it through - all tests have turned purple and the log is full of "data flooding from 127.0.0.1, closing connection". I even reverted the client directory to the default from the installer, with no change. Pointers to how to further diagnose this, or even better resolve it, appreciated. -- Please keep list traffic on the list. Rob MacGregor Whoever fights monsters should see to it that in the process he doesn't become a monster. Friedrich Nietzsche

2 2

[hobbit] Hobbit Server performance
by xbgmsharp＠gmail.com 04 Sep '07

04 Sep '07

Hi, Here is a solution in order to reduce I/O which is a lot consuming. I load all the rrd and hist and webserver (cgi-bin,www,web,secure-cgi) into different tmpfs (http://en.wikipedia.org/wiki/TMPFS) This way everything is load into memory (cached). # free -k total used free shared buffers cached Mem: 6229400 5570816 658584 0 466896 4685444 I have 6G of memory and i set 2G for rrd and the other 2G for hist and a very small one for the webserver 64M. I check 2896 hosts. I have reduce my iowait% for 60-80% load to 0%. The webinterface is much faster. I order to write data on the disk rsync data twice a day. All of this does'nt execpt for having a local caching DNS daemon and many other tunning parameters. You can also: * reduce the timeout TCP/IP: echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout echo 1800 > /proc/sys/net/ipv4/tcp_keepalive_intvl echo 3000 > /proc/sys/net/ipv4/tcp_keepalive_time echo 0 > /proc/sys/net/ipv4/tcp_window_scaling echo 0 > /proc/sys/net/ipv4/tcp_sack echo 0 > /proc/sys/net/ipv4/tcp_timestamps * increase TCP/IP buffer echo 262144 > /proc/sys/net/core/rmem_max echo 262144 > /proc/sys/net/core/wmem_max * mount hobbit partition with noatime option and for more security nodev,noexec,nosuid * unilimit cpu,mem,files for hobbit user * increase inodes number on partition because of rrd and hist and clientdata directory. * Controls the memory # Controls the default maxmimum size of a mesage queue kernel.msgmax = 65536 # Controls the maximum shared segment size, in bytes kernel.shmmax = 4294967295 # Controls the maximum number of shared memory segments, in pages kernel.shmall = 268435456 kernel.shmmax = 536870912 Regards, KaYa On 8/29/07, Henrik Stoerner <henrik (at) hswn.dk> wrote: > On Wed, Aug 29, 2007 at 04:14:58PM -0500, Jeff Newman wrote: > > The server is a DL360 G3 with two 3ghz processors and 2gb of RAM. > It does only > > hobbit and cricket (10 or so routers, so not a huge hit) > > > > I've noticed the following two things: > > > > #1. I am constantly using 100% of my physical ram (no swap really) > > > > Mem: 2075040k total, 2051928k used, 23112k free, 52172k buffers > > Swap: 0k total, 0k used, 0k free, 1744168k cached > > > > is this normal? Not sure how to tell which process specifically is > > eating up the most RAM. > > I think you're running Linux, right? > > The "1744168k cached" normally goes in the "Mem:" line, not the "Swap:" > line. And that's the clue to what's using your RAM: Linux uses available > RAM as a variable-size disk cache; the disk cache grows until just about > all of RAM is used. It usually makes sense, the cached data is a copy of > data which is stored on disk, so when an application needs more memory, > the diskcache memory can be freed instantly and allocated to the > application. > > Therefore, Linux boxes usually have very little "free" memory. > > The "top" utility can sort processes by memory-usage. Also, "ps -vax" > will tell you how much memory each process is using. > > > > #2. I am consistantly running with 2-300 ports in TIME_WAIT > involving port 1984 > > > > # netstat -an | grep TIME | grep 1984 | wc -l > > 253 > > > > Im not sure if this is normal either, nor am I sure if it is bad or > > not? > > It's not unusual, and quite harmless. The TIME_WAIT state happens when a > socket is closed; the operating system keeps the socket around for some > time (usually 20-30 secs, cannot remember what the Linux default is) to > make sure that all packets destined for this socket have been received > (there might be some duplicated/retransmitted packets still in transit > when the socket is closed). This makes sure that a new connection using > the same port will not see packets from the old connection. > > > > Here is the output from some of the "bb" buttons at the top of my > > hobbit page. If anything stands out, any help would be apprieciated. > > > > DNS statistics: > > # hostnames resolved : 138 > > # succesful : 134 > > # failed : 4 > > DNS lookups completed 1188421854.099983 > 9.239654 > > Your DNS lookups are a bit slow - 9 seconds for 138 DNS lookups. Nothing > critical, just a bit slower than I'd expect. Installing a local caching > DNS daemon is a good way to eliminate this problem. > > > Regards, > Henrik > > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe (at) hswn.dk > > >

2 1

New beta hobbit - rrd disk cache and graph display
by wxxx333＠gmail.com 04 Sep '07

04 Sep '07

Hi, As always kudos Henrik for the excellent work. I didn't start do play with the new hobbit beta version, who good improvements about the disk I/O limitations on huge load systems. I have a specific question about what's hobbit behavior when disk cache for rrd is active, and i request the graph for a specific metric. Will i not see the last (around) 30 minutes of info on the graph? Or is there some kind "trap" activated when i request a specific graph, who'll request that the info for that host/service should be written to the RRD before the graph is generated? Regards Werner

3 2

Msgs issue on solaris 10 and a dns issue on centos 4.4
by joshuakrause＠triad.rr.com 03 Sep '07

03 Sep '07

First Issue: I have a couple of solaris machines that range from 8-10 in versions. They all have the hobbit client installed on them and I am retrieving data back from all of them except for one. On that one I get all of the tests I am checking for except the messages. And I've looked on the server and there is a /var/adm/messages file and it has 644 permissions like all of the other Hobbit Clients I run but this one doesn't report anything back. Second Issue: I am monitoring 2 dns servers running CentOS 4.4. They are running the Hobbit Client and were working fine until I added a couple of DNS entries via webmin and had Webmin restart dns. After dns restarted and came back up hobbit has it marked as down and it hasn't come back up in 4 days. I have tried removing the hosts and making sure the logs are all removed. But nothing has worked.

1 0

rrd network traffic graphs
by joshuakrause＠triad.rr.com 03 Sep '07

03 Sep '07

I noticed that the network graphs on the trends column show up in bits or bytes how can i change them to show mb. So instead of 40000000.0 it wil show 40.0. As always thanks in advance

2 3

monitoring a file
by joshuakrause＠triad.rr.com 03 Sep '07

03 Sep '07

I am wanting to monitor a file to make sure it doesn't get any smaller than 500k. This is on a remote machine that has the client software setup on it. On the remote machine i put this entry in the localclient.cfg that looked like this: FILE /export/home/dncs/mystro/bfs/Compass_OOB/rt.grp.z SIZE<800 I am not getting anything to report back on the BBDISPLAY though. any ideas would be appreciated.

2 1