November 2007 - Xymon - Xymon mailman web

[hobbit] strange graph behavior - random machines & graphs
by gumby3203＠gmail.com 30 Nov '07

30 Nov '07

I have recently noticed a strange thing happening with some of the rrd graphs generated by Hobbit. When you look at the graph, it looks as though the rrd data is one one format (gauge), but the graph is generating it in a different format (derive). I can't seem to find any pattern to the hosts or tests that are exhibiting this strange behavior, and it is only happening on a handful of graphs. I have attached a picture of one of these graphs, since I'm not really sure how to describe it. Note the huge numbers displayed on the curr/min/avg/max line. Any idea what's going on here? When I dump the RRD file manually, everything looks okay. I'm running Hobbit 4.2.0 with the 2007-02-09 allinone patch (I believe the latest). This has only happened in the past few weeks, though when exactly it started, I don't know. Any ideas?

6 36

[hobbit] Hobbit server doesn't start on CentOS 5 with hobbit-4.2.0 installed
by hobbit＠razorsedge.org 30 Nov '07

30 Nov '07

So, for the record, there is nothing wrong with the RPM? -- -m Sebastian wrote: > I looked at that already and it seemed high. Mine is also: > kernel.shmmax = 4294967295 > > I found the problem though: > max number of segments = 4096 > segments allocated 4096 > > Fixed with: > sysctl -w kernel.shmmni=8192 > kernel.shmmni = 8192 > > Thanks for making me look at ipcs again though. > > Sebastian > > > > _____ > > From: Josh Luthman [mailto:josh at imaginenetworksllc.com] > Sent: 30 November 2007 15:18 > To: hobbit at hswn.dk > Subject: Re: [hobbit] Hobbit server doesn't start on CentOS 5 with > hobbit-4.2.0 installed > > > FTR my Hobbit server: > > -> sysctl -a|grep shmmax > kernel.shmmax = 4294967295 > > > On 11/30/07, HAMPL Pavel <phampl at csob.cz <mailto:phampl at csob.cz> > wrote: > > Hi, > > What about to increase shmmax in the system when it complains that cannot > get allocated new 262kB? > You can check what is your size: # sysctl -a|grep shmmax > This cmd says you what is already used: # ipcs -m > > You can increase it in /etc/sysctl (just put there something big, it is > just > a restriction, kernel.shmmax = XXXX) and then apply it by : # sysctl -p > > Cheers, > > Pavel Hampl > > > _____ > > From: Sebastian [mailto:spa at syntec.co.uk] > Sent: Friday, November 30, 2007 3:47 PM > To: hobbit at hswn.dk > Subject: RE: [hobbit] Hobbit server doesn't start on CentOS 5 with > hobbit-4.2.0 installed > > > > I uninstalled hobbit-4.2.0-7.el5.re.i386.rpm and compiled hobbit-4.2.0 > from > source with the allinone.patch and get exactly the same error. > > It might be worth noting that this was originally a minimum CentOS 5 > install > - I had to install gcc-c++ (and rrd-devel and pcre-devel) to compile > hobbit, > and I previously had to install gcc for compiling some other software - > all > from yum. > > The server has 2 GB of RAM but is running several other services. > > Any other ideas? > > Sebastian > > > > _____ > > From: Josh Luthman [mailto:josh at imaginenetworksllc.com] > Sent: 29 November 2007 18:44 > To: hobbit at hswn.dk > Subject: Re: [hobbit] Hobbit server doesn't start on CentOS 5 with > hobbit-4.2.0-7.el5.re.i386.rpm installed > > > Just compile it from source. If you don't know how to do that I can give > you instructions command by command, just ask. I use CentOS5 on my boxes > but I've never bothered with the RPM. > > > On 11/29/07, Sebastian <spa at syntec.co.uk> wrote: > > I installed an RPM of hobbit-4.2.0 called hobbit-4.2.0-7.el5.re.i386.rpm, > which I probably got from here: > > <http://rpm.razorsedge.org/centos-5/RE/repodata/repoview/hobbit-4-4.2.0-7.el > 5.re.html> > http://rpm.razorsedge.org/centos-5/RE/repodata/repoview/hobbit-4-4.2.0-7.el5 > .re.html > It includes the All-in-one patch. So I configured the cfg files and > started > hobbit, but I get this in hobbitlaunch.log: > > 2007-11-29 17:33:47 hobbitlaunch starting > 2007-11-29 17:33:47 Loading tasklist configuration from > /usr/libexec/hobbit/server/etc/hobbitlaunch.cfg > 2007-11-29 17:33:47 Loading hostnames > 2007-11-29 17:33:47 Loading saved state > 2007-11-29 17:33:47 Cannot access checkpoint file > /usr/libexec/hobbit/server/tmp/hobbitd.chk for restore > 2007-11-29 17:33:47 Setting up network listener on 0.0.0.0:1984 > 2007-11-29 17:33:47 Setting up signal handlers > 2007-11-29 17:33:47 Setting up hobbitd channels > 2007-11-29 17:33:47 Could not get shm of size 262144: No space left on > device > 2007-11-29 17:33:47 Cannot setup status channel > 2007-11-29 17:33:47 Task hobbitd terminated, status 1 > > This is the first time I've started it, so I presume that this is normal: > "Cannot access checkpoint file /usr/libexec/hobbit/server/tmp/hobbitd.chk > for restore". The problem looks like this: "Could not get shm of size > 262144: No space left on device". I see that there are two similar issues > listed on <http://www.hswn.dk/hobbit/help/known-issues.html> > http://www.hswn.dk/hobbit/help/known-issues.html that relate to FreeBSD, > but > none of the sysctl keys listed are in this version of Linux: > > # cat /proc/version > Linux version 2.6.18-8.1.15.el5.centos.plus ( > <mailto:mockbuild at builder6.centos.org> mockbuild at builder6.centos.org ) > (gcc > version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Thu Oct 25 06:49:40 EDT > 2007 > > Google returns 1 result for "Could not get shm of size" "No space left on > device" and that was the Known Issues page I've already checked. Any > ideas? > > Also, I noticed that the hobbitd.pid file did not get deleted when hobbitd > terminated. It seems like it should have been. > > Many thanks, > > Sebastian >

3 4

[hobbit] RE: CPU Utilization, Not Good
by jkwade＠futurefrontiers.com 30 Nov '07

30 Nov '07

Added the sar descriptions Thanks.James _____ From: James Wade [mailto:jkwade at futurefrontiers.com] Sent: Friday, November 30, 2007 12:07 PM To: hobbit at hswn.dk Subject: [hobbit] RE: CPU Utilization, Not Good I thought I'd give some more information. I'm using solaris, and cpu utilization used vmstat. If I do a sar I get this: %usr %sys %wio %idle 10:10:00 23 5 6 66 10:20:00 46 5 4 44 10:30:00 69 7 7 18 10:40:00 76 7 4 13 10:50:01 81 6 2 11 11:00:00 69 7 4 20 11:10:01 71 9 5 15 11:20:01 88 7 1 4 Average 23 5 6 66 Notice that cpu utilization from about 10:30 to 11:20 is high. However, with Hobbit, I get a solid 23% average for everything. the hobbit-client-sunos.sh uses: vmstat 300 2 which gives met this: $ vmstat 300 2 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s3 s6 s9 in sy cs us sy id 0 0 0 166920152 16811328 878 1814 1902 55 55 0 0 0 0 0 0 1755 3374 6188 16 4 80 0 0 0 166919832 16811024 878 1814 1902 55 55 0 0 0 0 0 0 1756 3376 6188 16 4 80 Notice that a 5 minute average from vmstat shows 80% idle, yet the sar above doesn't show that. Is there anyway to get Hobbit to use sar for CPU utilization instead of vmstat on solaris? Thanks.James _____ From: James Wade [mailto:jkwade at futurefrontiers.com] Sent: Friday, November 30, 2007 11:21 AM To: 'hobbit at hswn.dk' Subject: CPU Utilization, Not Good Hi, I've asked this several times, but I really have not received a good answer. Hobbit doesn't really seem to monitor cpu utilization correctly. For example, I can look at the load, login to the box, and then run top. Top shows that cpu utilization is at 100%, 0% idle time. However, the "trends" utilization graph shows 20% cpu utilization. Can someone tell me why? Thanks..James James Wade

1 0

CPU Utilization, Not Good
by jkwade＠futurefrontiers.com 30 Nov '07

30 Nov '07

I thought I'd give some more information. I'm using solaris, and cpu utilization used vmstat. If I do a sar I get this: 10:10:00 23 5 6 66 10:20:00 46 5 4 44 10:30:00 69 7 7 18 10:40:00 76 7 4 13 10:50:01 81 6 2 11 11:00:00 69 7 4 20 11:10:01 71 9 5 15 11:20:01 88 7 1 4 Average 23 5 6 66 Notice that cpu utilization from about 10:30 to 11:20 is high. However, with Hobbit, I get a solid 23% average for everything. the hobbit-client-sunos.sh uses: vmstat 300 2 which gives met this: $ vmstat 300 2 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s3 s6 s9 in sy cs us sy id 0 0 0 166920152 16811328 878 1814 1902 55 55 0 0 0 0 0 0 1755 3374 6188 16 4 80 0 0 0 166919832 16811024 878 1814 1902 55 55 0 0 0 0 0 0 1756 3376 6188 16 4 80 Notice that a 5 minute average from vmstat shows 80% idle, yet the sar above doesn't show that. Is there anyway to get Hobbit to use sar for CPU utilization instead of vmstat on solaris? Thanks.James _____ From: James Wade [mailto:jkwade at futurefrontiers.com] Sent: Friday, November 30, 2007 11:21 AM To: 'hobbit at hswn.dk' Subject: CPU Utilization, Not Good Hi, I've asked this several times, but I really have not received a good answer. Hobbit doesn't really seem to monitor cpu utilization correctly. For example, I can look at the load, login to the box, and then run top. Top shows that cpu utilization is at 100%, 0% idle time. However, the "trends" utilization graph shows 20% cpu utilization. Can someone tell me why? Thanks..James James Wade

1 0

[hobbit] aix help
by rebecca_henderson＠link.freedom.com 30 Nov '07

30 Nov '07

Thanks I got everything installed. -----Original Message----- From: hobbit at hswn.dk [mailto:hobbit at hswn.dk] Sent: Friday, November 30, 2007 8:54 AM To: Rebecca Henderson; hobbit at hswn.dk Subject: Re: [hobbit] aix help Importance: Low Also, if you *know* that all the dependencies are met, you can force it to install... "rpm -i --nodeps <RPMFILE>" Peter On Wed, 2007-11-28 at 12:33 -0800, Rebecca Henderson wrote: >> I'm trying to install the RPM for AIX client for hobbit and It >> complains about >> >> /bin/bash is needed by hobbit-client-4.2.0-7.aix5.3.re >> >> /bin/sh is needed by hobbit-client-4.2.0-7.aix5.3.re >> >> libc.a(shr.o) is needed by hobbit-client-4.2.0-7.aix5.3.re >> >> >> >> they are all there, can anyone help with this? >> >> Thanks in advance, >> >> >> >> > > To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

Hobbit server doesn't start on CentOS 5 with hobbit-4.2.0-7.el5.re.i386.rpm installed
by spa＠syntec.co.uk 30 Nov '07

30 Nov '07

I installed an RPM of hobbit-4.2.0 called hobbit-4.2.0-7.el5.re.i386.rpm, which I probably got from here: http://rpm.razorsedge.org/centos-5/RE/repodata/repoview/hobbit-4-4.2.0-7.el5 .re.html It includes the All-in-one patch. So I configured the cfg files and started hobbit, but I get this in hobbitlaunch.log: 2007-11-29 17:33:47 hobbitlaunch starting 2007-11-29 17:33:47 Loading tasklist configuration from /usr/libexec/hobbit/server/etc/hobbitlaunch.cfg 2007-11-29 17:33:47 Loading hostnames 2007-11-29 17:33:47 Loading saved state 2007-11-29 17:33:47 Cannot access checkpoint file /usr/libexec/hobbit/server/tmp/hobbitd.chk for restore 2007-11-29 17:33:47 Setting up network listener on 0.0.0.0:1984 2007-11-29 17:33:47 Setting up signal handlers 2007-11-29 17:33:47 Setting up hobbitd channels 2007-11-29 17:33:47 Could not get shm of size 262144: No space left on device 2007-11-29 17:33:47 Cannot setup status channel 2007-11-29 17:33:47 Task hobbitd terminated, status 1 This is the first time I've started it, so I presume that this is normal: "Cannot access checkpoint file /usr/libexec/hobbit/server/tmp/hobbitd.chk for restore". The problem looks like this: "Could not get shm of size 262144: No space left on device". I see that there are two similar issues listed on http://www.hswn.dk/hobbit/help/known-issues.html that relate to FreeBSD, but none of the sysctl keys listed are in this version of Linux: # cat /proc/version Linux version 2.6.18-8.1.15.el5.centos.plus (mockbuild at builder6.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Thu Oct 25 06:49:40 EDT 2007 Google returns 1 result for "Could not get shm of size" "No space left on device" and that was the Known Issues page I've already checked. Any ideas? Also, I noticed that the hobbitd.pid file did not get deleted when hobbitd terminated. It seems like it should have been. Many thanks, Sebastian

6 16

aix help
by rebecca_henderson＠link.freedom.com 30 Nov '07

30 Nov '07

I'm trying to install the RPM for AIX client for hobbit and It complains about /bin/bash is needed by hobbit-client-4.2.0-7.aix5.3.re /bin/sh is needed by hobbit-client-4.2.0-7.aix5.3.re libc.a(shr.o) is needed by hobbit-client-4.2.0-7.aix5.3.re they are all there, can anyone help with this? Thanks in advance,

5 5

view question
by rebecca_henderson＠link.freedom.com 29 Nov '07

29 Nov '07

How do I take the bbgen and bbtest headers out of the view for my clients? Thanks! -----Original Message----- From: hobbit at hswn.dk [mailto:hobbit at hswn.dk] Sent: Thursday, November 29, 2007 12:43 PM To: Rebecca Henderson; hobbit at hswn.dk Subject: RE: [hobbit] File existence monitoring Importance: Low Well, I finally have file existence working with changing filenames. Thanks for the suggestion Charles. It wouldn't exactly work for me though since the files may never be created. The documentation of client-local.cfg says "Putting wildcards directly into the file: entry will not work." But, you can do something like this: file:`echo /usr/local/scripts/logs/logfile.*` Since the beginning of the filename is always the same, doing the echo command on the filename still reported a name that Hobbit could check against. Then, in the hobbit-client.cfg, I have this: FILE %^\/usr\/local\/scripts\/logs\/logfile.* SIZE>1K This checks that the file is there and that it's over 1K in size. Thanks! D. Stuffle ======================================================================== === From: Charles Jones [mailto:jonescr at cisco.com] Sent: Wednesday, November 28, 2007 7:23 PM To: hobbit at hswn.dk Subject: Re: [hobbit] File existence monitoring It won't work the way you are doing it now. You are using an ls command to dynamically create the list of files to check in your client-local, so when that file is missing, the ls command doesn't output it as a filename to check, so the test drops off when that file dissappears, Since the files are dynamic, I assume created every day, you are going to have to figure out a way to tell if one of them goes away. Maybe you could have a cron job that runs every morning, and does an ls of the files and saves the output somewhere like /var/tmp/todays_files.txt, and then in your client-local you could do something like: [hostname] file:`head -1 /var/tmp/todays_files.txt|tail -1` file:`head -2 /var/tmp/todays_files.txt|tail -1` file:`head -3 /var/tmp/todays_files.txt|tail -1` ... That would add a file entry for every line of todays_files.txt. So if one of the files got deleted, it would still be referenced in todays_files.txt, and thus continue to be monitored (including notification that it was no longer there). That is a frightful way to do it, but I can't think of anything better at the moment :) Dealing with randomly generated filenames is a pain. -Charles Stuffle, David (Corporate) wrote: I found part of my problem of why the file existence isn't working. I wasn't specifying the full path in the hobbit-clients.cfg file. I also had to escape the forward slashes and underscores with backslashes. Plus, the "red" parameter wasn't valid from what the log said. But still, I want to know if a file doesn't exist. The hobbitcleint.log file on the client just keeps saying "ls: /usr/local/scripts/logs/dcsmnj8yecotnwbcevyym9t8c_4d7i*: No such file or directory". Therefore, it doesn't report anything about it. But I want to be alerted that it doesn't exist. Is anyone successfully monitoring file existence? D. Stuffle ======================================================================== === From: Stuffle, David (Corporate) [[ mailto:David.Stuffle at adesa.com ]mailto:David.Stuffle at adesa.com] Sent: Tuesday, November 27, 2007 5:15 PM To: [ mailto:hobbit at hswn.dk ]hobbit at hswn.dk Subject: RE: [hobbit] File existence monitoring Thanks Greg, The filenames do change, they have the date appended to them. I'm wanting to check that the files are there and that they're not 0 bytes. I know that when I was first setting it up I messed up something and it said "File missing" when it didn't find the file. Now it doesn't seem to care. D. Stuffle ======================================================================== === From: Hubbard, Greg L [[ mailto:greg.hubbard at eds.com ]mailto:greg.hubbard at eds.com] Sent: Tuesday, November 27, 2007 4:49 PM To: hobbit at hswn.dk Subject: RE: [hobbit] File existence monitoring If the file names never change, then specify the file name in the client-local.cfg instead of running the ls command. If you delete the file, then the ls command will return nothing. That is what I would try first, anyway. Or, it could be your check -- you are only asking for an alarm if the file size is bigger than 1K. A missing file is not bigger than 1K is it? And it seems like you need to do some other things as well to get this correct. I am working from memory... GLH ======================================================================== === From: Stuffle, David (Corporate) [[ mailto:David.Stuffle at adesa.com ]mailto:David.Stuffle at adesa.com] Sent: Tuesday, November 27, 2007 2:59 PM To: hobbit at hswn.dk Subject: [hobbit] File existence monitoring Hi, I'm wanting to check the existence of specific files in a directory and that they are at leat 1K in size. I have this in the hobbit-clients.cfg file: HOST=hostname1 PROC httpd FILE %^sdc_.* MTIME<3600 TIME=W:0730:2200 red FILE %^dcs1xaz9saotnw713cmtaut8c_6w3j.* red size>1K TIME=*:0730:1800 FILE %^dcsaplg0kbotnwvrj16x28msg_1i2n.* red size>1K TIME=*:0730:1800 FILE %^dcsdq1p60cotnw32xcywlbisg_9i5q.* red size>1K TIME=*:0730:1800 FILE %^dcsmnj8yecotnwbcevyym9t8c_4d7i.* red size>1K TIME=*:0730:1800 FILE %^dcsp6ro5kbotnwvjh6yh0dmsg_3j1y.* red size>1K TIME=*:0730:1800 FILE %^dcsu5rokrbotnwb01tmd19mat_4w7s.* red size>1K TIME=*:0730:1800 And this in client-local.cfg: [hostname1] file:`find /usr/local/webtrends/sdc/weblog -type f | xargs ls -1tr | tail -1` file:`ls -1 /usr/local/scripts/logs/dcs1xaz9saotnw713cmtaut8c_6w3j*` file:`ls -1 /usr/local/scripts/logs/dcsaplg0kbotnwvrj16x28msg_1i2n*` file:`ls -1 /usr/local/scripts/logs/dcsdq1p60cotnw32xcywlbisg_9i5q*` file:`ls -1 /usr/local/scripts/logs/dcsmnj8yecotnwbcevyym9t8c_4d7i*` file:`ls -1 /usr/local/scripts/logs/dcsp6ro5kbotnwvjh6yh0dmsg_3j1y*` file:`ls -1 /usr/local/scripts/logs/dcsu5rokrbotnwb01tmd19mat_4w7s*` They all show up under the files column. To test it, I deleted the file specified in the last line from the directory, but all Hobbit does is remove it from displaying. I want it to go red saying that the file does not exist. The man pages says "By default, a warning is triggered for files that have a FILE entry, but which do not exist." But, I'm not seeing this occur. Any ideas? D. Stuffle DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this document and any attachments. DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this document and any attachments. DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this document and any attachments. DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this document and any attachments.

1 0

File existence monitoring
by David.Stuffle＠adesa.com 29 Nov '07

29 Nov '07

Hi, I'm wanting to check the existence of specific files in a directory and that they are at leat 1K in size. I have this in the hobbit-clients.cfg file: HOST=hostname1 PROC httpd FILE %^sdc_.* MTIME<3600 TIME=W:0730:2200 red FILE %^dcs1xaz9saotnw713cmtaut8c_6w3j.* red size>1K TIME=*:0730:1800 FILE %^dcsaplg0kbotnwvrj16x28msg_1i2n.* red size>1K TIME=*:0730:1800 FILE %^dcsdq1p60cotnw32xcywlbisg_9i5q.* red size>1K TIME=*:0730:1800 FILE %^dcsmnj8yecotnwbcevyym9t8c_4d7i.* red size>1K TIME=*:0730:1800 FILE %^dcsp6ro5kbotnwvjh6yh0dmsg_3j1y.* red size>1K TIME=*:0730:1800 FILE %^dcsu5rokrbotnwb01tmd19mat_4w7s.* red size>1K TIME=*:0730:1800 And this in client-local.cfg: [hostname1] file:`find /usr/local/webtrends/sdc/weblog -type f | xargs ls -1tr | tail -1` file:`ls -1 /usr/local/scripts/logs/dcs1xaz9saotnw713cmtaut8c_6w3j*` file:`ls -1 /usr/local/scripts/logs/dcsaplg0kbotnwvrj16x28msg_1i2n*` file:`ls -1 /usr/local/scripts/logs/dcsdq1p60cotnw32xcywlbisg_9i5q*` file:`ls -1 /usr/local/scripts/logs/dcsmnj8yecotnwbcevyym9t8c_4d7i*` file:`ls -1 /usr/local/scripts/logs/dcsp6ro5kbotnwvjh6yh0dmsg_3j1y*` file:`ls -1 /usr/local/scripts/logs/dcsu5rokrbotnwb01tmd19mat_4w7s*` They all show up under the files column. To test it, I deleted the file specified in the last line from the directory, but all Hobbit does is remove it from displaying. I want it to go red saying that the file does not exist. The man pages says "By default, a warning is triggered for files that have a FILE entry, but which do not exist." But, I'm not seeing this occur. Any ideas? D. Stuffle DISCLAIMER: The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this document and any attachments.

3 5

Looking to monitor APC, SonicWall, and Apple Xraids
by eric＠iconla.com 29 Nov '07

29 Nov '07

If anyone has any experience monitoring APC UPS, SonicWall, or Apple Xraids with hobbit, please let me know. I'd like to know what is available to monitor. -Eric

1 0