August 2005 - Xymon - Xymon mailman web

[hobbit] Re: hobbit_rrd stops working after about 1 hour
by tpr＠pics.com 24 Aug '05

24 Aug '05

Olivier, Why don't you try the approach of putting your rrd files in a tmpfs filesystem? This seems to have resolved my rrd problem. At least you can try to see if this resolves your issue and then you know for sure it is related to disk IO. This is what I did: 1. mkdir /usr/local/bbvar/rrd_orig 2. mv /usr/local/bbvar/rrd /usr/local/bbvar/rrd_orig 3. mkdir /usr/local/bbvar/rrd 4. Add the following line to /etc/fstab: tmpfs /usr/local/bbvar/rrd tmpfs mode=755,rw,size=2G 0 0 5. mount /usr/local/bbvar/rrd; chown <id of bb user> /usr/local/bbvar/rrd 6. cp -pr /usr/local/bbvar/rrd_orig/rrd/* /usr/local/bbvar/rrd 7. Start hobbit If you want to keep this as a permanent solution, then you will need to setup a cronjob to periodically copy the rrd files from the tmpfs filesystem back to disk. This is because if you unmount the tmpfs FS all data will be lost. You can put a line in crontab as shown below to run at 8:30 PM daily: 30 20 * * * rsync -av /usr/local/bbvar/rrd /usr/local/bbvar/rrd_orig Remember that everytime you reboot, you will need to copy the files from disk to the tmpfs filesystem. You can put a line in /etc/rc.local to do this for you. Hope this helps. -Naeem Olivier Beau <olivier at qalpit.c om> To hobbit at hswn.dk 08/22/2005 11:28 cc PM Subject Re: [hobbit] Re: hobbit_rrd stops Please respond to working after about 1 hour hobbit at hswn.dk Hi, it happened a third time for me this night (3 times in 3 weeks) : syptoms: hobbitd seems to slow down and stops graphing. i think Naeem and me are hitting a bug. i've looked closer this night, and i saw that hobbitd_rrd was running at 100% on the cpu it was on; i tried to strace the procees, but strace wouldnt give me any ouptut ! i finally killed hobbitd_rrd, and everything went back to normal. hobbitd.log has : Task rrdstatus terminated, status 1 rrd_status.log has : Worker process died with exit code 1, terminating during normal running, vmstat shows a i/o wait of 25% my problems happened always at night, exactly at the time legato starts -> something strange is happening whith hobbitd_rrd when the server is under very heavy i/o.. henrik, could this be a OS issue or more a hobbitd_rrd problem ? Olivier Selon Naeem.Maqsud at sybase.com: > Well, as nobody has suggested anything to my problem I guess that I'm the > only one having this issue. I have managed to find the root cause. The > hobbitd_rrd process was showing to be in "uninterruptible sleep" state most > of the time with high iowait associated with the CPU it was running on. I > suspected that the problem may be due to disk IO while updating rrds for > the 2000 hosts. > I created a tmpfs filesystem and copied the rrd directory into it. Since > then (48 hours ago) my rrd graphs have been updating continuously. I do > however need to write back to disk periodically to avoid loss of data after > a reboot. > > This is OK as a temporary fix but I would like to have a permanent > solution. I would like to hear from other hobbit users who have more than > 1000 hosts monitored. What type of servers and disk subsystems are they > using? Perhaps my problem is to do with RedHat and Dell server combination. > Perhaps I need to stripe over multiple spindles. > > -Naeem > > > > > Naeem > Maqsud/SYBASE > To > 08/18/2005 05:02 hobbit at hswn.dk > PM cc > > Subject > hobbit_rrd stops working after > about 1 hour > > > > > > > > > > > Hi, > > I'm testing out hobbit 4.1.1 for possible migration from big brother (with > bbgen). I suspected scalability issues with BB as my rrd graphs were > updated intermittently. However, hobbit is exhibiting similar problems. > After about 1 hr of restarting hobbit, the rrd graphs stop updating except > for the cpu utilization for the hobbit server itself. > > The hobbit server is running RedHat Linux AS 3.0. It has 2 x 2.4 GHz Xeon > processors and 1GB of memory. About 800 servers are sending updates to the > hobbit server. Another 1200 servers are getting remote tests. > > Load average has stayed below 1 most of the time. CPU usage has been low > with 75% idle. 4 CPUs show up due to hyperthreading and I've noticed that > after the restart of hobbit server, hobbitd_rrd process stays on CPU3 with > 100% utilization for the one hour that it is busy. > > I hope someone can shed some light on this. > > Thanks, > Naeem > > > > > To unsubscribe from the hobbit list, send an e-mail to > hobbit-unsubscribe at hswn.dk > > > -- Olivier Beau To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

[hobbit] Still here ...
by tpr＠pics.com 24 Aug '05

24 Aug '05

Just to make people stop worrying: No, I haven't dropped off the planet. I'm still here, and plan on getting some work done on Hobbit the coming days. I've just been kept busy with other work the past weeks, and on top of that a major birthday event happened this past week-end that kept me away from any kind of programming. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

BBtest
by hobbitsam＠gmail.com 24 Aug '05

24 Aug '05

Hi, We are having some servers duplicate (beause some tests). What can we do to stop the messages below ? we have set prefer flag. The other question is about RPCs tests.... This can be reduce using --concurrency=N ? Can This delay be caused if some hosts is down while hobbit is testing ? Thanks Ps.: What about AIX test ??? bbtest-net version 4.0.4 Statistics: Hosts total : 151 Hosts with no tests : 0 Total test count : 610 Status messages : 611 Alert status msgs : 0 Transmissions : 10 DNS statistics: # hostnames resolved : 131 # succesful : 129 # failed : 2 # calls to dnsresolve : 140 TCP test statistics: # TCP tests total : 339 # HTTP tests : 0 # Simple TCP tests : 339 # Connection attempts : 339 # bytes written : 2838 # bytes read : 14241 Error output: Host xxxx appears twice in bb-hosts! This may cause strange results Host xxxx appears twice in bb-hosts! This may cause strange results Host yyyy appears twice in bb-hosts! This may cause strange results Host yyyy appears twice in bb-hosts! This may cause strange results bbtest-net: Cannot resolve IP for host Flexlm WARNING: Runtime 415 longer than time limit (300) TIME SPENT Event Starttime Duration bbtest-net startup 1124913114.080027 - Service definitions loaded 1124913114.083415 0.003388 Tests loaded 1124913114.111832 0.028417 DNS lookups completed 1124913119.129270 5.017438 Test engine setup completed 1124913119.145836 0.016566 TCP tests completed 1124913129.999272 10.853436 PING test completed (141 hosts) 1124913135.028123 5.028851 PING test results sent 1124913135.042905 0.014782 Test result collection completed 1124913135.043258 0.000353 LDAP test engine setup completed 1124913135.043261 0.000003 LDAP tests executed 1124913135.043263 0.000002 LDAP tests result collection completed 1124913135.043264 0.000001 RPC tests executed 1124913528.915263 393.871999 NSLOOKUP tests executed 1124913528.918001 0.002738 Test results transmitted 1124913528.974950 0.056949 bbtest-net completed 1124913529.191534 0.216584 TIME TOTAL 415.111507

1 0

client logfile monitoring.
by tlp-hobbit＠holme-pedersen.dk 24 Aug '05

24 Aug '05

As I can read from the changes file, logfile monitoring is no longer supported nativly in the client. What do others do then for logfile monitoring the B style ? BR Thomas

4 4

Fw: [hobbit] hobbitclient diskcapacity
by lars.ebeling＠leopg9.no-ip.org 24 Aug '05

24 Aug '05

Hi, below is an extract from the clientdata: [who] root pts/ta Aug 12 10:10 root pts/tb Aug 16 05:48 [df] [vmstat] procs memory page faults cpu r b w avm free re at pi po fr de sr in sy cs us sy id 2 0 0 25363 11390 24 10 0 0 0 0 0 111 1013 73 8 3 89 3 0 0 24585 11168 28 4 0 0 0 0 0 110 1553 66 3 1 96 Filesystem 1024-blocks Used Available Capacity Mounted on /dev/vg01/lvol11 8714948 3292890 5422058 38% /dunihd99 /dev/vg00/lvol4 2548360 1068200 1480160 42% /home /dev/vg00/lvol5 2249944 1868272 381672 84% /opt /dev/vg00/lvol6 307200 217976 89224 71% /tmp /dev/vg00/lvol7 1635856 1303064 332792 80% /usr /dev/vg00/lvol8 1030528 287952 742576 28% /var /dev/vg00/lvol1 265520 45064 220456 17% /stand /dev/vg00/lvol3 203976 91640 112336 45% / [memory] Total:512 Seems like the headings are wrong. Regards Lars ----- Original Message ----- From: lars ebeling To: hobbit Sent: Tuesday, August 16, 2005 2:39 PM Subject: [hobbit] hobbitclient diskcapacity Hi, got this some times: Tue Aug 16 13:56:37 METDST 2005 - Filesystems NOT ok Expected string (Capacity and Mounted) not found in df output header System: leopg9 Tue Aug 16 13:56:37 2005 Regards Lars Ebeling http://leopg9.no-ip.org Hobbithobbyist "Never run after your own hat. Others will be delighted to do it; why spoil their fun?" -- Mark Twain

5 9

[hobbit] Client for AIX
by CHRIS.MORRIS＠RWEnpower.com 24 Aug '05

24 Aug '05

On Wed, Aug 24 at 09:48, Henrik wrote: >> But now the servers complains "No client backend for OS 'aix'". >Try installing the latest Hobbit snapshot on your server. >http://www.hswn.dk/beta/ I have installed the latest Hobbit snapshot and am running the AIX client config on an AIX 5.1 system but I am having a problem with the netstat data. In the larrd-data.log I get the following message : - 2005-08-24 16:11:19 RRD error updating /usr/local/hobbit/data/rrd/tru6707/netstat.rrd from 10.175.4.217: expected 16 data source readings (got 15) from 1124896279:31882533:4760259:0:8141380:16032146:U:U:3145266384:3098051523:426 6285002:493162550:3214756185:220948442:6077963:339657:.. . I count 16 fields there!! Any ideas? Thnx Chris **************************************************************************** The information contained in this email is intended only for the use of the intended recipient at the email address to which it has been addressed. If the reader of this message is not an intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination or copying of the message or associated attachments is strictly prohibited. If you have received this email in error, please contact the sender by return email or call 01793 877777 and ask for the sender and then delete it immediately from your system.Please note that neither RWE npower nor the sender accepts any responsibility for viruses and it is your responsibility to scan attachments (if any). *****************************************************************************

1 0

Client for AIX
by hj.beckers＠kreis-steinfurt.de 24 Aug '05

24 Aug '05

Has anybody a full working version? What I did: Used hobbitclient-linux as base. Had to set LANG=C (Henrik, maybe you should set this for all scripts) Had to substitute "df -P -x none ..." with "df -P | grep -v ":/" -> AIX 4/5-df knows no -x and the grep gets rid of NFS-mounts Added Henriks suggestions from the latest AIX-thread. Has anybody a working "free"-utility for AIX? But now the servers complains "No client backend for OS 'aix'". Yours hjb _________________________________________ Diese E-Mail wurde geprüft von Symantec Antivirus

2 1

AIX-module
by hj.beckers＠kreis-steinfurt.de 24 Aug '05

24 Aug '05

If someone programs an OS-module for AIX, please note that AIX-vmstat delivers one more parameter than LINUX-vmstat. Linux: procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 24804 3856 17472 132828 0 0 41 60 198 286 2 1 96 1 AIX: System Configuration: lcpu=2 mem=3072MB kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 2 1 660034 123 0 2 5 110 282 0 519 26588 764 12 8 71 9 Yours hjb _________________________________________ Diese E-Mail wurde geprüft von Symantec Antivirus

1 0

Still here ...
by henrik＠hswn.dk 23 Aug '05

23 Aug '05

Just to make people stop worrying: No, I haven't dropped off the planet. I'm still here, and plan on getting some work done on Hobbit the coming days. I've just been kept busy with other work the past weeks, and on top of that a major birthday event happened this past week-end that kept me away from any kind of programming. Regards, Henrik

1 0

[hobbit] hobbit_rrd Fatal signal caught! with rrdtools 1.2.11
by lgrilli＠be.tiauto.com 23 Aug '05

23 Aug '05

Hi Henrik, Seems that I give you the answer :-) Is it going to be supported in the future, or should I downgrade my freebsd ports tree to got the 1.2.10 version. Laurent -----Original Message----- From: henrik at hswn.dk [mailto:henrik at hswn.dk] Sent: Tuesday, August 23, 2005 12:30 PM To: hobbit at hswn.dk Subject: Re: [hobbit] hobbit_rrd Fatal signal caught! with rrdtools 1.2.11 On Tue, Aug 23, 2005 at 01:02:07AM +0200, Grilli, Laurent wrote: > /home/hobbit/data/rrd/X.X.X.X/users.rrd from X.X.X.X: > '/home/hobbit/data/rrd/X.X.X.X/users.rrd' is not an RRD file > 2005-08-22 23:45:18 RRD error updating > /home/hobbit/data/rrd/X.X.X.X/disk,C.rrd from X.X.X.X: reading the cookie > off /home/hobbit/data/rrd/X.X.X.X/disk,C.rrd faild > ..... > ..... > 2005-08-22 23:49:26 Worker process died with exit code 134, terminating > > Plenty of these entries, of course X.X.X.X is normal information > > Is it due to the version of rrd ? > > The same data send by the client to a Hobbit 4.1.1 run fine but with the > rrdtools 1.2.10 Since it works with the 1.2.10 version of the RRD library, but fails with 1.2.11 I'd say that yes - this is an RRDtool library problem. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. This communication is from TI Automotive.

2 1