- Xymon - Xymon mailman web

Hobbit Server performance
by jeffnewman75＠gmail.com 31 Aug '07

31 Aug '07

All, Have been taking a look at my hobbit server for the past week or so. Wanted to ask a couple things. The server is a DL360 G3 with two 3ghz processors and 2gb of RAM. It does only hobbit and cricket (10 or so routers, so not a huge hit) I've noticed the following two things: #1. I am constantly using 100% of my physical ram (no swap really) Mem: 2075040k total, 2051928k used, 23112k free, 52172k buffers Swap: 0k total, 0k used, 0k free, 1744168k cached is this normal? Not sure how to tell which process specifically is eating up the most RAM. #2. I am consistantly running with 2-300 ports in TIME_WAIT involving port 1984 # netstat -an | grep TIME | grep 1984 | wc -l 253 Im not sure if this is normal either, nor am I sure if it is bad or not? Here is the output from some of the "bb" buttons at the top of my hobbit page. If anything stands out, any help would be apprieciated. If I can provide any other info, let me know. Thanks, Jeff bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7a Feb 19 2003 LDAP library: OpenLDAP 20213 Statistics: Hosts total : 137 Hosts with no tests : 0 Total test count : 141 Status messages : 141 Alert status msgs : 0 Transmissions : 4 DNS statistics: # hostnames resolved : 138 # succesful : 134 # failed : 4 # calls to dnsresolve : 141 TCP test statistics: # TCP tests total : 4 # HTTP tests : 2 # Simple TCP tests : 2 # Connection attempts : 4 # bytes written : 272 # bytes read : 6108 TIME SPENT Event Starttime Duration bbtest-net startup 1188421844.836442 - Service definitions loaded 1188421844.837799 0.001357 Tests loaded 1188421844.860329 0.022530 DNS lookups completed 1188421854.099983 9.239654 Test engine setup completed 1188421854.101787 0.001804 TCP tests completed 1188421854.249955 0.148168 PING test completed (137 hosts) 1188421867.657468 13.407513 PING test results sent 1188421867.659863 0.002395 Test result collection completed 1188421867.659878 0.000015 LDAP test engine setup completed 1188421867.659880 0.000002 LDAP tests executed 1188421867.659882 0.000002 LDAP tests result collection completed 1188421867.659883 0.000001 Test results transmitted 1188421867.660531 0.000648 bbtest-net completed 1188421867.662248 0.001717 TIME TOTAL 22.825806 bbgen for Hobbit version 4.2.0 Statistics: Hosts : 243 Status messages : 1255 Purple messages : 0 Pages : 22 TIME SPENT Event Starttime Duration Startup 1188421967.748593 - Load links done 1188421967.748887 0.000294 Load bbhosts done 1188421967.752720 0.003833 ACK removal done 1188421967.752787 0.000067 Load STATE done 1188421967.772438 0.019651 Color calculation done 1188421967.772690 0.000252 Hobbit pagegen start 1188421967.772717 0.000027 Hobbit pagegen done 1188421967.794090 0.021373 BB2 generation done 1188421967.800862 0.006772 BBNK generation done 1188421967.801386 0.000524 Summary transmission done 1188421967.801388 0.000002 Run completed 1188421967.801389 0.000001 TIME TOTAL 0.052796 Statistics for Hobbit daemon Up since 29-Aug-2007 13:49:58 (0 days, 02:20:00) Incoming messages : 152683 - status : 97745 - combo : 16461 - page : 10 - summary : 0 - data : 22090 - client : 16205 - notes : 0 - enable : 0 - disable : 0 - ack : 0 - config : 0 - query : 0 - hobbitdboard : 140 - hobbitdlog : 4 - drop : 0 - rename : 0 - dummy : 28 - ping : 0 - notify : 0 - schedule : 0 - download : 0 - Bogus/Timeouts : 0 Incoming messages/sec : 17 (average last 300 seconds) status channel messages: 97839 (1 readers) stachg channel messages: 1113 (1 readers) page channel messages: 293 (1 readers) data channel messages: 22090 (1 readers) notes channel messages: 0 (0 readers) enadis channel messages: 0 (0 readers) client channel messages: 16195 (1 readers) clichg channel messages: 0 (1 readers)

6 7

e-mail reply ending up as new e-mail
by yf.liang＠yahoo.ca 31 Aug '07

31 Aug '07

I'm using yahoo mail, and all my submissions to the Hobbit Group are interpreted as new e-mails. I thought the e-mail receiver would interpret e-mails with "RE: " in Subject as replies to existing e-mails. Alas, no. How do I submit a reply to existing e-mails in Hobbit Group mail, as a Follow-Up? Thank you --------------------------------- Be smarter than spam. See how smart SpamGuard is at giving junk email the boot with the All-new Yahoo! Mail

1 0

No events reported...
by sholmes42＠mac.com 31 Aug '07

31 Aug '07

Fellow hobbits, This morning my hobbit summery page (bb2.html) says "No events received in the last 240 minutes" when actually several have been received over the last couple of hours. It shows 2 acks, but no event history. Where could this be getting messed up? Thanks, Steve Holmes

2 2

[hobbit] Disk Space & Alerting Groups
by jkwade＠futurefrontiers.com 31 Aug '07

31 Aug '07

Sorry, I should have given an example like this: /export 85% 90% alert usergroup at email.com /export 96% 98% alert admins at email.com ...where the two values are Yellow and Red. Yellow emails, and Red pages. Also, this is on two systems in our infrastructure. Thanks.James _____ From: James Wade [mailto:jkwade at futurefrontiers.com] Sent: Friday, August 31, 2007 9:01 AM To: hobbit at hswn.dk Subject: [hobbit] Disk Space & Alerting Groups Hello All, Is there a way in hobbit to specify to alert to different groups based on disk space. For example: /export 90% Full alert usergroup at email.com /export 98% Full alert admins at email.com The user group I'm working with wants to be paged at 90%, but our admin team doesn't want to be paged for a filesystem that's only 90% full. Thanks.James

1 0

Disk Space & Alerting Groups
by jkwade＠futurefrontiers.com 31 Aug '07

31 Aug '07

Hello All, Is there a way in hobbit to specify to alert to different groups based on disk space. For example: /export 90% Full alert usergroup at email.com /export 98% Full alert admins at email.com The user group I'm working with wants to be paged at 90%, but our admin team doesn't want to be paged for a filesystem that's only 90% full. Thanks.James

1 0

[hobbit] shared memory/semaphores after shutdown, stale alerts, other errors
by david.gore＠verizonbusiness.com 31 Aug '07

31 Aug '07

Hobbit: 4.2.0 with allinone patch. OS: Fedora Core 5, Dell Optiplex, dual core with 4G of memory MAXMSG_STATUS=2048 # maximum size of a "status" message in kB, default: 256 MAXMSG_CLIENT=4096 # maximum size of a "client" message in kB, default: 512 MAXMSG_DATA=2048 # maximum size of a "data" message in kB, default: 256 [hobbit at hobbit2 etc]$ ipcs -lm After shutting down the hobbit server: [hobbit at hobbit2 ~]$ ipcs ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0901727f 5668872 hobbit 600 131072 0 ------ Semaphore Arrays -------- key semid owner perms nsems 0x0901727f 7176201 hobbit 600 3 ------ Message Queues -------- key msqid owner perms used-bytes messages [hobbit at hobbit2 ~]$ ipcrm -s 7176201 [hobbit at hobbit2 ~]$ ipcrm -m 5668872 The snapshot does not suffer from the left-over shared memory/segments issue, but unfortunately with the description or comments being displayed as part of the host name on bb2.html it is unusable. Here is my second issue: Page.log: 2007-08-31 01:29:22 Stale alert for host08:ts-delay dropped 2007-08-31 01:29:22 Stale alert for hostf:cpu dropped 2007-08-31 01:29:22 Stale alert for host2:ocs dropped 2007-08-31 01:29:22 Stale alert for host06:procs dropped 2007-08-31 01:29:22 Stale alert for host08:tl1am-mir dropped 2007-08-31 01:29:22 Stale alert for host12:procs dropped 2007-08-31 01:29:22 Stale alert for host15:tl1am-osi dropped 2007-08-31 01:29:22 Stale alert for host16:tl1am-osi dropped 2007-08-31 01:29:22 Stale alert for host6:procs dropped 2007-08-31 01:29:22 Stale alert for host2:memory dropped 2007-08-31 01:29:22 Stale alert for host3:memory dropped 2007-08-31 01:29:22 Stale alert for ns:sins-out dropped 2007-08-31 01:29:22 Stale alert for hostxx:procsLSE dropped 2007-08-31 01:24:10 hobbitd_alert: Got message 49083, expected 49082 2007-08-31 01:24:17 hobbitd_alert: Got message 49085, expected 49084 2007-08-31 01:25:05 hobbitd_alert: Got message 49097, expected 49090 2007-08-31 01:25:22 hobbitd_alert: Got message 49104, expected 49103 2007-08-31 01:25:42 hobbitd_alert: Got message 49118, expected 49117 2007-08-31 01:26:17 hobbitd_alert: Got message 49129, expected 49126 2007-08-31 01:26:17 hobbitd_alert: Got message 49132, expected 49130 2007-08-31 01:27:04 Dropping (more) garbled data . . . 2007-08-31 09:43:18 hobbitd_alert: Got message 6950, expected 6946 Done 2007-08-31 09:43:40 hobbitd_alert: Got message 6965, expected 6955 stty: : Invalid argument stty: : Invalid argument 2007-08-31 10:38:21 hobbitd_alert: Got message 7693, expected 7692 stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument . . . 2007-08-31 12:34:49 Stale alert for host3:ocs dropped 2007-08-31 12:34:50 Stale alert for host4:ocs dropped 2007-08-31 12:34:50 Stale alert for host5:ocs dropped 2007-08-31 12:34:50 Stale alert for host09:procs dropped 2007-08-31 12:34:50 Stale alert for host10:procs dropped 2007-08-31 12:34:50 Stale alert for host14:tl1am-osi dropped 2007-08-31 12:34:50 Stale alert for host15:tl1am-mir dropped 2007-08-31 12:34:50 Stale alert for host103:se dropped 2007-08-31 12:34:50 Stale alert for host17a:procs dropped I think some of this may be caused by a server side external script, that ssh's to a remote host, runs a restart script, and emails to management, developers and support that processes have been restarted. Some of the ssh'ing is captured in the page.log file. Regardless, what is the deal with these stale alerts? I restarted hobbit to dump them 11 hours ago, and had to restart hobbit again. Unfortunately, these stale alerts are causing unnecessary page-outs. Any hints of debugging would be appreciated. I suspect the restart script, but it is needed to keep 19 hosts healthy until they are patched next week, so I cannot stop using it. Regardless that is not related to the shared memory and semaphores because that was happening before the restart script was installed. I would even try to use our second hobbit server instance but it suffers from 'Whoops ! bb failed to send message - timeout' every couple hours with the same configuration, also resolved if I could use the snapshot. ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 768000 max total shared memory (kbytes) = 8388608 min seg size (bytes) = 1 David Gore (v965-3670) Network Management Systems (NMS) IMPACT Transport Team Lead - SCSA, SCNA Page: 1-800-PAG-eMCI pin 1406090 Vnet: 965-3676

1 0

Diagnosing client data (vmstat)
by yf.liang＠yahoo.ca 31 Aug '07

31 Aug '07

Two questions: 1. what's the format of [vmstat] ouput that the Hobbit server is expecting? The sar command is not available, so I'm substituting with iostat output that looks like: [vmstat] cpu utilizaton stat using iostat: --------------------------------- 0 0 0 0 0 0 0 0 0 0 16 1 0 13 0 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 where the 2nd last tine is the actual iostat output (iostat 300 2) and last line is what I think Hobbit is expecting. Tthe last 4 numbers, 0 0 100 0 are usr, sys, idle and iowait, that I parsed from iostat output. 2. on server side, how do we debug/trace when client data have arrived? There's hist, histlogs, hostdata, logs. I would like to see some reponse to events like: data arriving, data interpreted/extracted, etc. Thank you, --------------------------------- Be smarter than spam. See how smart SpamGuard is at giving junk email the boot with the All-new Yahoo! Mail

3 3

[hobbit] Diagnosing client data (vmstat)
by yf.liang＠yahoo.ca 30 Aug '07

30 Aug '07

Wow! Your response is faster than any commercial service I've seen. Anyway, the OS is SunOS; I'm customizing hobbitclient-sunos.sh, so it's sending iostat output instead of vmstat under the vmstat section. Could I format the output so the server receiving it will think it's coming from vmstat? The vmstat on Sun server does not display the io-wait time, so I substituted using sar, except for this one which does not have sar installed. SunOS server 5.10 Generic_118833-17 sun4v sparc SUNW,Sun-Fire-T200 $ vmstat 2 2 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr m1 m3 m4 m5 in sy cs us sy id 0 0 0 29045048 14792424 8 30 6 0 0 0 0 0 0 0 0 683 294 694 0 0 100 0 0 0 28957904 14589248 0 5 0 0 0 0 0 0 0 0 0 808 514 797 0 0 100 --------------------------------- All new Yahoo! Mail --------------------------------- Get news delivered. Enjoy RSS feeds right on your Mail page.

2 1

Reg Exp matching in hobbit-clients.cfg
by bill.matthews＠gmail.com 29 Aug '07

29 Aug '07

Greetings, I've been able to do some basic regular expression matching using the examples in hobbit-clients.cfg. For example, this works fine: HOST=%(server1|server2|server3).domain.com But this doesn't: HOST=%(server*).domain.com I'm sure I'm just missing the syntax for wildcard matching? Also is there a way to define a test for every server except some hosts? I have a lot of tests I would love to put in the DEFAULT section, but I have a couple hosts that the tests would fail for. I'm wondering if something like this would be possible? HOST=%(!badhost1|badhost2).domain.com Thanks for the help, BM

4 5

BigBrother scripts with Hobbit Client?
by shane.presley＠gmail.com 29 Aug '07

29 Aug '07

Hello, We've running a Hobbit server, but still mostly BigBrother clients on our Linux servers. I love the features of the Hobbit client compared to BB, but I'm hesitant to switch because of the number of BB scripts we use. For example, we use bb-raid.sh to monitor our RAID controller and openmanage.sh to monitor our Dell hardware. Both were provided from Deadcat. Should these scripts work under the Hobbit client? If not, are there any tips on how to retrofit them? Thanks Shane

2 2