August 2007 - Xymon - Xymon mailman web

[hobbit] shared memory/semaphores after shutdown, stale alerts, other errors
by david.gore＠verizonbusiness.com 31 Aug '07

31 Aug '07

Hobbit: 4.2.0 with allinone patch. OS: Fedora Core 5, Dell Optiplex, dual core with 4G of memory MAXMSG_STATUS=2048 # maximum size of a "status" message in kB, default: 256 MAXMSG_CLIENT=4096 # maximum size of a "client" message in kB, default: 512 MAXMSG_DATA=2048 # maximum size of a "data" message in kB, default: 256 [hobbit at hobbit2 etc]$ ipcs -lm After shutting down the hobbit server: [hobbit at hobbit2 ~]$ ipcs ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0901727f 5668872 hobbit 600 131072 0 ------ Semaphore Arrays -------- key semid owner perms nsems 0x0901727f 7176201 hobbit 600 3 ------ Message Queues -------- key msqid owner perms used-bytes messages [hobbit at hobbit2 ~]$ ipcrm -s 7176201 [hobbit at hobbit2 ~]$ ipcrm -m 5668872 The snapshot does not suffer from the left-over shared memory/segments issue, but unfortunately with the description or comments being displayed as part of the host name on bb2.html it is unusable. Here is my second issue: Page.log: 2007-08-31 01:29:22 Stale alert for host08:ts-delay dropped 2007-08-31 01:29:22 Stale alert for hostf:cpu dropped 2007-08-31 01:29:22 Stale alert for host2:ocs dropped 2007-08-31 01:29:22 Stale alert for host06:procs dropped 2007-08-31 01:29:22 Stale alert for host08:tl1am-mir dropped 2007-08-31 01:29:22 Stale alert for host12:procs dropped 2007-08-31 01:29:22 Stale alert for host15:tl1am-osi dropped 2007-08-31 01:29:22 Stale alert for host16:tl1am-osi dropped 2007-08-31 01:29:22 Stale alert for host6:procs dropped 2007-08-31 01:29:22 Stale alert for host2:memory dropped 2007-08-31 01:29:22 Stale alert for host3:memory dropped 2007-08-31 01:29:22 Stale alert for ns:sins-out dropped 2007-08-31 01:29:22 Stale alert for hostxx:procsLSE dropped 2007-08-31 01:24:10 hobbitd_alert: Got message 49083, expected 49082 2007-08-31 01:24:17 hobbitd_alert: Got message 49085, expected 49084 2007-08-31 01:25:05 hobbitd_alert: Got message 49097, expected 49090 2007-08-31 01:25:22 hobbitd_alert: Got message 49104, expected 49103 2007-08-31 01:25:42 hobbitd_alert: Got message 49118, expected 49117 2007-08-31 01:26:17 hobbitd_alert: Got message 49129, expected 49126 2007-08-31 01:26:17 hobbitd_alert: Got message 49132, expected 49130 2007-08-31 01:27:04 Dropping (more) garbled data . . . 2007-08-31 09:43:18 hobbitd_alert: Got message 6950, expected 6946 Done 2007-08-31 09:43:40 hobbitd_alert: Got message 6965, expected 6955 stty: : Invalid argument stty: : Invalid argument 2007-08-31 10:38:21 hobbitd_alert: Got message 7693, expected 7692 stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument stty: : Invalid argument . . . 2007-08-31 12:34:49 Stale alert for host3:ocs dropped 2007-08-31 12:34:50 Stale alert for host4:ocs dropped 2007-08-31 12:34:50 Stale alert for host5:ocs dropped 2007-08-31 12:34:50 Stale alert for host09:procs dropped 2007-08-31 12:34:50 Stale alert for host10:procs dropped 2007-08-31 12:34:50 Stale alert for host14:tl1am-osi dropped 2007-08-31 12:34:50 Stale alert for host15:tl1am-mir dropped 2007-08-31 12:34:50 Stale alert for host103:se dropped 2007-08-31 12:34:50 Stale alert for host17a:procs dropped I think some of this may be caused by a server side external script, that ssh's to a remote host, runs a restart script, and emails to management, developers and support that processes have been restarted. Some of the ssh'ing is captured in the page.log file. Regardless, what is the deal with these stale alerts? I restarted hobbit to dump them 11 hours ago, and had to restart hobbit again. Unfortunately, these stale alerts are causing unnecessary page-outs. Any hints of debugging would be appreciated. I suspect the restart script, but it is needed to keep 19 hosts healthy until they are patched next week, so I cannot stop using it. Regardless that is not related to the shared memory and semaphores because that was happening before the restart script was installed. I would even try to use our second hobbit server instance but it suffers from 'Whoops ! bb failed to send message - timeout' every couple hours with the same configuration, also resolved if I could use the snapshot. ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 768000 max total shared memory (kbytes) = 8388608 min seg size (bytes) = 1 David Gore (v965-3670) Network Management Systems (NMS) IMPACT Transport Team Lead - SCSA, SCNA Page: 1-800-PAG-eMCI pin 1406090 Vnet: 965-3676

1 0

Diagnosing client data (vmstat)
by yf.liang＠yahoo.ca 31 Aug '07

31 Aug '07

Two questions: 1. what's the format of [vmstat] ouput that the Hobbit server is expecting? The sar command is not available, so I'm substituting with iostat output that looks like: [vmstat] cpu utilizaton stat using iostat: --------------------------------- 0 0 0 0 0 0 0 0 0 0 16 1 0 13 0 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 where the 2nd last tine is the actual iostat output (iostat 300 2) and last line is what I think Hobbit is expecting. Tthe last 4 numbers, 0 0 100 0 are usr, sys, idle and iowait, that I parsed from iostat output. 2. on server side, how do we debug/trace when client data have arrived? There's hist, histlogs, hostdata, logs. I would like to see some reponse to events like: data arriving, data interpreted/extracted, etc. Thank you, --------------------------------- Be smarter than spam. See how smart SpamGuard is at giving junk email the boot with the All-new Yahoo! Mail

3 3

[hobbit] Diagnosing client data (vmstat)
by yf.liang＠yahoo.ca 30 Aug '07

30 Aug '07

Wow! Your response is faster than any commercial service I've seen. Anyway, the OS is SunOS; I'm customizing hobbitclient-sunos.sh, so it's sending iostat output instead of vmstat under the vmstat section. Could I format the output so the server receiving it will think it's coming from vmstat? The vmstat on Sun server does not display the io-wait time, so I substituted using sar, except for this one which does not have sar installed. SunOS server 5.10 Generic_118833-17 sun4v sparc SUNW,Sun-Fire-T200 $ vmstat 2 2 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr m1 m3 m4 m5 in sy cs us sy id 0 0 0 29045048 14792424 8 30 6 0 0 0 0 0 0 0 0 683 294 694 0 0 100 0 0 0 28957904 14589248 0 5 0 0 0 0 0 0 0 0 0 808 514 797 0 0 100 --------------------------------- All new Yahoo! Mail --------------------------------- Get news delivered. Enjoy RSS feeds right on your Mail page.

2 1

Reg Exp matching in hobbit-clients.cfg
by bill.matthews＠gmail.com 29 Aug '07

29 Aug '07

Greetings, I've been able to do some basic regular expression matching using the examples in hobbit-clients.cfg. For example, this works fine: HOST=%(server1|server2|server3).domain.com But this doesn't: HOST=%(server*).domain.com I'm sure I'm just missing the syntax for wildcard matching? Also is there a way to define a test for every server except some hosts? I have a lot of tests I would love to put in the DEFAULT section, but I have a couple hosts that the tests would fail for. I'm wondering if something like this would be possible? HOST=%(!badhost1|badhost2).domain.com Thanks for the help, BM

4 5

BigBrother scripts with Hobbit Client?
by shane.presley＠gmail.com 29 Aug '07

29 Aug '07

Hello, We've running a Hobbit server, but still mostly BigBrother clients on our Linux servers. I love the features of the Hobbit client compared to BB, but I'm hesitant to switch because of the number of BB scripts we use. For example, we use bb-raid.sh to monitor our RAID controller and openmanage.sh to monitor our Dell hardware. Both were provided from Deadcat. Should these scripts work under the Hobbit client? If not, are there any tips on how to retrofit them? Thanks Shane

2 2

hobbit quick search for FireFox 2 ?
by tj_yang＠hotmail.com 28 Aug '07

28 Aug '07

I used to be able to switch the default Google search to internal BB server search in FireFox 1. But my old quick external F.F. search module doesn't work in F.F.2 anymore. Anyone has hobbit search module for FF 2 ? T.J. Yang _________________________________________________________________ See what you�re getting into�before you go there http://newlivehotmail.com/?ocid=TXT_TAGHM_migration_HM_viral_preview_0507

4 8

Latest snapshot stable
by Jason.Chambers＠geosoft.com 28 Aug '07

28 Aug '07

I was wondering if the latest snapshot is stable enough for install. We have a few servers that been flapping for a few hours now. Would love to test the flap resistant notice. Jason Chambers I.T. Helpdesk Associate [cid:image001.jpg at 01C7E957.2B2881D0] <http://www.geosoft.com/> 85 Richmond St W<http://maps.google.com/maps?f=q&hl=en&q=geosoft&sll=43.650833,-79.382853&ss…> - Toronto, ON, CA - M5H 2C9 - Tel: 416.369.0111<skype:+14163690111> ext 344 - Fax: 416.369.9599

3 3

Hobbit Client for SCO
by trent.melcher＠sitel.com 28 Aug '07

28 Aug '07

Anyone out there have a compiled SCO client? Im looking for a version that would run on 5.0.4, 5.0.5, 5.0.6 Thanks Trent

5 10

Todays snapshot - bbgen crashes
by jonescr＠cisco.com 28 Aug '07

28 Aug '07

I know that snapshots are not considered stable, but I just wanted to report that I grabbed todays snapshot (dated Aug 10 00:05), and noticed that bbgen is crashing. If there is anything I can do to help troubleshoot, let me know. P.S. I "upgraded" to this snapshot in an attempt to resolve a problem I was having with a previous snapshot, where I was not receiving recovery pages.

7 8

Help with a custom test script
by q＠tippett.com 28 Aug '07

28 Aug '07

Hello, I am new to writing custom scripts and looking for advice on how to write a test to monitor the status of a machine in our render farm. Each machine is broken down into a number of CPU cores. We have a command that displays the status of each core in the format per line of "<machine name>_<core number> in|out <partition name>" (i.e. testbox_0 in testpartition). So here is the part I an trying to work out. A machine with 4 cores will produce 4 lines each with it's own status. I would like to to include all cores in one test. I would like to produce a YELLOW status if any core is labeled as out and RED if they are all listed as out. I would also like to make rrd graphs that shows the in and out time for each core. Any help would be greatly appreciated. Q

2 2