October 2009 - Xymon - Xymon mailman web

[hobbit] IPv6
by henrik＠hswn.dk 30 Oct '09

30 Oct '09

In <ea6cc0c70811281053n3c27f49fpf78fd2b800a52bf5 at mail.gmail.com> "David A. Bandel" <david.bandel at gmail.com> writes: >Searched the archives and found a few IPv6 inquiries, but only that >there are no plans to support it. While I'm not a programmer, I >suspect it would actually be fairly simple to implement (please >correct me if I'm wrong). It's not entirely trivial, but it can be done (of course). The actual changes needed to "talk" IPv6 in the network service tests is probably not a whole lot; may real concern is going through all of the places throughout Xymon where the code assumes that an IP- address looks like "1.2.3.4". >This is starting to become a priority for me. I need to monitor my >ssh servers, but they all only respond now to ipv6, so I can't :-(. It is moving up on my priority list as well - it seems that IPv6 is (finally) seeing some real adoption. It won't be in the next version, but probably the one after that. Regards, Henrik

4 3

Alert mails with the Ack-Code
by Vernon.Everett＠woodside.com.au 30 Oct '09

30 Oct '09

Hi all Is there an easy way to send alerts without the ACK-CODE? By "easy", I mean not hacking the source code, and not redirecting all mails through a custom script. Cheers V NOTICE: This email and any attachments are confidential. They may contain legally privileged information or copyright material. You must not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages and all attachments.

3 4

Any info on hobbitd_locator?
by shea_greg＠emc.com 29 Oct '09

29 Oct '09

I admit I've only breezed through the code and now I'm looking for a cheat sheet on how to use the hobbitd_locator. Thanks Gregory R Shea EMC Corporation

1 0

monitor processes by owner
by josie_wells＠yahoo.com 29 Oct '09

29 Oct '09

Hi, I am new to this mailing list so apologies if I am breaking any protocol by sending this mail out to this address - I have searched the faqs and can find no answer. What I am trying to do is to monitor processes by the owner. In hobbit-clients.cfg I can specify a string to search for in the process name but I can find no way to look in the USER column. If I could, then I could get a count of processes owned by oracle and graph it out. Is this possible? Many thanks Josie

4 3

[hobbit] bbproxy over stunnel
by rower.master＠gmail.com 28 Oct '09

28 Oct '09

Hi, > > I'm trying to do the same thing that you did. > > Do I have to copy the cert generated on the server to the bbproxy? I need > to change the port 1984 to 11984 on the hobbitserver.cfg of the bbporxy? > I followed the doc on wiki but I think I,ve missed something. > > Thanks in advance, > > Mario. > > > > > On Mon, Nov 24, 2008 at 8:58 AM, Darrin Khan <medavian at gmail.com> wrote: > >> Hello All, >> >> I have a problem getting stunnel and bbproxy to hobbitd to play nicely. I >> am not sure if this has been covered before, however I have found a few bits >> and peices of information about hobbit and stunnel, but nothing like what I >> have configured. >> >> Here is the idea.. >> >> client -> bbproxy -> stunnel -> stunnel ->hobbitd >> >> Client is a SQL server that can't see the world. bbproxy is running on a >> server behind a firewall that the SQL server can reach. bbproxy is >> configured to send data to 127.0.0.1:11984. stunnel is listening on >> 127.0.0.1:11984 this then forwards out through the firewall to another >> server running stunnel listening on 11984 this in turn dumps traffic on the >> remote server to port 1984 (hobbitd). >> >> This config works great, all my messages are encrypted and I am getting >> all the updates to hobbitd on the remote server no worries. Has been working >> like this for a few weeks now. >> >> The issue I have been trying to nut out is that the clients are not >> receiving any config in the other direction, particularly the >> log:/var/log/messages:10240 entries from client-local.cfg. >> >> If I remove the stunnel(s) and tell the bbproxy to connect to the hobbitd >> directly, the clients get the config data in the reverse direction and they >> in turn send back the log data they are supposed to. >> >> Has anyone been able to get this to work ? >> >> I suspect it may be my stunnel configs. they are below. Any help would be >> greatly appreciated. >> >> Darrin >> >> ----- bbproxy server ----- >> chroot = /var/run/stunnel/ >> setuid = nobody >> setgid = nobody >> socket = l:TCP_NODELAY=1 >> socket = r:TCP_NODELAY=1 >> debug = 5 >> output = /var/log/stunnel.log >> foreground=no >> >> [hobbit] >> accept = 11984 >> connect = xx.xx.xxx.xxx:11984 >> TIMEOUTbusy = 5 >> TIMEOUTclose = 2 >> TIMEOUTconnect = 2 >> TIMEOUTidle = 5 >> >> ----- hobbitd server ----- >> cert = /etc/pki/tls/certs/nms.ext.example.net.pem >> chroot = /var/run/stunnel/ >> setuid = nobody >> setgid = nobody >> socket = l:TCP_NODELAY=1 >> socket = r:TCP_NODELAY=1 >> debug = 5 >> output = /var/log/stunnel.log >> foreground=no >> >> [hobbit] >> accept = 11984 >> connect = 1984 >> TIMEOUTbusy = 5 >> TIMEOUTclose = 2 >> TIMEOUTconnect = 2 >> TIMEOUTidle = 5 >> >> -- >> Darrin Khan >> medavian at gmail.com >> "If you save the world too often, it begins to expect it..." >> - Unknown >> > >

1 0

Solaris Zones
by jkwade＠futurefrontiers.com 28 Oct '09

28 Oct '09

Hello, I sent an earlier email asking if anyone had written anything for Solaris Zones so that it sends xymon information about the zone, instead of the global zone. I have zonestat loaded on, and it give information on the trends section. I received no answers, so I've started writing one myself. I have some questions that I was hopping someone might be able to help with, so I can make the test easier. What I'd like to do is have my test send status in the same format as xymon does for a client. For example, memory, physical and swap, along with CPU load. Is there a way to accomplish this? Thanks.James

2 1

Monitoring Exchange
by ejacobs＠thomaspublishing.com 28 Oct '09

28 Oct '09

Apparently, last night our Exchange server stopped receiving email until it was rebooted this morning. Everyone is up in arms. Hobbit showed nothing unusual. Does anyone have any way to monitor Exchange? Eric Jacobs Infrastructure and Operations Information Technology Group Phone:215-682-5354 Email:ejacobs at thomaspublishing.com

7 6

Dual BBDISPLAY lines
by d.tom.schmitt＠L-3com.com 27 Oct '09

27 Oct '09

I am working on the BBWin XML configuration file. Server is running Xymon 4.3.0-0.beta2 I added these two lines to the beginning of the configs after the comments: <setting name="bbdisplay" value="128.170.1.151" /> or monitor1.csw.l-3com.com <setting name="bbdisplay" value="128.170.1.161" /> or monitor2.csw.l-3com.com Only the first Xymon gets the updates, the second just looks like it is doing the standard connectivity icons. The name in both of the bb-hosts entries are identical. MONITOR1: bb-hosts entry: 128.170.1.150 dschmitt.csw.l-3com.com # conn BBDISPLAY: MONITOR2: bb-hosts entry: 128.170.1.150 dschmitt.csw.l-3com.com # conn BBDISPLAY: I have restarted the service and monitor2 xymon. Thanks, Tom Schmitt Senior IT Staff - R&D

2 1

[hobbit] performance help needed
by olivier＠audry.fr 27 Oct '09

27 Oct '09

hi all, do you have a lot of memory ? If yes you can create tmpfs for your rrd and sync the tmpfs every couple hours. We do it for a little hobbit server with 3500+ devices. For rrd hist and www dir. Regards Olivier AUDRY ----- Mail Original ----- De: "shea greg" <shea_greg at emc.com> À: bgmilne at staff.telkomsa.net, hobbit at hswn.dk Cc: "shea greg" <shea_greg at emc.com> Envoyé: Mardi 27 Octobre 2009 14h24:47 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne Objet: RE: [hobbit] performance help needed >>On Monday, 26 October 2009 20:55:15 shea_greg at emc.com wrote: >> Hi all, >> >> >> >> First off, sorry for the long post, I'm trying to supply as much data as >> possible for analysis. >> >> >> >> I have a single Hobbit server with approximately 3500 hosts, a mixture >> of windows and unix, some DB tests, >> >> some BEA tests and a few custom tests. I have over 70000 RRD files >> which seems to be causing Hobbit performance >> >> problems, most specifcally clock offset. I have a cron job that >> restarts Hobbit every 30 minutes otherwise the offset >> >> grows so large it eats all memory and OOM kill starts. NTP is fine, it >> seems to be the time it takes for Hobbit to process >> >> the client data. OS resides on RAID1 146GB drives SAS 15K RPM, second >> drive for RRDs is a single 300GB SAS 15K RPM. >> >> At the end is a graph showing the clock offset. What else can I try? > >Add more spindles. > >70 000 RRD files will result in a minimum of 233 IOPS (assuming they are all >being updated at 5-minute intervals). The EMC people I've spoken to say a 15k >FC disk shouldn't really be averaging much more than 180 IOPS, 15k SAS or 15k >SCSI wouldn't be any better. The 311 you seem to be doing isn't significant >overhead for the minumum of 233, so it is unlikely that any tuning will help. > >If you can't add spindles, you could look at the 4.3 branch, which has some >features that allow scaling out to more hosts, or streamlining RRD writes >(which may allow you to lose the clock offset, but will likely not reduce the >load average much). > >Regards, >Buchan Hi Buchan, Thanks for your response. I bounced around the idea of external storage, but even here at EMC there is a cost associated with external storage, that's why I tried the second drive. I've read about the enhancements in 4.3, but thought I should upgrade from RH 4.7 to RH 5.3 first (RH is the official supported Linux) as there were IO improvements in the kernel. I also tried a newer version of RRD 1.2.30 and 1.3.8. RRD 1.3.8 doesn't work Hobbit 4.2. On to the storage requisition process.... Thanks -Grs- Gregory R Shea EMC Corporation

1 0

performance help needed
by shea_greg＠emc.com 27 Oct '09

27 Oct '09

Hi all, First off, sorry for the long post, I'm trying to supply as much data as possible for analysis. I have a single Hobbit server with approximately 3500 hosts, a mixture of windows and unix, some DB tests, some BEA tests and a few custom tests. I have over 70000 RRD files which seems to be causing Hobbit performance problems, most specifcally clock offset. I have a cron job that restarts Hobbit every 30 minutes otherwise the offset grows so large it eats all memory and OOM kill starts. NTP is fine, it seems to be the time it takes for Hobbit to process the client data. OS resides on RAID1 146GB drives SAS 15K RPM, second drive for RRDs is a single 300GB SAS 15K RPM. At the end is a graph showing the clock offset. What else can I try? I moved the RRDs off to a separate drive hoping this would help, but the write per second is high. I've tried reducing read-ahead, mounting noatime,nodiratime, changing IO scheduling to deadline, nothing seems to help. Here's a sample output from iostat -xd 60 10: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 68.08 0.17 20.02 1.33 704.78 0.67 352.39 34.98 4.25 210.36 3.47 7.01 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 68.08 0.17 20.02 1.33 704.78 0.67 352.39 34.98 4.25 210.36 3.47 7.01 sdb 0.00 674.60 1.53 311.04 12.27 7887.05 6.13 3943.52 25.27 24.50 78.38 1.91 59.70 sdb1 0.00 674.60 1.53 311.04 12.27 7887.05 6.13 3943.52 25.27 24.50 78.38 1.91 59.70 sdb2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.17 88.10 1.33 704.78 0.67 352.39 8.00 20.31 230.09 0.79 7.01 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Drive sdb1 is housing the RRD files Memory seems fine: Memory Used Total Percentage Physical 7645M 7973M 95% Actual 4688M 7973M 58% Swap 64M 9983M 0% [hobbit at hobbitmon rrd]$ uname -a Linux hobbitmon 2.6.9-78.0.8.ELsmp #1 SMP Wed Nov 5 07:14:58 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [hobbit at hobbitmon rrd]$ cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Output from bbgen: bbgen for Hobbit version 4.2.0 Statistics: Hosts : 3506 Status messages : 41934 Purple messages : 0 Pages : 171 Output from bbtest: bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7a Feb 19 2003 LDAP library: OpenLDAP 20213 Statistics: Hosts total : 3511 Hosts with no tests : 2390 Total test count : 1470 Status messages : 1596 Alert status msgs : 0 Transmissions : 18 DNS statistics: # hostnames resolved : 358 # succesful : 339 # failed : 19 # calls to dnsresolve : 530 TCP test statistics: # TCP tests total : 411 # HTTP tests : 161 # Simple TCP tests : 250 # Connection attempts : 411 # bytes written : 24722 # bytes read : 543706 TIME SPENT Event Starttime Duration bbtest-net startup 1256584823.382254 - Service definitions loaded 1256584823.383506 0.001252 Tests loaded 1256584823.468743 0.085237 DNS lookups completed 1256584828.565010 5.096267 Test engine setup completed 1256584828.572444 0.007434 TCP tests completed 1256584839.000192 10.427748 PING test completed (1082 hosts) 1256584881.612835 42.612643 PING test results sent 1256584890.617168 9.004333 Test result collection completed 1256584890.617453 0.000285 LDAP test engine setup completed 1256584890.617453 0.000000 LDAP tests executed 1256584890.617454 0.000001 LDAP tests result collection completed 1256584890.617455 0.000001 NTP tests executed 1256584894.477007 3.859552 RPC tests executed 1256584894.988810 0.511803 Test results transmitted 1256584895.016358 0.027548 bbtest-net completed 1256584895.018441 0.002083 TIME TOTAL 71.636187 Output for hobbitd: Statistics for Hobbit daemon Up since 26-Oct-2009 15:00:11 (0 days, 00:25:02) Incoming messages : 398039 - status : 367373 - combo : 5193 - page : 183 - summary : 75 - data : 15310 - client : 9595 - notes : 0 - enable : 0 - disable : 0 - ack : 0 - config : 0 - query : 50 - hobbitdboard : 63 - hobbitdlog : 180 - drop : 0 - rename : 0 - dummy : 5 - ping : 0 - notify : 0 - schedule : 1 - download : 0 - Bogus/Timeouts : 11 Incoming messages/sec : 262 (average last 300 seconds) status channel messages: 366410 (1 readers) stachg channel messages: 34214 (1 readers) page channel messages: 5600 (1 readers) data channel messages: 15310 (1 readers) notes channel messages: 0 (0 readers) enadis channel messages: 0 (0 readers) client channel messages: 9565 (1 readers) clichg channel messages: 17 (1 readers)

2 2