August 2006 - Xymon - Xymon mailman web

Hobbit monitor security bugfix release - 4.1.2p2
by henrik＠hswn.dk 02 Aug '06

02 Aug '06

Version 4.1.2p2 of Hobbit has just been uploaded to SourceForge, and is available at http://sourceforge.net/project/showfiles.php?group_id=128058&package_id=140… This release fixes a security bug reported by Jason Kruse earlier today: File access via the Hobbit "config" method failed to restrict access to the Hobbit configuration directory. It is therefore possible for anyone with access to the Hobbit network daemon on port 1984 to read any file on the Hobbit server which is readable by the user running the hobbitd daemon. This problem affects all versions of Hobbit 4.0 and 4.1. Users are encouraged to update their Hobbit installation, or restrict network access to Hobbit network daemon (e.g. by implementing firewall rules restricting access to TCP port 1984). This version also includes all non-security patches that have been published since the release of version 4.1.2p1 in November 2005. Regards, Henrik Storner

1 0

bbgen - Program crashed on service temperature
by Michael.Nagel＠forsa.de 02 Aug '06

02 Aug '06

Hi, I'm trying to implement a script to read temperature data from a primergy via snmp. Sending the results via bin/bb I get: - a core file in BBHOME/tmp - RED on the bbgen page "- Program crashed Fatal signal caught!" - hobbitlaunch.log reports "Task bbdisplay terminated by signal 6" I can reproduce the error with: /bin/bb localhost "status cati-b2.temperature green Wed Aug 2 17:39:50 CEST 2006 - temperature ok BLABLA" Using the service name "temperature" it crashes. Using other service names both, my script and the test works fine. My Hobbit version is 4.2-beta-20060605. Any ideas? Michael Nagel

2 1

Security issue with Hobbit "config" command.
by henrik＠hswn.dk 02 Aug '06

02 Aug '06

On Wed, Aug 02, 2006 at 10:11:42AM -0500, Kruse, Jason K. wrote: > The config directive does not stay locked into the etc directory but > will follow ../.. to allow access to any file the hobbit user has access > to on the system. Thanks, this was not meant to happen. There was actually a security check in the code, but it got two parameters in the wrong order, so it would always grant access. A patch is attached. Regards, Henrik

2 1

[hobbit] Hobbit newbie from BB: differences and what may I lose from migrating?
by neil.camp＠deca.mil 02 Aug '06

02 Aug '06

This is an awesome write up. Maybe you should consider adding it to your webpage? -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Tuesday, August 01, 2006 5:13 PM To: hobbit at hswn.dk Subject: Re: [hobbit] Hobbit newbie from BB: differences and what may I lose from migrating? Hi Jordan, I'll try to answer your questions. Since I also develop Hobbit I am probably slightly biased when it comes to the "is-this-more-difficult- to-do-than-with-BB" type of questions, but I am sure others will voice their opinions on that. On Tue, Aug 01, 2006 at 12:36:29PM -0700, Jordan Mendler wrote: > > First, after reading through whatever I could find on the website I am > still a little bit confused about configuration and setup. With BB, you > install and configure each client and server on the local machine, > except for the universal bb-hosts. Is this the same on Hobbit, or does > Hobbit use a central configuration file that is modified only on the > server to configure clients? I am trying to figure out the difference > between installing, maintaining and configuring BB and Hobbit setups. First, let me stress that Hobbit is fully compatible with your existing BB clients. You can keep your current client setup and just switch to Hobbit on the server side, and all of your clients will continue to work as they do with BB as the server. So you can migrate the server side first, and then migrate clients when you find that it is convenient to do so - or you want to take advantage of some of the new stuff that is in Hobbit. The Hobbit client configuration is maintained on the Hobbit server. Clients in Hobbit are designed to be *really* dumb; they just collect data, and all of the configuration of what to monitor, what thresholds to use for e.g. disk utilisation and so on is configured only on the Hobbit server. This is a major difference between Hobbit and BB. With BB you have delegated the client administration to whoever manages each server. Hobbit centralizes the monitoring configuration, so you will probably have a group of people who take more control of the monitoring setup. > Hobbit looks alot more complex to setup, but once I get my feet wet is > it any harder than BB? I think it is easier, once you get used to the Hobbit way of doing things. But as I said, I am biased. > Second is performance. I know this list may be biased toward Hobbit, but > is it actually faster? We have about 50-100 clients on BB and I did not > notice any performance issues. With that number of systems monitored, you probably will not see a huge difference. BB works quite well for a small number of systems, but when you move beyond a couple of hundred boxes the overhead of generating webpages through shell scripts becomes very noticeable. On my setup, the servers were simply choking on the disk I/O caused by BB saving every status in a separate file, and from the huge number of small cut-grep-awk-sed etc. commands that ran to generate webpages. > Hobbit looks like it is very complex, so does this mean it uses a lot > of resources on the client and server? What speed/ram server is usually > the minimum recommended for a dedicated Hobbit server? Would something > like a dual Pentium II 266mhz have any performance issues as a server, > if it does nothing else? What about for clients? We have still have > some testing, stating and production servers left that are singe chip > Pentium III 700-850 mhz, and even a couple Pentium II's. Just need to > make sure all the resources used for things like graphs are taken from > the server and not each client. The Hobbit server uses fewer ressources than the BB server. The main ressource usage is memory; Hobbit keeps everything in memory except the history logs and the RRD files used for graphs. That doesn't mean a whole lot, though: Here's a ps listing of the Hobbit processes running on my main monitoring system - it handles about 2500 hosts: $ ps vax|cut -c1-100|egrep "PID|hobbit" PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND 732 ? Ss 1:24 0 101 1802 696 0.0 hobbitlaunch 735 ? S 2434:37 1 162 31357 29784 2.8 hobbitd 1470 ? S 14:50 0 99 2332 1088 0.1 hobbitd_channel --channel=stachg 1471 ? S 25:18 0 108 2515 1048 0.1 hobbitd_history 1472 ? S 964:26 0 99 2332 1264 0.1 hobbitd_channel --channel=page 1473 ? S 1227:34 0 154 5661 3912 0.3 hobbitd_alert 1474 ? S 4090:05 0 99 2332 1264 0.1 hobbitd_channel --channel=status 1475 ? D 2962:15 0 178 7381 4392 0.4 hobbitd_rrd 1476 ? S 259:55 0 99 2332 1208 0.1 hobbitd_channel --channel=data 1477 ? S 494:13 0 178 5141 2128 0.2 hobbitd_rrd 1478 ? S 126:20 0 99 2844 1832 0.1 hobbitd_channel --channel=client 1480 ? S 291:20 0 146 4485 2792 0.2 hobbitd_client 5552 ? S 0:00 0 669 2002 1352 0.1 sh -c vmstat 300 2 1>/usr/lib/hobbit/client/ As you can see, the biggest chunk of memory goes to the "hobbitd" process which is the one that keeps all state information. It's currently using some 31 MB of memory. (This box has 1 GB RAM). A rough estimate of how much memory Hobbit needs would be the size of your bbvar/logs/ directory, plus 30 MB. As for CPU usage, your PII/266 should be adequate for 50-100 servers. The box I'm running on is an old (7-8 years) Solaris server with a 900 MHz UltraSparc II processor. That's roughly comparable to a PII running at 1.2 GHz. And it handles 25 times as many hosts as you are aiming for. > Third is plugins. Are BB plugins compatible with Hobbit? Yes. > Also how hard are plugins to write for Hobbit? Plugins that run on the monitored client systems are as easy to write as for BB, since it is basically the same thing. Hobbit also allows you to write plugins for the Hobbit server, which receive events from the Hobbit server daemon. This is used by the core Hobbit tools - e.g. the hobbitd_rrd processes you see in the ps-listing above are a plugin that handle updating of the RRD files from the status- and data-messages that are sent to Hobbit. There aren't any third-party plugins that use this yet (at least, I don't know of any), but writing them is fairly simple since it basically involves reading data from a pipe and processing it in whatever way you want. > I don't know if these even exist for > bb, but I ultimately would like to integrate plugins that 1) monitor > legato tape backup, Dont know about this. > 2) run nmap to see what ports are open/can be seen from an external > machine, The Hobbit client in version 4.2 (about to be released soon) reports details about the network services running on a host. So you can check for which ports are open/listening for connections, and trigger alerts if any unwanted ports show up. > 3) run 'lshw -html' to show a list of all the hardware on the system, This would typically be a client-side test. > 4) monitor uptime, This is standard. > 5) monitor OS and kernel versions (uname -a and head -n 1 /etc/issue), This data is collected by the Hobbit client. > 6) maybe some more router/network monitoring stuff and Hobbit comes with built-in network service monitoring. There is also an SNMP add-on which can be used for monitoring devices such as routers. > Fourth is relay. By this I mean monitoring systems on a private > subnetwork that are only accessible to the Hobbit server by going > through an intermediate server. Is this possible with Hobbit and is it > any more difficult to do than on BB? Two ways of doing that. First, there is a proxy utility which is used to forward Hobbit messages from one network to another. This is used if your client systems on the private subnet are allowed to make outgoing connections to the proxy, and the proxy can connect to the real Hobbit server. Second, Hobbit 4.2 includes a set of tools where it's the server that contacts clients to pick up the data they have collected (i.e. the traffic is initiated by the server, where the normal BB setup is for the client to initiate the connection). Useful for DMZ style setups where clients are not allowed to generate outbound connections. > Fifth is portability. BB is very portable, I can make a 'model' client > for say Red Hat and tar it and distribute it very easily to every server > I have using only a few commands. Is Hobbit the same, or are there > client dependencies or other things that may make this more difficult. The Hobbit client uses only the system libraries and standard utilities found on your client systems. You will need at least one system where you can compile the client binaries (that's similar to the BB requirements), since a few of the client-side tools are written in C. Once you have a client compiled for an OS, it is as portable as any binary that is dynamically linked on your platform. I.e. you can just copy it over as long as the same run-time libraries are available. So far, we haven't managed to find any unix-like system that couldn't run the Hobbit client. Including some rather odd ones. The current list of client-side data collectors are hobbitclient-aix.sh hobbitclient-darwin.sh hobbitclient-freebsd.sh hobbitclient-hp-ux.sh hobbitclient-irix.sh hobbitclient-linux.sh hobbitclient-netbsd.sh hobbitclient-openbsd.sh hobbitclient-osf1.sh hobbitclient-sunos.sh > Sixth is development. How active is the development of Hobbit, how big > is the community, etc? How many people can attest to having fully > functional hobbit setups, how long has it been around and how often are > new releases usually made? Hobbit started back in late 2002 when it was called the "bbgen toolkit". It was renamed to Hobbit in March 2005 when it had developed into a complete replacement for BB. More details in the hobbit(7) man-page available online at http://www.hswn.dk/hobbit/help/manpages/ It is actively being developed by me, but people on this list have made contributions of code. Some have picked up special projects like the Windows client and run that completely on their own. I'd say Hobbit currently has a very active user community, and the development community is slowly growing beyond just myself. There are currently 433 subscribers to the Hobbit mailing list. According to the Sourceforge download statistics, it is downloaded about 1000 times per month. http://sourceforge.net/project/stats/?group_id=128058&ugn=hobbitmon&type =&mode=year There was a thread on the mailing list back in May about who uses Hobbit. The results were summarized here: http://en.wikibooks.org/wiki/System_Monitoring_with_Hobbit/User_Guide#Wh o_use_Hobbit_.3F New releases have usually happened frequently - 2-4 times a year. The current interval between the 4.1.2 release and version 4.2 is unusually long - a whole year. I don't expect that to happen again. > Also I saw something this morning about a Windows client -- how > stable is that? >From what I hear it should be usable. But you can stick with the current BBNT client until it reaches version 1.0. > How stable is the Solaris version? Rock-solid. > Is there a client for Mac OSX? Yes. It will run the Hobbit server also, if you want to. > Is Hobbit like BB in the sense that you can change paths to system > binaries like grep and sed to allow easy use on other UNIXes like OSX? Adding a client for a new OS will require implementing both a client-side script to collect whatever data is interesting for this system, and implementing the data parsing on the Hobbit server-side. So it is somewhat more challenging. But since Hobbit already supports all of the common Unix systems, I doubt that you will need to worry about that. If you do have a system which is not on the list, I will help you with adding support for it. > When will 4.2 be officially released as a production version? Probably by the end of this week. > Since we have a working BB setup for now, I need to > decide if I should try to start migrating now or if I should wait some > time for Hobbit to develop more before I migrate from BB. I don't think you have to wait. But it's for You to decide. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

3 3

[hobbit] rrd-data.log
by brodie＠mcw.edu 02 Aug '06

02 Aug '06

I am stabbing in the dark here, but the duplicate data on my end seems to be caused by parsing the output of the netstat -s command on *AIX*. Here, what is different is that the netstat -s command on aix is much more verbose, showing stuff for ipv4 and ipv6. Perhaps the "icmp:" and "icmpv6:" or other similar items is where the parsing breaks, and supposed duplicates are detected? --------------------------------------------------------- Kent C. Brodie - brodie at phys.mcw.edu Department of Physiology Medical College of Wisconsin (414) 456-8590 -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Wednesday, August 02, 2006 9:53 AM To: hobbit at hswn.dk Subject: Re: [hobbit] rrd-data.log On Wed, Aug 02, 2006 at 01:50:28PM +0200, Beau Olivier wrote: > Hi, > > yes, this is interesting, and i think it points out a new problem, 802.1q on nics : > > eth1 Link encap:Ethernet HWaddr 00:0D:9D:4E:11:9C > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:2798842 errors:0 dropped:0 overruns:0 frame:0 > TX packets:8950695 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:217776970 (207.6 MiB) TX bytes:4275403340 (3.9 GiB) > Interrupt:201 > > eth1.9 Link encap:Ethernet HWaddr 00:0D:9D:4E:11:9C > inet addr:192.168.250.33 Bcast:192.168.250.0 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:2226941 errors:0 dropped:0 overruns:0 frame:0 > TX packets:3441485 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:520111630 (496.0 MiB) TX bytes:410431496 (391.4 MiB) Perhaps, but these data should not get anywhere near the code that prints out this message. The code that generates that message is the one that parses the output from "netstat -s" which should look like Ip: 3017099 total packets received 1 with invalid addresses 0 forwarded 0 incoming packets discarded 3017058 incoming packets delivered 3154813 requests sent out Icmp: 51081 ICMP messages received 0 input ICMP message failed. What does this command report on your host? Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

3 2

[hobbit] rrd-data.log
by brodie＠mcw.edu 02 Aug '06

02 Aug '06

Aha...! I asked this very question last week- nobody gave me help... (sob, sob..). Anyway, here are two separate chunks of data that is CAUSING the duplicate data error like Beau is having. Henrik, what in this data is considered to be "duplicate"? (I do notice that the netstat data in AIX is um... QUITE verbose........) Data segments follow. Help? @@data#810|1154529466.624163|999.888.777.202||fred.moo.mcw.edu|ifstat data fred,moo,mcw,edu.ifstat aix ------------------------------------------------------------- ETHERNET STATISTICS (ent0) : Device Type: 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Hardware Address: 00:02:55:4f:a9:f4 Elapsed Time: 0 days 0 hours 0 minutes 0 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 0 Packets: 0 Bytes: 0 Bytes: 0 Interrupts: 2 Interrupts: 0 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 0 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 0 Broadcast Packets: 0 Multicast Packets: 0 Multicast Packets: 0 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 200 Driver Flags: Up Broadcast Running Simplex AlternateAddress 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Specific Statistics: -------------------------------------------------------------------- Link Status: Down Media Speed Selected: Auto negotiation Media Speed Running: Unknown Receive Pool Buffer Size: 1024 Free Receive Pool Buffers: 1024 No Receive Pool Buffer Errors: 0 Receive Buffer Too Small Errors: 0 Entries to transmit timeout routine: 0 Transmit IPsec packets: 0 Transmit IPsec packets dropped: 0 Receive IPsec packets: 0 Receive IPsec packets dropped: 0 Inbound IPsec SA offload count: 0 Transmit Large Send packets: 0 Transmit Large Send packets dropped: 0 Packets with Transmit collisions: 1 collisions: 0 6 collisions: 0 11 collisions: 0 2 collisions: 0 7 collisions: 0 12 collisions: 0 3 collisions: 0 8 collisions: 0 13 collisions: 0 4 collisions: 0 9 collisions: 0 14 collisions: 0 5 collisions: 0 10 collisions: 0 15 collisions: 0 ------------------------------------------------------------- ETHERNET STATISTICS (ent1) : Device Type: 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Hardware Address: 00:02:55:4f:a9:f3 Elapsed Time: 0 days 0 hours 0 minutes 0 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 0 Packets: 0 Bytes: 0 Bytes: 0 Interrupts: 2 Interrupts: 0 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 0 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 0 Broadcast Packets: 0 Multicast Packets: 0 Multicast Packets: 0 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 200 Driver Flags: Up Broadcast Running Simplex AlternateAddress 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Specific Statistics: -------------------------------------------------------------------- Link Status: Down Media Speed Selected: Auto negotiation Media Speed Running: Unknown Receive Pool Buffer Size: 1024 Free Receive Pool Buffers: 1024 No Receive Pool Buffer Errors: 0 Receive Buffer Too Small Errors: 0 Entries to transmit timeout routine: 0 Transmit IPsec packets: 0 Transmit IPsec packets dropped: 0 Receive IPsec packets: 0 Receive IPsec packets dropped: 0 Inbound IPsec SA offload count: 0 Transmit Large Send packets: 0 Transmit Large Send packets dropped: 0 Packets with Transmit collisions: 1 collisions: 0 6 collisions: 0 11 collisions: 0 2 collisions: 0 7 collisions: 0 12 collisions: 0 3 collisions: 0 8 collisions: 0 13 collisions: 0 4 collisions: 0 9 collisions: 0 14 collisions: 0 5 collisions: 0 10 collisions: 0 15 collisions: 0 ------------------------------------------------------------- ETHERNET STATISTICS (ent2) : Device Type: 10/100/1000 Base-TX PCI-X Adapter (14106902) Hardware Address: 00:02:55:53:c2:3e Elapsed Time: 196 days 15 hours 56 minutes 59 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 365802341 Packets: 1036460447 Bytes: 1683156286637 Bytes: 112378387074 Interrupts: 0 Interrupts: 614513005 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 30 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 16681 Broadcast Packets: 210557123 Multicast Packets: 0 Multicast Packets: 283080 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 381 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 2000 Driver Flags: Up Broadcast Running Simplex 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 10/100/1000 Base-TX PCI-X Adapter (14106902) Specific Statistics: -------------------------------------------------------------------- Link Status: Up Media Speed Selected: Auto negotiation Media Speed Running: 1000 Mbps Full Duplex PCI Mode: PCI-X (100-133) PCI Bus Width: 64-bit Latency Timer: 144 Cache Line Size: 128 Jumbo Frames: Disabled TCP Segmentation Offload: Enabled TCP Segmentation Offload Packets Transmitted: 113291719 TCP Segmentation Offload Packet Errors: 0 Transmit and Receive Flow Control Status: Enabled XON Flow Control Packets Transmitted: 0 XON Flow Control Packets Received: 430 XOFF Flow Control Packets Transmitted: 0 XOFF Flow Control Packets Received: 430 Transmit and Receive Flow Control Threshold (High): 45056 Transmit and Receive Flow Control Threshold (Low): 24576 Transmit and Receive Storage Allocation (TX/RX): 16/48 @@ @@data#811|1154529466.624871|999.777.666.202||phred.mrr.mcw.edu|netstat data phred,mrr,mcw,edu.netstat aix icmp: 597 calls to icmp_error 0 errors not generated because old message was icmp Output histogram: echo reply: 58031 destination unreachable: 537 1 message with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 2 destination unreachable: 562 echo: 58031 time exceeded: 12 58031 message responses generated igmp: 282998 messages received 0 messages received with too few bytes 0 messages received with bad checksum 282998 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 2 membership reports sent tcp: 449406211 packets sent 419904427 data packets (1612944301 bytes) 176131 data packets (218422100 bytes) retransmitted 20439309 ack-only packets (8520363 delayed) 6 URG only packets 420337 window probe packets 2131650 window update packets 6334351 control packets 113291719 large sends 3038842766 bytes sent using largesend 64240 bytes is the biggest largesend 909296772 packets received 857387271 acks (for 1835107293 bytes) 7844383 duplicate acks 0 acks for unsent data 307390334 packets (2464690667 bytes) received in-sequence 45039 completely duplicate packets (1602267 bytes) 0 old duplicate packets 5 packets with some dup. data (624 bytes duped) 2441887 out-of-order packets (440550 bytes) 8 packets (8 bytes) of data after window 8 window probes 2587766 window update packets 3902 packets received after close 0 packets with bad hardware assisted checksum 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 1440 discarded by listeners 0 discarded due to listener's queue full 23840405 ack packet headers correctly predicted 35343642 data packet headers correctly predicted 399880 connection requests 5548229 connection accepts 5947965 connections established (including accepts) 5953023 connections closed (including 25563 drops) 0 connections with ECN capability 0 times responded to ECN 137 embryonic connections dropped 404559742 segments updated rtt (of 404578267 attempts) 0 segments with congestion window reduced bit set 0 segments with congestion experienced bit set 0 resends due to path MTU discovery 10019 path MTU discovery terminations due to retransmits 29115 retransmit timeouts 1 connection dropped by rexmit timeout 2 fast retransmits 0 when congestion window less than 4 segments 3 newreno retransmits 0 times avoided false fast retransmits 427603 persist timeouts 0 connections dropped due to persist timeout 8957 keepalive timeouts 8928 keepalive probes sent 29 connections dropped by keepalive 0 times SACK blocks array is extended 0 times SACK holes array is extended 0 packets dropped due to memory allocation failure 0 connections in timewait reused 0 delayed ACKs for SYN 0 delayed ACKs for FIN 0 send_and_disconnects 0 spliced connections 0 spliced connections closed 0 spliced connections reset 0 spliced connections timeout 0 spliced connections persist timeout 0 spliced connections keepalive timeout udp: 13767017 datagrams received 0 incomplete headers 0 bad data length fields 0 bad checksums 597 dropped due to no socket 6762980 broadcast/multicast datagrams dropped due to no socket 0 dropped due to full socket buffers 7003440 delivered 6994616 datagrams output ip: 923406868 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 923121828 packets for this host 283574 packets for unknown/unsupported protocol 0 packets forwarded 1396 packets not forwardable 0 redirects sent 456499667 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 69 IP Multicast packets dropped due to no receiver 0 successful path MTU discovery cycles 0 path MTU rediscovery cycles attempted 0 path MTU discovery no-response estimates 0 path MTU discovery response timeouts 0 path MTU discovery decreases detected 0 path MTU discovery packets sent 0 path MTU discovery memory allocation failures 0 ipintrq overflows 0 with illegal source 0 packets processed by threads 0 packets dropped by threads 0 packets dropped due to the full socket receive buffer 0 dead gateway detection packets sent 0 dead gateway detection packet allocation failures 0 dead gateway detection gateway allocation failures ipv6: 3 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with incorrect version number 0 with illegal source 0 input packets without enough memory 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 0 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded 3 packets not forwardable 0 too big packets not forwarded 0 packets sent from this host 0 packets sent with fabricated ipv6 header 0 output packets dropped due to no bufs 0 output packets without enough memory 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 packets dropped due to full socket receive buffer 0 packets not delivered due to bad raw IPv6 checksum icmpv6: 0 calls to icmp6_error 0 errors not generated because old message was icmpv6 Output histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 redirects: 0 echo requests: 0 echo replies: 0 group queries: 0 group reports: 0 group terminations: 0 router solicitations: 0 router advertisements: 0 neighbor solicitations: 0 neighbor advertisements: 0 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 echo requests: 0 echo replies: 0 group queries: 0 bad group queries: 0 group reports: 0 bad group reports: 0 our groups' reports: 0 group terminations: 0 bad group terminations: 0 router solicitations: 0 bad router solicitations: 0 router advertisements: 0 bad router advertisements: 0 neighbor solicitations: 0 bad neighbor solicitations: 0 neighbor advertisements: 0 bad neighbor advertisements: 0 redirects: 0 bad redirects: 0 mobility calls when not started: 0 home agent address discovery requests: 0 bad home agent address discovery requests: 0 bad home agent address discovery replys: 0 bad home agent address discovery replys: 0 prefix solicitations: 0 bad prefix solicitations: 0 prefix advertisements: 0 bad prefix advertisements: 0 0 message responses generated @@ @@data#1228|1154529774.054469|141.106.224.202||jordan.hmgc.mcw.edu|netst at data jordan,hmgc,mcw,edu.netstat aix icmp: 29 calls to icmp_error 0 errors not generated because old message was icmp Output histogram: echo reply: 16024 destination unreachable: 17 45 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 7 destination unreachable: 58 echo: 16024 16024 message responses generated igmp: 79096 messages received 0 messages received with too few bytes 0 messages received with bad checksum 79096 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 2 membership reports sent tcp: 256987083 packets sent 233786265 data packets (146658044 bytes) 30885 data packets (6116690 bytes) retransmitted 16701059 ack-only packets (3980875 delayed) 0 URG only packets 193 window probe packets 17752 window update packets 6450950 control packets 45166554 large sends 740111354 bytes sent using largesend 64240 bytes is the biggest largesend 496262560 packets received 450701474 acks (for 151098725 bytes) 9591183 duplicate acks 0 acks for unsent data 195459972 packets (949784396 bytes) received in-sequence 23755 completely duplicate packets (8371 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 3375737 out-of-order packets (104293 bytes) 0 packets (0 bytes) of data after window 0 window probes 4869090 window update packets 3508 packets received after close 0 packets with bad hardware assisted checksum 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 1117 discarded by listeners 0 discarded due to listener's queue full 10028924 ack packet headers correctly predicted 24673885 data packet headers correctly predicted 100631 connection requests 6257521 connection accepts 6358083 connections established (including accepts) 6359382 connections closed (including 24024 drops) 0 connections with ECN capability 0 times responded to ECN 65 embryonic connections dropped 240375423 segments updated rtt (of 240404633 attempts) 0 segments with congestion window reduced bit set 0 segments with congestion experienced bit set 0 resends due to path MTU discovery 14379 path MTU discovery terminations due to retransmits 31032 retransmit timeouts 5 connections dropped by rexmit timeout 3 fast retransmits 0 when congestion window less than 4 segments 10 newreno retransmits 5 times avoided false fast retransmits 195 persist timeouts 0 connections dropped due to persist timeout 1407 keepalive timeouts 1393 keepalive probes sent 14 connections dropped by keepalive 0 times SACK blocks array is extended 0 times SACK holes array is extended 0 packets dropped due to memory allocation failure 0 connections in timewait reused 0 delayed ACKs for SYN 0 delayed ACKs for FIN 0 send_and_disconnects 0 spliced connections 0 spliced connections closed 0 spliced connections reset 0 spliced connections timeout 0 spliced connections persist timeout 0 spliced connections keepalive timeout udp: 6773177 datagrams received 0 incomplete headers 0 bad data length fields 0 bad checksums 29 dropped due to no socket 471205 broadcast/multicast datagrams dropped due to no socket 0 socket buffer overflows 6301943 delivered 6301977 datagrams output ip: 504670433 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 503051772 packets for this host 79154 packets for unknown/unsupported protocol 0 packets forwarded 1539510 packets not forwardable 0 redirects sent 263316001 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 IP Multicast packets dropped due to no receiver 0 successful path MTU discovery cycles 0 path MTU rediscovery cycles attempted 0 path MTU discovery no-response estimates 0 path MTU discovery response timeouts 0 path MTU discovery decreases detected 0 path MTU discovery packets sent 0 path MTU discovery memory allocation failures 0 ipintrq overflows 0 with illegal source 0 packets processed by threads 0 packets dropped by threads 0 packets dropped due to the full socket receive buffer 0 dead gateway detection packets sent 0 dead gateway detection packet allocation failures 0 dead gateway detection gateway allocation failures ipv6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with incorrect version number 0 with illegal source 0 input packets without enough memory 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 0 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded 0 packets not forwardable 0 too big packets not forwarded 0 packets sent from this host 0 packets sent with fabricated ipv6 header 0 output packets dropped due to no bufs 0 output packets without enough memory 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 packets dropped due to full socket receive buffer 0 packets not delivered due to bad raw IPv6 checksum icmpv6: 0 calls to icmp6_error 0 errors not generated because old message was icmpv6 Output histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 redirects: 0 echo requests: 0 echo replies: 0 group queries: 0 group reports: 0 group terminations: 0 router solicitations: 0 router advertisements: 0 neighbor solicitations: 0 neighbor advertisements: 0 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 echo requests: 0 echo replies: 0 group queries: 0 bad group queries: 0 group reports: 0 bad group reports: 0 our groups' reports: 0 group terminations: 0 bad group terminations: 0 router solicitations: 0 bad router solicitations: 0 router advertisements: 0 bad router advertisements: 0 neighbor solicitations: 0 bad neighbor solicitations: 0 neighbor advertisements: 0 bad neighbor advertisements: 0 redirects: 0 bad redirects: 0 mobility calls when not started: 0 home agent address discovery requests: 0 bad home agent address discovery requests: 0 bad home agent address discovery replys: 0 bad home agent address discovery replys: 0 prefix solicitations: 0 bad prefix solicitations: 0 prefix advertisements: 0 bad prefix advertisements: 0 0 message responses generated @@ --------------------------------------------------------- Kent C. Brodie - brodie at phys.mcw.edu Department of Physiology Medical College of Wisconsin (414) 456-8590 -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Wednesday, August 02, 2006 5:59 AM To: hobbit at hswn.dk Subject: Re: [hobbit] rrd-data.log On Wed, Aug 02, 2006 at 12:23:53PM +0200, Beau Olivier wrote: > > I'm having "Internal error: Duplicate match ignored" in my rrd-data.log, > what could cause this ? It means your netstat data doesn't look like what Hobbit expects. Basically that it found two or more values for the same piece of data. The best way of identifying which data causes this is probably to run two things at the same time: 1) login as the hobbit user, and run bbcmd hobbitd_channel --channel=data tee /tmp/data.log 2) Run "tail -f" on the rrd-data.log file. When you see that error message in the rrd-data.log file, terminate the first command. You should then have the "guilty" data at the end of the /tmp/data.log file. I'd obviously be interested to see what it looks like. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

1 0

[hobbit] Oratns
by JMSmith＠stvincenthealth.com 02 Aug '06

02 Aug '06

Ha! I know what you mean. My son is a programmer for a government agency. ________________________________ From: Everett, Vernon [mailto:Vernon.Everett at dpi.wa.gov.au] Sent: Tuesday, August 01, 2006 9:16 PM To: hobbit at hswn.dk Subject: RE: [hobbit] Oratns Thanks Jim Sounds like an idea. If you have time I would apreciate it. But let's give it some time, somebody may still have the answer. Henrik could probably check the code and tell us - and he should be up in just a few hours. The irony is, I am a DBA as well as a SysAdmin, and could do exactly as you suggest easily enough, but I work for a government department, hired as a contract SysAdmin!!!!!! The outcry it would cause if a SysAdmin did any DBA work is just not worth it. The senior DBA is on leave, and the junior one can't/won't help. (The senior DBA is a mate of mine, so he would help, but isn't back for 2 weeks) Amazing how easily politics can prevent you having a good day. :-( Cheers Vernon ________________________________ From: Smith, Jim [mailto:JMSmith at stvincenthealth.com] Sent: Wednesday, 2 August 2006 9:42 AM To: hobbit at hswn.dk Subject: RE: [hobbit] Oratns Yes, I'm sure it WOULD be a CLM! Unfortunately, I don't have multiple listeners running on any of my servers so I can't be of much help. Do you have a test server running Oracle that you can play with? I do and am willing to give this a try if I can get my DBA to assist with setting up some dummy listeners. Jim Smith -----Original Message----- From: Everett, Vernon [mailto:Vernon.Everett at dpi.wa.gov.au] Sent: Tuesday, August 01, 2006 8:36 PM To: hobbit at hswn.dk Subject: [hobbit] Oratns Hi all Some assistance with bb-hosts please. If I want to check for an oracle listener, my line looks like this 1.2.3.4 name.domain.com # oratns:1545 This will check for an Oracle listener on port 1545 How do I make it check for more than one Oracle listener? For reasons unknown to me, we are running multiple listeners on separate ports on a server. Should I have multiple oratns entries (so my line looks like this) 1.2.3.4 name.domain.com # oratns:1545 oratns 1546 Or can we do it like this 1.2.3.4 name.domain.com # oratns:1545:1546 Ordinarily, I would just run a few tests, but bring down one of the listeners on a production server as a test could prove to be a serious CLM. Regards Vernon To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk NOTICE: This email contains confidential or proprietary information which may be legally privileged. It is intended only for the named recipient(s). If an addressing error has misdirected the email, please notify the author by replying to this message. If you are not the named recipient, you are not authorized to use, disclose, distribute, copy, print or rely on this email, and should immediately delete it from your computer system. NOTICE: This email contains confidential or proprietary information which may be legally privileged. It is intended only for the named recipient(s). If an addressing error has misdirected the email, please notify the author by replying to this message. If you are not the named recipient, you are not authorized to use, disclose, distribute, copy, print or rely on this email, and should immediately delete it from your computer system.

1 0

client info
by kolbjorn.barmen＠uninett.no 02 Aug '06

02 Aug '06

Is it possible to add more information to the "info" page? One thing I miss on that page is "Client version:" (especially now as I upgrade BBWin client). -- Kolbjørn Barmen UNINETT Driftsenter

2 1

[hobbit] Shire: update
by brodie＠mcw.edu 02 Aug '06

02 Aug '06

So long as we can "search" on the module or item description(s), we should be fine. --------------------------------------------------------- Kent C. Brodie - brodie at phys.mcw.edu Department of Physiology Medical College of Wisconsin (414) 456-8590 -----Original Message----- From: Charles Goyard [mailto:cgoyard at cvf.fr] Sent: Tuesday, August 01, 2006 9:56 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Shire: update Hi, talking about categories, one thing that deadcat misses is the display (and search) of the column name generated. People sometime use really generic names, such as "files", "size", "modules" or "test" (you bet) for very application-specific matters. It can also prevent name clashes. Just a idea. -- Charles Goyard - cgoyard at cvf.fr - (+33) 1 45 38 01 31 To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

3 2

[hobbit] RE: Default groupings
by henrik＠hswn.dk 02 Aug '06

02 Aug '06

Hi Jason, On Thu, Jul 20, 2006 at 10:15:40AM -0500, Kruse, Jason K. wrote: > > If you set a local test without a GROUP label it will cause all alerts > that match to fire, regardless of whether they have a GROUP label or > not. > > To use the GROUP functionality properly every test that requires a > different group will require a label for every entry for a host/service, > depending on how you've configured your hobbit-alerts.cfg I would like > the default to be to send a GROUP label with every set of tests and have > the hobbit-alerts.cfg to have a default group for matching when no group > is assigned. If I understand you correctly, you have a setup like this. In hobbit-clients.cfg you tag some e.g. disk statuses with a group label, and some do not have a group label: HOST server1 DISK / 80 90 groupid=admins HOST server2 DISK / 85 95 And in hobbit-alerts.cfg you then have the "admins" group alerts going to one recipient and the other alerts going to someone else: SERVICE=disk GROUP=admins MAIL admins at foo.com SERVICE=disk MAIL someone at foo.com I think the real problem is different - when you have a test that does not have any GROUP association, then alert rules with a specific GROUP label should not be matched. And the current code doesn't behave quite as it should; there's a patch attached to this mail. So with that patch let me walk through how I think it should work. If server1 gets a disk alert, it will be tagged with the "admins" group label. This will currently match both of the alert rules; the first one obviously because the group label matches, and the second one because it doesn't have any group filter at all. This doesn't seem right - and it's the problem you describe. I think the best solution here is to change the last alert rule to SERVICE=DISK EXGROUP=* MAIL someone at foo.com so it is clear that this rule is not used whenever the status is tagged with a group label. Does that sound like a reasonable solution? If server2 gets a disk alert, it won't have any group label. So it won't match the first alert rule (group doesn't match), and it will match the second one - both in the form with and without the EXGROUP setting. So that behaves as one would expect. Regards, Henrik

1 0