Aha...! I asked this very question last week- nobody gave me help... (sob, sob..). Anyway, here are two separate chunks of data that is CAUSING the duplicate data error like Beau is having. Henrik, what in this data is considered to be "duplicate"? (I do notice that the netstat data in AIX is um... QUITE verbose........) Data segments follow. Help? @@data#810|1154529466.624163|999.888.777.202||fred.moo.mcw.edu|ifstat data fred,moo,mcw,edu.ifstat aix ------------------------------------------------------------- ETHERNET STATISTICS (ent0) : Device Type: 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Hardware Address: 00:02:55:4f:a9:f4 Elapsed Time: 0 days 0 hours 0 minutes 0 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 0 Packets: 0 Bytes: 0 Bytes: 0 Interrupts: 2 Interrupts: 0 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 0 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 0 Broadcast Packets: 0 Multicast Packets: 0 Multicast Packets: 0 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 200 Driver Flags: Up Broadcast Running Simplex AlternateAddress 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Specific Statistics: -------------------------------------------------------------------- Link Status: Down Media Speed Selected: Auto negotiation Media Speed Running: Unknown Receive Pool Buffer Size: 1024 Free Receive Pool Buffers: 1024 No Receive Pool Buffer Errors: 0 Receive Buffer Too Small Errors: 0 Entries to transmit timeout routine: 0 Transmit IPsec packets: 0 Transmit IPsec packets dropped: 0 Receive IPsec packets: 0 Receive IPsec packets dropped: 0 Inbound IPsec SA offload count: 0 Transmit Large Send packets: 0 Transmit Large Send packets dropped: 0 Packets with Transmit collisions: 1 collisions: 0 6 collisions: 0 11 collisions: 0 2 collisions: 0 7 collisions: 0 12 collisions: 0 3 collisions: 0 8 collisions: 0 13 collisions: 0 4 collisions: 0 9 collisions: 0 14 collisions: 0 5 collisions: 0 10 collisions: 0 15 collisions: 0 ------------------------------------------------------------- ETHERNET STATISTICS (ent1) : Device Type: 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Hardware Address: 00:02:55:4f:a9:f3 Elapsed Time: 0 days 0 hours 0 minutes 0 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 0 Packets: 0 Bytes: 0 Bytes: 0 Interrupts: 2 Interrupts: 0 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 0 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 0 Broadcast Packets: 0 Multicast Packets: 0 Multicast Packets: 0 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 200 Driver Flags: Up Broadcast Running Simplex AlternateAddress 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 10/100 Mbps Ethernet PCI Adapter II (1410ff01) Specific Statistics: -------------------------------------------------------------------- Link Status: Down Media Speed Selected: Auto negotiation Media Speed Running: Unknown Receive Pool Buffer Size: 1024 Free Receive Pool Buffers: 1024 No Receive Pool Buffer Errors: 0 Receive Buffer Too Small Errors: 0 Entries to transmit timeout routine: 0 Transmit IPsec packets: 0 Transmit IPsec packets dropped: 0 Receive IPsec packets: 0 Receive IPsec packets dropped: 0 Inbound IPsec SA offload count: 0 Transmit Large Send packets: 0 Transmit Large Send packets dropped: 0 Packets with Transmit collisions: 1 collisions: 0 6 collisions: 0 11 collisions: 0 2 collisions: 0 7 collisions: 0 12 collisions: 0 3 collisions: 0 8 collisions: 0 13 collisions: 0 4 collisions: 0 9 collisions: 0 14 collisions: 0 5 collisions: 0 10 collisions: 0 15 collisions: 0 ------------------------------------------------------------- ETHERNET STATISTICS (ent2) : Device Type: 10/100/1000 Base-TX PCI-X Adapter (14106902) Hardware Address: 00:02:55:53:c2:3e Elapsed Time: 196 days 15 hours 56 minutes 59 seconds Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 365802341 Packets: 1036460447 Bytes: 1683156286637 Bytes: 112378387074 Interrupts: 0 Interrupts: 614513005 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 30 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 16681 Broadcast Packets: 210557123 Multicast Packets: 0 Multicast Packets: 283080 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 381 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 2000 Driver Flags: Up Broadcast Running Simplex 64BitSupport ChecksumOffload PrivateSegment LargeSend DataRateSet 10/100/1000 Base-TX PCI-X Adapter (14106902) Specific Statistics: -------------------------------------------------------------------- Link Status: Up Media Speed Selected: Auto negotiation Media Speed Running: 1000 Mbps Full Duplex PCI Mode: PCI-X (100-133) PCI Bus Width: 64-bit Latency Timer: 144 Cache Line Size: 128 Jumbo Frames: Disabled TCP Segmentation Offload: Enabled TCP Segmentation Offload Packets Transmitted: 113291719 TCP Segmentation Offload Packet Errors: 0 Transmit and Receive Flow Control Status: Enabled XON Flow Control Packets Transmitted: 0 XON Flow Control Packets Received: 430 XOFF Flow Control Packets Transmitted: 0 XOFF Flow Control Packets Received: 430 Transmit and Receive Flow Control Threshold (High): 45056 Transmit and Receive Flow Control Threshold (Low): 24576 Transmit and Receive Storage Allocation (TX/RX): 16/48 @@ @@data#811|1154529466.624871|999.777.666.202||phred.mrr.mcw.edu|netstat data phred,mrr,mcw,edu.netstat aix icmp: 597 calls to icmp_error 0 errors not generated because old message was icmp Output histogram: echo reply: 58031 destination unreachable: 537 1 message with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 2 destination unreachable: 562 echo: 58031 time exceeded: 12 58031 message responses generated igmp: 282998 messages received 0 messages received with too few bytes 0 messages received with bad checksum 282998 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 2 membership reports sent tcp: 449406211 packets sent 419904427 data packets (1612944301 bytes) 176131 data packets (218422100 bytes) retransmitted 20439309 ack-only packets (8520363 delayed) 6 URG only packets 420337 window probe packets 2131650 window update packets 6334351 control packets 113291719 large sends 3038842766 bytes sent using largesend 64240 bytes is the biggest largesend 909296772 packets received 857387271 acks (for 1835107293 bytes) 7844383 duplicate acks 0 acks for unsent data 307390334 packets (2464690667 bytes) received in-sequence 45039 completely duplicate packets (1602267 bytes) 0 old duplicate packets 5 packets with some dup. data (624 bytes duped) 2441887 out-of-order packets (440550 bytes) 8 packets (8 bytes) of data after window 8 window probes 2587766 window update packets 3902 packets received after close 0 packets with bad hardware assisted checksum 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 1440 discarded by listeners 0 discarded due to listener's queue full 23840405 ack packet headers correctly predicted 35343642 data packet headers correctly predicted 399880 connection requests 5548229 connection accepts 5947965 connections established (including accepts) 5953023 connections closed (including 25563 drops) 0 connections with ECN capability 0 times responded to ECN 137 embryonic connections dropped 404559742 segments updated rtt (of 404578267 attempts) 0 segments with congestion window reduced bit set 0 segments with congestion experienced bit set 0 resends due to path MTU discovery 10019 path MTU discovery terminations due to retransmits 29115 retransmit timeouts 1 connection dropped by rexmit timeout 2 fast retransmits 0 when congestion window less than 4 segments 3 newreno retransmits 0 times avoided false fast retransmits 427603 persist timeouts 0 connections dropped due to persist timeout 8957 keepalive timeouts 8928 keepalive probes sent 29 connections dropped by keepalive 0 times SACK blocks array is extended 0 times SACK holes array is extended 0 packets dropped due to memory allocation failure 0 connections in timewait reused 0 delayed ACKs for SYN 0 delayed ACKs for FIN 0 send_and_disconnects 0 spliced connections 0 spliced connections closed 0 spliced connections reset 0 spliced connections timeout 0 spliced connections persist timeout 0 spliced connections keepalive timeout udp: 13767017 datagrams received 0 incomplete headers 0 bad data length fields 0 bad checksums 597 dropped due to no socket 6762980 broadcast/multicast datagrams dropped due to no socket 0 dropped due to full socket buffers 7003440 delivered 6994616 datagrams output ip: 923406868 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 923121828 packets for this host 283574 packets for unknown/unsupported protocol 0 packets forwarded 1396 packets not forwardable 0 redirects sent 456499667 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 69 IP Multicast packets dropped due to no receiver 0 successful path MTU discovery cycles 0 path MTU rediscovery cycles attempted 0 path MTU discovery no-response estimates 0 path MTU discovery response timeouts 0 path MTU discovery decreases detected 0 path MTU discovery packets sent 0 path MTU discovery memory allocation failures 0 ipintrq overflows 0 with illegal source 0 packets processed by threads 0 packets dropped by threads 0 packets dropped due to the full socket receive buffer 0 dead gateway detection packets sent 0 dead gateway detection packet allocation failures 0 dead gateway detection gateway allocation failures ipv6: 3 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with incorrect version number 0 with illegal source 0 input packets without enough memory 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 0 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded 3 packets not forwardable 0 too big packets not forwarded 0 packets sent from this host 0 packets sent with fabricated ipv6 header 0 output packets dropped due to no bufs 0 output packets without enough memory 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 packets dropped due to full socket receive buffer 0 packets not delivered due to bad raw IPv6 checksum icmpv6: 0 calls to icmp6_error 0 errors not generated because old message was icmpv6 Output histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 redirects: 0 echo requests: 0 echo replies: 0 group queries: 0 group reports: 0 group terminations: 0 router solicitations: 0 router advertisements: 0 neighbor solicitations: 0 neighbor advertisements: 0 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 echo requests: 0 echo replies: 0 group queries: 0 bad group queries: 0 group reports: 0 bad group reports: 0 our groups' reports: 0 group terminations: 0 bad group terminations: 0 router solicitations: 0 bad router solicitations: 0 router advertisements: 0 bad router advertisements: 0 neighbor solicitations: 0 bad neighbor solicitations: 0 neighbor advertisements: 0 bad neighbor advertisements: 0 redirects: 0 bad redirects: 0 mobility calls when not started: 0 home agent address discovery requests: 0 bad home agent address discovery requests: 0 bad home agent address discovery replys: 0 bad home agent address discovery replys: 0 prefix solicitations: 0 bad prefix solicitations: 0 prefix advertisements: 0 bad prefix advertisements: 0 0 message responses generated @@ @@data#1228|1154529774.054469|141.106.224.202||jordan.hmgc.mcw.edu|netst at data jordan,hmgc,mcw,edu.netstat aix icmp: 29 calls to icmp_error 0 errors not generated because old message was icmp Output histogram: echo reply: 16024 destination unreachable: 17 45 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 7 destination unreachable: 58 echo: 16024 16024 message responses generated igmp: 79096 messages received 0 messages received with too few bytes 0 messages received with bad checksum 79096 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 2 membership reports sent tcp: 256987083 packets sent 233786265 data packets (146658044 bytes) 30885 data packets (6116690 bytes) retransmitted 16701059 ack-only packets (3980875 delayed) 0 URG only packets 193 window probe packets 17752 window update packets 6450950 control packets 45166554 large sends 740111354 bytes sent using largesend 64240 bytes is the biggest largesend 496262560 packets received 450701474 acks (for 151098725 bytes) 9591183 duplicate acks 0 acks for unsent data 195459972 packets (949784396 bytes) received in-sequence 23755 completely duplicate packets (8371 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 3375737 out-of-order packets (104293 bytes) 0 packets (0 bytes) of data after window 0 window probes 4869090 window update packets 3508 packets received after close 0 packets with bad hardware assisted checksum 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 1117 discarded by listeners 0 discarded due to listener's queue full 10028924 ack packet headers correctly predicted 24673885 data packet headers correctly predicted 100631 connection requests 6257521 connection accepts 6358083 connections established (including accepts) 6359382 connections closed (including 24024 drops) 0 connections with ECN capability 0 times responded to ECN 65 embryonic connections dropped 240375423 segments updated rtt (of 240404633 attempts) 0 segments with congestion window reduced bit set 0 segments with congestion experienced bit set 0 resends due to path MTU discovery 14379 path MTU discovery terminations due to retransmits 31032 retransmit timeouts 5 connections dropped by rexmit timeout 3 fast retransmits 0 when congestion window less than 4 segments 10 newreno retransmits 5 times avoided false fast retransmits 195 persist timeouts 0 connections dropped due to persist timeout 1407 keepalive timeouts 1393 keepalive probes sent 14 connections dropped by keepalive 0 times SACK blocks array is extended 0 times SACK holes array is extended 0 packets dropped due to memory allocation failure 0 connections in timewait reused 0 delayed ACKs for SYN 0 delayed ACKs for FIN 0 send_and_disconnects 0 spliced connections 0 spliced connections closed 0 spliced connections reset 0 spliced connections timeout 0 spliced connections persist timeout 0 spliced connections keepalive timeout udp: 6773177 datagrams received 0 incomplete headers 0 bad data length fields 0 bad checksums 29 dropped due to no socket 471205 broadcast/multicast datagrams dropped due to no socket 0 socket buffer overflows 6301943 delivered 6301977 datagrams output ip: 504670433 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 503051772 packets for this host 79154 packets for unknown/unsupported protocol 0 packets forwarded 1539510 packets not forwardable 0 redirects sent 263316001 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 IP Multicast packets dropped due to no receiver 0 successful path MTU discovery cycles 0 path MTU rediscovery cycles attempted 0 path MTU discovery no-response estimates 0 path MTU discovery response timeouts 0 path MTU discovery decreases detected 0 path MTU discovery packets sent 0 path MTU discovery memory allocation failures 0 ipintrq overflows 0 with illegal source 0 packets processed by threads 0 packets dropped by threads 0 packets dropped due to the full socket receive buffer 0 dead gateway detection packets sent 0 dead gateway detection packet allocation failures 0 dead gateway detection gateway allocation failures ipv6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with incorrect version number 0 with illegal source 0 input packets without enough memory 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 0 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded 0 packets not forwardable 0 too big packets not forwarded 0 packets sent from this host 0 packets sent with fabricated ipv6 header 0 output packets dropped due to no bufs 0 output packets without enough memory 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 packets dropped due to full socket receive buffer 0 packets not delivered due to bad raw IPv6 checksum icmpv6: 0 calls to icmp6_error 0 errors not generated because old message was icmpv6 Output histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 redirects: 0 echo requests: 0 echo replies: 0 group queries: 0 group reports: 0 group terminations: 0 router solicitations: 0 router advertisements: 0 neighbor solicitations: 0 neighbor advertisements: 0 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: unreachable: 0 packets too big: 0 time exceeded: 0 parameter problems: 0 echo requests: 0 echo replies: 0 group queries: 0 bad group queries: 0 group reports: 0 bad group reports: 0 our groups' reports: 0 group terminations: 0 bad group terminations: 0 router solicitations: 0 bad router solicitations: 0 router advertisements: 0 bad router advertisements: 0 neighbor solicitations: 0 bad neighbor solicitations: 0 neighbor advertisements: 0 bad neighbor advertisements: 0 redirects: 0 bad redirects: 0 mobility calls when not started: 0 home agent address discovery requests: 0 bad home agent address discovery requests: 0 bad home agent address discovery replys: 0 bad home agent address discovery replys: 0 prefix solicitations: 0 bad prefix solicitations: 0 prefix advertisements: 0 bad prefix advertisements: 0 0 message responses generated @@ --------------------------------------------------------- Kent C. Brodie - brodie at phys.mcw.edu Department of Physiology Medical College of Wisconsin (414) 456-8590 -----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: Wednesday, August 02, 2006 5:59 AM To: hobbit at hswn.dk Subject: Re: [hobbit] rrd-data.log On Wed, Aug 02, 2006 at 12:23:53PM +0200, Beau Olivier wrote:
I'm having "Internal error: Duplicate match ignored" in my
rrd-data.log,
what could cause this ?
It means your netstat data doesn't look like what Hobbit expects. Basically that it found two or more values for the same piece of data. The best way of identifying which data causes this is probably to run two things at the same time: 1) login as the hobbit user, and run bbcmd hobbitd_channel --channel=data tee /tmp/data.log 2) Run "tail -f" on the rrd-data.log file. When you see that error message in the rrd-data.log file, terminate the first command. You should then have the "guilty" data at the end of the /tmp/data.log file. I'd obviously be interested to see what it looks like. Regards, Henrik To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk