Buchan- you should really capture the incoming data segment while simultaneously watching the rrd-data.log file; that way you can 100% let Henrik know which data chunks are causing RRD issues. In my case, I only had to "watch" for 5 minutes to capture all the instances of duplicztes happening.... --------------------------------------------------------- Kent C. Brodie - brodie at phys.mcw.edu Department of Physiology Medical College of Wisconsin (414) 456-8590 -----Original Message----- From: Buchan Milne [mailto:bgmilne at staff.telkomsa.net] Sent: Thursday, August 03, 2006 9:46 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Some more clear RRD duplicating-causing samples On Thursday 03 August 2006 16:28, Brodie, Kent wrote:

Henrik, here you go: Two more netstat data chunks that cause the Duplication error (parsing issue?) in rrd/hobbit.

My simple observation is the one thing these two hosts have in common is virtual network interfaces, so instead of simpler items like eth0, eth1 and so on, you have eth0, eth0:0, and so forth. Linux in this case, but that would apply to solaris and so on as well. Just a hunch.................

@@data#98185|1154614736.024527|192.168.224.202||dunn.hmgc.mcw.edu|ifstat

data dunn,hmgc,mcw,edu.ifstat linux eth0 Link encap:Ethernet HWaddr 00:09:3D:13:DC:AB inet addr:192.168.224.105 Bcast:192.168.224.255 Mask:255.255.255.0 inet6 addr: fe80::209:3dff:fe13:dcab/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:40318794 errors:0 dropped:0 overruns:0 frame:0 TX packets:15352953 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2942042461 (2.7 GiB) TX bytes:21803054108 (20.3 GiB) Interrupt:185

eth0:0 Link encap:Ethernet HWaddr 00:09:3D:13:DC:AB inet addr:192.168.224.107 Bcast:192.168.224.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:185

eth0:1 Link encap:Ethernet HWaddr 00:09:3D:13:DC:AB inet addr:192.168.224.160 Bcast:192.168.224.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:185

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:37737 errors:0 dropped:0 overruns:0 frame:0 TX packets:37737 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6042388 (5.7 MiB) TX bytes:6042388 (5.7 MiB)

@@

@@data#98366|1154614859.203728|192.168.224.202||clifford.hmgc.mcw.edu|if

stat data clifford,hmgc,mcw,edu.ifstat linux eth0 Link encap:Ethernet HWaddr 00:09:3D:13:C8:78 inet addr:192.168.224.104 Bcast:192.168.224.255 Mask:255.255.255.0 inet6 addr: fe80::209:3dff:fe13:c878/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:146220628 errors:0 dropped:0 overruns:0 frame:0 TX packets:42544430 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:14048692258 (13.0 GiB) TX bytes:57506442876 (53.5 GiB) Interrupt:185

eth0:0 Link encap:Ethernet HWaddr 00:09:3D:13:C8:78 inet addr:192.168.224.106 Bcast:192.168.224.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:185

eth0:1 Link encap:Ethernet HWaddr 00:09:3D:13:C8:78 inet addr:192.168.224.159 Bcast:192.168.224.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:185

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1189289 errors:0 dropped:0 overruns:0 frame:0 TX packets:1189289 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:347179798 (331.0 MiB) TX bytes:347179798 (331.0 MiB)

@@

Hmm, now that I think about this more carefully, this issue could be affecting me as well, resulting in some gaps in various rrd files (which I first thought might be affected by running everything via bbproxy and a rather complex setup). In our environment we typically use bonding/teaming (eg with eth0 connected to one switch, eth1 connected to another, with the two interfaces set up in passive failover, appearing as one "bond0" device). Then, to ensure access to the right networks, we have VLANs (eg bond0.4, bond0.54, bond0106) on top of the bonded interface. In some cases (HA setups), there are virtual interfaces (eg bond0.4:0)on top of the bonded interface. I've included sample ifstat contents (but, not necessarily at the time I've seen gaps in the rrds): [ifstat] bond0 Link encap:Ethernet HWaddr 00:16:35:7E:8D:59 inet6 addr: fe80::216:35ff:fe7e:8d59/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:1353358109 errors:0 dropped:0 overruns:0 frame:0 TX packets:1414161412 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3466938915 (3.2 GiB) TX bytes:1361756811 (1.2 GiB) bond0.4 Link encap:Ethernet HWaddr 00:16:35:7E:8D:59 inet addr:192.168.211.41 Bcast:192.168.211.255 Mask:255.255.255.0 inet6 addr: fe80::216:35ff:fe7e:8d59/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:1353172280 errors:0 dropped:0 overruns:0 frame:0 TX packets:1414125729 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3747147076 (3.4 GiB) TX bytes:2930351274 (2.7 GiB) bond0.4:0 Link encap:Ethernet HWaddr 00:16:35:7E:8D:59 inet addr:192.168.211.203 Bcast:192.168.211.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 bond0.54 Link encap:Ethernet HWaddr 00:16:35:7E:8D:59 inet addr:192.168.12.18 Bcast:192.168.15.255 Mask:255.255.252.0 inet6 addr: fe80::216:35ff:fe7e:8d59/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:15562 errors:0 dropped:0 overruns:0 frame:0 TX packets:220 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:952935 (930.6 KiB) TX bytes:9408 (9.1 KiB) bond0.106 Link encap:Ethernet HWaddr 00:16:35:7E:8D:59 inet addr:192.168.106.68 Bcast:192.168.106.255 Mask:255.255.255.0 inet6 addr: fe80::216:35ff:fe7e:8d59/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:173127 errors:0 dropped:0 overruns:0 frame:0 TX packets:38287 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:10379598 (9.8 MiB) TX bytes:3224568 (3.0 MiB) eth0 Link encap:Ethernet HWaddr 00:16:35:7E:8D:59 inet6 addr: fe80::216:35ff:fe7e:8d59/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1353358109 errors:0 dropped:0 overruns:0 frame:0 TX packets:1414161412 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3466938915 (3.2 GiB) TX bytes:1361756811 (1.2 GiB) Interrupt:201 eth1 Link encap:Ethernet HWaddr 00:16:35:7E:8D:59 inet6 addr: fe80::216:35ff:fe7e:8d59/64 Scope:Link UP BROADCAST NOARP SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:209 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1668785 errors:0 dropped:0 overruns:0 frame:0 TX packets:1668785 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:720576454 (687.1 MiB) TX bytes:720576454 (687.1 MiB) Regards, Buchan -- Buchan Milne ISP Systems Specialist B.Eng,RHCE(803004789010797),LPIC-2(LPI000074592)