monitoring aggregated network traffic
Hi,
is there any "usual" way, to monitor and graph the traffic of a bulk of network interfaces? The reason is, some of our hosts are connected to iSCSI storages via multipath over two or four independent interfaces. Since the traffic is balanced equally, every interface shows (more or less) the same amount of traffic. This leads to very useless rrdtool graphics as there is only one multicolored thick line visible. It would be really helpful to see a graph which shows the SUM of the particular interfaces. I've recently tried to hack an own ifconfig script which does the calculation and adds an fake "aggr0" interface, but for some reason, that fake interface doesn't show up. It's visible under the [ifstat] section, but i assume the rrdtool relies on some other info.
Cheers,
Stephan
On 13-10-2011 16:44, netz-haut - stephan seitz wrote:
is there any "usual" way, to monitor and graph the traffic of a bulk of network interfaces? The reason is, some of our hosts are connected to iSCSI storages via multipath over two or four independent interfaces. Since the traffic is balanced equally, every interface shows (more or less) the same amount of traffic. This leads to very useless rrdtool graphics as there is only one multicolored thick line visible. It would be really helpful to see a graph which shows the SUM of the particular interfaces.
This can be done by creating an RRD graph definition to do the summation for you.
In RRDtool, you first use one or more "DEF" lines to grab specific data items from one or more RRD files, and put them into some "variables". You can then either use these variables directly to make a line on the graph, or you can use them for calculating other variables via a "CDEF" line.
The default ifstat graph has these lines:
DEF:inbytes at RRDIDX@=@RRDFN@:bytesReceived:AVERAGE
CDEF:in at RRDIDX@=inbytes at RRDIDX@,8,*(The @RRDIDX@ and @RRDFN@ are because this graph repeats over all the interfaces - Xymon replaces these with a number and the RRD filename).
What this does is to first define a variable "inbytes at RRDIDX@" which has the "bytesReceived" value from the RRD. The it computes another variable, "in at RRDIDX@" as 8*inbytes - to convert from bytes to bits. The notation here is Reverse Polish (RPN).
So if you have 4 RRD files, one for each interface, you can have RRDtool do the math for you like this:
[ifsum] TITLE Network Traffic YAXIS Bits/second DEF:inbytes0=ifstat.eth0.rrd:bytesReceived:AVERAGE DEF:inbytes1=ifstat.eth1.rrd:bytesReceived:AVERAGE DEF:inbytes2=ifstat.eth2.rrd:bytesReceived:AVERAGE DEF:inbytes3=ifstat.eth3.rrd:bytesReceived:AVERAGE CDEF:intotal=inbytes0,inbytes1,inbytes2,inbytes3,+,+,+ CDEF:inbits=intotal,8,* LINE:inbits#FF0000:Aggregate inbound
The "LINE" definition then uses the "inbits" value to draw a (red) line with this data.
Have a look at graphs.cfg, and I'm sure you can figure out to add a line for the outgoing data also.
Add "ifsum" to the GRAPHS setting in xymonserver.cfg, and the graph should show up on the "trends" page.
Regards, Henrik
PS: I haven't actually tested any of this, so feel free to provide corrections :-)
How would that work when there are varying numbers of interfaces? I have 14 on one, and 10 on another, each interface in its own RRD being collected by MRTG. If I put all 14 in, won't it break on the 10?
Scot Kreienkamp skreien at la-z-boy.com
-----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Henrik Størner Sent: Thursday, October 13, 2011 1:42 PM To: xymon at xymon.com Subject: Re: [Xymon] monitoring aggregated network traffic
On 13-10-2011 16:44, netz-haut - stephan seitz wrote:
is there any "usual" way, to monitor and graph the traffic of a bulk of network interfaces? The reason is, some of our hosts are connected to iSCSI storages via multipath over two or four independent interfaces. Since the traffic is balanced equally, every interface shows (more or less) the same amount of traffic. This leads to very useless rrdtool graphics as there is only one multicolored thick line visible. It would be really helpful to see a graph which shows the SUM of the particular interfaces.
This can be done by creating an RRD graph definition to do the summation for you.
In RRDtool, you first use one or more "DEF" lines to grab specific data items from one or more RRD files, and put them into some "variables". You can then either use these variables directly to make a line on the graph, or you can use them for calculating other variables via a "CDEF" line.
The default ifstat graph has these lines:
DEF:inbytes at RRDIDX@=@RRDFN@:bytesReceived:AVERAGE
CDEF:in at RRDIDX@=inbytes at RRDIDX@,8,*(The @RRDIDX@ and @RRDFN@ are because this graph repeats over all the interfaces - Xymon replaces these with a number and the RRD filename).
What this does is to first define a variable "inbytes at RRDIDX@" which has the "bytesReceived" value from the RRD. The it computes another variable, "in at RRDIDX@" as 8*inbytes - to convert from bytes to bits. The notation here is Reverse Polish (RPN).
So if you have 4 RRD files, one for each interface, you can have RRDtool do the math for you like this:
[ifsum] TITLE Network Traffic YAXIS Bits/second DEF:inbytes0=ifstat.eth0.rrd:bytesReceived:AVERAGE DEF:inbytes1=ifstat.eth1.rrd:bytesReceived:AVERAGE DEF:inbytes2=ifstat.eth2.rrd:bytesReceived:AVERAGE DEF:inbytes3=ifstat.eth3.rrd:bytesReceived:AVERAGE CDEF:intotal=inbytes0,inbytes1,inbytes2,inbytes3,+,+,+ CDEF:inbits=intotal,8,* LINE:inbits#FF0000:Aggregate inbound
The "LINE" definition then uses the "inbits" value to draw a (red) line with this data.
Have a look at graphs.cfg, and I'm sure you can figure out to add a line for the outgoing data also.
Add "ifsum" to the GRAPHS setting in xymonserver.cfg, and the graph should show up on the "trends" page.
Regards, Henrik
PS: I haven't actually tested any of this, so feel free to provide corrections :-)
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On 13-10-2011 19:52, Scot Kreienkamp wrote:
How would that work when there are varying numbers of interfaces? I have 14 on one, and 10 on another, each interface in its own RRD being collected by MRTG. If I put all 14 in, won't it break on the 10?
I was afraid You were going to ask that question. Yes, it will break because some of those files just do not exist, and RRDtool cannot handle that.
I don't have an automatic solution for that. You can define multiple graph definitions - one for each number of interfaces - and then use the "TRENDS" setting for each host to determine which one will be shown. Not pretty, but it works.
Regards, Henrik
Yeah, when you wrote that solution like that I was afraid that's what you'd say. :)
No problem, at least it's workable even if it is a PITA.
Scot Kreienkamp Senior Systems Engineer skreien at la-z-boy.com
-----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Henrik Størner Sent: Thursday, October 13, 2011 4:48 PM To: xymon at xymon.com Subject: Re: [Xymon] monitoring aggregated network traffic
On 13-10-2011 19:52, Scot Kreienkamp wrote:
How would that work when there are varying numbers of interfaces? I have 14 on one, and 10 on another, each interface in its own RRD being collected by MRTG. If I put all 14 in, won't it break on the 10?
I was afraid You were going to ask that question. Yes, it will break because some of those files just do not exist, and RRDtool cannot handle that.
I don't have an automatic solution for that. You can define multiple graph definitions - one for each number of interfaces - and then use the "TRENDS" setting for each host to determine which one will be shown. Not pretty, but it works.
Regards, Henrik
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On Friday 14 October 2011, Scot Kreienkamp wrote:
Yeah, when you wrote that solution like that I was afraid that's what you'd say. :) What you can do is create a subdirectory 'overview1' in the data/rrd directory. In that directory you can then create symlinks for the data you want to graph.
I use this to graph the memory usage of AIX lpars running on the same hardware box. So I create a subdirectory 'data/rrd/pSerie1' and in the subdirectory I create symlinks for each server running on that pSerie: vmstat-server1.rrd -> ../server1/vmstat.rrd vmstat-server2.rrd -> ../server2/vmstat.rrd vmstat-server3.rrd -> ../server3/vmstat.rrd
I then added a section to hobbit-graph.cfg to process all 'vmstat-(.*).rrd' files and stack the data on top of each other:
[vmstat-avm-total] FNPATTERN vmstat-(.+).rrd TITLE Accessed Virtual Memory (Stacked) YAXIS Size kB -b 1024 DEF:avmt at RRDIDX@=@RRDFN@:mem_avm:AVERAGE CDEF:avm at RRDIDX@=avmt at RRDIDX@,4048,* LINE1:avm at RRDIDX@#@COLOR@:@RRDPARAM@:STACK GPRINT:avm at RRDIDX@:LAST:AVM \: %5.1lf%s (cur) GPRINT:avm at RRDIDX@:MAX: \: %5.1lf%s (max) GPRINT:avm at RRDIDX@:MIN: \: %5.1lf%s (min) GPRINT:avm at RRDIDX@:AVERAGE: \: %5.1lf%s (avg)\n
The result is a stacked graph with 1 line per server.
For the graph, the url is something like hobbitgraph.sh?host=pSerie1&service=vmstat-avm-total&action=menu
I hope this info can help you.
Stef
My current firewall only has 32 bit counters and they reset a few times per day, which is what got me into this problem. I ended up writing a quick script that finds the RRD's I want to combine, grabs the last value from each of them, then sums them up. I then set that as a target in my MRTG definition with the gauge option. I take advantage of the logic built into MRTG and RRDTool so I don't have to worry about counter resets that way. Looks much better now.... I actually have lines and my trends graph doesn't look like it was colored by my 3 year old. It also works with my current graphs configuration.
Scot Kreienkamp skreien at la-z-boy.com
-----Original Message----- From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Stef Coene Sent: Friday, October 14, 2011 1:01 PM To: xymon at xymon.com Subject: Re: [Xymon] monitoring aggregated network traffic
On Friday 14 October 2011, Scot Kreienkamp wrote:
Yeah, when you wrote that solution like that I was afraid that's what you'd say. :) What you can do is create a subdirectory 'overview1' in the data/rrd directory. In that directory you can then create symlinks for the data you want to graph.
I use this to graph the memory usage of AIX lpars running on the same hardware box. So I create a subdirectory 'data/rrd/pSerie1' and in the subdirectory I create symlinks for each server running on that pSerie: vmstat-server1.rrd -> ../server1/vmstat.rrd vmstat-server2.rrd -> ../server2/vmstat.rrd vmstat-server3.rrd -> ../server3/vmstat.rrd
I then added a section to hobbit-graph.cfg to process all 'vmstat-(.*).rrd' files and stack the data on top of each other:
[vmstat-avm-total] FNPATTERN vmstat-(.+).rrd TITLE Accessed Virtual Memory (Stacked) YAXIS Size kB -b 1024 DEF:avmt at RRDIDX@=@RRDFN@:mem_avm:AVERAGE CDEF:avm at RRDIDX@=avmt at RRDIDX@,4048,* LINE1:avm at RRDIDX@#@COLOR@:@RRDPARAM@:STACK GPRINT:avm at RRDIDX@:LAST:AVM \: %5.1lf%s (cur) GPRINT:avm at RRDIDX@:MAX: \: %5.1lf%s (max) GPRINT:avm at RRDIDX@:MIN: \: %5.1lf%s (min) GPRINT:avm at RRDIDX@:AVERAGE: \: %5.1lf%s (avg)\n
The result is a stacked graph with 1 line per server.
For the graph, the url is something like hobbitgraph.sh?host=pSerie1&service=vmstat-avm-total&action=menu
I hope this info can help you.
Stef
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
participants (4)
-
henrik＠hswn.dk
-
s.seitz＠netz-haut.de
-
SKreien＠la-z-boy.com
-
stef.coene＠docum.org