On Fri, Oct 13, 2006 at 04:09:59PM -0400, Epp, Matthew Contractor PEO EIS AKO wrote:

Searched the archives and found other mentions of this error but no solution. I have a Solaris 8 server with the Hobbit client reporting to the Hobbit server, and everything works except the netstat trend graph. I made sure Hobbit generates the netstat.rrd file itself, and verified that it has the correct fields:

$ rrdtool dump netstat.rrd |grep "<name>"

            <name> udpInDatagrams </name>
            <name> udpOutDatagrams </name>
            <name> udpInErrors </name>
            <name> tcpActiveOpens </name>
            <name> tcpPassiveOpens </name>
            <name> tcpAttemptFails </name>
            <name> tcpEstabResets </name>
            <name> tcpCurrEstab </name>
            <name> tcpOutDataBytes </name>
            <name> tcpInInorderBytes </name>
            <name> tcpInUnorderBytes </name>
            <name> tcpRetransBytes </name>
            <name> tcpOutDataPackets </name>
            <name> tcpInInorderPackets </name>
            <name> tcpInUnorderPackets </name>
            <name> tcpRetransPackets </name>

But viewing a debug output of rrd-data.log shows that something is making it think there should only be 11 fields:

2006-10-13 19:18:22 RRD update param 03: 'udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActiveOpens:tcpPassiveOpens:t cpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataBytes:tcpInInorderBytes :tcpInUnorderBytes'

11 items here, yes.

2006-10-13 19:18:22 RRD update param 04: '1160767102:1580411:5594723:0:113952:19256340:17330:36082:7:653169856:440119 035:9398449:159635980:26380336:13186666:30559:612744'

And 16 values there. Most odd.

Where is "param 03" coming from? From some configuration on the client side?

Param 03 is the "-t" option, which tells rrd_update() that param 04 defines the sequence of data set values.

Hmm ... could you try attaching gdb to the running hobbitd_rrd process. There are two of them - one is handling "status" messages, the other "data" messages. You'll need to find the process-ID of the one handling "data" messages. Do something like

$ ps -u hobbit -f|egrep "hobbitd_rrd|hobbitd_channel"

and look for the lines with

hobbit 25196 25150 0 Oct04 ? 00:00:02 hobbitd_channel --channel=data --log=/var/log/hobbit/rrd-data.log hobbitd_rrd --rrddir=/var/lib/hobbit/rrd hobbit 25219 25196 0 Oct04 ? 00:00:53 hobbitd_rrd --rrddir=/var/lib/hobbit/rrd

where the PID of the hobbitd_channel process (25196 here) matches the parent-PID of the hobbitd_rrd process (25219). The "25219" is what you need. Now run gdb like this (as the hobbit user):

$ gdb ~/server/bin/hobbitd_rrd (gdb) attach 25219 [... lots of gdb printout ...] (gdb) p netstat_tpl

I'm interested in what the netstat_tpl output shows, since that is what is used to build that "param 04" value. Here it looks like

(gdb) p netstat_tpl $1 = 0x8076a68 "udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActiveOpens:tcpPassiveOpens:tcpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataBytes:tcpInInorderBytes:tcpInUnorderBytes:tcpRetransBytes:tcpOutDataPack"... (gdb) p netstat_tpl+200 $2 = 0x8076b30 "ets:tcpInInorderPackets:tcpInUnorderPackets:tcpRetransPackets"

After that, just ctrl-d to exit gdb. If it asks if you want to "detach", answer "y".

Also, could you run this command (as the hobbit user)

$ bbcmd hobbitd_channel --channel=data hobbitd_capture --hosts=localhost

(replace "localhost" with the hostname of your solaris 8 client) ? Let it run for 10 minutes and send me the output.

Thanks, Henrik