Problems with writing client data while running several clients
Hi,
we are running 1 to 3 clients on each host, one for $hostname and one for each virtual host defined on the machine in order to get client data like disk, cpu, etc. for $hostname and for each virtual host. Each client is started using $HOBBITCLIENTHOME/bin/hobbitlaunch --config=$HOBBITCLIENTHOME/etc/clientlaunch.${MACHINE}.cfg --log=$HOBBITCLIENTHOME/logs/clientlaunch.${MACHINE}.log --pidfile=$HOBBITCLIENTHOME/logs/clientlaunch.${MACHINE}.pid
Now I have the problem that client data of one or both clients is neglected by the server. In rrd-data.log it shows: 2006-01-18 13:03:40 RRD error updating /pica/ffm/com/sys/hobbit/hobbit/data/rrd/lbscgi2/netstat.rrd from 141.2.164.215: illegal attempt to update using time 1137585820 when last update time is 1137585820 (minimum one second step) In rrd-status.log are similar entries like 2006-01-18 12:48:38 RRD error updating /pica/ffm/com/sys/hobbit/hobbit/data/rrd/lbscgi2/memory.real.rrd from 141.2.164.215: illegal attempt to update using time 1137584918 when last update time is 1137584918 (minimum one second step)
The client data (disk, etc) for both becomes purple in bb.html. Date for both clients is stored (sometimes) correctly: hobbit/hobbit/data/rrd/otter -rw-r--r-- 1 bb prod 19576 Jan 18 12:43 memory.real.rrd hobbit/hobbit/data/rrd/lbscgi2 -rw-r--r-- 1 bb prod 19576 Jan 18 12:58 memory.real.rrd
I tried starting the clients with some seconds difference, that helps for a certain time and then the problem reappears. What can I do?
kind regards Rolf
-- Mit freundlichen Gruessen Rolf Schrittenlocher
HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 798 2881 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at rz.uni-frankfurt.de
Hi Rolf,
have you looked into your hobbit-log files? New with hobbit i had the same problem and found out that this happens when to identical (in my case conn-tests) run on the server and collect the information for the same machine:
page core-router 1.2.3.4 router1 # [..] page locationA 1.2.3.4 router1 # [..] Maybe you find identical lines testing the same in your bb-hosts.
Are the service you are monitoring for the virtual hosts equaly named?
Best Mathias
In message <43CE350C.7020404 at rz.uni-frankfurt.de>, Rolf Schrittenlocher writes:
Hi,
we are running 1 to 3 clients on each host, one for $hostname and one for each virtual host defined on the machine in order to get client data like disk, cpu, etc. for $hostname and for each virtual host. Each client is started using $HOBBITCLIENTHOME/bin/hobbitlaunch --config=$HOBBITCLIENTHOME/etc/clientlaunch.${MACHINE}.cfg --log=$HOBBITCLIENTHOME/logs/clientlaunch.${MACHINE}.log --pidfile=$HOBBITCLIENTHOME/logs/clientlaunch.${MACHINE}.pid
Now I have the problem that client data of one or both clients is neglected by the server. In rrd-data.log it shows: 2006-01-18 13:03:40 RRD error updating /pica/ffm/com/sys/hobbit/hobbit/data/rrd/lbscgi2/netstat.rrd from 141.2.164.215: illegal attempt to update using time 1137585820 when last update time is 1137585820 (minimum one second step) In rrd-status.log are similar entries like 2006-01-18 12:48:38 RRD error updating /pica/ffm/com/sys/hobbit/hobbit/data/rrd/lbscgi2/memory.real.rrd from 141.2.164.215: illegal attempt to update using time 1137584918 when last update time is 1137584918 (minimum one second step)
The client data (disk, etc) for both becomes purple in bb.html. Date for both clients is stored (sometimes) correctly: hobbit/hobbit/data/rrd/otter -rw-r--r-- 1 bb prod 19576 Jan 18 12:43 memory.real.rrd hobbit/hobbit/data/rrd/lbscgi2 -rw-r--r-- 1 bb prod 19576 Jan 18 12:58 memory.real.rrd
I tried starting the clients with some seconds difference, that helps for a certain time and then the problem reappears. What can I do?
kind regards Rolf
-- Mit freundlichen Gruessen Rolf Schrittenlocher
HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 798 2881 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at rz.uni-frankfurt.de
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Hi Mathias,
there are no identical entries in bb-hosts. I'm almost sure that the client info send is using the correct names, once for the hostname and once for each alias, because there is the accurate data in hobbit/hobbit/data for both names before and after the period the log indicates failure. So it is a temporary problem (it occurred for about half an hour, when everything went green again). I have the impression that if two hosts are sending data at almost the same time hobbit doesn't like that. If they do so for a period>status bbdisplay goes purple. Once the point of time those two are sending differs for at least one second (if my interpretation of the message in the log is right) data is stored and everything goes back green. My proposition is to make the server wait for 1 second before writing if he can't instead of writing the information to a log but perhaps this is infeasable.
kind regards Rolf
Hi Rolf,
have you looked into your hobbit-log files? New with hobbit i had the same problem and found out that this happens when to identical (in my case conn-tests) run on the server and collect the information for the same machine:
page core-router 1.2.3.4 router1 # [..] page locationA 1.2.3.4 router1 # [..] Maybe you find identical lines testing the same in your bb-hosts.
Are the service you are monitoring for the virtual hosts equaly named?
Best Mathias
In message <43CE350C.7020404 at rz.uni-frankfurt.de>, Rolf Schrittenlocher writes:
Hi,
we are running 1 to 3 clients on each host, one for $hostname and one for each virtual host defined on the machine in order to get client data like disk, cpu, etc. for $hostname and for each virtual host. Each client is started using $HOBBITCLIENTHOME/bin/hobbitlaunch --config=$HOBBITCLIENTHOME/etc/clientlaunch.${MACHINE}.cfg --log=$HOBBITCLIENTHOME/logs/clientlaunch.${MACHINE}.log --pidfile=$HOBBITCLIENTHOME/logs/clientlaunch.${MACHINE}.pid
Now I have the problem that client data of one or both clients is neglected by the server. In rrd-data.log it shows: 2006-01-18 13:03:40 RRD error updating /pica/ffm/com/sys/hobbit/hobbit/data/rrd/lbscgi2/netstat.rrd from 141.2.164.215: illegal attempt to update using time 1137585820 when last update time is 1137585820 (minimum one second step) In rrd-status.log are similar entries like 2006-01-18 12:48:38 RRD error updating /pica/ffm/com/sys/hobbit/hobbit/data/rrd/lbscgi2/memory.real.rrd from 141.2.164.215: illegal attempt to update using time 1137584918 when last update time is 1137584918 (minimum one second step)
The client data (disk, etc) for both becomes purple in bb.html. Date for both clients is stored (sometimes) correctly: hobbit/hobbit/data/rrd/otter -rw-r--r-- 1 bb prod 19576 Jan 18 12:43 memory.real.rrd hobbit/hobbit/data/rrd/lbscgi2 -rw-r--r-- 1 bb prod 19576 Jan 18 12:58 memory.real.rrd
I tried starting the clients with some seconds difference, that helps for a certain time and then the problem reappears. What can I do?
kind regards Rolf
-- Mit freundlichen Gruessen Rolf Schrittenlocher
HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 798 2881 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at rz.uni-frankfurt.de
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Mit freundlichen Gruessen Rolf Schrittenlocher
HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 798 2881 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at rz.uni-frankfurt.de
participants (2)
-
brandste＠hebis.de
-
Schrittenlocher＠rz.uni-frankfurt.de