i enabled all disabled status that are used from graphing (cpu,disk,procs,...) and i have not had a single crash in the last 36 hours (before, crashes would happen at least twice per day) -> from my user point of view, it looks like disabled status can crash hobbitd_rrd olivier On 22/01/2009 21:28, Olivier Beau wrote:
Henrik,
Here are 2 other extracts from crashes :
2009-01-20 16:36:06 hobbitd_rrd: Got message 517875 @@status#517875/sw01.courrierinternational|1232465766.838715|192.168.255.32 ||sw01.courrierinternational|if_load|1232467566|green||green|1225102669|0||0||0|0||network/switch-dedie
2009-01-20 16:36:06 startpos 162634, fillpos 166552, endpos -1 2009-01-20 16:36:06 Want msg 517876, startpos 162634, fillpos 166552, endpos -1, usedbytes=3918, bufleft=361831 2009-01-20 16:36:06 Want msg 517876, startpos 162634, fillpos 170333, endpos -1, usedbytes=7699, bufleft=358050 2009-01-20 16:36:06 hobbitd_rrd: Got message 517876 @@status#517876/sw01.ctoutvert|1232465766.838761|192.168.255.32||sw01.ctout vert|memory|1234247285|blue||blue|1231828085|0||1234247285|Disabled by 2009-01-20 16:36:06 startpos 172884, fillpos 172884, endpos -1 2009-01-20 16:36:06 Peer at 0.0.0.0:0 failed: Broken pipe 2009-01-20 16:36:06 Peer not up, flushing message queue 2009-01-20 16:36:06 Opening file /data/hobbit/server/etc/hobbit-rrddefinitions.cfg 2009-01-20 16:36:06 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=528383 2009-01-20 16:36:06 hobbitd_rrd: Got message 517913 @@status#517913/sw01.excenteurofac|1232465766.929692|192.168.255.32||sw01.e xcenteurofac|if_err|1232467566|green||green|1231866461|0||0||0|0||network/switch-dedie
if_load and if_err are status from devmon, that i do not graph using ncv/extra-test.. memory is also generate from devmon, and is graphes by default in xymon
2009-01-22 17:14:20 hobbitd_rrd: Got message 343666 @@status#343666/logicimmo-netapp2|1232640859.848737|127.0.0.1||logicimmo-ne tapp2|disk|2147483647|blue||blue|1232479545|0||-1|Disabled by 2009-01-22 17:14:20 startpos 417512, fillpos 419047, endpos -1 2009-01-22 17:14:20 Peer at 0.0.0.0:0 failed: Broken pipe 2009-01-22 17:14:20 Peer not up, flushing message queue 2009-01-22 17:14:20 Opening file /data/hobbit/server/etc/hobbit-rrddefinitions.cfg 2009-01-22 17:14:20 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=528383 2009-01-22 17:14:20 hobbitd_rrd: Got message 343677 @@status#343677/tif-netapp1|1232640860.884630|127.0.0.1||tif-netapp1|disk|1 232644460|green||green|1230710616|0||0||0|0|stockage|unix/infrasys/stockage 2009-01-22 17:14:20 startpos 1335, fillpos 3954, endpos 2589
disk is generate by netapp.pl (from the hobbit-client-perl)
-> i noticed that in my 3 extracts, the last log before the crash is disabled. Looks like this could be a problem ? (i've check 2 other crashes, and there again, the last log is a disabled status)
i checked those 3 disabled status : those hosts are up and running (so normal status are sent to hobbitd) we have disabled them for migration purpose, that might happen in a few days, or weeks...
For your mysql question : yes i do graph mysql using NVC NCV_mysql="Questions:DERIVE,Threadsconnected:GAUGE,*:NONE"
Olivier
On 22/01/2009 15:29, Henrik Størner wrote:
In <4974AE8B.80706 at gmail.com> Olivier Beau <obeau79 at gmail.com> writes:
It happened again today at 17:00:22. Nothing new when doing a bt on the coredump. An extract of rrd-status.log from 16h55 to 17h05 is available at http://www.qalpit.com/~olivier/tmp/rrd-status.log.gz
OK, the interesting part is here when it crashes:
2009-01-19 17:00:22 hobbitd_rrd: Got message 181436 @@status#181436/cedratnet-bdd1|1232380822.602633|127. 0.0.1||cedratnet-bdd1|mysql|1232398822|green||green|1231215890|0||0||1232380812|0|linuxmysql|unix/mysql
2009-01-19 17:00:22 startpos 342639, fillpos 378880, endpos 342991 2009-01-19 17:00:22 hobbitd_rrd: Got message 181437 @@status#181437/moniteur-ora2|1232380822.618847|10.12 .0.67||moniteur-ora2|cpu|1255363113|blue||blue|1228751913|0||1255363113|Disabled by 2009-01-19 17:00:22 startpos 342995, fillpos 378880, endpos -1 2009-01-19 17:00:22 Peer at 0.0.0.0:0 failed: Broken pipe 2009-01-19 17:00:22 Peer not up, flushing message queue 2009-01-19 17:00:22 Opening file /data/hobbit/server/etc/hobbit-rrddefinitions.cfg 2009-01-19 17:00:22 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=528383 2009-01-19 17:00:22 hobbitd_rrd: Got message 181450 @@status#181450/nurun-etam-bdd1|1232380822.807004|127 .0.0.1||nurun-etam-bdd1|mysql|1232398822|green||green|1231768476|0||0||1232380582|0|linuxmysql|unix/mysql
2009-01-19 17:00:22 startpos 17100, fillpos 19357, endpos 17846 2009-01-19 17:00:22 Opening file /data/hobbit/server/etc/bb-hosts
It appears to be a "mysql" status from either cedratnet-bdd1 or nurun-etam-bdd1 that causes the crash (I cannot tell exactly, because output buffering comes into play when there's a crash). It *could* also be the cpu-report from moniteur-ora2, but I doubt that - the cpu-status is tested a lot more than the mysql-status.
In fact, "mysql" isn't part of hobbitd_rrd by default. So is this something you've added ? Is it something that you generate graphs for ? Or is it just a status that hobbitd_rrd should ignore ?
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk