funny errors in the logs and weird random http monitoring failures
Hello all, At risk of covering an already well worn topic (I did check the archives first, I swear) I'm getting two strange behaviors that may be related.
I've inherited a Hobbit server that was working fine until the unit was reset (rudely), and since then it's been working but has been a little funky.
In my bb-display.log and bb-network I see: 2005-10-11 16:10:39 Could not connect to bbd at 127.0.0.1:1984 - Connection refused 2005-10-11 16:10:39 Whoops ! bb failed to send message - Connection failed 2005-10-11 16:10:39 hobbitd status-board not available 2005-10-11 16:10:39 Could not connect to bbd at 127.0.0.1:1984 - Connection refused 2005-10-11 16:10:39 Whoops ! bb failed to send message - Connection failed 2005-10-11 16:10:43 Could not connect to bbd at 127.0.0.1:1984 - Connection refused 2005-10-11 16:10:43 Whoops ! bb failed to send message - Connection failed 2005-10-11 16:10:43 hobbitd status-board not available 2005-10-11 16:10:43 Could not connect to bbd at 127.0.0.1:1984 - Connection refused 2005-10-11 16:10:43 Whoops ! bb failed to send message - Connection failed
When I do a "netstat -anp" I do see
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN 6237/hobbitd
So I'm not sure what gives. It also doesn't seem to be a consistent condition.
Also, perhaps related, I have a few servers in my bb-hosts file that are configured to check http which go into alarm for periods for no apparent reason. I'm obviously pretty new to Hobbit, so any pointers would be GREATLY appreciated.
Thanks! -Paul
On Wed, Oct 12, 2005 at 04:56:56PM -0400, Paul Bliss wrote:
At risk of covering an already well worn topic (I did check the archives first, I swear) I'm getting two strange behaviors that may be related.
I've inherited a Hobbit server that was working fine until the unit was reset (rudely), and since then it's been working but has been a little funky.
What version of Hobbit is this ? If you don't known, running "~hobbit/server/bin/bbgen --version" should tell you.
In my bb-display.log and bb-network I see: 2005-10-11 16:10:39 Could not connect to bbd at 127.0.0.1:1984 - Connection refused 2005-10-11 16:10:39 Whoops ! bb failed to send message - Connection failed 2005-10-11 16:10:39 hobbitd status-board not available
Could you check the "hobbitlaunch.log" file in the same directory if there's any mention of the "hobbitd" task terminating with a signal or a non-zero exit code ?
Also, perhaps related, I have a few servers in my bb-hosts file that are configured to check http which go into alarm for periods for no apparent reason.
Hmm - I don't see any relation in this to the other problem you have, but let's handle one problem at a time :-)
Regards, Henrik
Hello,
I've only just started migrating from the BB client to the hobbit client, starting with my hobbit server.
Everything looks fine, except the netstat and vmstat columns are now purple. The odd thing is that the RRD files are still being updated!
Under BB I was running the LARRD external scripts netstat-bf.sh and vmstat-bf.sh to generate the pages. But I see there are sections for netstat and vmstat in the ~/client/bin/hobbitclient-sunos.sh script so I'm not keen to add these back in to the clientlaunch.cfg file.
Also, although my vmstat.rrd file is being generated, the graph is broken. Oddly enough, the graph on the netstat page is generating fine with the current data even though the page status is purple.
If anyone can shed some light on what I may have missed I'd appreciate it!
Thanks and regards, Andy.
#####################################################################################
This email is intended for the person to whom it is addressed only. If you are not the intended recipient, do not read, copy or use the contents in any way. The opinions expressed may not necessarily reflect those of ZESPRI Group of Companies ('ZESPRI').
While every effort has been made to verify the information contained herein, ZESPRI does not make any representations as to the accuracy of the information or to the performance of any data, information or the products mentioned herein. ZESPRI will not accept liability for any losses, damage or consequence, however, resulting directly or indirectly from the use of this e-mail/attachments. #####################################################################################
On Thu, Oct 13, 2005 at 12:30:13PM +1300, Andy France wrote:
I've only just started migrating from the BB client to the hobbit client, starting with my hobbit server.
Everything looks fine, except the netstat and vmstat columns are now purple. The odd thing is that the RRD files are still being updated!
Hobbit doesn't generate a specific column for the netstat and vmstat data - similar to how LARRD worked if you had LARDCOMM=DATA configured in your bf-larrd.sh script. You probably had LARRDCOMM=STATUS ...
Also, although my vmstat.rrd file is being generated, the graph is broken.
It shouldn't break just because of switching from the BB client to the Hobbit client. It WILL break, if you migrate your server from BB to Hobbit - the BB vmstat data was a terrible mess with different data layouts for each operating system, so the format of the vmstat RRD files just isn't compatible between BB and Hobbit.
Any errors in your rrd-data.log (or larrd-data.log) file ?
Regards, Henrik
henrik at hswn.dk wrote on 13/10/2005 06:39:05 p.m.:
On Thu, Oct 13, 2005 at 12:30:13PM +1300, Andy France wrote:
I've only just started migrating from the BB client to the hobbit client, starting with my hobbit server.
Everything looks fine, except the netstat and vmstat columns are now purple. The odd thing is that the RRD files are still being updated!
Hobbit doesn't generate a specific column for the netstat and vmstat data - similar to how LARRD worked if you had LARDCOMM=DATA configured in your bf-larrd.sh script. You probably had LARRDCOMM=STATUS ...
Oh! That's probably it :-) I've dropped the columns and will rely on trends to view the graphs from now on.
Also, although my vmstat.rrd file is being generated, the graph is broken.
It shouldn't break just because of switching from the BB client to the Hobbit client. It WILL break, if you migrate your server from BB to Hobbit - the BB vmstat data was a terrible mess with different data layouts for each operating system, so the format of the vmstat RRD files just isn't compatible between BB and Hobbit.
Any errors in your rrd-data.log (or larrd-data.log) file ?
Only the odd "illegal attempt to update using time" message for various hosts, nothing specific to errors in vmstat.
This server has only ever run hobbit and not BB. I have tried removing the vmstat.rrd file and letting it get re-created but this still results in a broken graph.
I guess I need to brush up on the rrdtool syntax to check if the data file looks as it should?
Regards, Henrik
Thanks, Andy.
#####################################################################################
This email is intended for the person to whom it is addressed only. If you are not the intended recipient, do not read, copy or use the contents in any way. The opinions expressed may not necessarily reflect those of ZESPRI Group of Companies ('ZESPRI').
While every effort has been made to verify the information contained herein, ZESPRI does not make any representations as to the accuracy of the information or to the performance of any data, information or the products mentioned herein. ZESPRI will not accept liability for any losses, damage or consequence, however, resulting directly or indirectly from the use of this e-mail/attachments. #####################################################################################
On Fri, Oct 14, 2005 at 09:47:38AM +1300, Andy France wrote:
Also, although my vmstat.rrd file is being generated, the graph is broken.
It shouldn't break just because of switching from the BB client to the Hobbit client.
This server has only ever run hobbit and not BB. I have tried removing the vmstat.rrd file and letting it get re-created but this still results in a broken graph.
"Broken" - do you mean there is no graph at all, or it just looks wrong ?
Could you send me the vmstat.rrd file that gets created, and a copy of the client data reported by the Hobbit client (you can pick it up from the "Client data" link on the cpu-, disk- or memory-status displays for that host).
Regards, Henrik
Hi,
Andy France schrieb:
Only the odd "illegal attempt to update using time" message for various hosts, nothing specific to errors in vmstat.
That's exactly what I get in my larrd-status.log. It's filling up rapidly but the memory graphs are ok.
2005-10-14 08:26:58 RRD error updating /data1/hobbit/data/rrd/xnode6/memory.actual.rrd from 1x.1x.1x.3x: illegal attempt to update using time 1129271218 when last update time is 1129271218 (minimum one second step)
Regards, Dirk
participants (4)
-
Andy＠zespri.com
-
Dirk.Kastens＠uni-osnabrueck.de
-
henrik＠hswn.dk
-
pbliss＠techvalleycom.com