Problems with hobbitping when the Network is unreachable (BUG)
Hi Henrik,
I rebooted my hobbit server and it lost its default gateway, unbeknown to me. The only symptoms of something wrong was a purple in a network test is was supposed to be doing and a purple in the server's bbtest column. Now, I reckon one or other or both of these should have both gone red and not purple, and the reason that they did not is due to hobbitping going into a spin: what looks like a fairly tight loop. Hobbitping was using quite a lot of CPU (I saw it at well over 30%) and server/tmp/ping-stdout.1407 was empty but server/tmp/ping-stderr.5499 was getting many (about 60) "Failed to send ICMP packet: Network is unreachable" written to it every second. I reckon this message should have been in the report of the connection test it was trying to do (although you could argue that the colour should be clear and not red, since one doesn't want dozens of alarms for a problem that is actually on the network-test server). It took me some time to figure out what was wrong because hobbit didn't report this problem in any of its logs (presumably because hobbitping never returned and so bbtest-net never returned its status) (and hobbitd_client stopped working for some other reason (core dump) yesterday and, although restarted, remains not reporting with nothing in the logs).
Kind regards,
Sebastian
Forgot to mention that I am using the latest version of hobbitping (snapshot from 18-Mar-08).
From: Sebastian [mailto:spa at syntec.co.uk] Sent: 22 May 2008 19:08 To: hobbit at hswn.dk Subject: [hobbit] Problems with hobbitping when the Network is unreachable (BUG)
Hi Henrik,
I rebooted my hobbit server and it lost its default gateway, unbeknown to me. The only symptoms of something wrong was a purple in a network test is was supposed to be doing and a purple in the server's bbtest column. Now, I reckon one or other or both of these should have both gone red and not purple, and the reason that they did not is due to hobbitping going into a spin: what looks like a fairly tight loop. Hobbitping was using quite a lot of CPU (I saw it at well over 30%) and server/tmp/ping-stdout.1407 was empty but server/tmp/ping-stderr.5499 was getting many (about 60) "Failed to send ICMP packet: Network is unreachable" written to it every second. I reckon this message should have been in the report of the connection test it was trying to do (although you could argue that the colour should be clear and not red, since one doesn't want dozens of alarms for a problem that is actually on the network-test server). It took me some time to figure out what was wrong because hobbit didn't report this problem in any of its logs (presumably because hobbitping never returned and so bbtest-net never returned its status) (and hobbitd_client stopped working for some other reason (core dump) yesterday and, although restarted, remains not reporting with nothing in the logs).
Kind regards,
Sebastian
Try fping. I think Henrik has not had time to fix problems with hobbitping. It may be one of those "it seemed a good idea at the time" things...
GLH
From: Sebastian [mailto:spa at syntec.co.uk]
Sent: Thursday, May 22, 2008 1:08 PM
To: hobbit at hswn.dk
Subject: [hobbit] Problems with hobbitping when the Network isunreachable (BUG)
Hi Henrik,
I rebooted my hobbit server and it lost its default gateway,unbeknown to me. The only symptoms of something wrong was a purple in a network test is was supposed to be doing and a purple in the server's bbtest column. Now, I reckon one or other or both of these should have both gone red and not purple, and the reason that they did not is due to hobbitping going into a spin: what looks like a fairly tight loop. Hobbitping was using quite a lot of CPU (I saw it at well over 30%) and server/tmp/ping-stdout.1407 was empty but server/tmp/ping-stderr.5499 was getting many (about 60) "Failed to send ICMP packet: Network is unreachable" written to it every second. I reckon this message should have been in the report of the connection test it was trying to do (although you could argue that the colour should be clear and not red, since one doesn't want dozens of alarms for a problem that is actually on the network-test server). It took me some time to figure out what was wrong because hobbit didn't report this problem in any of its logs (presumably because hobbitping never returned and so bbtest-net never returned its status) (and hobbitd_client stopped working for some other reason (core dump) yesterday and, although restarted, remains not reporting with nothing in the logs).
Kind regards,
Sebastian
On Thu, May 22, 2008 at 07:07:42PM +0100, Sebastian wrote:
I rebooted my hobbit server and it lost its default gateway, unbeknown to me. The only symptoms of something wrong was a purple in a network test is was supposed to be doing and a purple in the server's bbtest column. Now, I reckon one or other or both of these should have both gone red and not purple, and the reason that they did not is due to hobbitping going into a spin: what looks like a fairly tight loop. Hobbitping was using quite a lot of CPU (I saw it at well over 30%) and server/tmp/ping-stdout.1407 was empty but server/tmp/ping-stderr.5499 was getting many (about 60) "Failed to send ICMP packet: Network is unreachable" written to it every second.
Thanks, the attached patch should solve this problem.
Regards, Henrik
Tested and it does solve the problem. (Network test goes red with: Service conn on XXXGW1 is not OK : Host responds to ping on 2 of 7 IP's
...
Traceroute results: socket: Network is unreachable )
Thanks for the quick patch.
Sebastian
-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: 23 May 2008 09:49 To: hobbit at hswn.dk Subject: Re: [hobbit] Problems with hobbitping when the Network isunreachable (BUG)
On Thu, May 22, 2008 at 07:07:42PM +0100, Sebastian wrote:
I rebooted my hobbit server and it lost its default gateway, unbeknown to me. The only symptoms of something wrong was a purple in a network test is was supposed to be doing and a purple in the server's bbtest column. Now, I reckon one or other or both of these should have both gone red and not purple, and the reason that they did not is due to hobbitping going into a spin: what looks like a fairly tight loop. Hobbitping was using quite a lot of CPU (I saw it at well over 30%) and server/tmp/ping-stdout.1407 was empty but server/tmp/ping-stderr.5499 was getting many (about 60) "Failed to send ICMP packet: Network is unreachable" written to it every second.
Thanks, the attached patch should solve this problem.
Regards, Henrik
participants (3)
-
greg.hubbard＠eds.com
-
henrik＠hswn.dk
-
spa＠syntec.co.uk