Damn that ICMP :)
On 12/31/07, Michael A. Price <mprice at sgt-inc.com> wrote:
Nice sleuthing...
It looks like the ball is back in my court. The trace command at the command line, never seems to end. I will do some research..
Thanks, michael
-----Original Message----- From: Josh Luthman [mailto:josh at imaginenetworksllc.com] Sent: Monday, December 31, 2007 10:27 AM To: hobbit at hswn.dk Subject: Re: [hobbit] bbtest - errors
Can you do a trace at the shell?
Josh,
I just figured out it's the #trace option. When I remove that option the errors go away...
Thanks, michael
-----Original Message----- From: Michael A. Price Sent: Monday, December 31, 2007 7:35 AM To: hobbit at hswn.dk Subject: RE: [hobbit] bbtest - errors
Josh,
Thanks for help, AGAIN.... One step closer...
I have one host down, and I have the trace option on all of my hosts listed in bb-hosts. When I comment out that downed host, the errors clear up in bb-test. Take a look...
Mon Dec 31 12:22:16 2007
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7m 23 Feb 2007 LDAP library: OpenLDAP 20213
Statistics: Hosts total : 310 Hosts with no tests : 7 Total test count : 307 Status messages : 308 Alert status msgs : 0 Transmissions : 5
DNS statistics:
hostnames resolved : 303
succesful : 303
failed : 0
calls to dnsresolve : 307
TCP test statistics:
TCP tests total : 2
HTTP tests : 1
Simple TCP tests : 1
Connection attempts : 2
bytes written : 135
bytes read : 553
TIME SPENT Event Starttime Duration bbtest-net startup 1199103736.384784
Service definitions loaded 1199103736.385887 0.001103 Tests loaded 1199103736.768919 0.383032 DNS lookups completed 1199103736.768928 0.000009 Test engine setup completed 1199103736.772261 0.003333 TCP tests completed 1199103736.773300 0.001039 PING test completed (303 hosts) 1199103755.089536 18.316236 PING test results sent 1199103755.091233 0.001697 Test result collection completed 1199103755.091241 0.000008 LDAP test engine setup completed 1199103755.091245 0.000004 LDAP tests executed 1199103755.091249 0.000004 LDAP tests result collection completed 1199103755.091252 0.000003 NSLOOKUP tests executed 1199103755.095923 0.004671 Test results transmitted 1199103755.098103 0.002180 bbtest-net completed 1199103755.099180 0.001077 TIME TOTAL 18.714396
But once I uncomment out the host and the hobbit server tries to do a traceroute to it, the errors come back again. Even if I disable the alerting of that host. Take a look....
Mon Dec 31 12:32:24 2007
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7m 23 Feb 2007 LDAP library: OpenLDAP 20213
Statistics: Hosts total : 311 Hosts with no tests : 7 Total test count : 308 Status messages : 309 Alert status msgs : 0 Transmissions : 5
DNS statistics:
hostnames resolved : 304
succesful : 304
failed : 0
calls to dnsresolve : 308
TCP test statistics:
TCP tests total : 2
HTTP tests : 1
Simple TCP tests : 1
Connection attempts : 2
bytes written : 135
bytes read : 553
Error output: Timeout waiting for data from child, killing it Timeout waiting for data from child, killing it Child process terminated with signal 15
TIME SPENT Event Starttime Duration bbtest-net startup 1199104344.425092
Service definitions loaded 1199104344.426152 0.001060 Tests loaded 1199104344.543955 0.117803 DNS lookups completed 1199104344.543964 0.000009 Test engine setup completed 1199104344.547454 0.003490 TCP tests completed 1199104344.548434 0.000980 PING test completed (304 hosts) 1199104369.082520 24.534086 PING test results sent 1199104399.089988 30.007468 Test result collection completed 1199104399.090003 0.000015 LDAP test engine setup completed 1199104399.090007 0.000004 LDAP tests executed 1199104399.090011 0.000004 LDAP tests result collection completed 1199104399.090015 0.000004 NSLOOKUP tests executed 1199104399.095563 0.005548 Test results transmitted 1199104399.097862 0.002299 bbtest-net completed 1199104399.098975 0.001113 TIME TOTAL 54.673883
Any ideas of why its doing it??? Or how to resolve it???
Thanks, michael
From: Josh Luthman [mailto:josh at imaginenetworksllc.com] Sent: Friday, December 28, 2007 5:30 PM To: hobbit at hswn.dk Subject: Re: [hobbit] bbtest - errors
Try Henrik's fping command at the bottom of this page:
http://www.hswn.dk/hobbiton/2007/11/msg00069.html
and stick a time in front to see how long it takes. On 12/28/07, Michael A. Price <mprice at sgt-inc.com> wrote: Josh,
Thanks for getting back to me so quickly, I updated my /etc/hosts file to have every single one of my monitored hosts, just as a test. I now have 'failed hosts' in my DNS statistic's, but my 'PING test results sent' are still off the charts. I still cant figure out the problem...
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7m 23 Feb 2007 LDAP library: OpenLDAP 20213
Statistics:
Hosts total : 311 Hosts with no tests : 7
Total test count : 308 Status messages : 309
Alert status msgs : 0 Transmissions : 5
DNS statistics:
hostnames resolved : 304
succesful : 304
failed : 0
calls to dnsresolve : 308
TCP test statistics:
TCP tests total : 2
HTTP tests : 1
Simple TCP tests : 1
Connection attempts : 2
bytes written : 135
bytes read : 553
Error output: Timeout waiting for data from child, killing it
Timeout waiting for data from child, killing it Child process terminated with signal 15
TIME SPENT Event Starttime Duration
bbtest-net startup 1198875012.330887
Service definitions loaded 1198875012.331984 0.001097 Tests loaded 1198875012.405015 0.073031 DNS lookups completed 1198875012.405024 0.000009
Test engine setup completed 1198875012.408543 0.003519 TCP tests completed 1198875012.409325 0.000782 PING test completed (304 hosts) 1198875037.083126 24.673801
PING test results sent 1198875067.092719 30.009593 Test result collection completed 1198875067.092733 0.000014 LDAP test engine setup completed 1198875067.092737 0.000004 LDAP tests executed 1198875067.092741 0.000004
LDAP tests result collection completed 1198875067.092745 0.000004 NSLOOKUP tests executed 1198875067.096007 0.003262 Test results transmitted 1198875067.098247 0.002240
bbtest-net completed 1198875067.099155 0.000908 TIME TOTAL 54.768268
Thanks, michael
From: Josh Luthman [mailto:josh at imaginenetworksllc.com] Sent: Thursday, December 27, 2007 11:15 AM
To: hobbit at hswn.dk Subject: Re: [hobbit] bbtest - errors
Michael,
Try adding "testip" after the comment in as many hosts as possible, IE:
10.0.0.250 myftp.server.com # testip
Josh On 12/27/07, Michael A. Price <mprice at sgt-inc.com> wrote: I just modified the /etc/nsswitch.conf file to remove DNS.
I find it interesting that no matter if the hobbit server uses DNS servers or local host files to look up the hosts the 'PING Test Results Sent' number is still off the charts.
Thanks so much for getting back to me
Thanks, michael
From: Josh Luthman [mailto: josh at imaginenetworksllc.com] Sent: Wednesday, December 26, 2007 6:00 PM
To: hobbit at hswn.dk Subject: Re: [hobbit] bbtest - errors
Your calls to dnsresolve went up one, how in the world did you "[update] the hobbit server to not use the DNS servers"?
It looks like it is still doing the exact same stuff concerning DNS to me... On 12/26/07, Michael A. Price <mprice at sgt-inc.com> wrote: Thanks for getting back to me on this.
I updated the hobbit server to not use the DNS servers and all that does is cause it to go from 100 failed hosts to 299 failed hosts.
I think it's the large "PING test results sent" number, what else could be the problem???
Here is another printout...
Thanks, michael
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7m 23 Feb 2007 LDAP library: OpenLDAP 20213
Statistics: Hosts total : 311 Hosts with no tests : 7
Total test count : 308
Status messages : 309
Alert status msgs : 0 Transmissions : 5
DNS statistics:
hostnames resolved : 304
succesful : 203
failed : 101
calls to dnsresolve : 308
TCP test statistics:
TCP tests total : 2
HTTP tests : 1
Simple TCP tests : 1
Connection attempts : 2
bytes written : 135
bytes read : 553
Error output:
Timeout waiting for data from child, killing it
Timeout waiting for data from child, killing it Child process terminated with signal 15
TIME SPENT
Event Starttime Duration
bbtest-net startup
1198691205.281738 - Service definitions loaded 1198691205.282850
0.001112Tests loaded 1198691205.316420 0.033570
DNS lookups completed 1198691215.446830
10.130410 Test engine setup completed
1198691215.450594 0.003764 TCP tests completed 1198691215.451393 0.000799 PING test completed (304 hosts) 1198691240.081987 24.630594
PING test results sent 1198691270.090627 30.008640
Test result collection completed 1198691270.090642
0.000015
LDAP test engine setup completed 1198691270.090656 0.000014
LDAP tests executed 1198691270.090660 0.000004
LDAP tests result collection completed
1198691270.090663 0.000003
NSLOOKUP tests executed 1198691270.146990 0.056327
Test results transmitted 1198691270.149410 0.002420
bbtest-net completed 1198691270.150271 0.000861
TIME TOTAL 64.868533
From: Josh Luthman [mailto: josh at imaginenetworksllc.com] Sent: Thursday, December 20, 2007 11:04 AM
To: hobbit at hswn.dk Subject: Re: [hobbit] bbtest - errors
If that was the only change you made recently try switching the DNS servers back to see if the problem disappears. On 12/20/07, Michael A. Price < mprice at sgt-inc.com> wrote: Thanks...
Actually, I updated my DNS servers and went from 300 failed lookups to
On 12/31/07, Michael A. Price <mprice at sgt-inc.com> wrote: - - - 100.
So I thought I was going to improve....
But it got worse!!!! Any other ideas???
Thanks, michael
From: Josh Luthman [mailto: josh at imaginenetworksllc.com] Sent: Thursday, December 20, 2007 8:10 AM To: hobbit at hswn.dk Subject: Re: [hobbit] bbtest - errors
failed : 100 <--- may be the cause, lots of
failed DNS queries On 12/19/07, Michael A. Price < mprice at sgt-inc.com> wrote:
My bbtest time went from 10 seconds to 89.0 ....
Has anyone seen this before???
Wed Dec 19 19:15:55 2007
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7m 23 Feb 2007 LDAP library: OpenLDAP 20213
Statistics: Hosts total : 310 Hosts with no tests : 7 Total test count : 307 Status messages : 308 Alert status msgs : 0 Transmissions : 5
DNS statistics:
hostnames resolved : 303
succesful : 203
failed : 100
calls to dnsresolve : 307
TCP test statistics:
TCP tests total : 2
HTTP tests : 1
Simple TCP tests : 1
Connection attempts : 2
bytes written : 135
bytes read : 553
Error output: Timeout waiting for data from child, killing it Timeout waiting for data from child, killing it Child process terminated with signal 15 Timeout waiting for data from child, killing it Timeout waiting for data from child, killing it Child process terminated with signal 15
TIME SPENT Event Starttime Duration bbtest-net startup 1198091755.294810
Service definitions loaded 1198091755.297812 0.003002 Tests loaded 1198091755.346908 0.049096 DNS lookups completed 1198091765.439050 10.092142 Test engine setup completed 1198091765.442685 0.003635 TCP tests completed 1198091765.443457 0.000772 PING test completed (303 hosts) 1198091790.084027 24.640570 PING test results sent 1198091850.102236 60.018209 Test result collection completed 1198091850.102455 0.000219 LDAP test engine setup completed 1198091850.102472 0.000017 LDAP tests executed 1198091850.102475 0.000003 LDAP tests result collection completed 1198091850.102482 0.000007 NSLOOKUP tests executed 1198091850.111523 0.009041 Test results transmitted 1198091850.118622 0.007099 bbtest-net completed 1198091850.120484 0.001862 TIME TOTAL 94.825674
Thanks, michael
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer