All,
Have been taking a look at my hobbit server for the past week or so. Wanted to ask a couple things.
The server is a DL360 G3 with two 3ghz processors and 2gb of RAM. It does only hobbit and cricket (10 or so routers, so not a huge hit)
I've noticed the following two things:
#1. I am constantly using 100% of my physical ram (no swap really)
Mem: 2075040k total, 2051928k used, 23112k free, 52172k buffers Swap: 0k total, 0k used, 0k free, 1744168k cached
is this normal? Not sure how to tell which process specifically is eating up the most RAM.
#2. I am consistantly running with 2-300 ports in TIME_WAIT involving port 1984
netstat -an | grep TIME | grep 1984 | wc -l
253
Im not sure if this is normal either, nor am I sure if it is bad or not? Here is the output from some of the "bb" buttons at the top of my hobbit page. If anything stands out, any help would be apprieciated. If I can provide any other info, let me know.
Thanks, Jeff
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7a Feb 19 2003 LDAP library: OpenLDAP 20213
Statistics: Hosts total : 137 Hosts with no tests : 0 Total test count : 141 Status messages : 141 Alert status msgs : 0 Transmissions : 4
DNS statistics:
hostnames resolved : 138
succesful : 134
failed : 4
calls to dnsresolve : 141
TCP test statistics:
TCP tests total : 4
HTTP tests : 2
Simple TCP tests : 2
Connection attempts : 4
bytes written : 272
bytes read : 6108
TIME SPENT Event Starttime Duration bbtest-net startup 1188421844.836442 - Service definitions loaded 1188421844.837799 0.001357 Tests loaded 1188421844.860329 0.022530 DNS lookups completed 1188421854.099983 9.239654 Test engine setup completed 1188421854.101787 0.001804 TCP tests completed 1188421854.249955 0.148168 PING test completed (137 hosts) 1188421867.657468 13.407513 PING test results sent 1188421867.659863 0.002395 Test result collection completed 1188421867.659878 0.000015 LDAP test engine setup completed 1188421867.659880 0.000002 LDAP tests executed 1188421867.659882 0.000002 LDAP tests result collection completed 1188421867.659883 0.000001 Test results transmitted 1188421867.660531 0.000648 bbtest-net completed 1188421867.662248 0.001717 TIME TOTAL 22.825806
bbgen for Hobbit version 4.2.0
Statistics: Hosts : 243 Status messages : 1255 Purple messages : 0 Pages : 22
TIME SPENT Event Starttime Duration Startup 1188421967.748593 - Load links done 1188421967.748887 0.000294 Load bbhosts done 1188421967.752720 0.003833 ACK removal done 1188421967.752787 0.000067 Load STATE done 1188421967.772438 0.019651 Color calculation done 1188421967.772690 0.000252 Hobbit pagegen start 1188421967.772717 0.000027 Hobbit pagegen done 1188421967.794090 0.021373 BB2 generation done 1188421967.800862 0.006772 BBNK generation done 1188421967.801386 0.000524 Summary transmission done 1188421967.801388 0.000002 Run completed 1188421967.801389 0.000001 TIME TOTAL 0.052796
Statistics for Hobbit daemon Up since 29-Aug-2007 13:49:58 (0 days, 02:20:00)
Incoming messages : 152683
- status : 97745
- combo : 16461
- page : 10
- summary : 0
- data : 22090
- client : 16205
- notes : 0
- enable : 0
- disable : 0
- ack : 0
- config : 0
- query : 0
- hobbitdboard : 140
- hobbitdlog : 4
- drop : 0
- rename : 0
- dummy : 28
- ping : 0
- notify : 0
- schedule : 0
- download : 0
- Bogus/Timeouts : 0 Incoming messages/sec : 17 (average last 300 seconds)
status channel messages: 97839 (1 readers) stachg channel messages: 1113 (1 readers) page channel messages: 293 (1 readers) data channel messages: 22090 (1 readers) notes channel messages: 0 (0 readers) enadis channel messages: 0 (0 readers) client channel messages: 16195 (1 readers) clichg channel messages: 0 (1 readers)
Linux tends to gobble up whatever memory thats not being used for buffers and cache, so on Linux boxes using nearly 100% of memory is normal. Use the "actual" reading to get the memory that is being used not including caching and buffers.
is normal, Hobbit is very network intensive.
On the bb-test results, it looks like your DNS lookups are a little slow and your ping tests are taking a little longer than I would consider normal. I am running Hobbit on similar hardware as yours, monitoring over 600 hosts and have similar times for the DNS and ping.
Thanks, Larry Barber
On 8/29/07, Jeff Newman <jeffnewman75 at gmail.com> wrote:
All,
Have been taking a look at my hobbit server for the past week or so. Wanted to ask a couple things.
The server is a DL360 G3 with two 3ghz processors and 2gb of RAM. It does only hobbit and cricket (10 or so routers, so not a huge hit)
I've noticed the following two things:
#1. I am constantly using 100% of my physical ram (no swap really)
Mem: 2075040k total, 2051928k used, 23112k free, 52172k buffers Swap: 0k total, 0k used, 0k free, 1744168k cached
is this normal? Not sure how to tell which process specifically is eating up the most RAM.
#2. I am consistantly running with 2-300 ports in TIME_WAIT involving port 1984
netstat -an | grep TIME | grep 1984 | wc -l
253
Im not sure if this is normal either, nor am I sure if it is bad or not? Here is the output from some of the "bb" buttons at the top of my hobbit page. If anything stands out, any help would be apprieciated. If I can provide any other info, let me know.
Thanks, Jeff
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7a Feb 19 2003 LDAP library: OpenLDAP 20213
Statistics: Hosts total : 137 Hosts with no tests : 0 Total test count : 141 Status messages : 141 Alert status msgs : 0 Transmissions : 4
DNS statistics:
hostnames resolved : 138
succesful : 134
failed : 4
calls to dnsresolve : 141
TCP test statistics:
TCP tests total : 4
HTTP tests : 2
Simple TCP tests : 2
Connection attempts : 4
bytes written : 272
bytes read : 6108
TIME SPENT
Event Starttime Duration bbtest-net startup 1188421844.836442
Service definitions loaded 1188421844.837799 0.001357 Tests loaded 1188421844.860329 0.022530 DNS lookups completed 1188421854.099983 9.239654 Test engine setup completed 1188421854.101787 0.001804 TCP tests completed 1188421854.249955 0.148168 PING test completed (137 hosts) 1188421867.657468 13.407513 PING test results sent 1188421867.659863 0.002395 Test result collection completed 1188421867.659878 0.000015 LDAP test engine setup completed 1188421867.659880 0.000002 LDAP tests executed 1188421867.659882 0.000002 LDAP tests result collection completed 1188421867.659883 0.000001 Test results transmitted 1188421867.660531 0.000648 bbtest-net completed 1188421867.662248 0.001717 TIME TOTAL 22.825806
bbgen for Hobbit version 4.2.0
Statistics: Hosts : 243 Status messages : 1255 Purple messages : 0 Pages : 22
TIME SPENT
Event Starttime Duration Startup 1188421967.748593
Load links done 1188421967.748887 0.000294 Load bbhosts done 1188421967.752720 0.003833 ACK removal done 1188421967.752787 0.000067 Load STATE done 1188421967.772438 0.019651 Color calculation done 1188421967.772690 0.000252 Hobbit pagegen start 1188421967.772717 0.000027 Hobbit pagegen done 1188421967.794090 0.021373 BB2 generation done 1188421967.800862 0.006772 BBNK generation done 1188421967.801386 0.000524 Summary transmission done 1188421967.801388 0.000002 Run completed 1188421967.801389 0.000001 TIME TOTAL 0.052796
Statistics for Hobbit daemon Up since 29-Aug-2007 13:49:58 (0 days, 02:20:00)
Incoming messages : 152683
- status : 97745
- combo : 16461
- page : 10
- summary : 0
- data : 22090
- client : 16205
- notes : 0
- enable : 0
- disable : 0
- ack : 0
- config : 0
- query : 0
- hobbitdboard : 140
- hobbitdlog : 4
- drop : 0
- rename : 0
- dummy : 28
- ping : 0
- notify : 0
- schedule : 0
- download : 0
- Bogus/Timeouts : 0 Incoming messages/sec : 17 (average last 300 seconds)
status channel messages: 97839 (1 readers) stachg channel messages: 1113 (1 readers) page channel messages: 293 (1 readers) data channel messages: 22090 (1 readers) notes channel messages: 0 (0 readers) enadis channel messages: 0 (0 readers) client channel messages: 16195 (1 readers) clichg channel messages: 0 (1 readers)
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On Wed, Aug 29, 2007 at 04:14:58PM -0500, Jeff Newman wrote:
The server is a DL360 G3 with two 3ghz processors and 2gb of RAM. It does only hobbit and cricket (10 or so routers, so not a huge hit)
I've noticed the following two things:
#1. I am constantly using 100% of my physical ram (no swap really)
Mem: 2075040k total, 2051928k used, 23112k free, 52172k buffers Swap: 0k total, 0k used, 0k free, 1744168k cached
is this normal? Not sure how to tell which process specifically is eating up the most RAM.
I think you're running Linux, right?
The "1744168k cached" normally goes in the "Mem:" line, not the "Swap:" line. And that's the clue to what's using your RAM: Linux uses available RAM as a variable-size disk cache; the disk cache grows until just about all of RAM is used. It usually makes sense, the cached data is a copy of data which is stored on disk, so when an application needs more memory, the diskcache memory can be freed instantly and allocated to the application.
Therefore, Linux boxes usually have very little "free" memory.
The "top" utility can sort processes by memory-usage. Also, "ps -vax" will tell you how much memory each process is using.
#2. I am consistantly running with 2-300 ports in TIME_WAIT involving port 1984
netstat -an | grep TIME | grep 1984 | wc -l
253
Im not sure if this is normal either, nor am I sure if it is bad or not?
It's not unusual, and quite harmless. The TIME_WAIT state happens when a socket is closed; the operating system keeps the socket around for some time (usually 20-30 secs, cannot remember what the Linux default is) to make sure that all packets destined for this socket have been received (there might be some duplicated/retransmitted packets still in transit when the socket is closed). This makes sure that a new connection using the same port will not see packets from the old connection.
Here is the output from some of the "bb" buttons at the top of my hobbit page. If anything stands out, any help would be apprieciated.
DNS statistics:
hostnames resolved : 138
succesful : 134
failed : 4
DNS lookups completed 1188421854.099983 9.239654
Your DNS lookups are a bit slow - 9 seconds for 138 DNS lookups. Nothing critical, just a bit slower than I'd expect. Installing a local caching DNS daemon is a good way to eliminate this problem.
Regards, Henrik
Two questions:
In one of the discussions I saw references to a HTML wrapper for email alert notifications. Does anyone have that available to share and also can you provide the install instructions?
How can Hobbit email alerts be customized? Is there a way to add some generic instructional text to the alert? All of ours go via email so there are no SMS type character limitations to be concerned with.
Please see attached script
It is a little bit modified script from Robert Taylor ( http://www.hswn.dk/hobbiton/2006/03/msg00067.html )
To install - just copy it to hobbit server ext folder (/home/hobbit/server/ext), do chown/chmod and add record like:
HOST=* COLOR=red SCRIPT /home/hobbit/server/ext/html-email-notification.sh e-mail at address.com REPEAT=2h RECOVERED
To the hobbit-alerts.cfg file to activate the notification
- sure - It can be customized
Alexander Levin
From: Meserole, Keith W. [mailto:km at keithm.us] Sent: Thursday, August 30, 2007 4:57 AM To: hobbit at hswn.dk; hobbit at hswn.dk Cc: Meserole, Keith W. Subject: [hobbit] Hobbit email alerts
Two questions: 1. In one of the discussions I saw references to a HTML wrapper for email alert notifications. Does anyone have that available to share and also can you provide the install instructions? 2. How can Hobbit email alerts be customized? Is there a way to add some generic instructional text to the alert? All of ours go via email so there are no SMS type character limitations to be concerned with.
Hi all,
In Hobbit 4.20 is there a way to set up a test from an internal network to test http of an outside host, such as a DSL router? Cavaet: A proxy server is required to get out and also the DSL is a dynamic IP. This is on Solaris10.
I currently have a line similar to this in the bbhost file. NSLOOKUP to the real host name works but ping fails (of course) since there is no host file entry.
0.0.0.0 DSL1 # noconn https://dsl1.dyndns.biz:8080
Can anyone provide a detailed example of how to accomplish this?
Thanks for the feedback everyone. I'll look into the DNS slowness, good to know about the memory/TIME_WAIT's, feel a bit better now :-)
-Jeff
On 8/29/07, Henrik Stoerner <henrik at hswn.dk> wrote:
On Wed, Aug 29, 2007 at 04:14:58PM -0500, Jeff Newman wrote:
The server is a DL360 G3 with two 3ghz processors and 2gb of RAM. It does only hobbit and cricket (10 or so routers, so not a huge hit)
I've noticed the following two things:
#1. I am constantly using 100% of my physical ram (no swap really)
Mem: 2075040k total, 2051928k used, 23112k free, 52172k buffers Swap: 0k total, 0k used, 0k free, 1744168k cached
is this normal? Not sure how to tell which process specifically is eating up the most RAM.
I think you're running Linux, right?
The "1744168k cached" normally goes in the "Mem:" line, not the "Swap:" line. And that's the clue to what's using your RAM: Linux uses available RAM as a variable-size disk cache; the disk cache grows until just about all of RAM is used. It usually makes sense, the cached data is a copy of data which is stored on disk, so when an application needs more memory, the diskcache memory can be freed instantly and allocated to the application.
Therefore, Linux boxes usually have very little "free" memory.
The "top" utility can sort processes by memory-usage. Also, "ps -vax" will tell you how much memory each process is using.
#2. I am consistantly running with 2-300 ports in TIME_WAIT involving port 1984
netstat -an | grep TIME | grep 1984 | wc -l
253
Im not sure if this is normal either, nor am I sure if it is bad or not?
It's not unusual, and quite harmless. The TIME_WAIT state happens when a socket is closed; the operating system keeps the socket around for some time (usually 20-30 secs, cannot remember what the Linux default is) to make sure that all packets destined for this socket have been received (there might be some duplicated/retransmitted packets still in transit when the socket is closed). This makes sure that a new connection using the same port will not see packets from the old connection.
Here is the output from some of the "bb" buttons at the top of my hobbit page. If anything stands out, any help would be apprieciated.
DNS statistics:
hostnames resolved : 138
succesful : 134
failed : 4
DNS lookups completed 1188421854.099983 9.239654
Your DNS lookups are a bit slow - 9 seconds for 138 DNS lookups. Nothing critical, just a bit slower than I'd expect. Installing a local caching DNS daemon is a good way to eliminate this problem.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Linux, right? I *think* some Linux distributions glom on to any available RAM for a disk cache, and then they give it up when the memory is needed by something else. I hope I am not getting Linux confused with Windows -- that would really start the flames!
As for the time_wait -- I have no idea. Some systems (Solaris) have a long time out when ports are released by default, and this can cause connections to hang around for a long time after the remote system has closed them. But this is just a shot in the dark.
GLH
-----Original Message----- From: Jeff Newman [mailto:jeffnewman75 at gmail.com] Sent: Wednesday, August 29, 2007 4:15 PM To: hobbit at hswn.dk Subject: [hobbit] Hobbit Server performance
All,
Have been taking a look at my hobbit server for the past week or so. Wanted to ask a couple things.
The server is a DL360 G3 with two 3ghz processors and 2gb of RAM. It does only hobbit and cricket (10 or so routers, so not a huge hit)
I've noticed the following two things:
#1. I am constantly using 100% of my physical ram (no swap really)
Mem: 2075040k total, 2051928k used, 23112k free, 52172k buffers Swap: 0k total, 0k used, 0k free, 1744168k cached
is this normal? Not sure how to tell which process specifically is eating up the most RAM.
#2. I am consistantly running with 2-300 ports in TIME_WAIT involving port 1984
netstat -an | grep TIME | grep 1984 | wc -l
253
Im not sure if this is normal either, nor am I sure if it is bad or not? Here is the output from some of the "bb" buttons at the top of my hobbit page. If anything stands out, any help would be apprieciated. If I can provide any other info, let me know.
Thanks, Jeff
bbtest-net version 4.2.0 SSL library : OpenSSL 0.9.7a Feb 19 2003 LDAP library: OpenLDAP 20213
Statistics: Hosts total : 137 Hosts with no tests : 0 Total test count : 141 Status messages : 141 Alert status msgs : 0 Transmissions : 4
DNS statistics:
hostnames resolved : 138
succesful : 134
failed : 4
calls to dnsresolve : 141
TCP test statistics:
TCP tests total : 4
HTTP tests : 2
Simple TCP tests : 2
Connection attempts : 4
bytes written : 272
bytes read : 6108
TIME SPENT Event Starttime Duration bbtest-net startup 1188421844.836442
Service definitions loaded 1188421844.837799 0.001357 Tests loaded 1188421844.860329 0.022530 DNS lookups completed 1188421854.099983 9.239654 Test engine setup completed 1188421854.101787 0.001804 TCP tests completed 1188421854.249955 0.148168 PING test completed (137 hosts) 1188421867.657468 13.407513 PING test results sent 1188421867.659863 0.002395 Test result collection completed 1188421867.659878 0.000015 LDAP test engine setup completed 1188421867.659880 0.000002 LDAP tests executed 1188421867.659882 0.000002 LDAP tests result collection completed 1188421867.659883 0.000001 Test results transmitted 1188421867.660531 0.000648 bbtest-net completed 1188421867.662248 0.001717 TIME TOTAL 22.825806
bbgen for Hobbit version 4.2.0
Statistics: Hosts : 243 Status messages : 1255 Purple messages : 0 Pages : 22
TIME SPENT Event Starttime Duration Startup 1188421967.748593
Load links done 1188421967.748887 0.000294 Load bbhosts done 1188421967.752720 0.003833 ACK removal done 1188421967.752787 0.000067 Load STATE done 1188421967.772438 0.019651 Color calculation done 1188421967.772690 0.000252 Hobbit pagegen start 1188421967.772717 0.000027 Hobbit pagegen done 1188421967.794090 0.021373 BB2 generation done 1188421967.800862 0.006772 BBNK generation done 1188421967.801386 0.000524 Summary transmission done 1188421967.801388 0.000002 Run completed 1188421967.801389 0.000001 TIME TOTAL 0.052796
Statistics for Hobbit daemon Up since 29-Aug-2007 13:49:58 (0 days, 02:20:00)
Incoming messages : 152683
- status : 97745
- combo : 16461
- page : 10
- summary : 0
- data : 22090
- client : 16205
- notes : 0
- enable : 0
- disable : 0
- ack : 0
- config : 0
- query : 0
- hobbitdboard : 140
- hobbitdlog : 4
- drop : 0
- rename : 0
- dummy : 28
- ping : 0
- notify : 0
- schedule : 0
- download : 0
- Bogus/Timeouts : 0 Incoming messages/sec : 17 (average last 300 seconds)
status channel messages: 97839 (1 readers) stachg channel messages: 1113 (1 readers) page channel messages: 293 (1 readers) data channel messages: 22090 (1 readers) notes channel messages: 0 (0 readers) enadis channel messages: 0 (0 readers) client channel messages: 16195 (1 readers) clichg channel messages: 0 (1 readers)
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
participants (6)
-
alevin＠reveredata.com
-
greg.hubbard＠eds.com
-
henrik＠hswn.dk
-
jeffnewman75＠gmail.com
-
km＠keithm.us
-
lebarber＠gmail.com