-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I have been battling false alerts with hobbit for quite some time (months or more), and am really starting to get quite frustrated. (Mostly in that I tend to ignore my SMS messages because there are so many FP's...
Anyway, the fault is that the hobbit client reports get truncated, yet the hobbit server uses the portion that it gets. This usually results in the procs, ports, or both columns going red due to non-running procs/non-open ports. In reality, the proc/port is fine, just the data was truncated so the hobbit server couldn't find it.
Initially I discovered my hobbit server was truncating some of this data, so I increased the relevant variables: MAXLINE="65535" MAXMSG_STATUS="2048" MAXMSG_CLIENT="2048" MAXMSG_DATA="4096"
However, I still get many red alerts, and when I check, the log files do not report any truncated or oversized messages. Also, when I examine the "Client data available" from the red hobbit report, I find the size of the message is nowhere near any value above, and in fact is always different... Some reports that work are longer than reports that don't work etc...
It isn't 100%, but generally (more than 98%) the clients with the problem are on bandwidth limited networks.
I would appreciate if anyone can provide any tips on how to make things more reliable?
Options I have considered:
- Get hobbit to compress it's data, which reduces network load, and hence should improve reliability.
- Add a "END" tag to the hobbit client data, and if the server doesn't get the END tag then ignore the whole file (or re-request it)
- Switch to polling mode (which effectively does 1 && 2 I suppose)
- Try and track down what is causing this, and fix it...
My hobbit server is behind a NAT router, so one possibility I have considered is the NAT router is dropping the map before the end of the TCP connection due to too many other connections or similar.
Basically, at this point I would like to see if anyone else has had similar problems, or can assist in resolving the problem. If I let it continue I will miss an urgent SMS report from hobbit, and/or cost stacks of money for sending all the SMS's....
Thanks for any suggestions...
Regards, Adam
Adam Goryachev Website Managers www.websitemanagers.com.au -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkkX3rsACgkQGyoxogrTyiX2PwCePnHjMUJHoMRH46zhEFxNFjGe azAAn0rHaDWOWbFfBoIpm+757j8tRWFH =FTnf -----END PGP SIGNATURE-----
On mán, 2008-11-10 at 18:11 +1100, Adam Goryachev wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I have been battling false alerts with hobbit for quite some time (months or more), and am really starting to get quite frustrated. (Mostly in that I tend to ignore my SMS messages because there are so many FP's...
Anyway, the fault is that the hobbit client reports get truncated, yet the hobbit server uses the portion that it gets. This usually results in the procs, ports, or both columns going red due to non-running procs/non-open ports. In reality, the proc/port is fine, just the data was truncated so the hobbit server couldn't find it.
Initially I discovered my hobbit server was truncating some of this data, so I increased the relevant variables: MAXLINE="65535" MAXMSG_STATUS="2048" MAXMSG_CLIENT="2048" MAXMSG_DATA="4096"
Anna added 2008-09-08 because of lots of truncations.
MAXLINE="32768" MAXMSG_STATUS="1024" MAXMSG_DATA="1024" MAXMSG_CLIENT="2048" MAXMSG_NOTES="1024"
After I added this, the problem was solved. I found the sizes from the truncations reported in the logs.
However, I still get many red alerts, and when I check, the log files do not report any truncated or oversized messages. Also, when I examine the "Client data available" from the red hobbit report, I find the size of the message is nowhere near any value above, and in fact is always different... Some reports that work are longer than reports that don't work etc...
Your logs - do they not report truncated or oversized messages like in the following message: http://www.hswn.dk/hobbiton/2006/05/msg00176.html
It isn't 100%, but generally (more than 98%) the clients with the problem are on bandwidth limited networks.
I would appreciate if anyone can provide any tips on how to make things more reliable?
Options I have considered:
- Get hobbit to compress it's data, which reduces network load, and hence should improve reliability.
- Add a "END" tag to the hobbit client data, and if the server doesn't get the END tag then ignore the whole file (or re-request it)
- Switch to polling mode (which effectively does 1 && 2 I suppose)
- Try and track down what is causing this, and fix it...
My hobbit server is behind a NAT router, so one possibility I have considered is the NAT router is dropping the map before the end of the TCP connection due to too many other connections or similar.
Have you considered setting up a Hobbit proxy. See: http://www.hswn.dk/hobbiton/2007/06/msg00080.html
-- Kindest Regards, Anna Jonna Ármannsdóttir, %& A: Because people read from top to bottom. Unix System Aministration, Computing Services, %& Q: Why is top posting bad? University of Iceland.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Anna Jonna Armannsdottir wrote:
On mán, 2008-11-10 at 18:11 +1100, Adam Goryachev wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I have been battling false alerts with hobbit for quite some time (months or more), and am really starting to get quite frustrated. (Mostly in that I tend to ignore my SMS messages because there are so many FP's...
Anyway, the fault is that the hobbit client reports get truncated, yet the hobbit server uses the portion that it gets. This usually results in the procs, ports, or both columns going red due to non-running procs/non-open ports. In reality, the proc/port is fine, just the data was truncated so the hobbit server couldn't find it.
Initially I discovered my hobbit server was truncating some of this data, so I increased the relevant variables: MAXLINE="65535" MAXMSG_STATUS="2048" MAXMSG_CLIENT="2048" MAXMSG_DATA="4096"
Anna added 2008-09-08 because of lots of truncations.
MAXLINE="32768" MAXMSG_STATUS="1024" MAXMSG_DATA="1024" MAXMSG_CLIENT="2048" MAXMSG_NOTES="1024"
I'm missing the MAXMSG_NOTES but I don't use notes anyway.
After I added this, the problem was solved. I found the sizes from the truncations reported in the logs.
I used to get log messages about truncations, but I don't anymore. After I added my above settings the problem went away for a while, but now it seems to have come back.
However, I still get many red alerts, and when I check, the log files do not report any truncated or oversized messages. Also, when I examine the "Client data available" from the red hobbit report, I find the size of the message is nowhere near any value above, and in fact is always different... Some reports that work are longer than reports that don't work etc...
Your logs - do they not report truncated or oversized messages like in the following message: http://www.hswn.dk/hobbiton/2006/05/msg00176.html
No, they don't.
It isn't 100%, but generally (more than 98%) the clients with the problem are on bandwidth limited networks.
I would appreciate if anyone can provide any tips on how to make things more reliable?
Options I have considered:
- Get hobbit to compress it's data, which reduces network load, and hence should improve reliability.
- Add a "END" tag to the hobbit client data, and if the server doesn't get the END tag then ignore the whole file (or re-request it)
- Switch to polling mode (which effectively does 1 && 2 I suppose)
- Try and track down what is causing this, and fix it...
My hobbit server is behind a NAT router, so one possibility I have considered is the NAT router is dropping the map before the end of the TCP connection due to too many other connections or similar.
Have you considered setting up a Hobbit proxy. See: http://www.hswn.dk/hobbiton/2007/06/msg00080.html
One remote site is running a hobbit proxy, and this site also has problems when forwarding the results to my hobbit server.
Here is the output from the bbproxy report: bbproxy for Hobbit version 4.2.0
Proxy statistics
Incoming messages : 623771 (0 msgs/second) Outbound messages : 385373
Incoming message distribution
- Combo messages : 4944
- Status messages : 607511 Messages merged : 518963 Resulting combos : 215412
- Page messages : 0
- Other messages : 14528
Proxy ressources
- Connection table size : 1
- Buffer space : 8 kByte
Timeout/failure details
- reading from client : 100
- connecting to server : 7012
- sending to server : 0
- recovered : 0
- reading from server : 0
- sending to client : 0
Average queue time : 1.139
I really do think it looks like the TCP connection is being lost when the client (or proxy) is attempting to send to the server. However, when this happens the client doesn't retry, and the server attempts to use the partially received message...
Further suggestions/comments?
Regards, Adam
Adam Goryachev Website Managers www.websitemanagers.com.au -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkkbveUACgkQGyoxogrTyiVj8gCgpAap2jz3IOfdnP283VVuYdAr QB0AoL+L9BFF+xajqfJfEX9ih3B5fXBq =Ol6O -----END PGP SIGNATURE-----
participants (2)
-
annaj＠hi.is
-
mailinglists＠websitemanagers.com.au