Hi,
I have bbwin 0.13 running in central mode on several boxes talking to xymon 4.3.7. Most of the boxes are reporting as advertised. I have 3 of them that intermittently stop reporting and turn purple. The servers still seem to be talking to xymon but I get the following errors in the eventvwr:
BBWin failed to send the client data successfully to the Xymon server. The error was : Can't send message : An existing connection was forcibly closed by the remote host..
BBWin failed to send the client data successfully to the Xymon server. The error was : Can't send message : The operation completed successfully..
The above errors are from 2 different machines.
tcpdump shows that the client and server are communicating. There is nothing in the xymon logs that gives any hints.
The bbwin.cfg is as follows:
<?xml version="1.0" encoding="utf-8" ?> <configuration> <bbwin> <setting name="bbdisplay" value="my.xymon.ip" /> <setting name="bbdisplay" value="my.other.xymon.ip" /> <!-- BB Pager Part --> <!--<setting name="usepager" value="false" /> <setting name="bbpager" value="yourfirstbbpager" /> <setting name="bbpager" value="yourfirstbbpager" /> <setting name="pagerlevels" value="red yellow" /> -->
<!-- proxy connection settings -->
<!--
<setting name="useproxy" value="false" />
<setting name="proxy" value="[user:password@]host[:port]'/>
-->
<!-- bbwin mode local or central -->
<setting name="mode" value="central" />
<setting name="configclass" value="win32" />
<setting name="autoreload" value="true" />
<setting name="timer" value="5m" />
<load name="bbwinupdate" value="bbwinupdate.dll"/>
<load name="cpu" value="cpu.dll"/>
<load name="disk" value="disk.dll"/>
<load name="externals" value="externals.dll"/>
<load name="filesystem" value="filesystem.dll"/>
<load name="memory" value="memory.dll"/>
<load name="msgs" value="msgs.dll"/>
<load name="procs" value="procs.dll"/>
<load name="stats" value="stats.dll"/>
<load name="svcs" value="svcs.dll"/>
<load name="uptime" value="uptime.dll"/>
<load name="who" value="who.dll"/>
<setting name="loglevel" value="3" />
<setting name="logpath" value="..\logs\BBWin.log"/>
<!-- If true, the agent will report reporting failures as warning events -->
<setting name="logreportfailure" value="false" />
</bbwin> </configuration>
I have the following in analysis.cfg:
CLASS=win32 LOAD 80 90 # Load threholds are in % MEMPHYS 99 101 MEMSWAP 90 97 MEMACT 97 99 SVC Dhcp startup=automatic status=started PROC svchost.exe 3 PORT STATE=LISTENING MIN=0 TRACK=Listen TEXT=Listen
Does anyone have any ideas how to troubleshoot this problem?
-- Tom me at tdiehl.org Spamtrap address me123 at tdiehl.org
On 04-08-2012 15:25, me at tdiehl.org wrote:
BBWin failed to send the client data successfully to the Xymon server. The error was : Can't send message : An existing connection was forcibly closed by the remote host..
Do you see a message about "data flooding" in the xymond.log file on the server ?
Try disabling the "msgs" module in the bbwin config - I've seen Windows servers generate multi-MB logs which could trigger xymond to close the connection.
Regards, Henrik
On Sat, 4 Aug 2012, Henrik Størner wrote:
On 04-08-2012 15:25, me at tdiehl.org wrote:
BBWin failed to send the client data successfully to the Xymon server. The error was : Can't send message : An existing connection was forcibly closed by the remote host..
Do you see a message about "data flooding" in the xymond.log file on the server ?
I do not see anything that says "data flooding" but I see a lot of the following:
2012-08-04 03:41:58 Oversize clichg msg from xymond for mail.mydomain.com truncated (n=6545894, limit=5242880) 2012-08-04 03:42:18 Oversize data/client msg from 75.xxx.xxx.106 truncated (n=6558881, limit 5242880) First line: mail.mydomain.com|bbwin|win32|
Is that what you are referring to?
Try disabling the "msgs" module in the bbwin config - I've seen Windows servers generate multi-MB logs which could trigger xymond to close the connection.
I just disabled the msgs module. I will report back after it runs a few hours.
Thanks for the help.
Regards,
-- Tom me at tdiehl.org Spamtrap address me123 at tdiehl.org
Henrik,
On 04-08-2012 15:25, me at tdiehl.org wrote:
BBWin failed to send the client data successfully to the Xymon server. The error was : Can't send message : An existing connection was forcibly closed by the remote host.. Do you see a message about "data flooding" in the xymond.log file on the server ? Just a suggestion on diagnosing these cases - could the "data flooding" message also include a) the type of message that is causing the data flooding (client/status/data/etc report) and b) the client name that is in that message. This info is in line 1 of the message, so easy enough to detect.
So rather than:
Data flooding from 192.168.1.1, closing connection
how about:
Data flooding from 192.168.1.1 - message type 'status' client 'test.example.com' - closing connection
I have recently had some cases where I am getting these messages from the xymon server itself, where event logs or syslogs are forwarded to the xymon server and an external script on the server is monitoring those logs and sending the message on behalf of the client. If the volume of messages is sustained at a flooding level, eventually the test for that client goes purple!
Try disabling the "msgs" module in the bbwin config - I've seen Windows servers generate multi-MB logs which could trigger xymond to close the connection.
Yes, it's an even worse cause of the above scenario, because filtering usually only gets done server-side not client side.
David.
-- David Baldwin - Senior Systems Administrator (Datacentres + Networks) Information and Communication Technology Services Australian Sports Commission http://ausport.gov.au Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616 david.baldwin at ausport.gov.au Leverrier Street Bruce ACT 2617
Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au
This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.
Hi David,
On 06-08-2012 01:24, David Baldwin wrote:
Do you see a message about "data flooding" in the xymond.log file on the server ? Just a suggestion on diagnosing these cases - could the "data flooding" message also include a) the type of message that is causing the data flooding (client/status/data/etc report) and b) the client name that is in that message. This info is in line 1 of the message, so easy enough to detect.
this was added back in 4.3.5 - please upgrade!
Regards, Henrik
participants (3)
-
david.baldwin@ausport.gov.au
-
henrik@hswn.dk
-
me@tdiehl.org