no client results arrive, lot of TIME_WAIT at server
Yes, that's what i checked at first. But i assume that also the info from the servers own client data are captured by the same client channel listener, and that obviously works.
Nothing in the clientdata.log, even if i activate the DEBUG mode with a kill -USR2 PID
My clientdata section:
The client back-end module. You need this if you are running the Xymon
client on any system.
[clientdata]
ENVFILE $XYMONSERVERHOME/etc/xymonserver.cfg
NEEDS xymond
CMD /home/xymon/server/bin/xymond_channel --channel=client
--log=$XYMONSERVERLOGS/clientdata.log xymond_client
--no-clear-files
--no-clear-ports
--no-port-listing
--uptime-status
INTERVAL 2m
ps listing: $ ps -ef|grep "xymond_channel --channel=client"|grep -v grep xymon 18865 18705 0 05:13 ? 00:00:00 /home/xymon/server/bin/xymond_channel --channel=client --log=/home/xymon/server/logs/clientdata.log xymond_client --no-clear-files --no-clear-ports --no-port-listing --uptime-status
I have a lot of TIME_WAITS from time to time, no idea why this goes up and down:
$ netstat -atn|grep 1984 tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN tcp 0 0 10.137.56.43:37752 10.137.56.43:1984 TIME_WAIT tcp 0 0 10.137.56.43:37822 10.137.56.43:1984 TIME_WAIT ...
10.137.56.43 is the Xymon server IP.
$ netstat -atn|grep 1984|wc -l 165 (sometimes much much more)
(But goes down to 7 after some mins)
From: Thomas Eckert <thomas.eckert at it-eckert.de> To: xymon <xymon at xymon.com> Date: 19/10/2017 11:14 Subject: Re: [Xymon] no client results arrive, lot of TIME_WAIT at server Sent by: "Xymon" <xymon-bounces at xymon.com>
Norbert,
I assume your manual (BB way) is something like xymon 1.2.3.4 “status example.com.test …”.
The xymon-client send clientdata that is processed by a “channel
listener” and _that_ sends status-messages similar to above (a bit
simplified).
That channel-listener may not be running/crashing:
Check our tasks.cfg for the section [clientdata] and verify, if the
xymond_channel-process is running (I suspect it’s not). Maybe the
clientdata.log helps with debugging. You may also run the task manually
to get further information.
What’s your problem with the "lot of TIME_WAIT”? Are these just for tcp/1984-connections?
Cheers Thomas
On 19 Oct 2017, at 07:30, Norbert Kriegenburg <
norbert.kriegenburg at de.ibm.com> wrote:
experts,
i have a new (not my first) Xymon installation (4.3.28 on RH 7.4,
self compiled).
All looks good, but the clients only show the network tests (conn,
ssh). Only the Xymon server itself shows everything (disk, cpu,...).
- all clients are RH 7.4, same install image used (i just copied over
the client part with client config, create a cronjob for automatic
start, so no root access nor extra install needed)
- nothing suspicious in server or client logs (even in debug mode)
- no ghosts
- if i send a test manually (the old BB way) this shows at Xymon
immediately (so the client settings are ok)
- if i shutdown the xymond at server, the client logs complains:
2017-10-19 05:09:18.051994 Whoops ! Failed to send message
(Connection failed)
2017-10-19 05:09:18.052104 -> Could not connect to Xymon
daemon at 10.137.56.43:1984 (Connection refused)
(so obviously they communicate)
Querying the xymond shows:
xymon 127.0.0.1 "query clientname.ssh"
green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok
is ok, but
xymon 127.0.0.1 "query clientname.disk"
shows nothing
Any ideas?
Norbert
_______________________________________________
Xymon mailing list
Xymon at xymon.com
http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On 19 Oct 2017, at 14:49, Norbert Kriegenburg <norbert.kriegenburg at de.ibm.com> wrote:
Yes, that's what i checked at first. But i assume that also the info from the servers own client data are captured by the same client channel listener, and that obviously works.
valid point
Nothing in the clientdata.log, even if i activate the DEBUG mode with a kill -USR2 PID
My clientdata section:
The client back-end module. You need this if you are running the Xymon client on any system.
[clientdata] ENVFILE $XYMONSERVERHOME/etc/xymonserver.cfg NEEDS xymond CMD /home/xymon/server/bin/xymond_channel --channel=client
--log=$XYMONSERVERLOGS/clientdata.log xymond_client
--no-clear-files
--no-clear-ports
--no-port-listing
--uptime-status INTERVAL 2m
The INTERVAL here is not im my and xymon.com’s-version of tasks.cfg— not sure if it does harm (it _may_ try to start a new channel every 2 minutes).
ps listing: $ ps -ef|grep "xymond_channel --channel=client"|grep -v grep xymon 18865 18705 0 05:13 ? 00:00:00 /home/xymon/server/bin/xymond_channel --channel=client --log=/home/xymon/server/logs/clientdata.log xymond_client --no-clear-files --no-clear-ports --no-port-listing --uptime-status
If this process does not change it should be fine; your xymon server was restarted 5:13 this morning?
I have a lot of TIME_WAITS from time to time, no idea why this goes up and down:
$ netstat -atn|grep 1984 tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN tcp 0 0 10.137.56.43:37752 10.137.56.43:1984 TIME_WAIT tcp 0 0 10.137.56.43:37822 10.137.56.43:1984 TIME_WAIT ...
10.137.56.43 is the Xymon server IP.
$ netstat -atn|grep 1984|wc -l 165 (sometimes much much more)
(But goes down to 7 after some mins)
Looks like internal xymon-communication. As long as the TIME_WAIT-count does not go >20k that should not be the cause.
Thing’s I’d try from here:
double-check the XYMSRV / XYMONSERVERS setting on the client(s), e.g. with:
xymoncmd /usr/bin/env | egrep -i 'xymsrv|xymonservers|bbdisp’just to double-check: tcpdump on xymon server and a client to check for tcp/1984 communication
run a channel-listener an the server to see if the client messages (other than the ones from the local client) make it
xymoncmd xymond_channel --channel=client cat | grep ‘^client ' xymoncmd xymond_channel --channel=client catThe latter might be very noisy — the former just prints the “header”/1st line of each connection. There should be one message for each client sending data every 5 minutes.
HTH Thomas
<graycol.gif>Thomas Eckert ---19/10/2017 11:14:56---Norbert, I assume your manual (BB way) is something like
xymon 1.2.3.4 “status example.com.test …”From: Thomas Eckert <thomas.eckert at it-eckert.de> To: xymon <xymon at xymon.com> Date: 19/10/2017 11:14 Subject: Re: [Xymon] no client results arrive, lot of TIME_WAIT at server Sent by: "Xymon" <xymon-bounces at xymon.com>
Norbert,
I assume your manual (BB way) is something like
xymon 1.2.3.4 “status example.com.test …”.The xymon-client send
clientdatathat is processed by a “channel listener” and _that_ sends status-messages similar to above (a bit simplified).That channel-listener may not be running/crashing: Check our
tasks.cfgfor the section[clientdata]and verify, if thexymond_channel-process is running (I suspect it’s not). Maybe theclientdata.loghelps with debugging. You may also run the task manually to get further information.What’s your problem with the "lot of TIME_WAIT”? Are these just for tcp/1984-connections?
Cheers Thomas On 19 Oct 2017, at 07:30, Norbert Kriegenburg <norbert.kriegenburg at de.ibm.com <mailto:norbert.kriegenburg at de.ibm.com>> wrote: experts,
i have a new (not my first) Xymon installation (4.3.28 on RH 7.4, self compiled). All looks good, but the clients only show the network tests (conn, ssh). Only the Xymon server itself shows everything (disk, cpu,...).
- all clients are RH 7.4, same install image used (i just copied over the client part with client config, create a cronjob for automatic start, so no root access nor extra install needed)
- nothing suspicious in server or client logs (even in debug mode)
- no ghosts
- if i send a test manually (the old BB way) this shows at Xymon immediately (so the client settings are ok)
- if i shutdown the xymond at server, the client logs complains:
2017-10-19 05:09:18.051994 Whoops ! Failed to send message (Connection failed) 2017-10-19 05:09:18.052104 -> Could not connect to Xymon daemon at 10.137.56.43 <mailto:daemon at 10.137.56.43>:1984 (Connection refused)
(so obviously they communicate)
Querying the xymond shows: xymon 127.0.0.1 "query clientname.ssh" green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok is ok, but xymon 127.0.0.1 "query clientname.disk" shows nothing
Any ideas?
Norbert
Xymon mailing list Xymon at xymon.com <mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon <http://lists.xymon.com/mailman/listinfo/xymon>_______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon <http://lists.xymon.com/mailman/listinfo/xymon>
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
participants (2)
-
norbert.kriegenburg@de.ibm.com
-
thomas.eckert@it-eckert.de