[Apologies in advance for the too-wordy message. Part 1 of 2 .]

Recently I violated one of the prime rules of being a SysAdmin - don't ever change two things at once.

At my work we were forced to migrate our data center to a new facility, so we bought a new monitoring PC to replace a RHEL 6.10 system that ran Xymon 4.3.12. The new monitoring PC runs RHEL 7.6 and Xymon 4.3.28 (using the Terabithia RPMs).

Ever since then, I have been running into two big problems that did not exist before. This is the first problem ("brown-outs"); I'll describe the 2nd in part 2.

Randomly, a group of systems (or some subset of the group) will report in as CRITICAL/RED due to failed xymonnet tests. Mostly SSH, but some SMTP and FTP as well. The hosts/services are all actually fine and the red alerts are incorrect/false positives.

The problem is getting worse - I'm now seeing several hundred red alerts a day from these "brown-outs". The hosts involved are more-or-less random - different buildings/OSes, etc. Sometimes all of them provoke alerts; most of the time it's just a subset of the list.

When they fail, the alert message is always

-- Service <service> on <host> is not OK : Service listening but unavailable (connect timeout)

To try and catch it in the act, I ran this test in a loop:

[root at mgmt xymon]# while true; do ( echo "["date"]" ; xymonnet --report --ping --checkresponse --timing --debug --no-update 2>&1 > /tmp/xymonnet.out ; grep 'err=[^0]' /tmp/xymonnet.out ); done

A couple of times I think I did catch it; here's an example:

-- Address=192.168.1.26:22, open=1, res=0, err=1, connecttime=0.004546, totaltime=11.653128, Address=192.168.1.25:22, open=1, res=0, err=1, connecttime=0.004510, totaltime=11.653092, Address=192.168.1.219:22, open=1, res=0, err=1, connecttime=0.003163, totaltime=11.651745, Address=192.168.1.151:22, open=1, res=0, err=1, connecttime=0.002923, totaltime=11.651505, Address=192.168.1.50:22, open=1, res=0, err=1, connecttime=0.002906, totaltime=11.651488, Address=137.78.80.38:22, open=1, res=0, err=1, connecttime=0.002819, totaltime=11.651401,

[... another 10 elided ...]

Address=192.168.1.184:22, open=1, res=0, err=1, connecttime=0.001098, totaltime=12.393879, Address=192.168.1.174:25, open=1, res=0, err=1, connecttime=0.000426, totaltime=12.364234, Address=192.168.1.182:25, open=1, res=0, err=1, connecttime=0.000418, totaltime=12.364226, Address=192.168.1.25:25, open=1, res=0, err=1, connecttime=0.000411, totaltime=12.364219, Address=192.168.1.25:21, open=1, res=0, err=1, connecttime=0.022773, totaltime=12.364044,

Notice the non-zero connecttime, but the exceeded-the-timeout totaltime values.

The services always immediately recover in the next test pass.

Are there any knobs I can turn on to help debug this problem? I'm assuming it's network/router/switch-related, but I need a smoking gun.

Failing that, is there any way in a .cfg file setting to turn these particular "Service listening but unavailable" statuses into a Yellow alert rather than Red? (I'd rather not have to resort to this but as a stop-gap, I would.)

	- Greg