From earle@isolar.DynDNS.ORG Wed Jun 24 08:23:50 2026
From: earle@isolar.DynDNS.ORG
To: xymon@xymon.com
Subject: [Xymon] Phantom red statuses (Fwd: Xymon [750466] mgmtconsole:msgs
	CRITICAL (RED))
Date: Thu, 11 Feb 2016 14:15:06 -0800
Message-ID: <4E9E2A92-8CD5-4371-85D4-AA802F228179@isolar.DynDNS.ORG>
In-Reply-To: <201602111957.u1BJvPkQ024499@miplmgmta.jpl.nasa.gov>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============8582410789628522985=="

--===============8582410789628522985==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

I'm running Xymon 4.3.12-2 server (yeah, I know ...) on my management system.
(RHEL 6.5 currently)

A couple of days ago I migrated our central syslog server over to the
Xymon server, so now "/var/log/messages" is getting a ton of stuff in
it that it never had before since all my systems are now reporting in
to it.

Ever since then I've seen something weird - every hour (for about 17+ hours)
I was getting RED alerts for the management console's own "msgs" status, but
the actual e-mail notifications don't show anything marked red in them!

It's either yellow or, as in the forwarded message below, green.  I have no
idea why I was getting RED alerts for this file if it thinks it's yellow or
green - any ideas?

The only other thing I can add is that when I go to the Web page for
mgmtconsole:msgs, it says "WARNING: Flapping status" at the top.

Is that a clue?

(Update: interestingly, it looks like the status has finally changed to
 green a few minutes ago - after having been red for nearly 17 1/2 hours.
 Still seeing "WARNING: Flapping status" on the svcstatus Web page, though.)

Thanks,

	- Greg

> Begin forwarded message:
>=20
> From: xymon Monitor <xymon at mgmtconsole.my.do.ma.in>
> Subject: Xymon [750466] mgmtconsole:msgs CRITICAL (RED)
> Date: February 11, 2016 at 11:57:25 AM PST
> To: sysadmins at mgmtconsole.my.do.ma.in
>=20
> green Thu Feb 11 11:57:24 PST 2016 - Log files ok
> <pre>
> </pre>
>=20
> No entries in <a href=3D"/xymon-cgi/svcstatus.sh?CLIENT=3Dmgmtconsole&amp;S=
ECTION=3Dmsgs:/var/log/messages">/var/log/messages</a>
>=20
>=20
> Full log <a href=3D"/xymon-cgi/svcstatus.sh?CLIENT=3Dmgmtconsole&amp;SECTIO=
N=3Dmsgs:/var/log/messages">/var/log/messages</a>
> <...SKIPPED...>
> Feb 11 11:57:17 host7 nrpe[20194]: [ID 927837 daemon.info] connect from mtf=
uji
>=20
> [... rest elided ... ]
>=20
> See http://mgmtconsole/xymon-cgi/svcstatus.sh?HOST=3Dmgmtconsole&SERVICE=3D=
msgs



--===============8582410789628522985==--


From cleaver@terabithia.org Wed Jun 24 08:23:50 2026
From: cleaver@terabithia.org
To: xymon@xymon.com
Subject: [Xymon] Phantom red statuses (Fwd: Xymon [750466] mgmtconsole:msgs
 CRITICAL (RED))
Date: Thu, 11 Feb 2016 15:07:41 -0800
Message-ID: <8d4a6f843f3b872ee3132782fd197a71.squirrel@mail.kkytbs.net>
In-Reply-To: <4E9E2A92-8CD5-4371-85D4-AA802F228179@isolar.DynDNS.ORG>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2327980075381254202=="

--===============2327980075381254202==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi Greg,

The flapping warning is what tips it off. Flap-detection in xymond
functions by looking at alternating alert states (eg, red/green) happening
within a certain period and "pegging" it at the higher status while it's
going back and forth. This prevents spurious recovery messages and
untimely pager death.

The thing is, though, for a normal functioning 'msgs' test it's almost
impossible for it to actually flap out of the box. The logfetch program
which controls the raw data sent to xymond_client for evaluation actually
walks back 6 "periods" (run cycles) in the log file and sends all
subsequent data up to xymond. This helps with mitigating any lost messages
by turning the 'msgs' test into a "Recent Errors in the Log" test instead
of a direct reflection of the event, since xymon is a state-based
monitoring system rather than a single-fire-and-forget (trap) based
system.

Out of the box, a single red event will cause the msgs test to remain red
for a solid 30m -- far too long for flapping to get triggered in most
cases.


Is there any chance you have multiple servers reporting in with the name
'mgmtconsole'? Especially if you're not using FQDN (which it doesn't seem
like you are), that seems like something that might cause this: Two
different servers with the same name, each sending their own red/green
states every few minutes.


HTH,
-jc



On Thu, February 11, 2016 2:15 pm, Greg Earle wrote:
> I'm running Xymon 4.3.12-2 server (yeah, I know ...) on my management
> system.
> (RHEL 6.5 currently)
>
> A couple of days ago I migrated our central syslog server over to the
> Xymon server, so now "/var/log/messages" is getting a ton of stuff in
> it that it never had before since all my systems are now reporting in
> to it.
>
> Ever since then I've seen something weird - every hour (for about 17+
> hours)
> I was getting RED alerts for the management console's own "msgs" status,
> but
> the actual e-mail notifications don't show anything marked red in them!
>
> It's either yellow or, as in the forwarded message below, green.  I have
> no
> idea why I was getting RED alerts for this file if it thinks it's yellow
> or
> green - any ideas?
>
> The only other thing I can add is that when I go to the Web page for
> mgmtconsole:msgs, it says "WARNING: Flapping status" at the top.
>
> Is that a clue?
>
> (Update: interestingly, it looks like the status has finally changed to
>  green a few minutes ago - after having been red for nearly 17 1/2 hours.
>  Still seeing "WARNING: Flapping status" on the svcstatus Web page,
> though.)
>
> Thanks,
>
> 	- Greg
>
>> Begin forwarded message:
>>
>> From: xymon Monitor <xymon at mgmtconsole.my.do.ma.in>
>> Subject: Xymon [750466] mgmtconsole:msgs CRITICAL (RED)
>> Date: February 11, 2016 at 11:57:25 AM PST
>> To: sysadmins at mgmtconsole.my.do.ma.in
>>
>> green Thu Feb 11 11:57:24 PST 2016 - Log files ok
>> <pre>
>> </pre>
>>
>> No entries in <a
>> href=3D"/xymon-cgi/svcstatus.sh?CLIENT=3Dmgmtconsole&amp;SECTION=3Dmsgs:/v=
ar/log/messages">/var/log/messages</a>
>>
>>
>> Full log <a
>> href=3D"/xymon-cgi/svcstatus.sh?CLIENT=3Dmgmtconsole&amp;SECTION=3Dmsgs:/v=
ar/log/messages">/var/log/messages</a>
>> <...SKIPPED...>
>> Feb 11 11:57:17 host7 nrpe[20194]: [ID 927837 daemon.info] connect from
>> mtfuji
>>
>> [... rest elided ... ]
>>
>> See
>> http://mgmtconsole/xymon-cgi/svcstatus.sh?HOST=3Dmgmtconsole&SERVICE=3Dmsgs
>
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>




--===============2327980075381254202==--


From novosirj@rutgers.edu Wed Jun 24 08:23:50 2026
From: novosirj@rutgers.edu
To: xymon@xymon.com
Subject: [Xymon] Phantom red statuses (Fwd: Xymon [750466]	mgmtconsole:msgs
 CRITICAL (RED))
Date: Thu, 11 Feb 2016 18:13:46 -0500
Message-ID: <4257BE9A-1B23-4095-AA70-830CA2BD02D8@rutgers.edu>
In-Reply-To: <4E9E2A92-8CD5-4371-85D4-AA802F228179@isolar.DynDNS.ORG>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5900051469693822008=="

--===============5900051469693822008==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Yes, flapping status is a sort
> On Feb 11, 2016, at 5:15 PM, Greg Earle <earle at isolar.DynDNS.ORG> wrote:
>=20
> I'm running Xymon 4.3.12-2 server (yeah, I know ...) on my management syste=
m.
> (RHEL 6.5 currently)
>=20
> A couple of days ago I migrated our central syslog server over to the
> Xymon server, so now "/var/log/messages" is getting a ton of stuff in
> it that it never had before since all my systems are now reporting in
> to it.
>=20
> Ever since then I've seen something weird - every hour (for about 17+ hours)
> I was getting RED alerts for the management console's own "msgs" status, but
> the actual e-mail notifications don't show anything marked red in them!
>=20
> It's either yellow or, as in the forwarded message below, green.  I have no
> idea why I was getting RED alerts for this file if it thinks it's yellow or
> green - any ideas?
>=20
> The only other thing I can add is that when I go to the Web page for
> mgmtconsole:msgs, it says "WARNING: Flapping status" at the top.
>=20
> Is that a clue?
>=20
> (Update: interestingly, it looks like the status has finally changed to
> green a few minutes ago - after having been red for nearly 17 1/2 hours.
> Still seeing "WARNING: Flapping status" on the svcstatus Web page, though.)

Yes, flapping status is essentially =E2=80=9Cpegged at red due to too many st=
atus changes.=E2=80=9D

--
____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
|| \\UTGERS      |---------------------*O*---------------------
||_// Biomedical | Ryan Novosielski - Senior Technologist
|| \\ and Health | novosirj at rutgers.edu - 973/972.0922 (2x0922)
||  \\  Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark
     `'


--===============5900051469693822008==--