So the memory usage on the machine is fairly high. This system is a VM, and was built with only 2GB of memory, of which about 1.8GB is in use. I have a maintenance window coming up this week where I am going to increase the available memory to the server, but I will also try to inject the 2 MALLOC debugging that you suggested into things as well to watch for additional issues as time goes on. Hopefully that can help to identify where the problem lies and I can be a testbed to help determine a resolution.
Greg.
On Mon, Oct 24, 2016 at 11:57 PM, Japheth Cleaver <cleaver at terabithia.org> wrote:
On 10/24/2016 10:59 AM, Greg Krpan wrote:
I haven't noticed any errors through xymond_client or from debug mode.
After running the xymoncmd line above I get the following:
./xymoncmd xymon 127.0.0.1 "xymondlog wstc-lr0dhcp1.svcs" wstc-lr0dhcp1|svcs|red||1477331698|1477331698|1477333498|0| 0|15.1.160.11|460159|||Y| red Mon Oct 24 11:55:55 2016 - Services NOT ok &red BBWin: No matching service - want started/automatic &green DHCPServer is started/automatic - want started/automatic &green McAfeeFramework is started/automatic - want started/automatic &green McShield is started/automatic - want started/automatic &green McTaskManager is started/automatic - want started/automatic &green VMTools is started/automatic - want started/automatic
Name StartupType Status DisplayName AeLookupSvc manual stopped Application Experience ALG manual stopped Application Layer Gateway Service AppIDSvc manual stopped Application Identity Appinfo manual sta]ted Application Information AppMgmt manual stopped Application Management AppReadiness manual stopped App Readiness AppXSvc manual stopped AppX Deployment Service (AppXSVC) AudioEndpointBuilder manual stopped Windows Audio Endpoint Builder Audiosrv ] manual st] ped Windows Audio ]BWin automatic started Big Brother Xymon Client BFE automatic started Base Filtering Engine
*snip*
Thanks; that confirms that the issue involved xymond_client or xymond, and isn't related to the web display.
Looking through the changes from 4.3.25 to 4.3.27, it's hard to see what might be causing this issue.
Is there any chance you're under a significant memory pressure on this machine? Would you be able to add some glibc debugging at all? If so, would you be able to add an: (export) MALLOC_CHECK_=3 (export) MALLOC_PERTURB_=1 ... into the environment? This might help trigger a memory issue that could otherwise go unnoticed.
Alternatively, the next step might be to downgrade to 4.3.25 and see if that fixes the problem (if so, that really indicated there's a specific hidden issue here). Also, it might be interesting to see if the el7 Terabithia RPMs show the same problem for you. There was a significant increase in lookup/buffer debugging in xymond_client in there that's also in the 4.x-master branch but isn't in 4.3.x when compiled from source.
Regards, -jc
--
In honor of those who lost their lives exploring the final frontier: Apollo 1; January 27, 1967 Virgil "Gus" Ivan Grissom, Edward Higgins White II, Roger Bruce Chaffee Space Shuttle Challenger, Mission STS-51-L; January 28, 1986 Francis R. Scobee, Michael J. Smith, Judith A. Resnik, Ellison S. Onizuka, Ronald E. McNair, Gregory B. Jarvis, Sharon Christa McAuliffe Space Shuttle Columbia, Mission STS-107; February 1, 2003 Rick D. Husband, William C. McCool, Michael P. Anderson, Kalpana Chawla, David M. Brown, Laurel Blair Salton Clark, Ilan Ramon