Hi,
I have a problem where a file check for a machine hosting an ELK stack is flapping between green and white, triggering alerts because of this. FYI, the host is sending its status information over authenticated HTTPS POST calls to xymoncgimsg.cgi.
I got to the point where I found out that the list of files that the client is reporting on is received from the server on every call and then, on subsequent runs, it is used to check the files and then report back. Upon further investigation I observed that on some runs, this HTTP POST returns an HTTP500 error, thus the client then points $LOGFETCHCFG to a file that contains the HTTP500 page instead of the list of files to check. This means that on the next run it will no longer report back to the server the status of the files. It will recover when the POST returns HTTP200, hence looping between green and white in between successful/unsuccessful POST requests.
Now, I would very much like to understand what happens BUT xymoncgimsg.cgi which is receiving the HTTP POST is a binary and I'm not aware of how to enable debugging to see what happens on those runs that fail with HTTP500.
FWIW, the calls only fail w/ HTTP500 for this particular host and I did consider the size of the status message - it's just around the 1MB mark. The MSG size set in the configs sits at 2MB currently so I thought I should be covered here but maybe you guys know better than me.
For now, my quick/dirty hack was to chattr +i on a working
$LOGFETCHCFG just so that I no longer get the flapping behavior but I would
like to see this get a proper fix.
Would anyone be able to advise on a next step? Looking through C source files is not exactly what I want to do since I don't have the expertise to do so but would be willing to give it some love.
So... help?!? :)
Thanks!