On Tue, December 1, 2015 9:14 am, John Thurston wrote:
How embarrassing. I was composing a note to mention a problem with the list archives not capturing all messages . . . when I discovered that the message for which I was searching was never sent to the list.
I composed the following message back in early October and then sent it only to myself :p No wonder it didn't generate any chatter.
On 8/28/2015 3:12 PM, J.C. Cleaver wrote:
On Fri, August 28, 2015 3:16 pm, John Thurston wrote:
On 8/28/2015 12:45 PM, John Thurston wrote:
On 6/10/2015 9:01 AM, Scot Kreienkamp wrote:
I have a xymon server running 4.3.21 that seems to be accumulating processes like these:
hobbit 28430 0.0 0.0 0 0 ? Z 12:50 0:00 [xymond_hostdata] <defunct> . . .
It seemed related to drop messages . . .
Hey, I think I'm seeing the same thing on Solaris with 4.3.21
I've ended up here after a customer let me know that email alerts were not working as expected. After a few hours of digging around, I decided that the alert daemon was failing to retrieve hostnames and failing miserably.
Have other people seen this behavior?
I have duplicated this behavior on another xymon server on Solaris. It certainly looks like this behavior breaks the alert daemon. Fortunately, I "drop" hosts in batches so can restart Xymon at that time, but this is still pretty icky.
J.C., do you know if your patch made it into the code-base?
Has anyone else tested this patch? If so, on what operating systems?
This patch took care of the defunct/zonebie processes on "drop" events, but I've just discovered that it does not solve the underlying problem. It still appears that xymond_hostdata does not behave correctly following a "drop" command. The effect is that alerts fail to be delivered for _some_ messages because hostnames can no longer be retrieved.
Example:
My xymon server is humming along. I have the alert module debug-logging to alerts.log. Immediately after issuing a "drop" command of the sort:
#xymon localhost "drop foo.bar.com sslcert"
the following sorts appear in the alerts.log. After this, some messages may result in alert emails being sent, but most quietly disappear. Currently, my resolution is to "xymon.sh restart" but that is much too heavy handed for long term use.
21178 2015-10-05 16:39:43.257559 get_xymond_message: Interrupted 21178 2015-10-05 16:39:43.257624 No files modified, skipping reload of /opt/xymon/server/etc/alerts.cfg 21178 2015-10-05 16:39:43.257680 No files modified, skipping reload of /opt/xymon/server/etc/holidays.cfg 21178 2015-10-05 16:39:43.257718 Checking criteria for host 'doadrbjnu-sp.bar.com', which is not defined 21178 2015-10-05 16:39:43.257773 Found a first matching rule 21178 2015-10-05 16:39:43.257802 Checking criteria for host 'doadrbjnu-sp.bar.com', which is not defined 21178 2015-10-05 16:39:43.257830 Checking criteria for host 'doadrbjnu-sp.bar.com', which is not defined 21178 2015-10-05 16:39:43.257854 Found a first matching rule 21178 2015-10-05 16:39:43.257879 Checking criteria for host 'doadrbjnu-sp.bar.com', which is not defined 21178 2015-10-05 16:39:43.257910 Checking criteria for host 'steam.bar.com', which is not defined 21178 2015-10-05 16:39:43.257935 Found a first matching rule 21178 2015-10-05 16:39:43.257960 Checking criteria for host 'steam.bar.com', which is not defined 21178 2015-10-05 16:39:43.257986 Checking criteria for host 'steam.bar.com', which is not defined 21178 2015-10-05 16:39:43.258010 Found a first matching rule 21178 2015-10-05 16:39:43.258035 Checking criteria for host 'steam.bar.com', which is not defined 21178 2015-10-05 16:39:43.258061 Checking criteria for host 'upsjdc.bar.com', which is not defined 21178 2015-10-05 16:39:43.258088 Found a first matching rule 21178 2015-10-05 16:39:43.258113 Checking criteria for host 'upsjdc.bar.com', which is not defined 21178 2015-10-05 16:39:43.258140 Checking criteria for host 'upsjdc.bar.com', which is not defined 21178 2015-10-05 16:39:43.258164 Found a first matching rule 21178 2015-10-05 16:39:43.258188 Checking criteria for host 'upsjdc.bar.com', which is not defined 21178 2015-10-05 16:39:43.258211 0 alerts to go 21178 2015-10-05 16:39:43.258270 Want msg 5039, startpos 134769, fillpos 134769, endpos -1, usedbytes=0, bufleft=131470 21178 2015-10-05 16:39:47.962032 Got 2831 bytes 21178 2015-10-05 16:39:47.962143 xymond_alert: Got message 5039 @@page#5039/soajnuexhs1.bar.com|1444091987.961845|10.2.3.40|soajnuexhs1.bar.com|msgs|0.0.0.0|1444093787|red|red|1444088306|ETS/MsgDir|540754|||| 21178 2015-10-05 16:39:47.962171 startpos 137600, fillpos 137600, endpos -1 21178 2015-10-05 16:39:47.962204 Got page message from soajnuexhs1.bar.com:msgs 21178 2015-10-05 16:39:47.962252 Want msg 5040, startpos 137600, fillpos 137600, endpos -1, usedbytes=0, bufleft=128639 21178 2015-10-05 16:39:58.022397 Got 297 bytes 21178 2015-10-05 16:39:58.022526 xymond_alert: Got message 5040 @@page#5040/doadofjdc-ea05p.bar.com|1444091998.022274|10.2.167.44|doadofjdc-ea05p.bar.com|msgs|0.0.0.0|1444093798|green|red|1444091998|DOA/IRIS||||| 21178 2015-10-05 16:39:58.022558 startpos 137897, fillpos 137897, endpos -1 21178 2015-10-05 16:39:58.022593 Got page message from doadofjdc-ea05p.bar.com:msgs 21178 2015-10-05 16:39:58.022630 Alert status changed from 1 to 0 21178 2015-10-05 16:39:58.022666 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022706 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022739 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022776 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022808 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022841 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022873 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022904 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022935 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022967 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.022998 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023028 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023059 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023089 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023120 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023151 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023187 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023221 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023252 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023282 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023313 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023342 Checking criteria for host 'doadofjdc-ea05p.bar.com', which is not defined 21178 2015-10-05 16:39:58.023369 Found no first matching rule 21178 2015-10-05 16:39:58.023402 Want msg 5041, startpos 137897, fillpos 137897, endpos -1, usedbytes=0, bufleft=128342 21178 2015-10-05 16:40:10.109262 get_xymond_message: Returning NULL due to EOF
Hmm. This seems to be fundamentally a different issue than the "hostdata module going rogue" thing, which was about zombies never being picked up. AFAICT, somehow the hosts tree structure is getting clobbered as a result of the drop (assuming all of those hosts are expected to be existing). There were a few patches for things in xymond.c at one point, and more error checking when going to POSIX btrees generally, but I hadn't encountered this in other intermittent hostlist readers. 1) Which version of Solaris is this? 2) Have you experienced this in other workers for xymon? (IE, xymond_client not being able to look up hostnames after a drop -- would probably lead to random purples) 3) Does issuing a "reload" command or -HUP to xymond_alert re-sync things? -jc