Remote hobbit server's ports & procs flapping Green/Clear on secondary BBDISPLAY
Per my email to the list last night to Neil entitled "Re: [hobbit] Securing Xymon Over Internet" I have some clients with their own local hobbit server which I have configured to also send to my xymon server as a secondary BBDISPLAY.
I have noticed that two tests on one remote server continually go green and then clear at the same time. That is, just now at 08:03:08 procs shows that it when from CLEAR to GREEN, then at the same time when BACK from GREEN to CLEAR. This continues with every 5 minute cycle.
--[snip from the Current non-green Systems screen]--
Tue Feb 17 08:03:08 2009 dns.myclient.org ports G C Tue Feb 17 08:03:08 2009 dns.myclient.org procs G C Tue Feb 17 08:03:08 2009 dns.myclient.org ports C G Tue Feb 17 08:03:08 2009 dns.myclient.org procs C G
--[snip from the Current non-green Systems screen]--
At the client site, their hobbit server currently reports "No events received in the last 240 minutes."
These two tests, ports and procs are actually ON my clients hobbit server. All other ports/procs tests on their other machines report back to me fine and are not flapping Green/Clear.
Where can I look to begin to determine what is causing this green/clear flapping?
THANKS!
-- Bill Arlofski Reverse Polarity, LLC
On Tue, Feb 17, 2009 at 08:20:32AM -0500, Bill Arlofski wrote:
I have noticed that two tests on one remote server continually go green and then clear at the same time. That is, just now at 08:03:08 procs shows that it when from CLEAR to GREEN, then at the same time when BACK from GREEN to CLEAR. This continues with every 5 minute cycle.
Check the BBDISP and BBDISPLAY settings in your hobbitserver.cfg. It sounds as if you have BBDISP=0.0.0.0 and BBDISPLAYS="ip1 ip2" - that causes hobbitd_client (the module analyzing client data) to send the results to both of your Xymon servers, causing double updates for each status. The reason the status is flapping between clear and green is that your hobbit-clients.cfg is different on the two servers.
What happens is:
- Client sends the client data to server1 and server2.
- server1 passes the client data to the local hobbitd_client task.
- server1's hobbitd_client analyses the data using the hobbit-clients.cfg found on server1. Then it generates status updates; these are sent to the servers listed in BBDISPLAYS, i.e. both server1 and server2.
- server2 does exactly the same, and sends another set of status updates to both server1 and server2.
The solution is to make sure you have identical hobbit-clients.cfg files on both servers, and to avoid updating everything twice you should only list the local server in hobbitserver.cfg's BBDISP setting. (But keep both servers in the hobbitclient.cfg's BBDISP / BBDISPLAYS setting).
Regards, Henrik
Henrik Størner wrote:
On Tue, Feb 17, 2009 at 08:20:32AM -0500, Bill Arlofski wrote:
I have noticed that two tests on one remote server continually go green and then clear at the same time. That is, just now at 08:03:08 procs shows that it when from CLEAR to GREEN, then at the same time when BACK from GREEN to CLEAR. This continues with every 5 minute cycle.
Check the BBDISP and BBDISPLAY settings in your hobbitserver.cfg. It sounds as if you have BBDISP=0.0.0.0 and BBDISPLAYS="ip1 ip2" - that causes hobbitd_client (the module analyzing client data) to send the results to both of your Xymon servers, causing double updates for each status. The reason the status is flapping between clear and green is that your hobbit-clients.cfg is different on the two servers.
Hi Henrik... Thanks for the fast and detailed reply.
Ahhh... OK.. I have now copied the hobbit-clients.cfg from my client's hobbit server to my central xymon server's hobbit-clients.cfg file and the flapping stopped.
However.. I still have a question based on your description:
What happens is:
- Client sends the client data to server1 and server2.
- server1 passes the client data to the local hobbitd_client task.
- server1's hobbitd_client analyses the data using the hobbit-clients.cfg found on server1. Then it generates status updates; these are sent to the servers listed in BBDISPLAYS, i.e. both server1 and server2.
- server2 does exactly the same, and sends another set of status updates to both server1 and server2.
At my client's site, their hobbit server monitors several other local servers with the hobbit client installed (cpu, disk, procs etc) and other network equipment (ping telnet, ssh etc)
Their server has my server and their server in the ~/server/etc/hobbitserver.cfg's BBDISPLAYS (with just 0.0.0.0 in BBDISP)
On my server I only have itself in the BBDISP, and BBDISPLAYS=""
I have the BBLOCATION set to their site name in their server and in front of all the tests in bb-hosts I have # NET="TheirSiteName" On my server, I have
NET:"MySiteName" in front of all of my local tests.
Their server also runs some local client scripts to gather NUT (Network UPS tools) data from their UPSes. So I found that I needed to make the same changes to BBDISP and BBDISPLAYS their ~/client/etc/hobbit-clients.cfg file otherwise MY Xymon server didn't get their UPS report data.
The solution is to make sure you have identical hobbit-clients.cfg files on both servers,
I think that this is what fixed it.
and to avoid updating everything twice you should only list the local server in hobbitserver.cfg's BBDISP setting. (But keep both servers in the hobbitclient.cfg's BBDISP / BBDISPLAYS setting).
If I do as you explained then I get the UPS data from the client scripts on my client's hobbit server, but all the other tests for my client's site go purple on my xymon system...
So in the end, I found that I needed to make sure that BBDISP=0.0.0.0 and BBDISPLAY="my.xymon.ip their.hobbit.ip" on their hobbit server's ~/server/etc/hobbitserver.cfg and ~/client/etc/hobbitclient.cfg files
So currently, with things configured as described it does not appear that any of their data is being sent twice to my xymon server, nothing is flapping, and I am seeing the UPS data from their client scripts too.
So I think ALL is well now. Thanks again for the quick reply!
-- Bill Arlofski Reverse Polarity, LLC
participants (2)
-
henrik＠hswn.dk
-
waa-hobbitml＠revpol.com