No...regional servers can't push to central. I have to pull from the regions. Egress rules are essentially open...ingress rules to inside the corporate network require a small miracle and help qualify for sainthood.

I'd prefer command and control happen from the central server but at a minimum it needs to manage the alerting. This also provides a layer of monitoring from outside our network within the regions themselves to allow us to show that alerts are potentially due to a network issue between local systems and remotely monitored instances (usually conn tests since ICMP seems to flake out occasionally).

=G=

From: J.C. Cleaver <cleaver at terabithia.org> Sent: Wednesday, April 6, 2016 7:52 PM To: Galen Johnson Cc: Xymon Mailing List Subject: Re: [Xymon] Regional Servers to Central

On Wed, April 6, 2016 3:48 pm, Galen Johnson wrote:

Hey,

I'm pretty sure this has been done (is possible). I'd like to change my setup to have regional Xymon servers that communicate to a central server. I know this is possible but...I can't open a connection back to the central server and have to use msgcache for my clients currently. Is it possible to set up the regional servers such that my central server manages the alerting and lets me see the remote servers using msgcache (or some other xymon server config that can use msgcache-like config)? I've attached a diagram for what I hope to do. If you can point me to a doc/web page, that would work to.

Can the regional servers push messages to the central server? If so, I believe you could use xymonfetch on the regional systems (similar to what you're doing now) reporting in (might have to use env variables on its command line) to a local xymonproxy port, which is itself configured to send both to the local xymond (on a non-standard port) as well as the regional ones.

Kind of depends what you're looking for the regional servers to do though. If all command and control remains central, then it's really just functioning as a proxy itself... (+ xymonnet polling?)

-jc

On Sun, Apr 24, 2016 at 5:44 PM Thomas Eckert <thomas.eckert at it-eckert.de> wrote:

Galen,

if egress SSH is allowed you could use an SSH tunnel from the central to the regional Servers opening say port 1985. Then use that for the communication between the Xymon servers.

This will weaken the intentions of the security policy of course ...

The tunnel can be managed by the ssh-tunnels extension by Padraig Lennon (on Xymonton) or my slightly extended version (on http://www.it-eckert.com/software/patches/ssh-tunnel/). There are also some blog posts on my site on setup and combining with xymonproxy.

Sorry to be so late to the scene. I have a similar requirement, but don't quite get the proposed solutions. I have a couple of headless "probe" Xymon servers located on less trusted networks, and a pair of Xymon servers that primarily probe devices (testing TCP services, ping checks etc) on our internal network. I want to be able to view the results of the probe servers on the internal server screens. I can't have the probe servers connect inbound to the internal Xymon servers, except perhaps via ssh tunnel.

JC suggested that xymonfetch could be run on the regional (probe) servers to send in to the internal servers. I haven't used xymonfetch before, so I'm not intimately familiar with how it's used. Nevertheless, in reading the documentation I can see that xymonfetch is intended to run on a Xymon server to connect to a Xymon client that has msgcache running. This doesn't seem to be the model described by JC, where msgcache isn't mentioned. Or am I misunderstanding something?

Perhaps relevant to a solution, the Xymon probe servers periodically connect to the Xymon clients over ssh, create a reverse tunnel, and run xymonclient.sh with suitable environment variables to push the client data through the reverse tunnel.

Perhaps what I need to do is something like this. I fire up a msgcache on each probe server, having everything feed into the msgcache instead of xymond, and I periodically run xymonfetch on each probe server to push the messages into the real xymond running there. (I'd probably have xymond listen on an alternative port, and have msgcache run on 1984.) I would then have the ability to run a second instance of xymonfetch on the probe servers, but being called via ssh from each internal server, complete with a reverse tunnel, so that the xymonfetch would inject into the xymond on the internal server. What I can't figure out here is how to allow a msgcache queue to be pushed into the probe server's xymond without being emptied and hence unavailable for the internal xymon server.

J

On 03/05/16 12:02, Jeremy Laidman wrote:

On Sun, Apr 24, 2016 at 5:44 PM Thomas Eckert <thomas.eckert at it-eckert.de <mailto:thomas.eckert at it-eckert.de>> wrote:

Galen,

if egress SSH is allowed you could use an SSH tunnel from the
central to the regional Servers opening say port 1985. Then use
that for the communication between the Xymon servers.

This will weaken the intentions of the security policy of course ...

The tunnel can be managed by the ssh-tunnels extension by Padraig
Lennon (on Xymonton) or my slightly extended version (on
http://www.it-eckert.com/software/patches/ssh-tunnel/). There are
also some blog posts on my site on setup and combining with
xymonproxy.

Sorry to be so late to the scene. I have a similar requirement, but don't quite get the proposed solutions. I have a couple of headless "probe" Xymon servers located on less trusted networks, and a pair of Xymon servers that primarily probe devices (testing TCP services, ping checks etc) on our internal network. I want to be able to view the results of the probe servers on the internal server screens. I can't have the probe servers connect inbound to the internal Xymon servers, except perhaps via ssh tunnel.

JC suggested that xymonfetch could be run on the regional (probe) servers to send in to the internal servers. I haven't used xymonfetch before, so I'm not intimately familiar with how it's used.
Nevertheless, in reading the documentation I can see that xymonfetch is intended to run on a Xymon server to connect to a Xymon client that has msgcache running. This doesn't seem to be the model described by JC, where msgcache isn't mentioned. Or am I misunderstanding something?

Perhaps relevant to a solution, the Xymon probe servers periodically connect to the Xymon clients over ssh, create a reverse tunnel, and run xymonclient.sh with suitable environment variables to push the client data through the reverse tunnel.

Perhaps what I need to do is something like this. I fire up a msgcache on each probe server, having everything feed into the msgcache instead of xymond, and I periodically run xymonfetch on each probe server to push the messages into the real xymond running there. (I'd probably have xymond listen on an alternative port, and have msgcache run on 1984.) I would then have the ability to run a second instance of xymonfetch on the probe servers, but being called via ssh from each internal server, complete with a reverse tunnel, so that the xymonfetch would inject into the xymond on the internal server. What I can't figure out here is how to allow a msgcache queue to be pushed into the probe server's xymond without being emptied and hence unavailable for the internal xymon server.

I would guess you can pass messages to xymonproxy, which will pass messages to the local xymon server plus xymonmsgcache, then the remote server will connect with ssh and collect the pending messages from the xymonmsgcache?

I hope that is what you were asking about.

Regards, Adam

-- Adam Goryachev Website Managers www.websitemanagers.com.au

On Tue, May 3, 2016 at 10:19 PM Thomas Eckert <thomas.eckert at it-eckert.de> wrote:

On 03 May 2016, at 08:46, Adam Goryachev < mailinglists at websitemanagers.com.au> wrote:

On 03/05/16 12:02, Jeremy Laidman wrote:

On Sun, Apr 24, 2016 at 5:44 PM Thomas Eckert <thomas.eckert at it-eckert.de> wrote:

...

Galen,

if egress SSH is allowed you could use an SSH tunnel from the central to the regional Servers opening say port 1985. Then use that for the communication between the Xymon servers.

This will weaken the intentions of the security policy of course ...

The tunnel can be managed by the ssh-tunnels extension by Padraig Lennon (on Xymonton) or my slightly extended version (on http://www.it-eckert.com/software/patches/ssh-tunnel/). There are also some blog posts on my site on setup and combining with xymonproxy.

Sorry to be so late to the scene. I have a similar requirement, but don't quite get the proposed solutions. I have a couple of headless "probe" Xymon servers located on less trusted networks, and a pair of Xymon servers that primarily probe devices (testing TCP services, ping checks etc) on our internal network. I want to be able to view the results of the probe servers on the internal server screens. I can't have the probe servers connect inbound to the internal Xymon servers, except perhaps via ssh tunnel.

JC suggested that xymonfetch could be run on the regional (probe) servers to send in to the internal servers. I haven't used xymonfetch before, so I'm not intimately familiar with how it's used. Nevertheless, in reading the documentation I can see that xymonfetch is intended to run on a Xymon server to connect to a Xymon client that has msgcache running. This doesn't seem to be the model described by JC, where msgcache isn't mentioned. Or am I misunderstanding something?

Perhaps relevant to a solution, the Xymon probe servers periodically connect to the Xymon clients over ssh, create a reverse tunnel, and run xymonclient.sh with suitable environment variables to push the client data through the reverse tunnel.

Perhaps what I need to do is something like this. I fire up a msgcache on each probe server, having everything feed into the msgcache instead of xymond, and I periodically run xymonfetch on each probe server to push the messages into the real xymond running there. (I'd probably have xymond listen on an alternative port, and have msgcache run on 1984.) I would then have the ability to run a second instance of xymonfetch on the probe servers, but being called via ssh from each internal server, complete with a reverse tunnel, so that the xymonfetch would inject into the xymond on the internal server. What I can't figure out here is how to allow a msgcache queue to be pushed into the probe server's xymond without being emptied and hence unavailable for the internal xymon server.

I would guess you can pass messages to xymonproxy, which will pass messages to the local xymon server plus xymonmsgcache, then the remote server will connect with ssh and collect the pending messages from the xymonmsgcache?

I hope that is what you were asking about.

Yep, that sort of thing.

I’ve only used msgcache on a single client up to now (for isolated hosts). But I _seem_ to remember that there are limits on the amount of messages it can cache (beside other drawbacks, listed here < http://www.it-eckert.com/blog/2014/remote-site-monitoring-with-ssh-tunnel/

...

).

That's a great point. Yes, I remember reading about the limitations of msgcache, now that you mention it.

As I understand the intended use is

Xymon (central) — Xymon (region1) — multiple clients region1 | — Xymon (region2) — multiple clients region2 | — …

Yes.

This would lead to a potential huge number of messages on the regional Xymon servers that might well go beyond the capacity of msgcache.

Agreed.

Using xymonproxy on the regional servers would allow to deliver the status-messages to the (local) regional server _and_ the central one.

Makes perfect sense.

This is outlined here < http://www.it-eckert.com/blog/2014/combine-ssh-tunnel-with-xymonproxy/> (the "remote-datacenter” would be a regional server). To have the data available on the regional server as well the xymond there has to listen on say 127.0.0.1:1986 and xymonproxy report to that location as well (xymonproxy-option for sending to multiple servers --server=SERVERIP[:PORT][,SERVER2IP[:PORT]] — according to xymonproxy(8) up to 3 servers are possible, configuration it pulled from the _last_ in the list!). The order of the xymonproxy “receivers” would also allow the configuration of the regions to be either from central or from the regional server.

I think what's missing from this plan is that the xymonnet probes need to be diverted to the xymonproxy instance also. But apart from that, I can't see why it wouldn't work for me.

The ssh-tunnel is handled by the central server using the ssh-tunnel

extension (either original < https://wiki.xymonton.org/doku.php/addons:ssh_tunnel> or extended version <http://www.it-eckert.com/software/patches/ssh-tunnel/>) that takes care the tunnel is up.

I'm not sure why, but I prefer to have transient ssh tunnels for the duration of the message transfer, rather than persistent tunnels that hang around forever. (Either way, ssh encryption and authentication is provided.) It looks like xymonproxy has a buffer, and so it may be able to queue messages destined for the central server for a short period of time, and the --lqueue size can be adjusted to manage this. Although I would be more comfortable if it had the ability to save its queue to disk so as to survive restarts, and also to allow for a much larger queue.

If xymonproxy offers the same queueing capability as msgcache, without the shortcomings, then it seems like an easy choice to make. I just don't know if it can queue things for long enough if my central server "checks in" every 5 minutes, instead of having a persistent tunnel. Come to think of it, I probably need to know what happens to the queue if a persistent tunnel goes down for a period of time (eg a firewall crashes and the link goes down for 30 minutes). Will the extra messages overflow? What would happen if a red-to-green transition was lost?

J

Thanks to all for your advice. A most interesting and useful thread.