[Xymon] Encrypted Xymon reporting over SSL using stunnel

On 14/3/19 16:00, Jeremy Laidman wrote:

On Wed, 13 Mar 2019 at 10:57, Adam Goryachev <mailinglists at websitemanagers.com.au <mailto:mailinglists at websitemanagers.com.au>> wrote:

IMHO, when using "security" tools, we really need to make sure we
rely
on a third party "library" to do this, security/encryption is a fast
moving area, and we really don't want to be using something 5
years old
because nobody understood how to update it from TLS1.1 to TLS1.2
or from
using 3DES to using AES or whatever the issue becomes.

I'm not sure I agree that encryption is fast-moving in all areas, or even most of them. Yes, there are bugs found in encryption software, short-cuts found in brute-forcing hashes, new algorithms every now and then, and the occasional implementation problem (wep, wpa, ssl); all of these scenarios require some kind of upgrade. However, the math(s) behind the basic crypto primitives is fundamentally sound (well-vetted and in some cases proven), particularly around simple use cases. Even with deprecated hashes (MD5, SHA1) they're not considered insecure, just that a well-resourced, and highly motivated and patient actor can brute-force them in time. With careful selection of hashing, encipherment and session key management algorithms, a good, strong solution can be implemented using >20-year-old crypto.

However, there's other value in being able to use a separate library, and allow upgrades to the library to improve security or squash bugs, without having to recompile Xymon.

Also, xymon may
not be updated as easily/quickly as what other system libraries
might,
so again, better to let the OS (system) provide the relevant tools
to do
this, and we just become another consumer.

Agreed.

As such, an obvious choice would be openssl, or something similar,
where
it is widely supported across many platforms, well maintained, and
likely to be well maintained in the future.

Not to mention, OpenSSL is already used by Xymon for things like https checks.

I think there are two separate requirements when referring to
encryption:
1) Encrypt the content so anyone listening can't see your log file

Yep.

2) Signature - ie, ensuring the message actually came from "one of my
clients" or taking this further it came from "this specific client".

Yep.

And there are situations where only signing is required. This not only ensures no tampering, but it also ensures that a large message has not been truncated by some process. On some systems, I often get a red "procs" test because the [ps] or [netstat] sections were so long that the data message was truncated. I'd prefer to skip a truncated message entirely than to react to conclusions based on incomplete data, at least in this case. A [signature] section at the end of the message (or heck, even just an [end] section) and appropriate handling in xymond, would fix this.

Having encryption also gives us signing, and having signing also ensures no tampering. If encryption and signing aren't required, the overhead of managing keys makes it burdensome to detect truncated messages. But if we bake in encryption, we get everything. If we use x509 certificates, we get all of this, plus potentially PKI (federated trust, revocation, expiry). It's a hierarchy, and the higher up we go, the harder it is to implement (both in the code, and rolling it out to nodes), but the more features we get.

(As an aside I like the idea that an x509 certificate can be used to tie a client data message to an entry in hosts.cfg, either allowing the commonName field to override the value in the client data message, or having a "CN" tag in the hosts.cfg file to match the client message to.)

Also, key management needs to be considered. It's a different task to encrypt traffic between a xymonproxy server and xymond server, to rolling out encryption to 500 devices each running a Xymon client.

- Do we need full PGP security for this? At least during setup, we
can
easily start with a known good/clean connection, or rely on the
admin to
ensure that some "shared secret" is valid on both ends.

So, like bootstrapping the key rollout procedure. This could leverage the existing "download" client message type. The client should know when her certificate is soon to expire, and can just ask for a new one with: "xymon download /meta/clientcert.x509" and the xymond process could sign a new cert on-the-fly and send it down the wire.

- We can already send data back to the client after the client
successfully delivers data to the server. If the "current" data the
client is sending is considered "clean/secure" then we can easily
send
back a renewed "certificate".

Yep, that's another option.

So...
a) Start with a simple shared secret that the admin can create (or is
auto-generated) on server install.
b) The admin configures this secret onto each client that is added,
along with adding the hostname to the server (xymon-hosts file).

b1) admin configures ssh public key on client

c) When the client connects to the server, it uses the SSH
"encryption"
to protect the conversation, sends the shared key plus the normal
"1st
set of data reports". The server responds with a client "certificate".
d) When the client next connects to the server, it uses the SSH
"encryption" to protect the conversation, and signs the data with the
certificate it received from the server. If the certificate is due to
expire in X days (configurable on server) then the server will
provide
an updated certificate to the client. The client can continue to
use the
"old" certificate until expiry, but should allow plenty of time to
transition to the new one. In reality, this could be one or two days
only, since most clients report every 5 mins.

e) Delete the shared secret.

I think there's too much going on here, but not sure where. It seems to me that there's already an authentication process going on using asymmetric keys via the ssh authentication mechanism. It would seem that an extra key to authenticate the client is redundant, because the client has already authenticated itself.

Actually, I was suggesting the SSH style encryption which would be used for encryption only, not signing, in my mind that was a separate process which works with the contents. In my thought process, the SSH encryption is kind of like HTTPS, where any client can connect to the server (without authentication) but still ensure that the content is encrypted/protected.

Within this, there would be another signature to sign the content which is used to confirm it came from client abc and not random unknown, nor from known client xyz.

Also, could the shared key be a problem if it gets out? Let's say I have 50 devices all using certs, but still with the shared key still in place, just in case of certificate corruption requiring a new bootstrapping procedure. Now, let's say someone hacks into one of the devices and retrieves the shared key. That key can now be used to bootstrap any new device to trust a rogue certificate authority, or at least to sign reports that are rejected by Xymon for being signed by the wrong certificate. Perhaps this could be used as part of a man-in-the-middle attack?

I think the shared key is only enough to "add" a client, and only if that client is already listed in the xymon-hosts file. So, the value of this shared key is pretty low. Worst case, it could allow a rogue client to get a valid certificate for a host that is listed but doesn't use encryption/certificates, either because it is a old client that doesn't support it, or it is a platform that isn't supported, etc...

I think I'd be more comfortable if the shared key was unique to each device. But ultimately, I think it shouldn't be required, because the ssh keys are per-device keys, and should be sufficient to give the same effect.

Or... we could configure a shared secret in the xymon-hosts file for
each client (different "password" for each client) and the client
simply
encrypts all data it returns with this shared secret. Advantage is
"simplicity" disadvantage is much less secure.

Why less secure? I'm not disagreeing, just want to understand why.

From my (limited) understanding, a lot of crypto attacks are based on dictionary attacks. Since pretty much 100% of our content is plain text, and a lot of that will be "recent dates, repeated a lot throughout the content", ie, log files... then it would make it easier to find out the "key" if you know (or can guess) the plain text that was encrypted. My understanding is that this is less relevant when you use some of the more modern encryption processes along with PKI. I guess everyone can know the content of the index.html that a user will get over https, so I assume that https encryption processes are able to overcome this issue.

Another option is to do the ssh host key thing, where the first time a connection is made to a host, everyone just accepts the new host key into their known_hosts file (presumably after checking the fingerprint out-of-band, lol).

So one way to do this for the Xymon client would be to have the client fetch its unique key the first time it connects to the server, and use it to sign from then on. I think this much the same as the "a" through "e" steps you outlined above, except not necessarily using certificates - but it could be a certificate, or a private key, or both client private key and server public key, or a unique per-client shared key, or a one-for-all-clients shared key.

Right, it could even be that some tool on the client connects to the server and requests the certificate, and you a second interactive tool on the server will receive/process that request, once the admin approves it, then the response is sent back to the client. I mean, who doesn't have two ssh sessions open (one client and one server) when adding a new client?

A few more comments to consider.

• It's easy to do crypto badly. Whatever is done here (if anything), we shouldn't be rolling our own crypto, and instead we need to use standard crypto primitives provided by a well-known and highly-regarded library such as OpenSSL. Even that is not sufficient; one could choose the wrong type of cipher (eg a block cipher where a stream cipher is appropriate), or use a hash function that requires sufficient source data volume to properly initialise, to has a password that could be only 8 characters. There are standardised ways of doing crypto that are recognised as optimal, or at least sound, such as PBKDF2 for deriving robust keys, salting hashes, and so on. If it's going to be built in, it needs to be done right.

• Certificates don't need to be x509. Secure shell (ssh) can support certificates for authentication, but they aren't x509 certs, and from what little I've read, their use seems to be further away from the "full-blown PKI" end of the spectrum and closer to simple asymmetric key pairs. The one thing that ssh's certs cannot do is revocation. This means if I use the same cert on all boxes I login to, and my private key is exposed, I need to change my key on each and every one of those boxes. One solution to this is suitable key management, eg storing my private key on an HSM or Yubikey; another is to use a unique key for each box I want to authenticate to (essentially giving myself multiple identities). This problem isn't necessarily relevant to the discussion about securing client data messages, but it highlights the fact that this is complex, and different requirements may benefit from having quite different solutions available.

• Certs give us the /potential /to use PKI, but to get all the benefits of PKI one also needs ancillary services such as an OCSP or CRL server and proper key management. However, many organisations don't have PKI infrastructure, so the benefits of PKI such as revocation lists, don't warrant the overhead, and self-signed certs or other forms of key pair technology, should be available for those who aren't able to afford or run a complete PKI solution. Non-PKI essentially becomes mutual authentication between pairs of nodes (one always being the Xymon server). If a client key is compromised, both ends need remediation. If a server key is compromised, all clients need remediation. This is usually good enough for most smaller organisations.

I mostly understand, and think I agree with most of the above. At the end of the day, the solution needs to be:

a) backwards compatible, so you can still accept plain, unsigned, unencrypted connections from the coffee maker in the corner

b) simple to install and maintain, and robust. If it's too hard to install or maintain, then people won't want to use it.

• I see no reason why we can't support ssh just with sufficient documentation. The same goes for or stunnel, IPSec, OpenVPN, etc. They're all useful to people in different circumstances. This just requires documenting each one. There may be cases where a script or two would benefit these different methods (an extreme case is xymon-rclient.sh, which IMHO is waaaay too big to be included as a simple "helper" script for one of these methods).

Possibly we could run the existing 1984 listener in a wrapper which handles all the "encryption", and then within xymon itself, there are a small number of additional "types" that can be sent from client, or returned to the client to add the additional certificate/signing stuff.

So, if we can easily add a command line flag that says enable "STARTSSL" command, that in itself would be a huge step forward. IMAP and POP and I think SMTP can all do this, so it should be a pretty well defined protocol, and should be well supported in common libraries.

Again, lets not try and jump and touch the moon, one step at a time...

• If we are to do this right, we should include support for both client-side and server-side authentication.

  PS, as for IPv6, I am yet to see any decent level of support here, I tried to start using it years ago, and it just never happened. Turns my routers (Cisco Meraki: www.meraki.com <http://www.meraki.com> details at https://documentation.meraki.com/zGeneral_Administration/Other_Topics/IPv6_D...)

  still don't support IPv6 either, so it still doesn't seem to be as important as claimed by everyone 10+ years ago when they said we would run out of IPv4 addresses.... It's hard to "change the world", I mean, we still have SMTP and DNS ;)

Yep, this is true, overall IPv6 take-up continues to be slow. However, the thing about Xymon is, it needs to be able to monitor a wide range of devices and services. So if you have 1000 nodes to monitor, and just one of them runs IPv6, your Xymon has to support IPv6 in order to detect a service outage. This is the situation I'm in. My work-around is to disable "conn" checks for such hosts, and run a script that does a ping6 and creates the "conn" status messages. Also, certain sectors of the industry are likely to have a relatively large IPv6 footprint, such as telcos, and IoT network providers.

True, there are many Meraki customers requesting IPv6, I'm just acknowledging that it is a much slower process than it seemed to be advertised. It originally seemed like we would all be forced to IPv6 within a couple of years.