Hi,
after reading an old thread about remote code execution on here [1], I wondered if something like this is still possible nowadays with xymon? In 2016 [2] this has been classified as a security risk, whereas in 2007, it was considered a feature/addon. I am wondering if remote code execution is still possible either via this mechanism or in some other way or if the development team has been taking precautions against attacks like that.
Thanks in advance.
Cheers Christoph
[1] https://lists.xymon.com/archive/2007-June/013587.html [2] https://packetstormsecurity.com/files/135758/xymon-execoverflowdisclsoe.txt
On 10/12/2021 6:11 AM, Christoph Zechner wrote:
after reading an old thread about remote code execution on here [1], I wondered if something like this is still possible nowadays with xymon?
Last time I looked, the Xymon client software permitted execution of arbitrary code supplied from the Xymon server. This default behavior could be changed by flipping a switch in the client's configuration (uh huh. Yeah, sure).
I have never considered this to be a "feature".
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Department of Administration State of Alaska
I think the "remote code execution" mentioned in the two references are two very different things: remotely configuring commands to run on the client, in the context of collecting data, versus remotely sending arbitrary pre-compiled code over the Internet to execute on the Xymon server, by leveraging the ability to overflow a memory buffer. Calling them both "remote code execution" is confabulating two different things.
The discussion about running arbitrary "code" on a client machine - commands that are determined by a configuration file on the server - is a feature of Xymon. It's not a buffer overflow bug. While this makes some people (myself included) feel a bit uncomfortable, the risk of exploitation is considerably less than that of a classic "remote code execution vulnerability" such as described in the Packetstorm post.
Here's a quick backgrounder, for those not familiar with this feature: The
"arbitrary commands" are the use of backticks in pseudo-filenames in log:,
file: or dir: entries in client-local.cfg. If I add an entry for a host
along the lines of "log:/bin/ls -t /var/log/my-logfile-* | head -1" means
that I can always look at the latest logfile, even though the filenames
have a datestamp suffix that changes. However, if I add an entry like
"log:rm -rf />/dev/null 2>&1" then I can attempt to do something nasty
(as the xymon user on the client machine). If there were a compiler on the
client machine, I could conceivably send actual code, have it compiled, and
then run it; this might give me more flexibility in my abuse of the client
machine (eg compile and run a remotely-accessible backdoor).
This feature is on by default. There appear to be two ways to it off: 1) don't run the clients in central mode, and 2) add the "--noexec" parameter to logfetch (in LOGFETCHOPTIONS in the file xymonclient.cfg).
(As an aside, I wonder if the same facility is available for Windows clients. Can anyone comment on this?)
For me to abuse this facility, I'd need write access to a configuration file on the Xymon server. I can't remotely exploit this from an arbitrary location, only from on the Xymon server. I can't somehow get the client to connect to my rogue server, and give it a fake payload to execute with backticks around my evil commands. The payload has to be present on the Xymon server as configured on the Xymon client.
To exploit this, I have to launch some sort of attack *from* the Xymon server, using an account with write access to the client-local.cfg file. This file wouldn't normally be writable by anyone except the xymon user and root. If Xymon was installed from a Terabithia package, this file is owned by root, so not even the xymon user could exploit this. There's no good reason for the xymon user to be able to write to this file, and it seems simple enough to change ownership of the file and ameliorate this risk. Certainly, if I could somehow get the xymond process (which accepts connections from anywhere) to write to an arbitrary file that is owned by the xymon user account, then this could be an attack vector on all machines that the Xymon server monitors via Xymon clients. But this does depend on an exploit on the Xymon server.
The Packetstorm post contains several vulnerabilities (which are all closed from Xymon version 4.3.25) that included a "remote code execution" vulnerability, CVE-2016-2054, that uses a buffer overflow. This is a vulnerability in the xymond process, which runs as the xymon user. This vulnerability allows an attacker to run code as the xymon user only. The attack can be launched from anywhere that has connectivity to the Xymon server. Quite a bad exploit, but it's not a remote root exploit.
I'm certainly less concerned about the ability to use backticks in a file on the Xymon server to run commands on Xymon clients as the xymon user, than I am about the ability for a remote attacker to run code on the Xymon server as the xymon user.
Of course, the backtick facility allows the buffer overflow exploit against the Xymon server to be leveraged to send commands to my whole fleet of Xymon clients.
So, in answer to Christoph's question: yes, remote commands can still be executed, under limited situations and context, but remote code execution is not possible since v4.3.25. My advice to anyone would be to implement Henrik's recommended mitigations in the Packetstorm post, and also add the "--noexec" parameter to logfetch. Those steps can then be reversed/adjusted where necessary, and after careful consideration of the risks and benefits, and alternatives.
Cheers Jeremy
On Wed, 13 Oct 2021 at 04:10, John Thurston <john.thurston at alaska.gov> wrote:
On 10/12/2021 6:11 AM, Christoph Zechner wrote:
after reading an old thread about remote code execution on here [1], I wondered if something like this is still possible nowadays with xymon?
Last time I looked, the Xymon client software permitted execution of arbitrary code supplied from the Xymon server. This default behavior could be changed by flipping a switch in the client's configuration (uh huh. Yeah, sure).
I have never considered this to be a "feature".
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Department of Administration State of Alaska
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Hi,
On 13/10/2021 04:04, Jeremy Laidman wrote:
I think the "remote code execution" mentioned in the two references are two very different things: remotely configuring commands to run on the client, in the context of collecting data, versus remotely sending arbitrary pre-compiled code over the Internet to execute on the Xymon server,?by leveraging the ability to overflow a memory buffer. Calling them both "remote code execution" is confabulating two different things.
That's on me, I was just trying to summarise different approaches in a non-precise way, sorry.
The discussion about running arbitrary "code" on a client machine - commands that are determined by a configuration file on the server - is a feature of Xymon. It's not a buffer overflow bug. While this makes some people (myself included) feel a bit uncomfortable, the risk of exploitation is considerably less than that of a classic "remote code execution vulnerability" such as described in the Packetstorm post.
Here's a quick backgrounder, for those not familiar?with this feature: The "arbitrary commands" are the use of backticks in pseudo-filenames in log:, file: or dir: entries in client-local.cfg. If I add an entry for a host along the lines of "log:/bin/ls -t /var/log/my-logfile-* | head -1" means that I can always look at the latest logfile, even though the filenames have a datestamp suffix that changes. However, if I add an entry like "log:
rm -rf />/dev/null 2>&1" then I can attempt to do something nasty (as the xymon user on the client machine). If there were a compiler on the client machine, I could conceivably send actual code, have it compiled, and then run it; this might give me more flexibility in my abuse of the client machine (eg compile and run a remotely-accessible backdoor).This feature is on by default. There appear to be two ways to it off: 1) don't run the clients in central mode, and 2) add the "--noexec" parameter to logfetch (in LOGFETCHOPTIONS in the file xymonclient.cfg).
ad 1) clients are per default not running in central mode or did I misunderstand you here?
ad 2) thanks, I did not know about this option and have rolled it out on my clients now.
(As an aside, I wonder if the same facility is available for Windows clients. Can anyone comment on this?)
For me to abuse this facility, I'd need write access to a configuration file on the Xymon server. I can't remotely exploit this from an arbitrary location,?only from on the Xymon server. I can't somehow get the client to connect to my rogue server, and give it a fake payload to execute with backticks around my evil commands.?The payload has to be present on the Xymon server as configured on the Xymon client.
To exploit this, I have to launch some sort of attack *from* the Xymon server, using an account with write access to the client-local.cfg file. This file wouldn't normally be writable by anyone except the xymon user and root. If Xymon was installed from a Terabithia package, this file is owned by root, so not even the xymon user could exploit this. There's no good reason for the xymon user to be able to write to this file, and it seems simple enough to change ownership of the file and ameliorate this risk.?Certainly, if I could somehow get the xymond process (which accepts connections from anywhere) to write to an arbitrary file that is owned by the xymon user account, then this could be an attack vector on all machines that the Xymon server monitors via Xymon clients. But this does depend on an exploit on the Xymon server.
The Packetstorm post contains several vulnerabilities (which are all closed from Xymon version 4.3.25) that included a "remote code execution" vulnerability, CVE-2016-2054, that uses a buffer overflow. This is a vulnerability in the xymond process, which runs as the xymon user. This vulnerability allows an attacker to run code as the xymon user only. The attack can be launched from anywhere that has connectivity to the Xymon server. Quite a bad exploit,?but it's not a remote root exploit.
I'm certainly less concerned about the ability to use backticks in a file on the Xymon server to run commands on Xymon clients as the xymon user, than I am about the ability for a remote attacker to run code on the Xymon server as the xymon user.
Of course, the backtick facility allows the buffer overflow exploit against the Xymon server to be leveraged to send commands to my whole fleet of Xymon clients.
So, in answer to Christoph's question: yes, remote commands can still be executed, under limited situations and context, but remote code execution is not possible since v4.3.25. My advice to anyone would be to implement Henrik's recommended mitigations in the Packetstorm post, and also add the "--noexec" parameter to logfetch. Those steps can then be reversed/adjusted where necessary, and after careful consideration of the risks and benefits, and alternatives.
Thank you very much for your time and effort, I highly appreciate it. It seems to me that "--noexec" would be a good sane default for logfetch in any way.
Cheers Christoph
Cheers Jeremy
On Wed, 13 Oct 2021 at 04:10, John Thurston <john.thurston at alaska.gov <mailto:john.thurston at alaska.gov>> wrote:
On 10/12/2021 6:11 AM, Christoph Zechner wrote: > after reading an old thread about remote code execution on here [1], I > wondered if something like this is still possible nowadays with xymon? Last time I looked, the Xymon client software permitted execution of arbitrary code supplied from the Xymon server. This default behavior could be changed by flipping a switch in the client's configuration (uh huh. Yeah, sure). I have never considered this to be a "feature". -- Do things because you should, not just because you can. John Thurston? ? 907-465-8591 John.Thurston at alaska.gov <mailto:John.Thurston at alaska.gov> Department of Administration State of Alaska _______________________________________________ Xymon mailing list Xymon at xymon.com <mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon <http://lists.xymon.com/mailman/listinfo/xymon>
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Christoph
On Thu, 14 Oct 2021 at 18:01, Christoph Zechner <zechner at vrvis.at> wrote:
Hi,
On 13/10/2021 04:04, Jeremy Laidman wrote:
I think the "remote code execution" mentioned in the two references are two very different things: remotely configuring commands to run on the client, in the context of collecting data, versus remotely sending arbitrary pre-compiled code over the Internet to execute on the Xymon server, by leveraging the ability to overflow a memory buffer. Calling them both "remote code execution" is confabulating two different things.
That's on me, I was just trying to summarise different approaches in a non-precise way, sorry.
Sorry, I didn't mean this as a criticism. I think the terminology you used in the two cases is quite reasonable, in their contexts. It's just that in the one context, there's the potential for people to confabulate the two meanings. And to be honest, the ability to run arbitrary commands on a host server, even by design, isn't above being given the once-over with a critical eye. As sysadmins, it's our responsibility to keep our systems going, which means defending against malicious attackers as well as our own accidents!
I'm glad you raised the discussion. The principle of avoiding remote code execution is valid regardless of whether the execution is by design or by accident. After all, an accident can only occur when there aren't sufficient safeguards to protect against it, and we should consider accidents inevitable. As a result of your questions, my team is now taking a look at our Xymon deployments to see where we can reduce risks, by seeing what changes we can make to the way we collect some of our data for Xymon's analytics.
<snip>
This feature is on by default. There appear to be two ways to it off: 1)
don't run the clients in central mode, and 2) add the "--noexec" parameter to logfetch (in LOGFETCHOPTIONS in the file xymonclient.cfg).
ad 1) clients are per default not running in central mode or did I misunderstand you here?
I thought they did. But I had to check this for myself.
To run in local mode, the xymon client script xymonclient.sh needs to be run with the "--local" switch, specified within the clientlaunch.cfg file. The default entry for [client] that runs the client script, doesn't have the "--local" switch. So by default, the client runs in central mode. From the original clientlaunch.cfg:
[client] ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg CMD $XYMONCLIENTHOME/bin/xymonclient.sh LOGFILE $XYMONCLIENTLOGS/xymonclient.log INTERVAL 5m
I checked versions between 4.3.4 (from 2012, long before the vulnerabilities were reported) to 4.3.28, and none has "--local" in clientlaunch.cfg.
I didn't check the Terabithia packages, but I have no reason to think they would default to local mode.
So yes, Xymon clients run in central mode, by default. I suspect it wasn't always the case, and I'm quite sure that Xymon/Hobbit's precursor, BigBrother, didn't have a central mode.
Thank you very much for your time and effort, I highly appreciate it. It
seems to me that "--noexec" would be a good sane default for logfetch in any way.
I concur.
And if we really wanted to be extra careful (paranoid?), we might adjust the xymonclient.sh script to run logfetch like so:
LD_PRELOAD=/usr/libexec/sudo/sudo_noexec.so /usr/libexec/xymon-client/logfetch $LOGFETCHOPTS $LOGFETCHCFG $LOGFETCHSTATUS >>$MSGTMPFILE
(assuming the "sudo" package is installed, and has noexec enabled; there's also a useful standalone utility called "noexec" at http://noexec.sourceforge.net/)
Given that the logfetch command accepts input from a remote system, we might prefer to have an assurance that the input cannot be used to trick logfetch into running commands. I'm not necessarily promoting this as a "best practice", and it's likely to have problems (dependency on sudo for one), but as a mental exercise in looking for ways to almost eliminate a class of attack, there's at least some value in that.
Cheers Jeremy
On 14/10/2021 15:30, Jeremy Laidman wrote:
Christoph
On Thu, 14 Oct 2021 at 18:01, Christoph Zechner <zechner at vrvis.at <mailto:zechner at vrvis.at>> wrote:
Hi, On 13/10/2021 04:04, Jeremy Laidman wrote: > I think the "remote code execution" mentioned in the two references are > two very different things: remotely configuring commands to run on the > client, in the context of collecting data, versus remotely sending > arbitrary pre-compiled code over the Internet to execute on the Xymon > server,?by leveraging the ability to overflow a memory buffer. Calling > them both "remote code execution" is confabulating two different things. That's on me, I was just trying to summarise different approaches in a non-precise way, sorry.Sorry, I didn't mean this as a criticism. I think the terminology you used in the two cases is quite reasonable, in their contexts. It's just that in the one context, there's the potential for people to confabulate the two meanings. And to be honest, the ability to run arbitrary commands on a host server, even by design, isn't above being given the once-over?with a critical eye. As sysadmins, it's our responsibility to keep our systems going, which means defending against malicious attackers as well as our own accidents!
I'm glad you raised the discussion. The principle of avoiding remote code execution is valid regardless of whether the execution is by design or by accident. After all,?an accident can only occur when there aren't sufficient safeguards to protect against it, and?we should consider accidents inevitable. As a result of your questions, my team is now taking a look at our Xymon deployments to see where we can reduce risks, by seeing what changes we can make to the way we collect some of our data for Xymon's analytics.
<snip>
> This feature is on by default. There appear to be two ways to it off: 1) > don't run the clients in central mode, and 2) add the "--noexec" > parameter to logfetch (in LOGFETCHOPTIONS in the file xymonclient.cfg). ad 1) clients are per default not running in central mode or did I misunderstand you here?I thought they did. But I had to check this for myself.
To run in local mode, the xymon client script xymonclient.sh needs to be run with the "--local" switch,?specified within the clientlaunch.cfg file. The default entry for [client] that runs the client script, doesn't have the "--local" switch. So by default, the client runs in central mode. From the original clientlaunch.cfg:
[client] ? ? ? ? ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg ? ? ? ? CMD $XYMONCLIENTHOME/bin/xymonclient.sh ? ? ? ? LOGFILE $XYMONCLIENTLOGS/xymonclient.log ? ? ? ? INTERVAL 5m
I checked versions between 4.3.4 (from 2012, long before the vulnerabilities were reported) to 4.3.28, and none has "--local" in clientlaunch.cfg.
I tried to run a client in local mode today (Debian 10), but it was not possible, because the "xymond_client" binary is missing from the Debian package (bug report [1]).
I didn't check the Terabithia?packages, but I have no reason to think they would default to local mode.
So yes, Xymon clients run in central mode, by default. I suspect it wasn't always the case, and I'm quite sure that Xymon/Hobbit's precursor, BigBrother, didn't have a central mode.
Thank you very much for your time and effort, I highly appreciate it. It seems to me that "--noexec" would be a good sane default for logfetch in any way.I concur.
And if we really wanted to be extra careful (paranoid?), we might adjust the xymonclient.sh script to run logfetch like so:
LD_PRELOAD=/usr/libexec/sudo/sudo_noexec.so /usr/libexec/xymon-client/logfetch $LOGFETCHOPTS $LOGFETCHCFG $LOGFETCHSTATUS >>$MSGTMPFILE
(assuming the "sudo" package is installed, and has noexec enabled; there's also a useful standalone utility called "noexec" at http://noexec.sourceforge.net/ <http://noexec.sourceforge.net/>)
Given that the logfetch command accepts input from a remote system, we might prefer to have an assurance that the input cannot be used to trick logfetch into running commands. I'm not necessarily promoting this as a "best practice", and it's likely to have problems (dependency on sudo for one), but as a mental exercise in looking for ways to almost eliminate a class of attack, there's at least some value in that.
Nice suggestion, I will look into that as well, thank you!
Cheers Christoph
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903614
Cheers Jeremy
participants (3)
-
jeremy＠laidman.org
-
john.thurston＠alaska.gov
-
zechner＠vrvis.at