I'm not successful filtering on the connection state associated with a port. None of the syntax variations I have tried have been successful. If I remove the STATE specifier, matches are found.
There are multiple hosts connecting to the same port:
ESTAB 0 0 10.160.8.130:61617 10.160.8.132:57765 ESTAB 0 0 10.160.8.130:61617 10.160.8.132:57766 ESTAB 0 0 10.160.8.130:61617 10.160.8.132:57768 ESTAB 0 0 10.160.8.130:61617 10.160.8.133:45096 ESTAB 0 0 10.160.8.130:61617 10.160.8.133:45104 ESTAB 0 0 10.160.8.130:61617 10.160.8.133:45107 ESTAB 0 0 10.160.8.130:61617 130.118.4.2:36141 ESTAB 0 0 10.160.8.130:61617 130.118.4.2:36150 ESTAB 0 0 10.160.8.130:61617 130.118.4.2:36151 ESTAB 0 0 10.160.8.130:61617 136.177.16.3:34320 ESTAB 0 0 10.160.8.130:61617 136.177.16.3:34321 ESTAB 0 0 10.160.8.130:61617 136.177.16.3:34324 ESTAB 0 0 10.160.8.130:61617 137.227.240.32:50726 ESTAB 0 0 10.160.8.130:61617 137.227.240.32:50727 ESTAB 0 0 10.160.8.130:61617 137.227.240.32:50729 LISTEN 0 0 *:61617 *:*
I've set up several port monitoring specifications, but none of them match the state (the first example where no state is specified succeeds):
PORT LOCAL=%: REMOTE=%10.160.8.132 MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-DHCP PORT LOCAL=%: REMOTE=%10.160.8.133 STATE=ESTABLISHED MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-nsp.er PORT LOCAL=%: REMOTE=%136.177.16.3 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.cr PORT LOCAL=%: REMOTE=%137.227.240.32 STATE=%ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.er PORT LOCAL=%: REMOTE=%130.118.4.2 STATE=%ESTAB* MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.wr
Note: On this server netstat does not exist and ss is being used,.
Observation: Discovering the syntax for REMOTE was trial and error. Specifying the IP address alone did not work, and I found no examples for the type of filtering above.
-- -- David Boldt <dboldt at usgs.gov>
"Discovery consists of seeing what everybody has seen and thinking what nobody has thought." --Albert Szent-Gyorgyi (1893 - 1986)
David
On Thu, Mar 10, 2016 at 1:15 AM Boldt, David <dboldt at usgs.gov> wrote:
I'm not successful filtering on the connection state associated with a port.
I couple of things to note, but first some explanation of how things work. The STATE and PORT rules are matched against the text in the [ports] section of the client data, which is generally the output of "netstat -na" in one form or another, but including only TCP sockets. The output of this command, and hence the required processing varies between operating systems, but in most cases the local IP and port is in column 3 (numbered from 0), the remote is in column 4 and the state is in column 5. (The exceptions to this are Solaris, using columns 0, 1 and 6 respectively; and Windows which uses columns 1, 2 and 3.)
The output of "ss" is most definitely different to any "netstat" that I've seen. In particular, the socket state is on column 0 - before either of the local or report addresses/ports. It might be helpful to know what OS you're using, that doesn't have a netstat command.
None of the syntax variations I have tried have been successful.
If I remove the STATE specifier, matches are found.
It's just a coincidence that the column numbers for local and remote are the same for netstat and ss. Had this not been the case, you'd have had more trouble getting matches even with STATE unspecified.
I've set up several port monitoring specifications, but none of them match the state
PORT LOCAL=%: REMOTE=%10.160.8.133 STATE=ESTABLISHED MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-nsp.er
The normal "netstat" output uses the string "ESTABLISHED". But "ss" uses "ESTAB". If [ports] shows ESTAB then that's what you need to match.
PORT LOCAL=%: REMOTE=%136.177.16.3 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.cr
Would work, if ESTAB was in the correct column.
PORT LOCAL=%: REMOTE=%137.227.240.32 STATE=%ESTAB MIN=3 MAX=3
I think this will match only when the state starts and ends with "ESTAB" and so it'd probably work, provided the state was in the first column.
COLOR=yellow TEXT=ActiveMQ-ns.er PORT LOCAL=%: REMOTE=%130.118.4.2 STATE=%ESTAB* MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.wr
The "%" marks the string that follows as a regular expression. The asterisk in a regular expression means to match zero or more of the previous character. So this would match "ESTAB" as well as "ESTA" and "ESTABB" and "ESTABBBBBB" but it wouldn't match "ESTABLISHED".
Note: On this server netstat does not exist and ss is being used,.
Yep, this is your problem. I'm assuming that you edited the xymonclient-<os>.sh script and replaced "netstat -na" with "ss"?
Observation: Discovering the syntax for REMOTE was trial and error.
Specifying the IP address alone did not work, and I found no examples for the type of filtering above.
The analysis.cfg file has several examples for the use of LOCAL. The REMOTE usage is the same.
So, what to do? The simplest thing would be to adjust the output of "ss" so that the columns match up. Simply replicating the state column would suffice. Something like so:
ss -nt | awk '{ print $0 $1 }'
Cheers Jeremy
On Mon, Mar 14, 2016 at 7:02 AM, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
David
On Thu, Mar 10, 2016 at 1:15 AM Boldt, David <dboldt at usgs.gov> wrote:
I'm not successful filtering on the connection state associated with a port.
I couple of things to note, but first some explanation of how things work. The STATE and PORT rules are matched against the text in the [ports] section of the client data, which is generally the output of "netstat -na" in one form or another, but including only TCP sockets. The output of this command, and hence the required processing varies between operating systems, but in most cases the local IP and port is in column 3 (numbered from 0), the remote is in column 4 and the state is in column 5. (The exceptions to this are Solaris, using columns 0, 1 and 6 respectively; and Windows which uses columns 1, 2 and 3.)
The output of "ss" is most definitely different to any "netstat" that I've seen. In particular, the socket state is on column 0 - before either of the local or report addresses/ports. It might be helpful to know what OS you're using, that doesn't have a netstat command.
Sapphire OS, a Linux derivative of unknown provenance, customized for use in IPAM/DHCP/DNS appliances sold by BT Diamond.
None of the syntax variations I have tried have been successful. If I remove the STATE specifier, matches are found.
It's just a coincidence that the column numbers for local and remote are the same for netstat and ss. Had this not been the case, you'd have had more trouble getting matches even with STATE unspecified.
I've set up several port monitoring specifications, but none of them match the state
PORT LOCAL=%: REMOTE=%10.160.8.133 STATE=ESTABLISHED MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-nsp.er
The normal "netstat" output uses the string "ESTABLISHED". But "ss" uses "ESTAB". If [ports] shows ESTAB then that's what you need to match.
PORT LOCAL=%: REMOTE=%136.177.16.3 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.cr
Would work, if ESTAB was in the correct column.
PORT LOCAL=%: REMOTE=%137.227.240.32 STATE=%ESTAB MIN=3 MAX=3
I think this will match only when the state starts and ends with "ESTAB" and so it'd probably work, provided the state was in the first column.
COLOR=yellow TEXT=ActiveMQ-ns.er PORT LOCAL=%: REMOTE=%130.118.4.2 STATE=%ESTAB* MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.wr
The "%" marks the string that follows as a regular expression. The asterisk in a regular expression means to match zero or more of the previous character. So this would match "ESTAB" as well as "ESTA" and "ESTABB" and "ESTABBBBBB" but it wouldn't match "ESTABLISHED".
Note: On this server netstat does not exist and ss is being used,.
I should have taken the documentation at its word, but was led astray by Internet examples which didn't make a lot of sense to me as regex expressions. Simplifying, now:.
PORT LOCAL=%:61617 REMOTE=%10.160.8.132 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-DHCP PORT LOCAL=%:61617 REMOTE=%10.160.8.133 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-nsp.er PORT LOCAL=%:61617 REMOTE=%136.177.16.3 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.cr PORT LOCAL=%:61617 REMOTE=%137.227.240.32 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.er PORT LOCAL=%:61617 REMOTE=%130.118.4.2 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.wr
and working
Yep, this is your problem. I'm assuming that you edited the xymonclient-<os>.sh script and replaced "netstat -na" with "ss"?
yes
Observation: Discovering the syntax for REMOTE was trial and error. Specifying the IP address alone did not work, and I found no examples for the type of filtering above.
The analysis.cfg file has several examples for the use of LOCAL. The REMOTE usage is the same.
ah, but none of the LOCAL examples specifies an IP address, only port distinctions.
So, what to do? The simplest thing would be to adjust the output of "ss" so that the columns match up. Simply replicating the state column would suffice. Something like so:
ss -nt | awk '{ print $0 $1 }'
Thanks for your thorough descriptions of the underlying process,and the example conversion.
-- -- David Boldt <dboldt at usgs.gov>
"... you cannot reason a person out of a position that the person has not been reasoned into." --Washington Post editorial comment on political positions
On Wed, Mar 16, 2016 at 10:00 AM Boldt, David <dboldt at usgs.gov> wrote:
It might be helpful to know what OS you're using, that doesn't have a netstat command.
Sapphire OS, a Linux derivative of unknown provenance, customized for use in IPAM/DHCP/DNS appliances sold by BT Diamond.
I'm somewhat familiar with the BT Diamond product, but not running on an appliance. It so annoys me when vendors do that. I cannot fathom why they think it's useful to anyone when they nobble an OS - I guess they call it "hardening". I suppose you're lucky that you can even get an agent installed and running on there.
I should have taken the documentation at its word, but was led astray
by Internet examples which didn't make a lot of sense to me as regex expressions. Simplifying, now:.
PORT LOCAL=%:61617 REMOTE=%10.160.8.132 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-DHCP
Yep, that should work (providing the STATE matches). Technically, a regular expression should be anchored at the beginning (with "^") and often at the end (with "$"), and special characters like "*" and "." should be escaped (with "\"), so as to ensure you don't get an accidental match. For example, your above line would match the IP address "210.160.8.132", and I'm confident you wouldn't want it to. If you need to match a whole string exactly, don't use the percent, something like "REMOTE=10.160.8.132". Otherwise, it's safest to anchor and escape; so to match a /24 subnet, you could do "REMOTE=%^10\.168\.8\."
A problem I once encountered is with a PROC test where I was monitoring the presence of the "named" process for BIND DNS, so I was matching "named" and requiring exactly one instance. An operator would occasionally run something like "tail /var/log/named" and this process string would also match, alerting that there were two named processes instead of one. Using "%named" was just as bad. I then switched to using a regular expression of "%^/usr/sbin/named", which improved things somewhat, but commands like "/usr/sbin/named-checkconf" also matched and caused alerts. After a few iterations, my PROC line is now:
PROC "%^/usr/sbin/named(\s*$|\s)" 1 1 "TEXT=/usr/sbin/named"This matches only when the command starts with /usr/sbin/named and either ends there (optionally with trailing spaces) or has a space (presumably followed by command-line arguments).
Anyhow, the moral of the story is, regular expressions should use adequate anchoring and escaping.
and working
I'm puzzled why this works if the STATE isn't in column 5, as is the case for the "ss" command. Unless you also modified the "ss" command to create column 5, as I suggested.
The analysis.cfg file has several examples for the use of LOCAL. The REMOTE usage is the same.
ah, but none of the LOCAL examples specifies an IP address, only port distinctions.
Except for this one:
Example: Check that ONLY ports 22, 80 and 443 are open for
incoming connections:
PORT STATE=LISTEN LOCAL=%0.0.0.0[.:].*
EXLOCAL=%.:$ MAX=0 "TEXT=Bad listeners"
Although this perhaps isn't the best example for what you were trying to do. (Not to mention that it didn't escape the dots, nor anchor. However in this example the matching happens to be unambiguous for the possible strings - it would be unusual to see any other IP address matching the regexp in the example.)
Nevertheless, it looks like you have a solution. It's always nice to see Xymon working with appliances - despite the best efforts of the vendors.
Cheers Jeremy
On Tue, Mar 15, 2016 at 7:58 PM, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
On Wed, Mar 16, 2016 at 10:00 AM Boldt, David <dboldt at usgs.gov> I'm somewhat familiar with the BT Diamond product, but not running on an appliance. It so annoys me when vendors do that. I cannot fathom why they think it's useful to anyone when they nobble an OS - I guess they call it "hardening". I suppose you're lucky that you can even get an agent installed and running on there.
It's a testament to xymon that I was able to extract a tar file into an arbitrary directory and have it run.
I should have taken the documentation at its word, but was led astray by Internet examples which didn't make a lot of sense to me as regex expressions. Simplifying, now:.
PORT LOCAL=%:61617 REMOTE=%10.160.8.132 STATE=ESTAB MIN=3 MAX=3 COLOR=yellow TEXT=ActiveMQ-DHCP
Yep, that should work (providing the STATE matches). Technically, a regular expression should be anchored at the beginning (with "^") and often at the end (with "$"), and special characters like "*" and "." should be escaped (with "\"), so as to ensure you don't get an accidental match. For example, your above line would match the IP address "210.160.8.132", and I'm confident you wouldn't want it to. If you need to match a whole string exactly, don't use the percent, something like "REMOTE=10.160.8.132". Otherwise, it's safest to anchor and escape; so to match a /24 subnet, you could do "REMOTE=%^10\.168\.8\."
A problem I once encountered is with a PROC test where I was monitoring the presence of the "named" process for BIND DNS, so I was matching "named" and requiring exactly one instance. An operator would occasionally run something like "tail /var/log/named" and this process string would also match, alerting that there were two named processes instead of one. Using "%named" was just as bad. I then switched to using a regular expression of "%^/usr/sbin/named", which improved things somewhat, but commands like "/usr/sbin/named-checkconf" also matched and caused alerts. After a few iterations, my PROC line is now:
PROC "%^/usr/sbin/named(\s*$|\s)" 1 1 "TEXT=/usr/sbin/named"This matches only when the command starts with /usr/sbin/named and either ends there (optionally with trailing spaces) or has a space (presumably followed by command-line arguments).
Anyhow, the moral of the story is, regular expressions should use adequate anchoring and escaping.
and working
I'm puzzled why this works if the STATE isn't in column 5, as is the case for the "ss" command. Unless you also modified the "ss" command to create column 5, as I suggested.
Very close to your recommendation, modified slightly to pick up LISTEN states, xymonclient-linux.sh using this for [ports]:
ss -ant | awk '{ print $0 $1 }'
The analysis.cfg file has several examples for the use of LOCAL. The REMOTE usage is the same.
ah, but none of the LOCAL examples specifies an IP address, only port distinctions.
Except for this one:
Example: Check that ONLY ports 22, 80 and 443 are open for incoming connections:
PORT STATE=LISTEN LOCAL=%0.0.0.0[.:].* EXLOCAL=%.:$ MAX=0 "TEXT=Bad listeners"
Although this perhaps isn't the best example for what you were trying to do. (Not to mention that it didn't escape the dots, nor anchor. However in this example the matching happens to be unambiguous for the possible strings - it would be unusual to see any other IP address matching the regexp in the example.)
Nevertheless, it looks like you have a solution. It's always nice to see Xymon working with appliances - despite the best efforts of the vendors.
indeed.
Thanks once again for all of your help.
-- -- David Boldt <dboldt at usgs.gov>
"... you cannot reason a person out of a position that the person has not been reasoned into." --Washington Post editorial comment on political positions
participants (2)
-
dboldt＠usgs.gov
-
jlaidman＠rebel-it.com.au