server-side message pre-processor?
Is there a mechanism through which I can dump certain host+test combinations to /dev/null ?
By way of example, let's hypothetically say I had a set of hosts whose owner steadfastly refused to clean up their data streams. They might be sending thousands of garbage test results to my server. The xymon server might be defined to not propagate the test results to the non-green screen, and there may be no alerts defined for the host+test combinations. But the xymon server is still passing thousands of garbage messages through its parsing, evaluating, graphing, and alerting handlers. Xymongen is still including these ignored messages in the list of "events received" at the bottom of the non-green page.
It would be cool if there were a way to create a flush-list for which: This host+test message combination will always be flushed. It will never get past the parsing handler. It won't be recorded. It wont be reported. It will not pass "go". It will not collect $200. It will just disappear.
Alternatively, the ability to prevent a host+test combination appearing on the "events received int the past x minutes" portion of the non-green page would be a big help. In the hypothetical situation outlined above, the thousands of messages from those hosts would push every other meaningful message off the bottom of that list of events.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
Can't you just set up iptables to drop the traffic so it never actually reaches the server? Something like
iptables -A INPUT -s 12.34.56.78 -p tcp --destination-port 1984 -j DROP
=G=
From: Xymon <xymon-bounces at xymon.com> on behalf of John Thurston <john.thurston at alaska.gov> Sent: Monday, November 2, 2015 7:59 PM To: xymon at xymon.com Subject: [Xymon] server-side message pre-processor?
Is there a mechanism through which I can dump certain host+test combinations to /dev/null ?
By way of example, let's hypothetically say I had a set of hosts whose owner steadfastly refused to clean up their data streams. They might be sending thousands of garbage test results to my server. The xymon server might be defined to not propagate the test results to the non-green screen, and there may be no alerts defined for the host+test combinations. But the xymon server is still passing thousands of garbage messages through its parsing, evaluating, graphing, and alerting handlers. Xymongen is still including these ignored messages in the list of "events received" at the bottom of the non-green page.
It would be cool if there were a way to create a flush-list for which: This host+test message combination will always be flushed. It will never get past the parsing handler. It won't be recorded. It wont be reported. It will not pass "go". It will not collect $200. It will just disappear.
Alternatively, the ability to prevent a host+test combination appearing on the "events received int the past x minutes" portion of the non-green page would be a big help. In the hypothetical situation outlined above, the thousands of messages from those hosts would push every other meaningful message off the bottom of that list of events.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
I guess removing these hosts entirely from hosts.cfg not possible because there are _some_ metrics from the evildoers you are interested in? (Which would rule out the iptables solution suggested by Galen)
To my knowledge there is no such mechanism you are looking for.
Thomas
On Nov 3, 2015 3:03 AM, Galen Johnson <Galen.Johnson at sas.com> wrote:
Can't you just set up iptables to drop the traffic so it never actually reaches the server? Something like
iptables -A INPUT -s 12.34.56.78 -p tcp --destination-port 1984 -j DROP
=G=
From: Xymon <xymon-bounces at xymon.com> on behalf of John Thurston <john.thurston at alaska.gov> Sent: Monday, November 2, 2015 7:59 PM To: xymon at xymon.com Subject: [Xymon] server-side message pre-processor?
Is there a mechanism through which I can dump certain host+test combinations to /dev/null ?
By way of example, let's hypothetically say I had a set of hosts whose owner steadfastly refused to clean up their data streams. They might be sending thousands of garbage test results to my server. The xymon server might be defined to not propagate the test results to the non-green screen, and there may be no alerts defined for the host+test combinations. But the xymon server is still passing thousands of garbage messages through its parsing, evaluating, graphing, and alerting handlers. Xymongen is still including these ignored messages in the list of "events received" at the bottom of the non-green page.
It would be cool if there were a way to create a flush-list for which: This host+test message combination will always be flushed. It will never get past the parsing handler. It won't be recorded. It wont be reported. It will not pass "go". It will not collect $200. It will just disappear.
Alternatively, the ability to prevent a host+test combination appearing on the "events received int the past x minutes" portion of the non-green page would be a big help. In the hypothetical situation outlined above, the thousands of messages from those hosts would push every other meaningful message off the bottom of that list of events.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Hypothetically, I'd have an alert that sends the test right back to the owner's email repeated every minute.
That or I'd use iptables to reject any message from the machine.
The problem isn't the data, it's the people. The fix needs to come there, everything else is masking a symptom not treating the problem.
-----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of John Thurston Sent: Monday, November 02, 2015 7:00 PM To: xymon at xymon.com Subject: [Xymon] server-side message pre-processor?
Is there a mechanism through which I can dump certain host+test combinations to /dev/null ?
By way of example, let's hypothetically say I had a set of hosts whose owner steadfastly refused to clean up their data streams. They might be sending thousands of garbage test results to my server. The xymon server might be defined to not propagate the test results to the non-green screen, and there may be no alerts defined for the host+test combinations. But the xymon server is still passing thousands of garbage messages through its parsing, evaluating, graphing, and alerting handlers. Xymongen is still including these ignored messages in the list of "events received" at the bottom of the non-green page.
It would be cool if there were a way to create a flush-list for which: This host+test message combination will always be flushed. It will never get past the parsing handler. It won't be recorded. It wont be reported. It will not pass "go". It will not collect $200. It will just disappear.
Alternatively, the ability to prevent a host+test combination appearing on the "events received int the past x minutes" portion of the non-green page would be a big help. In the hypothetical situation outlined above, the thousands of messages from those hosts would push every other meaningful message off the bottom of that list of events.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
Den 03-11-2015 01:59, John Thurston skrev:
Is there a mechanism through which I can dump certain host+test combinations to /dev/null ?
Remove the host from your hosts.cfg, then Xymon will ignore updates from it. Yes, it will still receive messages from it, but dropping them immediately when the hostname is not found doesn't cost much processing.
Too bad if you do want to monitor *some* elements, but dropping the host entirely from Xymon at least gives you some bargaining power towards the people who mis-manage their monitored server.
Firewalling can also be a solution, if the IP is unique.
Regards, Henrik
On 11/3/2015 6:11 AM, Henrik Størner wrote:
Den 03-11-2015 01:59, John Thurston skrev:
Is there a mechanism through which I can dump certain host+test combinations to /dev/null ?
Remove the host from your hosts.cfg, then Xymon will ignore updates from it. Yes, it will still receive messages from it, but dropping them immediately when the hostname is not found doesn't cost much processing.
Too bad if you do want to monitor *some* elements, but dropping the host entirely from Xymon at least gives you some bargaining power towards the people who mis-manage their monitored server.
Therein lies the rub. Xymon accepts all messages from a client, or it accepts none.
A host which has one business-critical test to report may also report an infinite number of bits of garbage. All of that garbage fills my logs and pushes everything else useful out of the "X events received in the past Y minutes" portion of the non-green screen. I beat, cajole, beg, but I can find no means by which to make an uncaring-host owner clean up their act.
re: Spam mail alerts back to the offending host owner That will just add load to my alert-handler and mailer. The recipient will create a mail-server-side rule to flush my noise to /dev/null. This is a very ineffective whip unless the target email audience is very large.
It would be cool if there were a per-host "accept" tag. I could stick it in a .default. line in hosts.cfg, "accept=disk,cpu,conn,http". Any other test reported for the host would be dropped.
The underlying problem is Xymon assumes a friendly environment. Some of my host-owners are not being friendly :(
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
On 11/3/2015 8:20 AM, John Thurston wrote:
- snip -
A host which has one business-critical test to report may also report an infinite number of bits of garbage. All of that garbage fills my logs and pushes everything else useful out of the "X events received in the past Y minutes" portion of the non-green screen. - snip -
A-ha! I found a slightly useful option to xymongen:
--eventignore=test[,test] Ignore these tests in the "All non-green" event log display.
The garbage will still arrive, and I'll have to handle it, but it won't be able to flush the rest of the useful messages off the event-list.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
On 11/3/2015 9:20 AM, John Thurston wrote:
On 11/3/2015 6:11 AM, Henrik Størner wrote:
Den 03-11-2015 01:59, John Thurston skrev:
Is there a mechanism through which I can dump certain host+test combinations to /dev/null ?
Remove the host from your hosts.cfg, then Xymon will ignore updates from it. Yes, it will still receive messages from it, but dropping them immediately when the hostname is not found doesn't cost much processing.
Too bad if you do want to monitor *some* elements, but dropping the host entirely from Xymon at least gives you some bargaining power towards the people who mis-manage their monitored server.
Therein lies the rub. Xymon accepts all messages from a client, or it accepts none.
A host which has one business-critical test to report may also report an infinite number of bits of garbage. All of that garbage fills my logs and pushes everything else useful out of the "X events received in the past Y minutes" portion of the non-green screen. I beat, cajole, beg, but I can find no means by which to make an uncaring-host owner clean up their act.
I guess one question would be whether these are being caused by poorly configured local settings, or by people coming up with new tests they're reporting in that you don't want to hear about?
Are you running in local config mode? If so, I might suggest using this as justification for migrating to central configuration. Put the thresholding control back into your hands unless the users are willing to accept responsibility for the types of things they're sending in.
re: Spam mail alerts back to the offending host owner That will just add load to my alert-handler and mailer. The recipient will create a mail-server-side rule to flush my noise to /dev/null. This is a very ineffective whip unless the target email audience is very large.
It would be cool if there were a per-host "accept" tag. I could stick it in a .default. line in hosts.cfg, "accept=disk,cpu,conn,http". Any other test reported for the host would be dropped.
At first, I was thinking that this would be a great use for an extension to xymonproxy, allowing it to function as a full-fledged filter of sorts -- expose it on your normal submission port and let it forward on a filtered view to xymond. The problem is that the more inspection xymonproxy does, the slower it's going to run. And this becomes more pronounced with higher volumes and when dealing with extcombo or compressed messages where we'd have to unpack arbitrary messages first and reconstruct things before sending them on.
A whitelisted accept/deny at the xymond level (passed as an --option on the command line) wouldn't be too difficult to implement, but dealing with host level configurations means more text scanning for each message coming in. On the plus side, xymond has to do that anyway since it's the final destination for all message types, but it's unfortunate that the message couldn't be blocked before then (network/tcp/cpu, etc).
Is the problem more along the lines of "I don't want to receive test 'xyz' on any host", or more "Here's a list of 38 different tests I want to reject on 18 / 400 servers". If it's the latter, a hosts.cfg text value (before an internal testname record is generated with which to assign accept/deny values to) might be the best option, performance concerns not withstanding.
The underlying problem is Xymon assumes a friendly environment. Some of my host-owners are not being friendly :(
Basically, yeah. There's a limit to how much can be done here without subverting the flexibility that's at the heart of how useful xymon can be. As a result, Xymon has a hosts.cfg, but no tests.cfg per se.
And even further-off features like SSL signing of messages (with a local cert controlled by appropriate unix privs) wouldn't alter the current assumption that someone with the authority to send one status message about a host had the authority to send any status message about that host.
As an initial step, I think adding hard-coded ignore-test records at xymond startup (by command option or by xymonserver.cfg) would probably be a pretty simple stop gap to create in the next rev.
HTH, -jc
On 11/3/2015 9:22 AM, Japheth Cleaver wrote:
- snip -
I guess one question would be whether these are being caused by poorly configured local settings, or by people coming up with new tests they're reporting in that you don't want to hear about?
These are poorly managed hosts whose owners see none of the cost of flooding the server with garbage. "What? Your stupid server can't handle my 1,000 events per host per day?! What kind of crappy system is that?!"
They are treating my Xymon server like it was an instance of Splunk. I keep trying to tell them it is an _alerting_ system, not a _logging_ system. If there is yellow on the page, they should be looking to see if there really is a problem. If there is a red on the page, they should be scrambling to fix the problem. Some of the owners are just ignoring all colors on the page until a customer calls with a problem. Then they are using the 'msgs' test result to see the most recent contents of the event-log.
Are you running in local config mode? If so, I might suggest using this as justification for migrating to central configuration. Put the thresholding control back into your hands unless the users are willing to accept responsibility for the types of things they're sending in.
But I don't know their business requirements ('course, it seems like they don't either). Nor do I have the time to personally handle configuration of 600+ hosts. Nor do I want to have access to those systems through the client distribution and management system.
Local-config mode works fine for us, except for the client's implicit ability to flood me with garbage. Its the "all or none" acceptance model which is painful.
- snip -
Is the problem more along the lines of "I don't want to receive test 'xyz' on any host", or more "Here's a list of 38 different tests I want to reject on 18 / 400 servers".
It's more the latter. "The owner of host=foo can't configure its client correctly. This has been going on for months. Fine. I will only accept disk and cpu from them."
I don't see the former condition as being realistic. Since any client can send a message containing any possible string as their test-name, it would impossible filter noise. Further, 90% of my system owners are politely handling their 'msgs' test. I don't want to punish those folks.
- snip -
As an initial step, I think adding hard-coded ignore-test records at xymond startup (by command option or by xymonserver.cfg) would probably be a pretty simple stop gap to create in the next rev.
I don't think this would be of any value to me. In general, I _want_ to accept messages of the name 'msgs'. There are just 10% of my hosts from which I want silence on messages of that type. The option of defining a per-host white-list is what I need.
But it is probably a very fringe need. I'll try getting my bat out again. Maybe I can beat some sense into the 10%.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
We are still trying to fix a HR issue with technology.
Your clients are misusing your system, and don't care. Go over their heads and make their management care.
-----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of John Thurston Sent: Tuesday, November 03, 2015 1:11 PM To: xymon at xymon.com Subject: Re: [Xymon] server-side message pre-processor?
On 11/3/2015 9:22 AM, Japheth Cleaver wrote:
- snip -
I guess one question would be whether these are being caused by poorly configured local settings, or by people coming up with new tests they're reporting in that you don't want to hear about?
These are poorly managed hosts whose owners see none of the cost of flooding the server with garbage. "What? Your stupid server can't handle my 1,000 events per host per day?! What kind of crappy system is that?!"
They are treating my Xymon server like it was an instance of Splunk. I keep trying to tell them it is an _alerting_ system, not a _logging_ system. If there is yellow on the page, they should be looking to see if there really is a problem. If there is a red on the page, they should be scrambling to fix the problem. Some of the owners are just ignoring all colors on the page until a customer calls with a problem. Then they are using the 'msgs' test result to see the most recent contents of the event-log.
Are you running in local config mode? If so, I might suggest using this as justification for migrating to central configuration. Put the thresholding control back into your hands unless the users are willing to accept responsibility for the types of things they're sending in.
But I don't know their business requirements ('course, it seems like they don't either). Nor do I have the time to personally handle configuration of 600+ hosts. Nor do I want to have access to those systems through the client distribution and management system.
Local-config mode works fine for us, except for the client's implicit ability to flood me with garbage. Its the "all or none" acceptance model which is painful.
- snip -
Is the problem more along the lines of "I don't want to receive test 'xyz' on any host", or more "Here's a list of 38 different tests I want to reject on 18 / 400 servers".
It's more the latter. "The owner of host=foo can't configure its client correctly. This has been going on for months. Fine. I will only accept disk and cpu from them."
I don't see the former condition as being realistic. Since any client can send a message containing any possible string as their test-name, it would impossible filter noise. Further, 90% of my system owners are politely handling their 'msgs' test. I don't want to punish those folks.
- snip -
As an initial step, I think adding hard-coded ignore-test records at xymond startup (by command option or by xymonserver.cfg) would probably be a pretty simple stop gap to create in the next rev.
I don't think this would be of any value to me. In general, I _want_ to accept messages of the name 'msgs'. There are just 10% of my hosts from which I want silence on messages of that type. The option of defining a per-host white-list is what I need.
But it is probably a very fringe need. I'll try getting my bat out again. Maybe I can beat some sense into the 10%.
-- Do things because you should, not just because you can.
John Thurston 907-465-8591 John.Thurston at alaska.gov Enterprise Technology Services Department of Administration State of Alaska
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
Hi,
Den 03-11-2015 kl. 19:22 skrev Japheth Cleaver:
On 11/3/2015 9:20 AM, John Thurston wrote:
It would be cool if there were a per-host "accept" tag. I could stick it in a .default. line in hosts.cfg, "accept=disk,cpu,conn,http". Any other test reported for the host would be dropped.
Is the problem more along the lines of "I don't want to receive test 'xyz' on any host", or more "Here's a list of 38 different tests I want to reject on 18 / 400 servers". If it's the latter, a hosts.cfg text value (before an internal testname record is generated with which to assign accept/deny values to) might be the best option, performance concerns not withstanding.
The attached patch against 4.x-master would do something like that - an "acceptonly:cpu,disk,conn,http" in hosts.cfg on the relevant hosts, and xymond will then match incoming tests against that entry and only process those that are permitted.
For hosts without an acceptonly-setting, the overhead should be fairly small.
(It compiles, but otherwise it is untested).
JC, feel free to veto it if you think it would be too much overhead.
Regards, Henrik
On 11/3/2015 2:20 PM, Henrik Størner wrote:
Hi,
Den 03-11-2015 kl. 19:22 skrev Japheth Cleaver:
On 11/3/2015 9:20 AM, John Thurston wrote:
It would be cool if there were a per-host "accept" tag. I could stick it in a .default. line in hosts.cfg, "accept=disk,cpu,conn,http". Any other test reported for the host would be dropped.
Is the problem more along the lines of "I don't want to receive test 'xyz' on any host", or more "Here's a list of 38 different tests I want to reject on 18 / 400 servers". If it's the latter, a hosts.cfg text value (before an internal testname record is generated with which to assign accept/deny values to) might be the best option, performance concerns not withstanding.
The attached patch against 4.x-master would do something like that - an "acceptonly:cpu,disk,conn,http" in hosts.cfg on the relevant hosts, and xymond will then match incoming tests against that entry and only process those that are permitted.
For hosts without an acceptonly-setting, the overhead should be fairly small.
(It compiles, but otherwise it is untested).
JC, feel free to veto it if you think it would be too much overhead.
The performance overhead is pretty small, as far as I can tell... It seems like it would fail with substrings after the first check, though (eg, "acceptonly:test123,test12,test1234" rejects 'test12').
How about this one?
-jc
Hi,
I think you are right about my first take not working if the first "strstr" wasn't the right match. Your version looks better.
Maybe John could take it for a spin?
/Henrik
Den 04-11-2015 kl. 01:46 skrev Japheth Cleaver:
On 11/3/2015 2:20 PM, Henrik Størner wrote:
Hi,
Den 03-11-2015 kl. 19:22 skrev Japheth Cleaver:
On 11/3/2015 9:20 AM, John Thurston wrote:
It would be cool if there were a per-host "accept" tag. I could stick it in a .default. line in hosts.cfg, "accept=disk,cpu,conn,http". Any other test reported for the host would be dropped.
Is the problem more along the lines of "I don't want to receive test 'xyz' on any host", or more "Here's a list of 38 different tests I want to reject on 18 / 400 servers". If it's the latter, a hosts.cfg text value (before an internal testname record is generated with which to assign accept/deny values to) might be the best option, performance concerns not withstanding.
The attached patch against 4.x-master would do something like that - an "acceptonly:cpu,disk,conn,http" in hosts.cfg on the relevant hosts, and xymond will then match incoming tests against that entry and only process those that are permitted.
For hosts without an acceptonly-setting, the overhead should be fairly small.
(It compiles, but otherwise it is untested).
JC, feel free to veto it if you think it would be too much overhead.
The performance overhead is pretty small, as far as I can tell... It seems like it would fail with substrings after the first check, though (eg, "acceptonly:test123,test12,test1234" rejects 'test12').
How about this one?
-jc
Another interesting thing was that none of the tests went purple... since even the "xymond" messages were getting blocked.
Running for a bit it seems to be working okay (with a fix for that); no real impact for hosts that don't have anything specified.
Seems useful to run an RC3 with it.
-jc
On 11/3/2015 10:53 PM, Henrik Størner wrote:
Hi,
I think you are right about my first take not working if the first "strstr" wasn't the right match. Your version looks better.
Maybe John could take it for a spin?
/Henrik
Den 04-11-2015 kl. 01:46 skrev Japheth Cleaver:
On 11/3/2015 2:20 PM, Henrik Størner wrote:
Hi,
Den 03-11-2015 kl. 19:22 skrev Japheth Cleaver:
On 11/3/2015 9:20 AM, John Thurston wrote:
It would be cool if there were a per-host "accept" tag. I could stick it in a .default. line in hosts.cfg, "accept=disk,cpu,conn,http". Any other test reported for the host would be dropped.
Is the problem more along the lines of "I don't want to receive test 'xyz' on any host", or more "Here's a list of 38 different tests I want to reject on 18 / 400 servers". If it's the latter, a hosts.cfg text value (before an internal testname record is generated with which to assign accept/deny values to) might be the best option, performance concerns not withstanding.
The attached patch against 4.x-master would do something like that - an "acceptonly:cpu,disk,conn,http" in hosts.cfg on the relevant hosts, and xymond will then match incoming tests against that entry and only process those that are permitted.
For hosts without an acceptonly-setting, the overhead should be fairly small.
(It compiles, but otherwise it is untested).
JC, feel free to veto it if you think it would be too much overhead.
The performance overhead is pretty small, as far as I can tell... It seems like it would fail with substrings after the first check, though (eg, "acceptonly:test123,test12,test1234" rejects 'test12').
How about this one?
-jc
participants (6)
-
cleaver＠terabithia.org
-
Galen.Johnson＠sas.com
-
henrik＠hswn.dk
-
john.thurston＠alaska.gov
-
Paul.Root＠CenturyLink.com
-
tom＠IT-Eckert.de