Hi,
We use powershell client on computers with localized windows (Czech language). EventLog messages are in Czech with non ASCII characters. I modified the winpsclient to send clientlog in UTF8. Messages are displayed on xymon web correctly, I was able filter messages, e.g.
LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
Exists some pitfalls, why not send clientlog in UTF8, but convert into US- ASCII?
Luboš Pavlíček
Hi Luboš
I think you’re talking about the ASCII encoding in function XymonSend.
I don’t know the reason for choosing ASCII encoding, that code was written before I was involved.
If the server can accept UTF-8 on all platforms then there is no reason why it should not be changed – maybe one of the server developers (Japheth ?) could comment?
Zak
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Luboš Pavlícek Sent: 22 March 2017 08:56 To: xymon at xymon.com Subject: [Xymon] Clientlog in UTF8
Hi, We use powershell client on computers with localized windows (Czech language). EventLog messages are in Czech with non ASCII characters. I modified the winpsclient to send clientlog in UTF8. Messages are displayed on xymon web correctly, I was able filter messages, e.g.
LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
Exists some pitfalls, why not send clientlog in UTF8, but convert into US-ASCII? Luboš Pavlíček
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
On 22/3/17 9:03 pm, zak.beck at accenture.com wrote:
Hi Luboš
I think you’re talking about the ASCII encoding in function XymonSend.
I don’t know the reason for choosing ASCII encoding, that code was written before I was involved.
I wrote that code. From memory, the xymon server could only handle the clientlog message in ASCII at the time, and forcing the translation to ASCII was a key part to getting it working. If the constraint of ASCII clientlog message has changed, then by all means feel free to convert to UTF8, however some users may still be running old servers that can only handle ASCII, so best to leave it configurable.
You'd need to test that all client-based tests (DISK, SVC, PORT, PROC, etc) are still working not just eventlog message matching.
If the server can accept UTF-8 on all platforms then there is no reason why it should not be changed – maybe one of the server developers (Japheth ?) could comment?
As above, need to verify server can handle UTF8. I suspect the clientlog message must be all one encoding - can't just do eventlog section in UTF8 and the rest in ASCII.
David.
Zak
*From:*Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Luboš Pavlícek *Sent:* 22 March 2017 08:56 *To:* xymon at xymon.com *Subject:* [Xymon] Clientlog in UTF8
Hi,
We use powershell client on computers with localized windows (Czech language). EventLog messages are in Czech with non ASCII characters. I modified the winpsclient to send clientlog in UTF8. Messages are displayed on xymon web correctly, I was able filter messages, e.g.
LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
That generally happens when someone leaves a remote session logged on and then changes their password elsewhere. ;)
Exists some pitfalls, why not send clientlog in UTF8, but convert into US-ASCII?
Luboš Pavlíček
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
-- David Baldwin - Senior Systems Administrator (Datacentres + Networks) Digital Information Management and Technology Australian Sports Commission http://ausport.gov.au Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616 david.baldwin at ausport.gov.au 1 Leverrier Street Bruce ACT 2617 Our Values: RESPECT + INTEGRITY + TEAMWORK + EXCELLENCE
Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au
This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.
Hi David
Ø some users may still be running old servers that can only handle ASCII, so best to leave it configurable
Thanks for your input – I was thinking the same as the above.
Zak
From: David Baldwin [mailto:david.baldwin at ausport.gov.au] Sent: 22 March 2017 22:11 To: Beck, Zak <zak.beck at accenture.com>; lubos.pavlicek at gmail.com; xymon at xymon.com Subject: Re: [Xymon] Clientlog in UTF8
On 22/3/17 9:03 pm, zak.beck at accenture.com<mailto:zak.beck at accenture.com> wrote: Hi Luboš
I think you’re talking about the ASCII encoding in function XymonSend.
I don’t know the reason for choosing ASCII encoding, that code was written before I was involved. I wrote that code. From memory, the xymon server could only handle the clientlog message in ASCII at the time, and forcing the translation to ASCII was a key part to getting it working. If the constraint of ASCII clientlog message has changed, then by all means feel free to convert to UTF8, however some users may still be running old servers that can only handle ASCII, so best to leave it configurable.
You'd need to test that all client-based tests (DISK, SVC, PORT, PROC, etc) are still working not just eventlog message matching.
If the server can accept UTF-8 on all platforms then there is no reason why it should not be changed – maybe one of the server developers (Japheth ?) could comment?
As above, need to verify server can handle UTF8. I suspect the clientlog message must be all one encoding - can't just do eventlog section in UTF8 and the rest in ASCII.
David.
Zak
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Luboš Pavlícek Sent: 22 March 2017 08:56 To: xymon at xymon.com<mailto:xymon at xymon.com> Subject: [Xymon] Clientlog in UTF8
Hi, We use powershell client on computers with localized windows (Czech language). EventLog messages are in Czech with non ASCII characters. I modified the winpsclient to send clientlog in UTF8. Messages are displayed on xymon web correctly, I was able filter messages, e.g.
LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red That generally happens when someone leaves a remote session logged on and then changes their password elsewhere. ;)
Exists some pitfalls, why not send clientlog in UTF8, but convert into US-ASCII? Luboš Pavlíček
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com<http://www.accenture.com>
Xymon mailing list
Xymon at xymon.com<mailto:Xymon at xymon.com>
--
David Baldwin - Senior Systems Administrator (Datacentres + Networks)
Digital Information Management and Technology
Tel 02 62147830 Fax 02 62141830 PO Box 176 Belconnen ACT 2616
david.baldwin at ausport.gov.au<mailto:david.baldwin at ausport.gov.au> 1 Leverrier Street Bruce ACT 2617
Our Values: RESPECT + INTEGRITY + TEAMWORK + EXCELLENCE
Keep up to date with what's happening in Australian sport visit www.ausport.gov.au<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ausport.gov.au&d=DwMDaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=56bdErr67jxvsqigxSp9k6anqyhtUKraJdOpHMsJlUI&s=XVgugJurmNjsulR2NzV0Jh7d_xw-_bOiWAWtWA5DxYo&e=>
This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.
Hi,
Thank you for your answers.
2017-03-22 23:11 GMT+01:00 David Baldwin <david.baldwin at ausport.gov.au>:
On 22/3/17 9:03 pm, zak.beck at accenture.com wrote:
Hi Luboš
I think you’re talking about the ASCII encoding in function XymonSend.
I don’t know the reason for choosing ASCII encoding, that code was written before I was involved.
I wrote that code. From memory, the xymon server could only handle the clientlog message in ASCII at the time, and forcing the translation to ASCII was a key part to getting it working. If the constraint of ASCII clientlog message has changed, then by all means feel free to convert to UTF8, however some users may still be running old servers that can only handle ASCII, so best to leave it configurable.
You'd need to test that all client-based tests (DISK, SVC, PORT, PROC, etc) are still working not just eventlog message matching.
I try multiple tests with client messages in UTF8. First, I will describe our environment. We use two Xymon servers (versions 4.3.25, 4.3.27), both are running on Linux boxes (CentOS, Debian) with LANG=en_US.UTF-8. Apache web server has set the default encoding to UTF-8.
Client tests DISK, UP,MEMHYS,MEMSWAP and MEMACT are without problems, because corresponding sections from clientlog are in English. CLOCK test are working correctly, because first line "epoch: <number>" from [clock] section isn't localized.
Tests LOG, PROC, SVC and PORT successfully filter non-ASCII strings. Examples:
LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
PROC "%Nový textový dokument"
PORT "LOCAL=%(:445)$" state=NASLOUCHÁNÍ TEXT=sdílení TRACK
PORT "REMOTE=%([.:]445)$" state=NAVÁZÁNO min=1 max=10 color=yellowTRACK=smb_servers TEXT=SMB_servers SVC "Xymončesky" status=started
FILE and DIR: I successfully test file and directory with non-ASCII chars in filename. File client-local.cfg on xymon server contains next lines:
file:d:\temp\čeština.txt dir:d:\temp\český
and file analysis.cfg contains tests:
FILE "D:\temp\čeština.txt" TYPE=file SIZE>100 MTIME<180 yellow TRACK DIR "D:\temp\český" SIZE<1000 TRACK
Bad filenames are in graph: "D:<tab>emp\čeština.txt" and "D:<tab>temp\český". But this si the slash problem.
Xymon server don't create some graph, e.g. netstat, user. This is a result of localized messages in output from windows programs netstat, ipconfig or qwinsta. See http://lists.xymon.com/archive/2014-January/038946.html and following discussion.
Attached patch adds local setting XymonAcceptUTF8. If value set to 1, then all client messages are converted to UTF8 before sending to xymon servers. If value set to 0 or not specified, then script removes diacritics (accents) from message, converts non-break spaces to normal spaces (question marks in eventlog summary) and converts to ASCII before sending to xymon server. The same setting is applied to all specified xymon servers.
Luboš Pavlíček
Hi Luboš
Thanks for your patch.
I will test it in my environment – I don’t expect any issues, it looks sensible – and then update the documentation and release – this will probably be next week.
The UTF8 option will be “at your own risk” as I don’t have any non-EN platforms to test on – as you have said, some graphs and maybe tests won’t work due to localisation, but I think these wouldn’t work even with ASCII encoding – e.g. netstat would return localised messages regardless of UTF8/ASCII.
Thanks
Zak
From: Luboš Pavlíček [mailto:lubos.pavlicek at gmail.com] Sent: 23 March 2017 17:00 To: David Baldwin <david.baldwin at ausport.gov.au> Cc: Beck, Zak <zak.beck at accenture.com>; xymon at xymon.com Subject: Re: [Xymon] Clientlog in UTF8
Hi,
Thank you for your answers.
2017-03-22 23:11 GMT+01:00 David Baldwin <david.baldwin at ausport.gov.au<mailto:david.baldwin at ausport.gov.au>>: On 22/3/17 9:03 pm, zak.beck at accenture.com<mailto:zak.beck at accenture.com> wrote: Hi Luboš
I think you’re talking about the ASCII encoding in function XymonSend.
I don’t know the reason for choosing ASCII encoding, that code was written before I was involved. I wrote that code. From memory, the xymon server could only handle the clientlog message in ASCII at the time, and forcing the translation to ASCII was a key part to getting it working. If the constraint of ASCII clientlog message has changed, then by all means feel free to convert to UTF8, however some users may still be running old servers that can only handle ASCII, so best to leave it configurable.
You'd need to test that all client-based tests (DISK, SVC, PORT, PROC, etc) are still working not just eventlog message matching.
I try multiple tests with client messages in UTF8. First, I will describe our environment. We use two Xymon servers (versions 4.3.25, 4.3.27), both are running on Linux boxes (CentOS, Debian) with LANG=en_US.UTF-8. Apache web server has set the default encoding to UTF-8.
Client tests DISK, UP,MEMHYS,MEMSWAP and MEMACT are without problems, because corresponding sections from clientlog are in English. CLOCK test are working correctly, because first line "epoch: <number>" from [clock] section isn't localized.
Tests LOG, PROC, SVC and PORT successfully filter non-ASCII strings. Examples:
LOG eventlog_System "%Zpracování zásad skupiny selhalo" COLOR=red
PROC "%Nový textový dokument"
PORT "LOCAL=%(:445)$" state=NASLOUCHÁNÍ TEXT=sdílení TRACK
PORT "REMOTE=%([.:]445)$" state=NAVÁZÁNO min=1 max=10 color=yellow TRACK=smb_servers TEXT=SMB_servers
SVC "Xymončesky" status=startedFILE and DIR: I successfully test file and directory with non-ASCII chars in filename. File client-local.cfg on xymon server contains next lines:
file:d:\temp\čeština.txt dir:d:\temp\český
and file analysis.cfg contains tests:
FILE "D:\temp\čeština.txt" TYPE=file SIZE>100 MTIME<180 yellow TRACK DIR "D:\temp\český" SIZE<1000 TRACK Bad filenames are in graph: "D:<tab>emp\čeština.txt" and "D:<tab>temp\český". But this si the slash problem.
Xymon server don't create some graph, e.g. netstat, user. This is a result of localized messages in output from windows programs netstat, ipconfig or qwinsta. See http://lists.xymon.com/archive/2014-January/038946.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_archive_2014-2DJanuary_038946.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Sa_ZLbF2_4uGXImwd3VJ8TZDXxPc282dRhwIiR9qnZE&s=_TGPeWA6RPMQx1TMlXKM-GfN6ER62LiEG0VfTAILFqM&e=> and following discussion.
Attached patch adds local setting XymonAcceptUTF8. If value set to 1, then all client messages are converted to UTF8 before sending to xymon servers. If value set to 0 or not specified, then script removes diacritics (accents) from message, converts non-break spaces to normal spaces (question marks in eventlog summary) and converts to ASCII before sending to xymon server. The same setting is applied to all specified xymon servers.
Luboš Pavlíček
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
participants (3)
-
david.baldwin＠ausport.gov.au
-
lubos.pavlicek＠gmail.com
-
zak.beck＠accenture.com