It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg: Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding 2019-01-29 11:43:06 Connecting to host x.x.x.x 2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
From: Xymon <xymon-bounces at xymon.com> On Behalf Of SebA Sent: Monday, 28 January 2019 20:16 To: Xymon Mailing List <xymon at xymon.com> Subject: [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:20:25 Connecting to host x.x.x.x <-- Yes, that took about 1 minute of 100% processing of 1 of the 8 CPUs/Cores. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
8 seconds for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg: Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com
Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding
2019-01-29 11:43:06 Connecting to host x.x.x.x
2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
*From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA *Sent:* Monday, 28 January 2019 20:16 *To:* Xymon Mailing List <xymon at xymon.com> *Subject:* [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg:
Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com
Zak,
I forgot to mention: regex may not cut it for the filename issue - it's placeholders for parts of the date that would be useful here, where that placeholder is substituted with that part of the date, e.g. in Python: %d -- Day of the month as a zero-padded decimal number - ref: http://strftime.org/ In .NET (PowerShell): https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-... We just need a way of specifying the date format we want to use and it could be interpreted by the
ToString
method of the date and time instance that holds the current datetime.
So, to add to what I had in my original post, in client-local.cfg: Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240 This would be interpreted at the client side as (when the date is 2019-01-30): C:\Temp\myServerLog30-01-19.*.log Which would find and process, e.g. C:\Temp\myServerLog30-01-19.0.log and C:\Temp\myServerLog30-01-19.1.log
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk> wrote:
Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding
2019-01-29 11:43:06 Connecting to host x.x.x.x
2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
*From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA *Sent:* Monday, 28 January 2019 20:16 *To:* Xymon Mailing List <xymon at xymon.com> *Subject:* [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg:
Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com
Hi
Hmm ok, that is quite a specific requirement, thanks for the detail, I understand it now – find files matching today’s date with a certain pattern. That might be a candidate for an external script where you can do whatever you want more precisely.
One problem is this part of the code is pretty old, I think I’ve hardly changed it, it’s called from multiple places and it already adds functionality like trying to expand environment variables like %APPDATA%. It then uses Resolve-Path to resolve the wildcards. Resolve-Path does not support regex or date filters so I’d have to change that to get a list of all the filenames and then compare to the regex / date filter. That will be slow (relatively) especially on a large directory with thousands of files. But it would filter down the filename candidates for log processing. I’ve added it to my todo list / tracker.
Zak
From: SebA <spah at syntec.co.uk> Sent: Wednesday, 30 January 2019 10:57 To: Beck, Zak <zak.beck at accenture.com> Cc: Xymon Mailing List <xymon at xymon.com> Subject: Re: [External] [Xymon] Xymon WinPSclient performance
Zak,
I forgot to mention: regex may not cut it for the filename issue - it's placeholders for parts of the date that would be useful here, where that placeholder is substituted with that part of the date, e.g. in Python: %d -- Day of the month as a zero-padded decimal number - ref: http://strftime.org/<https://urldefense.proofpoint.com/v2/url?u=http-3A__strftime.org_&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=7rSbEYzyMxSQsFDzuOl4TLyppuTOTD3ngLvvt5Igm8k&s=8cSsaCRMHG12uRID-D1Td43XDN5p08pGzW2PfWw9D70&e=> In .NET (PowerShell): https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-time-format-strings<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.microsoft.com_en-2Dus_dotnet_standard_base-2Dtypes_custom-2Ddate-2Dand-2Dtime-2Dformat-2Dstrings&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=7rSbEYzyMxSQsFDzuOl4TLyppuTOTD3ngLvvt5Igm8k&s=kq5ilMgfhKTR0NMJOjqktRYO4v7XNRBQouliKtiAZ5M&e=> We just need a way of specifying the date format we want to use and it could be interpreted by the
ToString method of the date and time instance that holds the current datetime.
So, to add to what I had in my original post, in client-local.cfg: Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240 This would be interpreted at the client side as (when the date is 2019-01-30): C:\Temp\myServerLog30-01-19.*.log Which would find and process, e.g. C:\Temp\myServerLog30-01-19.0.log and C:\Temp\myServerLog30-01-19.1.log
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk<mailto:spah at syntec.co.uk>> wrote: Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com<mailto:zak.beck at accenture.com>> wrote: Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding 2019-01-29 11:43:06 Connecting to host x.x.x.x 2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
From: Xymon <xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com>> On Behalf Of SebA Sent: Monday, 28 January 2019 20:16 To: Xymon Mailing List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:20:25 Connecting to host x.x.x.x <-- Yes, that took about 1 minute of 100% processing of 1 of the 8 CPUs/Cores. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
8 seconds for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg: Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com<http://www.accenture.com>
Well, the other options to solve our use case (my first option I'll refer to as (a)) are to either: (b) to allow backtick like processing in client-local.cfg like in the *nix Xymon client - I think a lot of people use the find command there to select the files they want sent over, us included; (c) to allow some other kind of selection of files, possibly something using: https://blogs.technet.microsoft.com/heyscriptingguy/2016/06/27/use-windows-p...
I think there is generally a use for people to be able to select files based on non-static filenames. In fact, in our current Windows use case I would like to select the newest (and only the newest) modified file that maches this: log:C:\Temp\myServerLog*.log:1024 Maybe there is a better way of representing that query, e.g.: log:C:\Temp\myServerLog*.log:1024*:newest* Or whatever. That would actually be the best for us - clear and deal with pseudo-randomly rotating log files.
Kind regards,
SebA
On Wed, 30 Jan 2019 at 13:57, Beck, Zak <zak.beck at accenture.com> wrote:
Hi
Hmm ok, that is quite a specific requirement, thanks for the detail, I understand it now – find files matching today’s date with a certain pattern. That might be a candidate for an external script where you can do whatever you want more precisely.
One problem is this part of the code is pretty old, I think I’ve hardly changed it, it’s called from multiple places and it already adds functionality like trying to expand environment variables like %APPDATA%. It then uses Resolve-Path to resolve the wildcards. Resolve-Path does not support regex or date filters so I’d have to change that to get a list of all the filenames and then compare to the regex / date filter. That will be slow (relatively) especially on a large directory with thousands of files. But it would filter down the filename candidates for log processing. I’ve added it to my todo list / tracker.
Zak
*From:* SebA <spah at syntec.co.uk> *Sent:* Wednesday, 30 January 2019 10:57 *To:* Beck, Zak <zak.beck at accenture.com> *Cc:* Xymon Mailing List <xymon at xymon.com> *Subject:* Re: [External] [Xymon] Xymon WinPSclient performance
Zak,
I forgot to mention:
regex may not cut it for the filename issue - it's placeholders for parts of the date that would be useful here, where that placeholder is substituted with that part of the date, e.g. in Python:
%d -- Day of the month as a zero-padded decimal number - ref: http://strftime.org/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__strftime.org_&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=7rSbEYzyMxSQsFDzuOl4TLyppuTOTD3ngLvvt5Igm8k&s=8cSsaCRMHG12uRID-D1Td43XDN5p08pGzW2PfWw9D70&e=>
In .NET (PowerShell): https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-... <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.microsoft.com_en-2Dus_dotnet_standard_base-2Dtypes_custom-2Ddate-2Dand-2Dtime-2Dformat-2Dstrings&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=7rSbEYzyMxSQsFDzuOl4TLyppuTOTD3ngLvvt5Igm8k&s=kq5ilMgfhKTR0NMJOjqktRYO4v7XNRBQouliKtiAZ5M&e=>
We just need a way of specifying the date format we want to use and it could be interpreted by the
ToString
method of the date and time instance that holds the current datetime.
So, to add to what I had in my original post, in client-local.cfg:
Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
This would be interpreted at the client side as (when the date is 2019-01-30):
C:\Temp\myServerLog30-01-19.*.log
Which would find and process, e.g. C:\Temp\myServerLog30-01-19.0.log and C:\Temp\myServerLog30-01-19.1.log
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk> wrote:
Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding
2019-01-29 11:43:06 Connecting to host x.x.x.x
2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
*From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA *Sent:* Monday, 28 January 2019 20:16 *To:* Xymon Mailing List <xymon at xymon.com> *Subject:* [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg:
Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com
Good news and bad news. The good news is that enabling XymonAcceptUTF8 makes the cycle run a lot faster and seems to work fine for us, without doubling the number of bytes sent in the report. I think the default though should probably be not to change the behaviour compared with the previous version (v2.19). Failing that, the default should probably be to enable XymonAcceptUTF8 - the default should not be something that slows down the cycle and uses lots of CPU time IMHO.
The bad news is I have found another thing to takes time: XymonDiskPart.
2019-01-30 13:39:13 XymonDiskPart start 2019-01-30 13:39:17 XymonDiskPart finished
I see this only runs every 'slow scan', but: (a) this also takes 4 seconds to complete; (b) I don't know what the use of it is, at least for most people, and it's not documented in XymonPSClient.doc - this is a possible example of feature bloat; (c) it causes a bad memory leak in the Virtual Disk service (vds.exe) on our failover cluster servers that have clustered disk.
Commenting out line 3875 in v2.28 r8017 has no negative consequences for us and removes this vds.exe memory leak. I have compared patched and unpatched processes running for several months on otherwise identical servers. We have had a conversation about this, see here: https://lists.xymon.com/archive/2018-March/045341.html But you didn't respond to that last message. Since the Virtual Disk service was last restarted on the unpatched server on 30/11/18 (so in 2 months), it has leaked up to today nearly 900 MB RAM and 9 GB of virtual memory.
Please can XymonDiskPart be made optional in the config?
We have another server (2012 R2, no failover cluster, no clustered disks) that is using 1 GB RAM and 2 GB virtual memory in the Xymon powershell process also (unpatched) v2.28 after 43 hours running. I haven't yet investigated the cause, but I'll try replacing it with the patched version without XymonDiskPart and see if that cures the memory leak.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk> wrote:
Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding
2019-01-29 11:43:06 Connecting to host x.x.x.x
2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
*From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA *Sent:* Monday, 28 January 2019 20:16 *To:* Xymon Mailing List <xymon at xymon.com> *Subject:* [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg:
Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com
Yes, I remember our conversation – very difficult for me to replicate.
I can definitely make this optional, and will most likely look to make this default to off.
Zak From: SebA <spah at syntec.co.uk> Sent: Wednesday, 30 January 2019 14:57 To: Beck, Zak <zak.beck at accenture.com> Cc: Xymon Mailing List <xymon at xymon.com> Subject: Re: [External] [Xymon] Xymon WinPSclient performance
Good news and bad news. The good news is that enabling XymonAcceptUTF8 makes the cycle run a lot faster and seems to work fine for us, without doubling the number of bytes sent in the report. I think the default though should probably be not to change the behaviour compared with the previous version (v2.19). Failing that, the default should probably be to enable XymonAcceptUTF8 - the default should not be something that slows down the cycle and uses lots of CPU time IMHO.
The bad news is I have found another thing to takes time: XymonDiskPart.
2019-01-30 13:39:13 XymonDiskPart start 2019-01-30 13:39:17 XymonDiskPart finished
I see this only runs every 'slow scan', but: (a) this also takes 4 seconds to complete; (b) I don't know what the use of it is, at least for most people, and it's not documented in XymonPSClient.doc - this is a possible example of feature bloat; (c) it causes a bad memory leak in the Virtual Disk service (vds.exe) on our failover cluster servers that have clustered disk.
Commenting out line 3875 in v2.28 r8017 has no negative consequences for us and removes this vds.exe memory leak. I have compared patched and unpatched processes running for several months on otherwise identical servers. We have had a conversation about this, see here: https://lists.xymon.com/archive/2018-March/045341.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.xymon.com_archive_2018-2DMarch_045341.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Yihzb9N4fl-Dl1BgOxMCrJsBMmLNdwI2ItiW5rRVcV8&s=-52Qhrm1-bvmWiTrhmSxyNmQdwM04ZP_KbPkNkWSIiA&e=> But you didn't respond to that last message. Since the Virtual Disk service was last restarted on the unpatched server on 30/11/18 (so in 2 months), it has leaked up to today nearly 900 MB RAM and 9 GB of virtual memory.
Please can XymonDiskPart be made optional in the config?
We have another server (2012 R2, no failover cluster, no clustered disks) that is using 1 GB RAM and 2 GB virtual memory in the Xymon powershell process also (unpatched) v2.28 after 43 hours running. I haven't yet investigated the cause, but I'll try replacing it with the patched version without XymonDiskPart and see if that cures the memory leak.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk<mailto:spah at syntec.co.uk>> wrote: Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com<mailto:zak.beck at accenture.com>> wrote: Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding 2019-01-29 11:43:06 Connecting to host x.x.x.x 2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
From: Xymon <xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com>> On Behalf Of SebA Sent: Monday, 28 January 2019 20:16 To: Xymon Mailing List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:20:25 Connecting to host x.x.x.x <-- Yes, that took about 1 minute of 100% processing of 1 of the 8 CPUs/Cores. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
8 seconds for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg: Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com<http://www.accenture.com>
The UTF8 / ASCII thing – interestingly, going right back to v1, the client has used an ASCII encoder. It has to use something because Socket.Send() expects a byte array, the data packet is constructed as a string and so you need to use an encoder from System.Text to convert the string to a byte array.
The change in v2.21 was to add the Remove-Diacritics function, which is to strip out non-ASCII characters and also the 0xa0 non-breaking space. I can see how Remove-Diacritics might be a performance bottleneck on a long string.
People with diacritics<https://en.wikipedia.org/wiki/Diacritic> may be using UTF8 anyway, I suspect most people using ASCII have no idea about the diacritic removal and have no need for it. Therefore I propose three values for the XymonAcceptUTF8 setting and this should impact on the minimum of people:
0 (the default) – use “original” ASCII, i.e. as per the original behaviour. This may cause diacritics to come through as unreadable multi-byte characters. 1 – use UTF8 2 – use “pure” ASCII, i.e. filter through Remove-Diacritics
Zak
From: Beck, Zak Sent: Wednesday, 30 January 2019 15:04 To: 'SebA' <spah at syntec.co.uk> Cc: Xymon Mailing List <xymon at xymon.com> Subject: RE: [External] [Xymon] Xymon WinPSclient performance
Yes, I remember our conversation – very difficult for me to replicate.
I can definitely make this optional, and will most likely look to make this default to off.
Zak From: SebA <spah at syntec.co.uk<mailto:spah at syntec.co.uk>> Sent: Wednesday, 30 January 2019 14:57 To: Beck, Zak <zak.beck at accenture.com<mailto:zak.beck at accenture.com>> Cc: Xymon Mailing List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: Re: [External] [Xymon] Xymon WinPSclient performance
Good news and bad news. The good news is that enabling XymonAcceptUTF8 makes the cycle run a lot faster and seems to work fine for us, without doubling the number of bytes sent in the report. I think the default though should probably be not to change the behaviour compared with the previous version (v2.19). Failing that, the default should probably be to enable XymonAcceptUTF8 - the default should not be something that slows down the cycle and uses lots of CPU time IMHO.
The bad news is I have found another thing to takes time: XymonDiskPart.
2019-01-30 13:39:13 XymonDiskPart start 2019-01-30 13:39:17 XymonDiskPart finished
I see this only runs every 'slow scan', but: (a) this also takes 4 seconds to complete; (b) I don't know what the use of it is, at least for most people, and it's not documented in XymonPSClient.doc - this is a possible example of feature bloat; (c) it causes a bad memory leak in the Virtual Disk service (vds.exe) on our failover cluster servers that have clustered disk.
Commenting out line 3875 in v2.28 r8017 has no negative consequences for us and removes this vds.exe memory leak. I have compared patched and unpatched processes running for several months on otherwise identical servers. We have had a conversation about this, see here: https://lists.xymon.com/archive/2018-March/045341.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.xymon.com_archive_2018-2DMarch_045341.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Yihzb9N4fl-Dl1BgOxMCrJsBMmLNdwI2ItiW5rRVcV8&s=-52Qhrm1-bvmWiTrhmSxyNmQdwM04ZP_KbPkNkWSIiA&e=> But you didn't respond to that last message. Since the Virtual Disk service was last restarted on the unpatched server on 30/11/18 (so in 2 months), it has leaked up to today nearly 900 MB RAM and 9 GB of virtual memory.
Please can XymonDiskPart be made optional in the config?
We have another server (2012 R2, no failover cluster, no clustered disks) that is using 1 GB RAM and 2 GB virtual memory in the Xymon powershell process also (unpatched) v2.28 after 43 hours running. I haven't yet investigated the cause, but I'll try replacing it with the patched version without XymonDiskPart and see if that cures the memory leak.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk<mailto:spah at syntec.co.uk>> wrote: Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com<mailto:zak.beck at accenture.com>> wrote: Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding 2019-01-29 11:43:06 Connecting to host x.x.x.x 2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
From: Xymon <xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com>> On Behalf Of SebA Sent: Monday, 28 January 2019 20:16 To: Xymon Mailing List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:20:25 Connecting to host x.x.x.x <-- Yes, that took about 1 minute of 100% processing of 1 of the 8 CPUs/Cores. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com<mailto:zak.beck at accenture.com> 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
8 seconds for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg: Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com<http://www.accenture.com>
I agree with your observations and proposal.
Kind regards,
SebA
On Wed, 30 Jan 2019 at 15:59, Beck, Zak <zak.beck at accenture.com> wrote:
The UTF8 / ASCII thing – interestingly, going right back to v1, the client has used an ASCII encoder. It has to use something because Socket.Send() expects a byte array, the data packet is constructed as a string and so you need to use an encoder from System.Text to convert the string to a byte array.
The change in v2.21 was to add the Remove-Diacritics function, which is to strip out non-ASCII characters and also the 0xa0 non-breaking space. I can see how Remove-Diacritics might be a performance bottleneck on a long string.
People with diacritics <https://en.wikipedia.org/wiki/Diacritic> may be using UTF8 anyway, I suspect most people using ASCII have no idea about the diacritic removal and have no need for it. Therefore I propose three values for the XymonAcceptUTF8 setting and this should impact on the minimum of people:
0 (the default) – use “original” ASCII, i.e. as per the original behaviour. This may cause diacritics to come through as unreadable multi-byte characters.
1 – use UTF8
2 – use “pure” ASCII, i.e. filter through Remove-Diacritics
Zak
*From:* Beck, Zak *Sent:* Wednesday, 30 January 2019 15:04 *To:* 'SebA' <spah at syntec.co.uk> *Cc:* Xymon Mailing List <xymon at xymon.com> *Subject:* RE: [External] [Xymon] Xymon WinPSclient performance
Yes, I remember our conversation – very difficult for me to replicate.
I can definitely make this optional, and will most likely look to make this default to off.
Zak
*From:* SebA <spah at syntec.co.uk> *Sent:* Wednesday, 30 January 2019 14:57 *To:* Beck, Zak <zak.beck at accenture.com> *Cc:* Xymon Mailing List <xymon at xymon.com> *Subject:* Re: [External] [Xymon] Xymon WinPSclient performance
Good news and bad news. The good news is that enabling XymonAcceptUTF8 makes the cycle run a lot faster and seems to work fine for us, without doubling the number of bytes sent in the report. I think the default though should probably be not to change the behaviour compared with the previous version (v2.19). Failing that, the default should probably be to enable XymonAcceptUTF8 - the default should not be something that slows down the cycle and uses lots of CPU time IMHO.
The bad news is I have found another thing to takes time: XymonDiskPart.
2019-01-30 13:39:13 XymonDiskPart start 2019-01-30 13:39:17 XymonDiskPart finished
I see this only runs every 'slow scan', but: (a) this also takes 4 seconds to complete; (b) I don't know what the use of it is, at least for most people, and it's not documented in XymonPSClient.doc - this is a possible example of feature bloat; (c) it causes a bad memory leak in the Virtual Disk service (vds.exe) on our failover cluster servers that have clustered disk.
Commenting out line 3875 in v2.28 r8017 has no negative consequences for us and removes this vds.exe memory leak. I have compared patched and unpatched processes running for several months on otherwise identical servers. We have had a conversation about this, see here: https://lists.xymon.com/archive/2018-March/045341.html <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.xymon.com_archive_2018-2DMarch_045341.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Yihzb9N4fl-Dl1BgOxMCrJsBMmLNdwI2ItiW5rRVcV8&s=-52Qhrm1-bvmWiTrhmSxyNmQdwM04ZP_KbPkNkWSIiA&e=> But you didn't respond to that last message. Since the Virtual Disk service was last restarted on the unpatched server on 30/11/18 (so in 2 months), it has leaked up to today nearly 900 MB RAM and 9 GB of virtual memory.
Please can XymonDiskPart be made optional in the config?
We have another server (2012 R2, no failover cluster, no clustered disks) that is using 1 GB RAM and 2 GB virtual memory in the Xymon powershell process also (unpatched) v2.28 after 43 hours running. I haven't yet investigated the cause, but I'll try replacing it with the patched version without XymonDiskPart and see if that cures the memory leak.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk> wrote:
Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding
2019-01-29 11:43:06 Connecting to host x.x.x.x
2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
*From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA *Sent:* Monday, 28 January 2019 20:16 *To:* Xymon Mailing List <xymon at xymon.com> *Subject:* [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg:
Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com
Sounds good. It would also be nice to get milliseconds in the xymonclient.log because on the Windows 2012 R2 server I mentioned in one of my e-mails today, it does (on collection number 2) the steps I was talking about much quicker (as you can see from these log snippets), but it does other tasks much slower:
2019-01-28 20:09:59 XymonCollectInfo: CPU info (WMI) 2019-01-28 20:10:01 Found 2 CPUs, total of 4 cores
The combination of just the normal tests takes up the time between the above and the below log snippets, although it looks like it may be: XymonCollectInfo: Service info (WMI), XymonCpu, XymonProcs, XymonRoute and XymonSvcs that take up most of the time...
2019-01-28 20:10:06 Using ASCII encoding 2019-01-28 20:10:06 Connecting to host x.x.x.x 2019-01-28 20:10:06 Sent 366488 bytes to server 2019-01-28 20:10:07 Received 0 bytes from server
Kind regards,
SebA
On Wed, 30 Jan 2019 at 15:03, Beck, Zak <zak.beck at accenture.com> wrote:
Yes, I remember our conversation – very difficult for me to replicate.
I can definitely make this optional, and will most likely look to make this default to off.
Zak
*From:* SebA <spah at syntec.co.uk> *Sent:* Wednesday, 30 January 2019 14:57 *To:* Beck, Zak <zak.beck at accenture.com> *Cc:* Xymon Mailing List <xymon at xymon.com> *Subject:* Re: [External] [Xymon] Xymon WinPSclient performance
Good news and bad news. The good news is that enabling XymonAcceptUTF8 makes the cycle run a lot faster and seems to work fine for us, without doubling the number of bytes sent in the report. I think the default though should probably be not to change the behaviour compared with the previous version (v2.19). Failing that, the default should probably be to enable XymonAcceptUTF8 - the default should not be something that slows down the cycle and uses lots of CPU time IMHO.
The bad news is I have found another thing to takes time: XymonDiskPart.
2019-01-30 13:39:13 XymonDiskPart start 2019-01-30 13:39:17 XymonDiskPart finished
I see this only runs every 'slow scan', but: (a) this also takes 4 seconds to complete; (b) I don't know what the use of it is, at least for most people, and it's not documented in XymonPSClient.doc - this is a possible example of feature bloat; (c) it causes a bad memory leak in the Virtual Disk service (vds.exe) on our failover cluster servers that have clustered disk.
Commenting out line 3875 in v2.28 r8017 has no negative consequences for us and removes this vds.exe memory leak. I have compared patched and unpatched processes running for several months on otherwise identical servers. We have had a conversation about this, see here: https://lists.xymon.com/archive/2018-March/045341.html <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.xymon.com_archive_2018-2DMarch_045341.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=S-aLwpx-PHBTBMIG_c2JczRC0SfuZCmsiH9Iams25FI&m=Yihzb9N4fl-Dl1BgOxMCrJsBMmLNdwI2ItiW5rRVcV8&s=-52Qhrm1-bvmWiTrhmSxyNmQdwM04ZP_KbPkNkWSIiA&e=> But you didn't respond to that last message. Since the Virtual Disk service was last restarted on the unpatched server on 30/11/18 (so in 2 months), it has leaked up to today nearly 900 MB RAM and 9 GB of virtual memory.
Please can XymonDiskPart be made optional in the config?
We have another server (2012 R2, no failover cluster, no clustered disks) that is using 1 GB RAM and 2 GB virtual memory in the Xymon powershell process also (unpatched) v2.28 after 43 hours running. I haven't yet investigated the cause, but I'll try replacing it with the patched version without XymonDiskPart and see if that cures the memory leak.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 15:28, SebA <spah at syntec.co.uk> wrote:
Hi Zak,
Thanks, yes, haha, I did notice the irony myself and was going to point it out, or change my first paragraph to put the emphasis on reducing CPU time spent, but I forgot! I noticed that powershell (on some versions of the PSclient) was using more CPU time than the process the server exists to run, which is clearly not a good situation.
So if I select UTF-8, it's not going to double the size of the messages sent to the Xymon server? I'll give it a go and see what happens.
I think being able to configure the scan interval for CPUs would be good.
The latest version of PSclient was not even completing one cycle before I was giving up and killing it, but I'll try UTF-8 and/or another server.
Kind regards,
SebA
On Tue, 29 Jan 2019 at 11:59, Beck, Zak <zak.beck at accenture.com> wrote:
Hi
Yes, there is no option for no encoding – I think the closest option to this is actually UTF-8 because it does not attempt to adjust the message body (remove diacritics and 0xa0 spaces) before sending. That’s what is happening between your two log messages. Personally I don’t see that behaviour albeit with a 10x smaller data packet (I checked on several servers):
2019-01-29 11:43:06 Using ASCII encoding
2019-01-29 11:43:06 Connecting to host x.x.x.x
2019-01-29 11:43:06 Sent 100537 bytes to server
We can add a third option to do no encoding I guess.
Detecting CPUs is entirely down to this WMI call: Get-WmiObject -Class Win32_Processor. Yes, it does take time. I have tried over the years to reduce the amount of WMI calls (I think we’re down to 4) because they do seem to take a lot of time and on occasion are unreliable. Unfortunately I have not been able to find an alternative to this call. And I have 1000 VMs which can have CPUs hot added without reboot, so there is use case for checking every time. The result of the WMI call is already cached so we could amend the script to optionally run only on slow scans (every 6 hours by default).
It may be simpler to add regex processing for your filename issue, I can take a look.
I had to chuckle, I hope you appreciate the irony of saying we’re suffering from feature bloat and then requesting more enhancements / features 😉
Zak
*From:* Xymon <xymon-bounces at xymon.com> *On Behalf Of *SebA *Sent:* Monday, 28 January 2019 20:16 *To:* Xymon Mailing List <xymon at xymon.com> *Subject:* [External] [Xymon] Xymon WinPSclient performance
It's great the way features have been added to Xymon WinPSclient, but I think it's started suffering from feature bloat now, illustrating the importance of making each feature optional like BBWin did very effectively. We are seeing memory leaks on Windows 2012 R2 using version 2.28 (I started another thread about memory leaks a while back - I'll come back to this issue another time) and more and more CPU time being used up during each cycle.
It seems that from version v2.21 a slow ASCII encoding conversion was added:
myServer1 - : xymonclient.ps1 2.21 2017-04-28 zak.beck at accenture.com 2019-01-28 19:19:26 Sending to server 2019-01-28 19:19:26 Using ASCII encoding 2019-01-28 19:*20*:25 Connecting to host x.x.x.x <-- Yes, that took about *1 minute of 100% processing of 1 of the 8 CPUs/Cores*. 2019-01-28 19:20:26 Sent 2007850 bytes to server <-- OK, yes, that's rather a lot of data, but still it's not good. 2019-01-28 19:20:26 Received 130 bytes from server
Compare with: myServer1 - : xymonclient.ps1 2.19 2016-12-28 zak.beck at accenture.com 2019-01-28 19:15:59 Sending to server 2019-01-28 19:15:59 Connecting to host x.x.x.x 2019-01-28 19:15:59 Sent 2006765 bytes to server 2019-01-28 19:16:00 Received 130 bytes from server
Is it possible that converting to ASCII step introduced in v2.21 be made optional? In the latest version it is possible to send via UTF8, which was new in v2.21, or convert to ASCII, but it's not possible via configuration alone, I believe, to have the old behaviour (from v2.19) that worked fine for us and was far faster.
Another slow process is detecting the number of CPUs (on the same server):
2019-01-28 19:29:05 XymonCollectInfo: Process info 2019-01-28 19:29:05 XymonCollectInfo: calling XymonProcsCPUUtilisation 2019-01-28 19:29:05 XymonCollectInfo: CPU info (WMI) 2019-01-28 19:29:13 Found 8 CPUs, total of 0 cores 2019-01-28 19:29:13 XymonCollectInfo: OS info (including memory) (WMI) 2019-01-28 19:29:13 XymonCollectInfo: Service info (WMI)
*8 seconds* for that. Why does it need to do this every time? The number of CPUs does not normally change without at least rebooting. It could do it just on the first cycle and cache it for future cycles.
Another thing that would really speed up processing for us is being able to dynamically specify the filenames using special characters or something for the date, e.g. &DD &MM &YY We have files that rotate with new filesnames and if we could narrow down the filenames to process it would speed up processing by several more seconds. E.g. C:\Temp\myServerLog29-12-18.0.log
In client-local.cfg:
Instead of: log:C:\Temp\myServerLog*.log:10240 If we could to this, it would mean we process a small fraction of the number of files: log:C:\Temp\myServerLog&DD-&MM-&YY.*.log:10240
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
www.accenture.com
participants (2)
-
spah＠syntec.co.uk
-
zak.beck＠accenture.com