Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
I have noticed a memory and thread leak in the Windows Virtual Disk service
- or vds.exe from the process list. This service is set to Manual by default and gets started when starting the XymonPSClient service on Windows 2008 R2. It starts off using 32 threads, but grows forever, the virtual memory and physical memory (which starts under 6 MB) used slowly grow too.
Has anyone noticed this problem and do they have a solution to it?
On Windows Server 2012, the Virtual Disk service does not get started by Xymon Windows Powershell client, so it's obviously not started directly by the client, but by a request for information. I looked into known memory leaks for vds.exe and the one I found was not applicable to 2008 R2, only 2008 and Vista. The version of vds.exe we have is later than the version with the fix: 6.1.7601.17514 from 2010.
The quick workaround is simply to stop the Virtual Disk Service - but as soon as the PSclient has used it, it does not stop properly and terminates unexpectedly, so one needs to set it not to restart on crash, or stop it immediately after starting, or to Disabled. But it would be good to know why it is starting, not stopping, and leaking threads / memory!
Kind regards,
SebA
Hi Seb
I’ve got hundreds of VMs running the Powershell client on Windows Server 2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same version of vds.exe too.
I’m wondering if function XymonDiskPart has something to do with it. This only runs on slow scans (every 6 hours-ish by default). This runs ‘diskpart’ and then
“list disk” For each disk returned, “select disk {n}” and then “detail disk”
Maybe you could try this from the command prompt on a server with vds stopped and see if it restarts it. You could also try commenting out this line (3875, towards the end) in the client and see what happens (insert a # at the beginning of the line):
$script:diskpartData = XymonDiskPartAlternatively, it could be the disk utilisation code which is using Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume, FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW, GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these doesn’t mention the Virtual Disk Service but I guess they could be using it.
Zak
From: SebA [mailto:spah at syntec.co.uk] Sent: Tuesday, 13 March 2018 18:14 To: Beck, Zak <zak.beck at accenture.com>; Xymon Mailing List <xymon at xymon.com> Subject: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
I have noticed a memory and thread leak in the Windows Virtual Disk service - or vds.exe from the process list. This service is set to Manual by default and gets started when starting the XymonPSClient service on Windows 2008 R2. It starts off using 32 threads, but grows forever, the virtual memory and physical memory (which starts under 6 MB) used slowly grow too.
Has anyone noticed this problem and do they have a solution to it?
On Windows Server 2012, the Virtual Disk service does not get started by Xymon Windows Powershell client, so it's obviously not started directly by the client, but by a request for information. I looked into known memory leaks for vds.exe and the one I found was not applicable to 2008 R2, only 2008 and Vista. The version of vds.exe we have is later than the version with the fix: 6.1.7601.17514 from 2010.
The quick workaround is simply to stop the Virtual Disk Service - but as soon as the PSclient has used it, it does not stop properly and terminates unexpectedly, so one needs to set it not to restart on crash, or stop it immediately after starting, or to Disabled. But it would be good to know why it is starting, not stopping, and leaking threads / memory!
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
Good morning,
i just noticed a memory leak caused bur our beloved ps client on win 2008r2: task manager reported a process “powershell” launched by System using 1.7GB. I uninstalled the ps client and the memory was immediately released.
Version in use was 2.19 (not so young).
Alessandro
Hi Alessandro
I think this is going to incredibly hard to track down as I haven’t had any other reports like this. I personally have the client running on 3-400 servers and am not aware of issues like this.
You sent me a couple of logs (thanks), could you explain the configuration differences between the server having the issue and others (client-local)?
Thanks
Zak
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Alessandro Tinivelli Sent: Friday, 23 March 2018 11:08 To: Xymon Mailing List <xymon at xymon.com> Subject: [Xymon] [EXCH Prio5]R: [EXCH Prio5]Re: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
Good morning,
i just noticed a memory leak caused bur our beloved ps client on win 2008r2: task manager reported a process “powershell” launched by System using 1.7GB. I uninstalled the ps client and the memory was immediately released.
Version in use was 2.19 (not so young).
Alessandro
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
Hi Zak,
Thanks for your swift reply and sorry for my own delay! It's good to know you have such extensive use of the PS client without this issue.
I think you are probably right. These issue has only appeared on servers using Windows Cluster Failover, where various iSCSI disks are clustered.
For the record, my workaround of disabling the Virtual Disk Service caused no bad effects on the server, but it does prevent diskpart from running. I set the service back to Manual, and it started as soon as I ran diskpart from an elevated Command Prompt. Exiting diskpart causes the VDS service and proces to stop.
When the Virtual Disk Service was set to manual, and started just by me running diskpart.exe, the client output has:
[diskpart] diskpart:::Clustered Unknown
When the service is disabled, the output is just an empty section:
[diskpart]
When VDS is started by me restarting the XymonPSclient service and it being on manual, the output is e.g.:
[diskpart] diskpart:Disk 0:278 GB:Not Clustered diskpart:Disk 1:1024 MB:Clustered Active diskpart:Disk 2:25 GB:Clustered Active diskpart:Disk 3:500 GB:Clustered Active diskpart:Disk 4:300 GB:Clustered Inactive diskpart:Disk 5:1024 MB:Clustered Inactive diskpart:Disk 6:500 GB:Clustered Inactive diskpart:Disk 7:200 GB:Clustered Inactive diskpart:Disk 8:1024 MB:Clustered Inactive diskpart:Disk 9:1024 MB:Clustered Inactive diskpart:Disk 10:200 GB:Clustered Inactive diskpart:Disk 11:200 GB:Clustered Inactive diskpart:Disk 12:500 GB:Clustered Active diskpart:Disk 13:5120 MB:Clustered Active diskpart:Disk 14:500 GB:Clustered Inactive
Compared with VDS running because of me running diskpart and running:
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”
for each disk (which does not leak memory) vds.exe uses about an extra MB of RAM when running due to the PS client.
I have also now commented out the line you suggest (' $script:diskpartData = XymonDiskPart ') and now when I restart the XymonPSclient service with VDS set to Manual it does not get restarted and left running (or restarted at all). So this looks like a better workaround than just disabling the service, but I will need to continue to monitor that.
Is the XymonDiskPart information used for anything? I mean, can it be used for anything without writing plug-ins?
We do not have memory leaks in powershell.exe on these servers (although I think I may have seen it on another server), so the issue above is not related to Alessandro's issue.
Thanks again for your help Zak, and I hope this knowledge is of benefit to you or others too.
Kind regards,
SebA
On 14 March 2018 at 09:21, Beck, Zak <zak.beck at accenture.com> wrote:
Hi Seb
I’ve got hundreds of VMs running the Powershell client on Windows Server 2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same version of vds.exe too.
I’m wondering if function XymonDiskPart has something to do with it. This only runs on slow scans (every 6 hours-ish by default). This runs ‘diskpart’ and then
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”
Maybe you could try this from the command prompt on a server with vds stopped and see if it restarts it. You could also try commenting out this line (3875, towards the end) in the client and see what happens (insert a # at the beginning of the line):
$script:diskpartData = XymonDiskPartAlternatively, it could be the disk utilisation code which is using Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume, FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW, GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these doesn’t mention the Virtual Disk Service but I guess they could be using it.
Zak
*From:* SebA [mailto:spah at syntec.co.uk] *Sent:* Tuesday, 13 March 2018 18:14 *To:* Beck, Zak <zak.beck at accenture.com>; Xymon Mailing List < xymon at xymon.com> *Subject:* [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
I have noticed a memory and thread leak in the Windows Virtual Disk service - or vds.exe from the process list. This service is set to Manual by default and gets started when starting the XymonPSClient service on Windows 2008 R2. It starts off using 32 threads, but grows forever, the virtual memory and physical memory (which starts under 6 MB) used slowly grow too.
Has anyone noticed this problem and do they have a solution to it?
On Windows Server 2012, the Virtual Disk service does not get started by Xymon Windows Powershell client, so it's obviously not started directly by the client, but by a request for information. I looked into known memory leaks for vds.exe and the one I found was not applicable to 2008 R2, only 2008 and Vista. The version of vds.exe we have is later than the version with the fix: 6.1.7601.17514 from 2010.
The quick workaround is simply to stop the Virtual Disk Service - but as soon as the PSclient has used it, it does not stop properly and terminates unexpectedly, so one needs to set it not to restart on crash, or stop it immediately after starting, or to Disabled. But it would be good to know why it is starting, not stopping, and leaking threads / memory!
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
Hi Seb
Thanks for confirming, that’s good news – I was fearing it may be the core disk utilisation functions which would be a much larger headache!
Is the XymonDiskPart information used for anything? I mean, can it be used for anything without writing plug-ins?
The [diskpart] section is used in one of our datacentres where they use clustering and the information is used primarily for reporting. So no, it is not used for anything by default.
I think the best solution here is for me to add a setting in xymonclient_config.xml allowing you to turn off the diskpart function for specific servers. You can then remove your comment and continue to use future updates.
Zak
From: SebA [mailto:spah at syntec.co.uk] Sent: Friday, 30 March 2018 14:04 To: Beck, Zak <zak.beck at accenture.com> Cc: Xymon Mailing List <xymon at xymon.com> Subject: Re: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
Hi Zak, Thanks for your swift reply and sorry for my own delay! It's good to know you have such extensive use of the PS client without this issue. I think you are probably right. These issue has only appeared on servers using Windows Cluster Failover, where various iSCSI disks are clustered. For the record, my workaround of disabling the Virtual Disk Service caused no bad effects on the server, but it does prevent diskpart from running. I set the service back to Manual, and it started as soon as I ran diskpart from an elevated Command Prompt. Exiting diskpart causes the VDS service and proces to stop. When the Virtual Disk Service was set to manual, and started just by me running diskpart.exe, the client output has:
[diskpart]
diskpart:::Clustered Unknown When the service is disabled, the output is just an empty section:
[diskpart] When VDS is started by me restarting the XymonPSclient service and it being on manual, the output is e.g.:
[diskpart]
diskpart:Disk 0:278 GB:Not Clustered
diskpart:Disk 1:1024 MB:Clustered Active
diskpart:Disk 2:25 GB:Clustered Active
diskpart:Disk 3:500 GB:Clustered Active
diskpart:Disk 4:300 GB:Clustered Inactive
diskpart:Disk 5:1024 MB:Clustered Inactive
diskpart:Disk 6:500 GB:Clustered Inactive
diskpart:Disk 7:200 GB:Clustered Inactive
diskpart:Disk 8:1024 MB:Clustered Inactive
diskpart:Disk 9:1024 MB:Clustered Inactive
diskpart:Disk 10:200 GB:Clustered Inactive
diskpart:Disk 11:200 GB:Clustered Inactive
diskpart:Disk 12:500 GB:Clustered Active
diskpart:Disk 13:5120 MB:Clustered Active
diskpart:Disk 14:500 GB:Clustered Inactive Compared with VDS running because of me running diskpart and running: “list disk” For each disk returned, “select disk {n}” and then “detail disk”
for each disk (which does not leak memory) vds.exe uses about an extra MB of RAM when running due to the PS client. I have also now commented out the line you suggest (' $script:diskpartData = XymonDiskPart ') and now when I restart the XymonPSclient service with VDS set to Manual it does not get restarted and left running (or restarted at all). So this looks like a better workaround than just disabling the service, but I will need to continue to monitor that. Is the XymonDiskPart information used for anything? I mean, can it be used for anything without writing plug-ins?
We do not have memory leaks in powershell.exe on these servers (although I think I may have seen it on another server), so the issue above is not related to Alessandro's issue. Thanks again for your help Zak, and I hope this knowledge is of benefit to you or others too.
Kind regards,
SebA
On 14 March 2018 at 09:21, Beck, Zak <zak.beck at accenture.com<mailto:zak.beck at accenture.com>> wrote: Hi Seb
I’ve got hundreds of VMs running the Powershell client on Windows Server 2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same version of vds.exe too.
I’m wondering if function XymonDiskPart has something to do with it. This only runs on slow scans (every 6 hours-ish by default). This runs ‘diskpart’ and then
“list disk” For each disk returned, “select disk {n}” and then “detail disk”
Maybe you could try this from the command prompt on a server with vds stopped and see if it restarts it. You could also try commenting out this line (3875, towards the end) in the client and see what happens (insert a # at the beginning of the line):
$script:diskpartData = XymonDiskPartAlternatively, it could be the disk utilisation code which is using Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume, FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW, GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these doesn’t mention the Virtual Disk Service but I guess they could be using it.
Zak
From: SebA [mailto:spah at syntec.co.uk<mailto:spah at syntec.co.uk>] Sent: Tuesday, 13 March 2018 18:14 To: Beck, Zak <zak.beck at accenture.com<mailto:zak.beck at accenture.com>>; Xymon Mailing List <xymon at xymon.com<mailto:xymon at xymon.com>> Subject: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
I have noticed a memory and thread leak in the Windows Virtual Disk service - or vds.exe from the process list. This service is set to Manual by default and gets started when starting the XymonPSClient service on Windows 2008 R2. It starts off using 32 threads, but grows forever, the virtual memory and physical memory (which starts under 6 MB) used slowly grow too.
Has anyone noticed this problem and do they have a solution to it?
On Windows Server 2012, the Virtual Disk service does not get started by Xymon Windows Powershell client, so it's obviously not started directly by the client, but by a request for information. I looked into known memory leaks for vds.exe and the one I found was not applicable to 2008 R2, only 2008 and Vista. The version of vds.exe we have is later than the version with the fix: 6.1.7601.17514 from 2010.
The quick workaround is simply to stop the Virtual Disk Service - but as soon as the PSclient has used it, it does not stop properly and terminates unexpectedly, so one needs to set it not to restart on crash, or stop it immediately after starting, or to Disabled. But it would be good to know why it is starting, not stopping, and leaking threads / memory!
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com<http://www.accenture.com>
Hi Zak,
Yes, I agree, a setting in a config file to enable or disable these checks would probably be best. Thanks for the additional information.
Kind regards,
SebA
On 16 April 2018 at 10:52, Beck, Zak <zak.beck at accenture.com> wrote:
Hi Seb
Thanks for confirming, that’s good news – I was fearing it may be the core disk utilisation functions which would be a much larger headache!
Is the XymonDiskPart information used for anything? I mean, can it be used for anything without writing plug-ins?
The [diskpart] section is used in one of our datacentres where they use clustering and the information is used primarily for reporting. So no, it is not used for anything by default.
I think the best solution here is for me to add a setting in xymonclient_config.xml allowing you to turn off the diskpart function for specific servers. You can then remove your comment and continue to use future updates.
Zak
*From:* SebA [mailto:spah at syntec.co.uk] *Sent:* Friday, 30 March 2018 14:04 *To:* Beck, Zak <zak.beck at accenture.com> *Cc:* Xymon Mailing List <xymon at xymon.com> *Subject:* Re: [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
Hi Zak,
Thanks for your swift reply and sorry for my own delay! It's good to know you have such extensive use of the PS client without this issue.
I think you are probably right. These issue has only appeared on servers using Windows Cluster Failover, where various iSCSI disks are clustered.
For the record, my workaround of disabling the Virtual Disk Service caused no bad effects on the server, but it does prevent diskpart from running. I set the service back to Manual, and it started as soon as I ran diskpart from an elevated Command Prompt. Exiting diskpart causes the VDS service and proces to stop.
When the Virtual Disk Service was set to manual, and started just by me running diskpart.exe, the client output has:
[diskpart]
diskpart:::Clustered Unknown
When the service is disabled, the output is just an empty section:
[diskpart]
When VDS is started by me restarting the XymonPSclient service and it being on manual, the output is e.g.:
[diskpart]
diskpart:Disk 0:278 GB:Not Clustered
diskpart:Disk 1:1024 MB:Clustered Active
diskpart:Disk 2:25 GB:Clustered Active
diskpart:Disk 3:500 GB:Clustered Active
diskpart:Disk 4:300 GB:Clustered Inactive
diskpart:Disk 5:1024 MB:Clustered Inactive
diskpart:Disk 6:500 GB:Clustered Inactive
diskpart:Disk 7:200 GB:Clustered Inactive
diskpart:Disk 8:1024 MB:Clustered Inactive
diskpart:Disk 9:1024 MB:Clustered Inactive
diskpart:Disk 10:200 GB:Clustered Inactive
diskpart:Disk 11:200 GB:Clustered Inactive
diskpart:Disk 12:500 GB:Clustered Active
diskpart:Disk 13:5120 MB:Clustered Active
diskpart:Disk 14:500 GB:Clustered Inactive
Compared with VDS running because of me running diskpart and running:
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”
for each disk (which does not leak memory) vds.exe uses about an extra MB of RAM when running due to the PS client.
I have also now commented out the line you suggest (' $script:diskpartData = XymonDiskPart ') and now when I restart the XymonPSclient service with VDS set to Manual it does not get restarted and left running (or restarted at all). So this looks like a better workaround than just disabling the service, but I will need to continue to monitor that.
Is the XymonDiskPart information used for anything? I mean, can it be used for anything without writing plug-ins?
We do not have memory leaks in powershell.exe on these servers (although I think I may have seen it on another server), so the issue above is not related to Alessandro's issue.
Thanks again for your help Zak, and I hope this knowledge is of benefit to you or others too.
Kind regards,
SebA
On 14 March 2018 at 09:21, Beck, Zak <zak.beck at accenture.com> wrote:
Hi Seb
I’ve got hundreds of VMs running the Powershell client on Windows Server 2008R2, 2012R2 and 2016 and I don’t see this issue. I just checked a couple of servers, vds.exe is between 2MB and 20MB and around 21 threads. Same version of vds.exe too.
I’m wondering if function XymonDiskPart has something to do with it. This only runs on slow scans (every 6 hours-ish by default). This runs ‘diskpart’ and then
“list disk”
For each disk returned, “select disk {n}” and then “detail disk”
Maybe you could try this from the command prompt on a server with vds stopped and see if it restarts it. You could also try commenting out this line (3875, towards the end) in the client and see what happens (insert a # at the beginning of the line):
$script:diskpartData = XymonDiskPartAlternatively, it could be the disk utilisation code which is using Windows API calls in kernel32.dll – FindFirstVolume, FindNextVolume, FindVolumeClose, GetVolumeInformation, GetVolumePathNamesForVolumeNameW, GetDiskFreeSpaceEx and GetDriveType. The MSDN documentation for these doesn’t mention the Virtual Disk Service but I guess they could be using it.
Zak
*From:* SebA [mailto:spah at syntec.co.uk] *Sent:* Tuesday, 13 March 2018 18:14 *To:* Beck, Zak <zak.beck at accenture.com>; Xymon Mailing List < xymon at xymon.com> *Subject:* [External] Memory leak in Windows Virtual Disk service caused by Xymon Windows Powershell client
I have noticed a memory and thread leak in the Windows Virtual Disk service - or vds.exe from the process list. This service is set to Manual by default and gets started when starting the XymonPSClient service on Windows 2008 R2. It starts off using 32 threads, but grows forever, the virtual memory and physical memory (which starts under 6 MB) used slowly grow too.
Has anyone noticed this problem and do they have a solution to it?
On Windows Server 2012, the Virtual Disk service does not get started by Xymon Windows Powershell client, so it's obviously not started directly by the client, but by a request for information. I looked into known memory leaks for vds.exe and the one I found was not applicable to 2008 R2, only 2008 and Vista. The version of vds.exe we have is later than the version with the fix: 6.1.7601.17514 from 2010.
The quick workaround is simply to stop the Virtual Disk Service - but as soon as the PSclient has used it, it does not stop properly and terminates unexpectedly, so one needs to set it not to restart on crash, or stop it immediately after starting, or to Disabled. But it would be good to know why it is starting, not stopping, and leaking threads / memory!
Kind regards,
SebA
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
participants (3)
-
Alessandro.Tinivelli＠faacgroup.com
-
spah＠syntec.co.uk
-
zak.beck＠accenture.com