04-04-2017 09:07 AM - edited 04-04-2017 09:08 AM
Hi everyone,
We have been experiencing VSS issues with VMware guests in regards to the installed Backup Exec Agent and a previously installed VMware Tools VSS option.
Uninstalling the VMware Tools VSS option in various ways including restarts did not fix our issues. If you search the internet for solutions, you find many attempts but no real solution or explanation.
One of our admins spend several hours with the Veritas support without a solution, he was about to escalate the issue with them, when we found the root cause and could actually fix our issues.
First the steps to solve this:
So what happened?
You uninstalled the VMware Tools VSS provider, but this manifest file did not get updated. We actually could see that it sometimes does get updated and sometimes does not. This seems to be some kind of issue with the VMware Tools uninstalled/installer.
But why this manifest.txt file?
As we found out, there scripts that get executed by Symantec/Veritas Backup Exec before the backup. You might find them in two locations, and it seems to depend a bit on the Windows version which script is executed (at which location). You could edit them both and just undo the checks in the scripts, but this wouldn't be correct. It is more correct to update the manifest.txt file. If you want to, you can check the date/time of the manifest.txt file before you change it - you might see it was not updated while you uninstalled the VMware Tools VSS provider (assuming you did only do this and not do additional installs/uninstalls within the VMware Tools / please note as well that this only is true when you still experienced those issues).
Now, back to those scripts, you find them here:
The name of the script that matters:
This script checks several DLLs, registry entries, paths and on Windows 2008 and newer the ProgramData-Path for this specific manifest.txt file and the two entries mentioned.
Once you uninstall the VMware VSS provider, and the file did not get updated, you might see this issue and wonder how to solve it. The solution is to simply update it to mirror the uninstallation of the VMware VSS provider (vcbprovider). We double checked this with several installations and could see if the file actually gets updated, those two values are set to FALSE, if it doesn't, at least one of the values remains true, what causes the pre-freeze-script.bat to write the registry key mentioned earlier and therefor causing the issue in the backup - exceptions.
If you still have the same issues after updating the manifest.txt, simply check all the DLL's that are mentioned in the script and make sure they don't exist. You might also consider to manually delete the registry-key (it seems to be just a dumy-key) to make sure there is no issue that prevents the script from deleting it. Make sure it does not re-appear after a backup! Otherwise you might still have some DLLs left on your system that cause the script to re-create the registry key.
Hope this helps a few of you out there. This was an ongoing issue for a while and I came accross those issues many times ever since Windows 2008. This applies to Windows 2008 R2, Windows 2012, Windows 2012 R2 and pretty sure to Windows 2016 as well.
It helped us getting rid of those issues completely and actually not even needing a single restart of the guest VM to solve the issues (removing the VMware VSS provide did not need a restart).
Regards
Florian Rossmark
Solved! Go to Solution.
05-02-2017 08:19 AM
Thanks to the information from Florian we have now updated an existing technote with regards to the issue
https://www.veritas.com/docs/000009506
For those reading this it is important to note that there have been changes in Backup Exec 16 FP1 when used in combination with VMware Tools 10.1.x (which we have included in the TN.)
04-04-2017 12:46 PM
Hi Florian,
This is valuable...too valuable to be left in the forums. I'd suggest adding this as an Article or even a Blog so that it doesn't get lost.
@AlexMatts is it possible to pin this to the forum too?
Thanks!
04-04-2017 06:53 PM
An easy way to ensure that the BE VSS provider is used in place of the VMTools provider is to re-install the BE remote agent in the VM's after any VMTools update.
04-05-2017 08:04 AM - edited 04-05-2017 08:04 AM
Hello, VOX community members:
@CraigV, thank you for highlighting this matter for our review; we are agreed on the import of the content shared by @frossmark, and I am now in the process of determining the best means of showcasing this solution, alongside the Veritas Support Engineering team.
I will update you all upon this thread with the results.
Again, we appreciate your dedication to the VOX community, and assisting its members in seeking and identifying solutions. They say it 'takes a village to raise a child,' though I'd also argue it 'takes a village to run effective infrastructure.' Not as catchy, perhaps, but no less accurate!
04-05-2017 09:02 AM
Hi Florian, this looks like very useful information
Can I clarify some points:
At what point in your steps do you re-install the VMware tools? (As they are uninstalled in Step 1)
Also when you re-installed the VMware tools did you choose the custom install option so that you can choose not to install the VMware VSS driver?
Thanks
Colin Weaver
Backup Exec Technical Support
04-05-2017 09:37 AM
Hi Colin,
The VMware Tools never get uninstalled, you only MODIFY/CHANGE them via Control Pannel / Programs and disable/uninstall the option Volume Shadow Copy Services Support.
This does not need a restart and you won't need to do a new installation of the tools and avoid installing the VMware VSS provider.
Please let me know if you have any further questions, and sorry for the confusion.
Regards
Florian Rossmark
04-05-2017 11:24 PM
04-25-2017 05:34 AM
Hi Florian - another couple of quick clarifications
Did you find you had to remove for the registry the key called
HKLM\System\CurrentControlSet\Services\BeVssProvid
?
or did adjusting the manifest.txt file content alone solve the condition?
Also I know you found you did not have to reboot the VM but between steps 8 and 9 did you have to restart the BE Remote Agent service or did you just find that the backup job could be run?
Thanks
Colin
04-25-2017 08:12 AM
Hi Colin,
To clarify this a bit - I did exactly what I wrote - I did not need to delete the reg-key since the script will take care of this automatically. If you look in to the script you will realize that it does checks and if they are sucessfull it will delete the reg-key in question.
While correcting the manifest.txt, the script will be successful. The issue is not BUEX, it is related to some circumstances that cause the manifest.txt not to change...
Therefor you don't need to restart any services as well, BUEX does exactly what it needs to do.
It was even more clear that this only happened when the manifest.txt wasn't updated after changing the installation (modify) and e.g. removing the VSS provider. The date of the file never changed.
Florian
04-26-2017 03:03 AM
Thanks Florian that helps
05-02-2017 08:19 AM
Thanks to the information from Florian we have now updated an existing technote with regards to the issue
https://www.veritas.com/docs/000009506
For those reading this it is important to note that there have been changes in Backup Exec 16 FP1 when used in combination with VMware Tools 10.1.x (which we have included in the TN.)