Not Able - Full Backup Using vSphere Agent BK2010 R3

[Normandy]

For unknown reasons one particular VM will no longer do a weekly full backup (*.vmdk files) using the vSphere agent addon in Backup Exec 2010 R3. 

Information About The VM:

• Windows 2008 x64
• Using Backup Exec agent (backup system state)
• Using vSphere agent to backup the entire VM (easier restore)

Error:

• An attempt to take a snapshot of a VM failed because it was unable to quiesce an application.
• Final error: 0xe000959A

Troubleshooting Steps I've Taken:

• Rebooted the VM
• Uninstalled VMware Tools & reinstalled
• Uninstalled VMware Tools & reinstalled but didn't install the VSS capability
• Deleted the selection list & backup job; then recreated them.
• Verified admin credentials for the backup/selection job (correct).

I'm kinda at my wits end troubleshooting this.  Additional troubleshooting tips/suggestion great appreciated.

[VJware]

Is this a GRT or a non GRT backup failure ? Alternatively, when one fails, does the other complete ?

Have you customized the post thaw and the pre freeze scripts ?

Did you also uninstall VMware Filesystem Sync Driver from VMware tools ? (This would require a reboot)

[pkh]

You should push out the BE remote ageent again to the VM.  This will disable the VMTools VSS provider and install the BE VSS provider which is the recommended provider.

[Normandy]

Is this a GRT or a non GRT backup failure?
-I did see any specific mention to it being a GRT failure.  Although the 'Use Backup Exec GRT' option is selected in the VMware vSphere agent backup job.

Alternatively, when one fails, does the other complete?
-The regular Windows server agent backup job does complete sucessfully (Incremental backup).

Have you customized the post thaw and the pre freeze scripts?
-No, I haven't customized any scrips.  Honestly I don't even know what those are.

Did you also uninstall VMware Filesystem Sync Driver from VMware tools ? (This would require a reboot)
-I think so, I first:

• Completely uninstalled VMware Tools; then rebooted the VM.
• Installed VMware Tools, but this time chose the custom install option and made sure VSS option was set to not install.  Then rebooted the VM.
• Tried the VMware agent backup again (full), but it failed.

[VJware]

Uncheck the usage of GRT from this VM's job & check if the backup completes successfully or not...

While installing VMWare tools, modify the install & Volume Shadow Copy Services Support along with VMware Filesystem Sync Driver can be uninstalled...

[Ben_L_]

Are there any errors in vCenter when the snapshot attempt fails?

[Normandy]

Uncheck the usage of GRT from this VM's job & check if the backup completes successfully or not...
-Unfortunately the backup job errored out again:

An attempt to take a snapshot of a VM failed because it was unable to quiesce an application.

While installing VMWare tools, modify the install & Volume Shadow Copy Services Support along with VMware Filesystem Sync Driver can be uninstalled...
-While doing this I noticed that Volume Shadow Copy Services Support is not installed.  I was unable to find VMware Filesystem Sync Driver option in the VMware Tools Installation window.

VMware Tools Installation window (what I see):

Toolbox

• WMI Performance Logging

VMware Device Driver

• Record/Replay Driver
• Paravirtual SCSI
• Virtual Printing (not installed)
• Memory Control Driver
• Mouse Driver
• SVGA Driver
• Audio Driver
• VMXNet NIC Driver
• VMXNet3 NIC Driver
• Volume Shadow Copy Services Support (not installed)
• VMCI Driver: vShield Driver & Shared Folders (both not installed)
• WYSE Multimedia Support Driver (not installed)

[Normandy]

Oddly enough when I looked at the vCenter logs nothing

%ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Logs\

However, when I left the vSphere client window open & ran the backup job I got an error:

The guest OS has reported an error during quiescing.  The error code was:

5 The error message was: 'VmBackupOnThaw' operation failed.

3 The error message was: Custom quiesce script failed.

Never seen these errors before... good thing I left vSphere client running.

[Normandy]

Is there a way to push out the agent again without uninstalling it from the server first?

[pkh]

Sure.

[Normandy]

Based on the error I posted in reply to Ben. L's question I guess the problem is related with a customized the post thaw and the pre freeze scripts?  Recommendations?

[fginacio]

The following tricks working without uninstalling the VMWare tools support for VSS. Please be advised that I've tried this method on all of my failed to backup of Windows Server 2008/2003 x64 VMs

The reason why I'm doing this again is that I feel uncomfortable in removing the VSS snapshot provider support in VMWare tools:

1. Backup the following regkey: *HKLM\Software\Microsoft\EventSystem*
2. Erase the regkey HKLM\Software\Microsoft\EventSystem{26c409cc-ae86-11d1-b616-00805fc79216} an below 
3. Reboot. 
4. Verify that things are working by running

vssadmin list writers
vssadmin list providers. 

and then in the services.msc list of services snap ins, make sure the following services are Started Automatically:

1. COM+ System Application
2. Distributed Transaction Coordinator
3. Volume Shadow Copy
4. VMware Physical Disk Helper Service

Automatically but not started:

1. Microsoft Software Shadow Copy Provider
2. VMware Snapshot Provider 

This is the solution that i tried around myself without uninstalling the VSS Provider support in VMWare tools.

See the following KB URL: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1009073&sl...

Take a look at this TN also:

http://www.symantec.com/business/support/index?page=content&id=TECH70486

Hope this can be a helpful post for you.

[Normandy]

Thanks for the reply.  Once I deleted that registry key things (VSS & other services) went south.  That's why we always backup.

[Lesta_G]

So, is your problem solved?

[fginacio]

Try to take some snapshot with quiesce mode enable at vcenter.

Remember to check the "quiesce mode" check box.

Maybe the root cause is the VMware.