Highlighted

BackupExec 2010 R3 Failure on vSphere/ESXi 5

I'm having an issue backing up virtual machines in our new vSphere 5 environment.

Our setup is:
2x Dell PowerEdge R710 ESXi 5 Hosts
1x Dell EqualLogic PS6100x SAN
1x Dell PowerEdge R510 Backup Server - Windows 2008 R2 & BackupExec 2010 R3 SP2
1x Dell PowerEdge R610 Windows 2008 R2 & vCenter Server 5

We have 4x Windows 2003 R2 virtual machines, 2 running CitriX XenApp 5, 1 running Oracle, and 1 running an IIS6 website. None of these VMs are currently in production, as we are trying to get the backup working first.

However, when I try to perform a backup of these VMs, they fail more often than they are successful. The last backup I just tried was of the 2 Citrix VMs, one was successful, one failed. Neither are in use, or have any users logged in. I have installed the BackupExec remote agent (RAWS) from the BE 2010 R3 server, and VMware tools from the vCenter/vSphere client. The backup is running using VMware SAN transport mode (see image). However the backup is failing with the following error:
---------------------------------------------
Completed status: Failed
Final error: 0xe000959a - An attempt to take a snapshot of a virtual machine failed because it was unable to quiesce an application.
Final error category: Resource Errors
-----------------------------------------------
In the Windows event logs of the VMs is the following error:
-----------------------------------------------
Event Type:      Error
Event Source:      VSS
Event Category:      None
Event ID:      12289
Description:
Volume Shadow Copy Service error: Unexpected error WaitForSingleObject(000002B8,40000) == [0x00000000].  hr = 0x00000000.
-----------------------------------------------
Event Type:      Error
Event Source:      VSS
Event Category:      None
Event ID:      12293
Description:
Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {564d7761-7265-2056-5353-2050726f7669}. Routine details CommitSnapshots({3533e446-e2aa-4ef8-9ae7-2daf81f3448e}) [hr = 0x800705b4].
-----------------------------------------------
Event Type:      Error
Event Source:      VSS
Event Category:      None
Event ID:      12298
Description:
Volume Shadow Copy Service error: The I/O writes cannot be held during the shadow copy creation period on volume \\?\Volume{ce659ab0-7f04-11e1-9c7b-806e6f6e6963}\. The volume index in the shadow copy set is 0. Error details: Open[0x00000000], Flush[0x00000000], Release[0x00000000], OnRun[0x8000ffff].
-----------------------------------------------
So far I have tried the following fixes:

Re-installing VMware Tools and removing the VMware Snapshot provider. I have also checked the VSS folder has been removed from the vmware tools\drivers folder, and the snapshot provider isn't listed in the registryas per the follwoing Symantec KB: http://www.symantec.com/business/support/index?page=content&id=TECH129724&actp=search&viewlocale=en_...

I've tried running "vssadmin add shadowstorage /for:=c: /on:=d: /maxsize=4gb" to incrase the snapshot storage space.

I've tried reinstalling the BE 2010 R3 RAWS agent, as well as the BE VSS Provider agent found under "C:\Program Files\Symantec\Backup Exec\Agents\BEVSSProvider"

I've also tried rebooting each VM serveral times!

 

7 Replies
Highlighted

Have you tried re-installing

Have you tried re-installing the RAWS in the problem VM's?

Highlighted

I've run a repair install. Do

I've run a repair install. Do you think I should completely uninstall & re-installed?

Highlighted

Before you do that, try

Before you do that, try re-installing the RAWS in the VM's first.

Highlighted

Hi,I uninstalled RWAS from

Hi,

I uninstalled RAWS from 10 of the servers, it appeared to work initially, as I was able to perform a successful backup (see attachment), but then when I ran a scheduled job of the same servers 30 mins later, it failed again on some of the servers with the 'e000959a - An attempt to take a snapshot of a virtual machine failed because it was unable to quiesce an application.' error message (see attachment2).

I did notice after re-installing that in Add/Remove Programs some of the install sizes are different on different servers, (see attachment 3 & 4), some at 184mb, some at 200mb, even though they are all Windows 2003 R2, all the same version of RAWS, and all installed from the same BE 2010 R3 server!

Ben

Highlighted

You can click on the

You can click on the properties of the beremote.exe and check that you have got the same version.

Highlighted

The file versions are the

The file versions are the same - 13.0.5204.125 - however the file sizes & dates are different - working server beremote - 1.76mb and 13th Feb 2012 vs failing server 1.21mb and 1st March 2012 (see attached pic)!

Highlighted

Check the beremote.exe on the

Check the beremote.exe on the media server.  Make sure that the beremote.exe on the remote server correspond to the beremote.exe on the media server.  If necessary, uninstall the RAWS and push out the RAWS again.