ESX 4.1 Backup Failing/Hanging At Exporting State

[Ben_Blackmore1]

Hi,

We have a VMWare ESX 4.1 infrastructure, which runs a number of our servers. One Friday we had an issue where one the the servers failed due to the datastore it used running out of space. This failure occured during our ESX backup job, which failed with:

 V-79-57344-38220 - Backup of the virtual machine 'VMS01' failed. VMware VCB framework reported the following error
 Error: Other error encountered: Snapshot creation failed: The operation cannot be allowed at the current time because the virtual machine has a question pending:
 'There is no more space for virtual disk VMS01-000003.vmdk. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.'
 

Once we'd moved the failed server to a new datastore with more free space, and got the server back up and running, we performed another backup. However, this failed with another error :

V-79-57344-38220 - Backup of the virtual machine 'VMS01' failed. VMware VCB framework reported the following error
 Error: Backup snapshot already exists.

So it seems that the initial failure has caused a VCB snapshot to get stuck in a failed state. Having looked into this error I have checked the option 'Delete any existing VCB snapshots. Backups of the virtual machine fail if a VCB snapshot exists’, and have tried to re-run the backup, however, this just seems to cause the backup job to sit in an 'Exporting' state, so far for several hours, and as of yet the byte count is still zero!

Can anyone suggest a solution to this issue, and a way to stop it from re-occuring should a failure happen in the future?

Many thanks

Ben
 

[CraigV]

Hi Ben,

Have you checked within ESX itself and deleted the snapshot there if it exists?

[Ben_Blackmore1]

HI,

Yes, there was a _VCB-BACKUP_ snapshot, which I deleted. However the date/time of that file says it was created when the current backup job started, not when the previous backup job that failed started.

I've tried running another last night, which I've just come into the office and found has been running for 4 hours, stuck in the exporting state, zero bytes backed up. Now there are 3 snapshots:
'Consolidate helper-0'
'_VCB-BACKUP_'
'SYMC-FULL 20-04-2011 08:49' (the date & time now, after cancelling the backup job)

[Ben_Blackmore1]

I found an article that suggested creating a snapshot, and then deleting all snapshots. This seems to have resolved the exporting problem, but now I get another error:

V-79-57344-38277 - Unable to open a disk of the virtual machine.

VixDiskLib_Open() reported the error:
V-79-57344-38277 - Unable to open a disk of the virtual machine.

VixDiskLib_Open() reported the error:

I'm not sure why or which file is open!

Any help would be appreciated!

Ben

[CraigV]

Ben: You might want to check the solution on the forum query below:

https://www-secure.symantec.com/connect/forums/backup-error-unable-open-disk-virtual-machine

Might assist you...

[Ben_Blackmore1 · Accepted solution]

Hi,

I managed to resolve this by rebooting the vCenter server. When it came back up it showed a load of snapshots it wasn't showing previously, or that it had said it had deleted. So I selected to delete all snapshots, and this time it took 2 hours 10 minutes to deleted (10 minutes to get from 5% - 99%, then 2 hours to finish). Previously it had just taken seconds to delete the snapshots, but looks like it wasn't actually doing anything.

After deleting the snapshots, I brought the VM back up, and took a successful backup!

Looks like it was the vCenter server that was holding the files open/locked.

Many thanks for the help

Ben