Problem with Backup Exec 2010 and Vsphere 4 snapshot processing

[Roob]

Hello, I have a problem with backing up some Windows server VMs in Vsphere 4. When I try to back up some VMs i get an error: Completed status: Failed Final error: 0xe0009574 - Unable to create a snapshot of the virtual machine. The virtual machine may be too busy to quiesce to take the snapshot. Final error category: Resource Errors For additional information regarding this error refer to link V-79-57344-38260 This link says only that I have to delete all snapshots for this VM, but in my case I do not have any pre made snapshots. I can see in Vsphere client that Backup Exec starts creating snapshot of VM, but after 8-9 minutes it gives up and I get an error above. If I look into Vsphere client, snapshot process is still not finished, and it sits on 95%. After few more minutes snapshot process finishes and snapshot is made, but it is too late because Backup Exec already fails with above error. It looks like there is some kind of time out somewhere, but I cannot find any solution all over the net. I am using agent installed on VMs. It is strange that this happens only on heavy load machines (like SQL with a lot of DBs), and other not heavy loaded machines do not have this problem and I can back them up without any problem. Let me say that I can create manual snapshot without any problem, so I do not think that problem is in Vsphere. Does anybody have same problem or any solution? Please help me I am desperate :(

[fiNal]

do you have this issue with just one VM or with every VM in your virtual datacenter?

this seems to be more of a vmware problem rather than a backup exec problem. you should figure out why it takes 10 minutes for your vm to create a snapshot ...

we are running an sql server with three instances and relatively heavy load, an exchange server with >300GB databases, and a file server with over a TB storage. none of those vms needs longer than 30 seconds to create a snapshot.

try deleting all snapshots of the VM and then create one via vsphere client. does it also take 10 minutes? if it does, try shutting down the vm and then power it on again. are you using thin provisioning disks? and what kind of storage are those disks on?

[Roob]

Your post made me think again.

So I decided to go through all points again. I patched ESX with latest updates and after that manual snapshots are faster than before. I experimented with VM with SQL that gave me most problems and I found out that if I have BE agent installed on that VM, than snapshot is not finished and backup fails. I also found out that if I back up only SQL databases I randomly I get SQLVDI errors in WMs applicaton log. So I decided to uninstall BE agent and now backups run without problem. I also tried other VMs and backup on them with agent installed runs ok, so I think that there must be a problem on particular VM. Anyway  I dont need granular restore of SQL so ...

Thanks for your help.

[plu34]

I have the exact same problem in my Vsphere 4 inviroment.
I get the error below some times on VM servers. The servers I get this on is random i the Vsphere 4 enviroment, and it is servers that have very litle load. If the problem below starts for one server, it continue until I reboot the server. After I reboot the backup works fine.

Completed status: Failed
Final error: 0xe0009574 - Unable to create a snapshot of the virtual machine. The virtual machine may be too busy to quiesce to take the snapshot.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-38260

This link says only that I have to delete all snapshots for this VM, but in my case I do not have any pre made snapshots.

I have no problems taking snapshots of the VM:s in Vsphere 4.

[Roob]

I suggest you try without BE agent installed in VM just to see if this is same problem I have. Looks to me like there is a problem with BE after all ...

[rbane]

I'm having the same issue.  Did anyone figure out a solution other than removing the BE agent?  Without the BE agent you cannot do the granular backup\restore...

[vfortier]

We are having the same issue with a customer.

The storage is Sun 7110 accessed in NFS.

VMWare is at version 4.0u1

Anybody got answers from Symantec?

[Donovan_Nederbi]

I'm also having the same issue.
The solutions that symantec offers is:
Delete the snapshots for the guest machines on the ESX / Vsphere 4.0 Host using the snapshot manager.

But the snapshots are being created by Backup Exec itself and are not removed: Snapshot for full backup created by BackupExec on 04-05-2010 03:49

Everyday I'm deleting the snapshots created and really getting frustrated about it. Last week I was free from work and when I came back on monday, there were 7 snapshots created on 1 VM.
I'm going to call support today for this issue and hope the will come with a solution shortly.

Donovan

[rbane]

For me it seems to happen even when there are no saved snapshots for the VM.  Let us know what you find out from support.  Thanks,

Rob

[Donovan_Nederbi]

For spending almost 2 hours on the phone with an technician we changed Transport mode priority list order setting:
- HotAdd
- NBD
Deselected SAN and NBDSSL.

We created a backup for a "problem" VM, by selecting just 1 VM and got an succesfull backup.
As the backup started we say in VC that the backup makes a snapshot of the backup and when finish it's being removed.

He concluded that when BE creates snapshots, maybe alot of snapshots are being created at the same time.
We have like 20 VM's so maybe the error is being generated that the server is indeed too busy making snapshots.
He said, maybe it's better to split the backup in more jobs then just one huge one.

I going to check whether the options I changed solved the problem otherwise I'm going to split them and will see the results.

I'm not saying this is the solution yet, I will keep in touch with support and will keep you guys posted.
BTW.. also made him read this thread.

Donovan

[rbane]

I'll give that a shot as well.  Thanks for the update.

Rob

[teiva-boy]

It's a good practice to have jobs setup individually, rather than one large job anyways...  It makes administration easier.  Thats right, you heard me, easier!

You can do concurrent jobs to disk, you can pause individual jobs for maintenance, you kick off backup jobs as needed without having to start a large one, or create a one-off for a particular task.  

And in the end, it improves your success rate where if you have 100 jobs for 100 servers, you can have a near 100% success rate.  Where as if you have 4 jobs with 25 servers in each job, one server can fail, failing one job, giving you only a 75% success rate in your backups.  Something that auditors will judge harshly against you.

[mhorner]

Hi Chaps, I've had this problem and worked through it, here is the playlist of fixes.

The error you're getting can be because the VMtools VVS provider is interfering with the BEVSSProvider. The solution is to do a custom reinstall of VMTools and deselect the File system sync driver and the VSS driver. A SAN based backup should then work.

If you get the same error then as others have said try a standard VM snapshot. The problem with this I've seen is that if you have a VM with some large RDM's attached then the block size of the VMFS will need to be increased and the total size of the VMFS MUST be bigger than the largest RDM (free space is not important only total size), this can cause the snapshot to fail.

[Rick100]

Any news if Symantec is coming our with a hotfix?? I have some VMs that work fine & others that get this error. It doesn't make sense. I've compared jobs, made all types of changes, tried some of the ideas listed.

No luck.

[Mifi01]

Hi all.

I have the same problem. After restarting the vm the backup works for 1 week fine. Ohter VM's on the same infrastrukture works allways perfect. It is realy frustrating with the new BEX 2010 Version at all :(

Michi

[gml542]

I had this problem with two different servers.

I have two different solutions.

1. Shut down the Virtual Server and run the VMware Agent backup while the server is off.

Power the server back on once the backup is complete and try the backup again while the server is on.  Very good chance this will fix the issue.

2. Same error, Different Problem:

C:\Program Files\VMware\VMware Tools\backupscripts.d\freeze.bat  was missing

Worked great once I loaded that file.

[stadt-frlg]

We encounter the same issue here. The interesting thing is that it just happens on our Win2003 Servers with MSSQL running.

An information beforehand: we got through all the advises and problem solutions on this topic. In detail: we actually stop all services which *could* result in a workload while doing the snapshots (with the pre-freeze and post-thaw scripts). We checked all the advises on http://www.symantec.com/business/support/index?page=content&id=TECH129724. But it didn't lead us to a long-term solution for now. The only "solution" is to restart the affected servers regularly.

The fact that Backup Exec doesn't do it's job is really frustrating for us. In the past we did our backups with our own scripts and there wasn't any need to come in for work at the weekends just because the backups failed sporadically. It nearly never failed in the past. That's the job of a backup software! Now we bought a promising solution and what did we get ...?

With Backup Exec we are really frustrated and have to think for an alternative product if the Backup Exec Team doesn't solve this and gives pseudo solutions like splitting all jobs in smaller ones (yes, I *would* do it if there was a good solution with the policies but at the current state I see no possibility for a Backup2Disk2Tape Backup if I split all the jobs).

Backup Exec Team: please find a solution for this without forcing us to turn our backup processes inside out (e.g. splitting all the jobs). Many people knock their heads on this topic and I didn't see any reply from Symantec. Backup's Exec Graphical Interface and possibilities are nice, but it should do one thing smoothly: Backups !!

I'm looking forward to get a support employee on the phone on monday and then we will see if there's another solution than all the things which didn't help on the web.

Regards

Christian

[stadt-frlg]

Maybe we found a solution by installing a Microsoft VSS update rollup package (KB940349) on our Windows 2003 servers. Since we applied this rollup package we didn't get a single backup failure for a few weeks now. We're happy and are looking forward to have this issue solved persistently.

The strange thing was that these updates obviously did not come with the normal WSUS updates. Therefore we had to apply the fixes manually. I wonder why microsoft does not provide these fixes within the normal windows updates.

I hope that this will be a good hint for some people.

Greetings
christian
 

[HandsOn]

Thanks for the tip Christian, I've installed this Rollup Package and will post the results.

GRML542, could you give us a bit more detail on your first solution? Do you mean run a backup of the Remote VM server when it's offline? 

Regards,

Sander