VM snapshot was not deleted from vCenter after Symantec Backup Exec 2010 R3 SP2 completed its backup

[haroldadalia]

Hi,

We are using Symantec Backup Exec 2010 R3 SP2 and we do backups of VMs. The settings of the backup job in backing up VMDK are as follows:

• Device:  Deduplication Storage Folder
• Transport mode priority list (SAN, Hotadd, NBD, NBDSSL)
• Back up virtual machines that are powered off (checked)
• Use GRT (checked)
• Enable application GRT (checked)
• AOFO (unchecked)

We do Full Backup of the VMDK every Friday and daily Differential Backup from Monday to Thursday and I also understand that the backup process of VMDK goes as described below based on http://www.symantec.com/docs/TECH155831:

1. Backup Exec backup job starts
2. Backup Exec tells VMware to create a snapshot of VMDK through vCenter
3. vCenter creates and mounts the snapshot to the datacenter and it now becomes the "active" snapshot (virtual machine is now running from that snaphsot)
4. Backup Exec then gets the ESX host and guest virtual machine information from vCenter it needs to access or backup
5. Backup Exec opens a connection with the ESX to ask for the virtual machine metadata
6. Using vStorage APIs, Backup Exec then opens a direct data connection to the “unknown” SAN volumes and the virtual machine data is offloaded directly to the media server for backup
   1. “Extracting the Backup Data from the Target Virtual Machine” 
   2. “Deleting the Temporary Snapshot” 
7. Once the backup completes, Backup Exec disconnects from the ESX host and vCenter
8. Backup Exec runs a verify (by default) of the data backed up
9. Backup job completes.

The problem starts here, when Symantec BE backup job starts, it will tell VMware to create a snapshot of the VMDK through vCenter and when the backup completes, Symantec will again tell VMware to delete the snapshot. What we observed is that, the created snapshot last 07/30/2012 was not deleted from vCenter, it is still existing until now. On the VMware side, a server running on snapshot is not a best practice because if the server is running on snapshot, storage vMotion is not possible. I would like to know if this issue is on Symantec BE or on VMware and what could be the possible cause of this incident? We wanted to remove this snapshot but we would definitely need any suggestions from any.

Any comments on this is greatly appreciated!! Thank a lot in advance!

Regards,

Harold Adalia

[Backup_Exec1]

Hi

Have you check the link for vmware which describes the process of consolidation

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&e...

• The remove snapshot process can take a long time to complete if the snapshots are large.
• There may me occasions where using the Consolidate option does not work. A snapshot configuration file (with .vmsd extension) in the virtual machine directory may interfere with the consolidation of the snapshots and may need to be renamed, moved to a new location, or deleted. For more information, see   Understanding virtual machine snapshots in VMware ESX (1015180)
• The commit process has no progress that you can follow. As long as the date on the files continues to update, the process is working. Also, if the virtual machine is off, you can use the file * command to see if any files are in use.
  If any of the files are locked by the consolidation process the following error message will be displayed:
  
  Can't read `filename' (Device or resource busy)

So you need to verify if this is happening for all virtual machine backup after successfull backup not able to delete the snapshot or it is happening for particular one.

Thanks

[haroldadalia]

Hi Backup_Exec,

Thank you very much for your prompt response!

I read the VMware article and it says that "If the consolidation fails, there were no snapshots shown in the Snapshot Manager, but the snapshot files were still being used on the datastore. This can cause the datastore to run out of space."

In our case, the snapshot is still visible in the snapshot manager. We are first running on VMware 4.1 Update 2 when we first encounter this issue. We have already upgraded to VMware 5 update 1 last August 23, 2012 and we are still seeing the snapshot in VM snapshot manager. 

Is it the consolidation part that could cause the incident of the snapshot not being deleted? As per Symantec knowledge base, the snapshot should get deleted after the backup completes.

Thanks!

Regards,

Harold Adalia

[pkh]

This is a long shot, but have you tried pushing out the remote agent again to the problem VM's?  Also, have you tried to select the VM's using the VM host rather than vCenter?

[Backup_Exec1]

Hi

In this case I would actually open a case with Vmware and check with them ,looks to be an issue from vmware side here

 Also to add on more here you can check the hostd.log on ESX and check if any error ad why it is not able to delete the snapshot

Moreover Try taking a snapshot by hand using the vSphere client  (quiesce checked, snap memory unchecked) and check for logs. You can continue to test this way. No need to run backups.& then try to delete the snapshot and check if you are able to delete it

Thanks