cancel
Showing results for 
Search instead for 
Did you mean: 

VMware VM "needs consolidation" after backup failure

elanmbx
Level 6

Has anyone else run into this issue?  VM backup fails with a "Status 13" or "Status 156" or "Status 40" - I have several hundred VMs and only a few end up in this state during a backup window.  No snapshots show up in Snapshot Manager on the VMs that have this error.  When consolidation is attempted VMware throws a "Unable to access file <unspecified filename> since it is locked" error.

The "fix" for us has been to run "services.sh restart" on the ESXi server - this unlocks the file - then manually consolidating the VM.

Sounds a lot like this issue: https://www-secure.symantec.com/connect/forums/problem-backup-virtual-machine-when-vm-convert-status-consolidate

Master: 7.6.0.2 on Solaris 10

Media: 2.6.0.2 on 5220 Appliances

11 REPLIES 11

elanmbx
Level 6

This appears to be similar as well: https://communities.vmware.com/message/2172747?tstart=2505

elanmbx
Level 6

More interesting info.  This is BackupExec-related: http://alpacapowered.wordpress.com/2012/06/04/snapshot-removal-issues-with-backupexec-and-locked-files/comment-page-1/

Jim-90
Level 6

All backup appplication that use the VADP api's have orphaned/zombie snapshot issues at some time.  We started using VADP backups early on and have found the orphaned snapshot problem is getting a lot less, 

The lessons we learnt was to:

  • Keep NBU up to date as much as possible through patches and EBB's to fix VMware issues.   
  • Monitor all the VM machines for the presence of snapshots either from RVTOOLS (vHealth tab) or a script that lists all the deltas present in the environment.

We decided against automated deletion as it was too risky.  If you can't consolidate the snapshots try cloning the guest, cloning forces consildation.  We successfully have done this when there were a very large number (hundrerds) of snapsots for a box. 

RVTOOLS is a good & free software for VMware reporting it will list other stuff like datastores and C: drives that are low of space etc if your software versions permit.  Only requires a read-only VM account.

Bmitche
Level 5

I run into this same thing occasionally. I do hotadd vm backups and sometimes the snapshots are still "hung-up" on the media server. I edit the media server vm and remove the "extra" disks....I can then run disk consolidation on the problem VM.

 

Anonymous
Not applicable

There is an Alarm definition in vSphere 5.x that could be used to setup email notification if you are wanting to closely monitor this.

vmconsolidatealarm.png

bcunha
Level 4
Partner Accredited Certified

Seeing the same problem with a 5230 2.6.0.2 master/media (already present in 2.6.0.1) and VMware 5.5. I've already a case opened for a long time now, without any progress.

GuillaumeB
Not applicable
Partner

Hey!

 

I am experiencing the same issue, does anyone has an answer already?


Thansk!

VirtualED
Level 5
Employee Accredited Certified

Please contact NetBackup support for a full investigation.  These issues are solvable.

Also review late breaking news, there is a patch that is needed at 7.6.0.2 and 7.6.0.3 that may help.

http://www.symantec.com/docs/TECH199999

Beyond that, look out for running too many backups/snapshot operations at the same time.  I/O contention is the major cause of consolidation failures.

Other issues I have found are:

1. Running multiple backups of the same VM at the same time.

2. Backups are hanging.

joyb81
Level 3
Certified

There are some methods which you may also do to fix it from vmware end http://aikitsupport.com/snapshot-consolidation-needed-fails-lock-message/

Let me know if any further need to lookup

ChristianWitzke
Level 3

SOLVED: with Information from the Website

1. migration to other Datastore

2. retry Consolidate

Social Media Sites:
https://de.linkedin.com/in/christian-witzke-2b311360
https://www.xing.com/profile/Christian_Witzke4
http://christian-witzke.de/

elanmbx
Level 6

We continue to suffer from this on a somewhat limited basis.  The "fix" at our site has been refined a bit - instead of restarting "services.sh" we simply restart hostd with the /etc/init.d/hostd script.  This allows us to consolidate the VM in question.

Our VMware team has expressed some interest lately in getting to the bottom of this, but for now we simply fix these manually when they crop up.