Highlighted

VMware VM "needs consolidation" after backup failure

Has anyone else run into this issue?  VM backup fails with a "Status 13" or "Status 156" or "Status 40" - I have several hundred VMs and only a few end up in this state during a backup window.  No snapshots show up in Snapshot Manager on the VMs that have this error.  When consolidation is attempted VMware throws a "Unable to access file <unspecified filename> since it is locked" error.

The "fix" for us has been to run "services.sh restart" on the ESXi server - this unlocks the file - then manually consolidating the VM.

Sounds a lot like this issue: https://www-secure.symantec.com/connect/forums/problem-backup-virtual-machine-when-vm-convert-status-consolidate

Master: 7.6.0.2 on Solaris 10

Media: 2.6.0.2 on 5220 Appliances

11 Replies
Highlighted

This appears to be similar as

This appears to be similar as well: https://communities.vmware.com/message/2172747?tstart=2505

Highlighted

More interesting info.  This

More interesting info.  This is BackupExec-related: http://alpacapowered.wordpress.com/2012/06/04/snapshot-removal-issues-with-backupexec-and-locked-files/comment-page-1/

Highlighted

All backup appplication that

All backup appplication that use the VADP api's have orphaned/zombie snapshot issues at some time.  We started using VADP backups early on and have found the orphaned snapshot problem is getting a lot less, 

The lessons we learnt was to:

  • Keep NBU up to date as much as possible through patches and EBB's to fix VMware issues.   
  • Monitor all the VM machines for the presence of snapshots either from RVTOOLS (vHealth tab) or a script that lists all the deltas present in the environment.

We decided against automated deletion as it was too risky.  If you can't consolidate the snapshots try cloning the guest, cloning forces consildation.  We successfully have done this when there were a very large number (hundrerds) of snapsots for a box. 

RVTOOLS is a good & free software for VMware reporting it will list other stuff like datastores and C: drives that are low of space etc if your software versions permit.  Only requires a read-only VM account.

Highlighted

I run into this same thing

I run into this same thing occasionally. I do hotadd vm backups and sometimes the snapshots are still "hung-up" on the media server. I edit the media server vm and remove the "extra" disks....I can then run disk consolidation on the problem VM.

 

Highlighted

There is an Alarm definition

There is an Alarm definition in vSphere 5.x that could be used to setup email notification if you are wanting to closely monitor this.

vmconsolidatealarm.png

Highlighted

Seeing the same problem with

Seeing the same problem with a 5230 2.6.0.2 master/media (already present in 2.6.0.1) and VMware 5.5. I've already a case opened for a long time now, without any progress.

Highlighted

Hey!   I am experiencing the

Hey!

 

I am experiencing the same issue, does anyone has an answer already?


Thansk!

Highlighted

Please contact NetBackup

Please contact NetBackup support for a full investigation.  These issues are solvable.

Also review late breaking news, there is a patch that is needed at 7.6.0.2 and 7.6.0.3 that may help.

http://www.symantec.com/docs/TECH199999

Beyond that, look out for running too many backups/snapshot operations at the same time.  I/O contention is the major cause of consolidation failures.

Other issues I have found are:

1. Running multiple backups of the same VM at the same time.

2. Backups are hanging.

There are some methods which

There are some methods which you may also do to fix it from vmware end http://aikitsupport.com/snapshot-consolidation-needed-fails-lock-message/

Let me know if any further need to lookup

Highlighted

SOLVED: with Information from

SOLVED: with Information from the Website

1. migration to other Datastore

2. retry Consolidate

Social Media Sites:
https://de.linkedin.com/in/christian-witzke-2b311360
https://www.xing.com/profile/Christian_Witzke4
http://christian-witzke.de/
Highlighted

We continue to suffer from

We continue to suffer from this on a somewhat limited basis.  The "fix" at our site has been refined a bit - instead of restarting "services.sh" we simply restart hostd with the /etc/init.d/hostd script.  This allows us to consolidate the VM in question.

Our VMware team has expressed some interest lately in getting to the bottom of this, but for now we simply fix these manually when they crop up.