Backup failure with error code 156-VM Backup

[ankur1809]

Backup failure with error code 156 .Failure on VM backup as Snapshot job is getting failed. Kindly confirm if this is something to be done @VM end itself. 01/12/2015 19:37:38 - Info nbjm (pid=10211) starting backup job (jobid=5530957) for client wpdbasn001, policy PROD_VMWARE_VW_Adhoc, schedule Differential-Inc 01/12/2015 19:37:38 - Info nbjm (pid=10211) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=5530957, request id:{C195FC52-9AC4-11E4-A7F2-03215F26AE7F}) 01/12/2015 19:37:38 - requesting resource STU_DDR1_DALMEDIA4 01/12/2015 19:37:38 - requesting resource nbutx2.NBU_CLIENT.MAXJOBS.wpdbasn001 01/12/2015 19:37:38 - requesting resource nbutx2.NBU_POLICY.MAXJOBS.PROD_VMWARE_VW_Adhoc 01/12/2015 19:37:38 - granted resource nbutx2.NBU_CLIENT.MAXJOBS.wpdbasn001 01/12/2015 19:37:38 - granted resource nbutx2.NBU_POLICY.MAXJOBS.PROD_VMWARE_VW_Adhoc 01/12/2015 19:37:38 - granted resource MediaID=@aaae7;DiskVolume=ddr1_vw;DiskPool=DDR1_DP;Path=ddr1_vw;StorageServer=ddr1;MediaServer=dalmedia4 01/12/2015 19:37:38 - granted resource STU_DDR1_DALMEDIA4 01/12/2015 19:37:38 - estimated 0 kbytes needed 01/12/2015 19:37:38 - begin Parent Job 01/12/2015 19:37:38 - begin VMware: Start Notify Script 01/12/2015 19:37:38 - Info RUNCMD (pid=26895) started 01/12/2015 19:37:38 - Info RUNCMD (pid=26895) exiting with status: 0 Operation Status: 0 01/12/2015 19:37:38 - end VMware: Start Notify Script; elapsed time 0:00:00 01/12/2015 19:37:38 - begin VMware: Step By Condition Operation Status: 0 01/12/2015 19:37:38 - end VMware: Step By Condition; elapsed time 0:00:00 01/12/2015 19:37:38 - begin VMware: Read File List Operation Status: 0 01/12/2015 19:37:38 - end VMware: Read File List; elapsed time 0:00:00 01/12/2015 19:37:38 - begin VMware: Create Snapshot 01/12/2015 19:37:38 - started process bpbrm (pid=4016) 01/12/2015 19:37:39 - Info bpbrm (pid=4016) wpdbasn001 is the host to backup data from 01/12/2015 19:37:39 - Info bpbrm (pid=4016) reading file list for client 01/12/2015 19:37:39 - Info bpbrm (pid=4016) start bpfis on client 01/12/2015 19:37:39 - Info bpbrm (pid=4016) Starting create snapshot processing 01/12/2015 19:37:40 - Info bpfis (pid=7756) Backup started 01/12/2015 19:37:41 - snapshot backup of client wpdbasn001 using method VMware_v2 01/12/2015 19:37:59 - Info bpbrm (pid=4016) INF - vmwareLogger: WaitForTaskCompleteEx: Unable to access file since it is locked <232> 01/12/2015 19:37:59 - Info bpbrm (pid=4016) INF - vmwareLogger: WaitForTaskCompleteEx: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE 01/12/2015 19:37:59 - Info bpbrm (pid=4016) INF - vmwareLogger: ConsolidateVMDisks: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE 01/12/2015 19:37:59 - Info bpbrm (pid=4016) INF - vmwareLogger: ConsolidateVMDisksAPI: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE 01/12/2015 19:37:59 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - VMware_freeze: VIXAPI freeze (VMware snapshot) failed with 36: SYM_VMC_TASK_REACHED_ERROR_STATE 01/12/2015 19:37:59 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - VMware error received: Unable to access file since it is locked 01/12/2015 19:38:00 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 01/12/2015 19:38:00 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - 01/12/2015 19:38:00 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 01/12/2015 19:38:00 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - 01/12/2015 19:38:00 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - snapshot processing failed, status 156 01/12/2015 19:38:00 - Critical bpbrm (pid=4016) from client wpdbasn001: FTL - snapshot creation failed, status 156 01/12/2015 19:38:00 - Warning bpbrm (pid=4016) from client wpdbasn001: WRN - ALL_LOCAL_DRIVES is not frozen 01/12/2015 19:38:00 - Info bpfis (pid=7756) done. status: 156 01/12/2015 19:38:00 - end VMware: Create Snapshot; elapsed time 0:00:22 01/12/2015 19:38:00 - Info bpfis (pid=7756) done. status: 156: snapshot error encountered 01/12/2015 19:38:00 - end writing Operation Status: 156 01/12/2015 19:38:00 - end Parent Job; elapsed time 0:00:22 01/12/2015 19:38:00 - begin VMware: Stop On Error Operation Status: 0 01/12/2015 19:38:00 - end VMware: Stop On Error; elapsed time 0:00:00 01/12/2015 19:38:00 - begin VMware: Delete Snapshot 01/12/2015 19:38:00 - started process bpbrm (pid=10372) 01/12/2015 19:38:01 - Info bpbrm (pid=10372) Starting delete snapshot processing 01/12/2015 19:38:02 - Info bpfis (pid=6292) Backup started 01/12/2015 19:38:02 - Critical bpbrm (pid=10372) from client wpdbasn001: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.wpdbasn001_1421113058.1.0 01/12/2015 19:38:02 - end writing Operation Status: 4207 Operation Status: 156 01/12/2015 19:38:03 - Info bpfis (pid=6292) done. status: 4207 01/12/2015 19:38:03 - end VMware: Delete Snapshot; elapsed time 0:00:03 01/12/2015 19:38:03 - Info bpfis (pid=6292) done. status: 4207: Could not fetch snapshot metadata or state files snapshot error encountered (156)

[inn_kam]

Snapshot files are not deleted from datastore folder after completion of VMware VM backups - Error in vCenter - "Unable to access file since it is locked <197>"

Article:TECH217279

|

Created: 2014-05-06

|

Updated: 2014-09-04

|

Article URL http://www.symantec.com/docs/TECH217279

 https://www-secure.symantec.com/connect/forums/vmware-snapshots-being-locked-netbackup-7602-upgrade

vStorage snapshots are orphaned due to "Unable to access file since it is locked" error during snapshot deletion

Article:TECH181118

|

Created: 2012-02-09

|

Updated: 2014-06-27

|

Article URL http://www.symantec.com/docs/TECH181118

[inn_kam]

STATUS CODE 156: VMware Consolidated Backup fails with status 156 (snapshot error encountered) reported

Article:TECH56887 Article URL http://www.symantec.com/docs/TECH56887

|

Created: 2008-01-18

|

Updated: 2010-01-26

|

https://www-secure.symantec.com/connect/forums/vmware-snapshot-error-encountered-156

[ankur1809]

Hard to understand as it is mainly related to VM level. Any document if it could be explained in simple language.

[Marianne · Accepted solution]

Anything that is 'mainly related to VM level' should be handed over to your VMware admin. 

Chat to VM admin to find out what the vCenter version is, then have a look at TNs posted above to see if any of them matches your environment.

TECH217279 may be applicable - speak to VM admin.
TECH181118 is probably not applicable since you are already on NBU 7.6. Check bpfis log on Backup Host and compare with TN.

TECH56887 is not applicable since VCB backups are no longer supported.
 

[ankur1809]

The scenario says if snapshot files in the datastore gets locked results in further backup keeps failing and they are to be unlocked and to be deleted manually. Or in some other case it needs disk consolidation as well. If there is a document or SOP related to the same. Kindly suggest.

[Marianne]

Are you saying you need documentation other than http://www.symantec.com/docs/TECH217279 ?

The TN provides a link to VMware KB article: http://kb.vmware.com/kb/2040846 .

Have you spoken to VM admins yet?
Do you know what the vCentre version is?

[ankur1809]

Yes, i do get in discussion with VM team on such issues as they used to take care of the same. VM version is 5.5. I just wanted to know if in general scenario is this backup admin responsibility or Vm admins take care of the same.

[Marianne]

If the issue is resolved with VMware (as stated in TECH217279 and VM KB 2040846, then the issue must be addressed by VM admins.

If issue is solved by NBU patch (as per TECH181118) then issue is addressed by backup admin. 

Extremely important that Backup Admin and VM Admin work as team (same as with any other backup agent).

When errors are seen, check logs on both sides, use Google and Support sites for both products to search detailed error messages.