snapshot error encountered (156)

[mrauf]

Hello Friends,

i am facing an issue in VMWare VSphere Backup issue with the error code snampshot error encountered - 156.

Master Server :- 7.6.1.1 / Windows 2008 Enterprise 64 bit

Media Server:- 7.6.1.1/Red Hat Linux

VMWare : - VMWare VSphere 6.0

Job Details: - 

5/25/2016 1:31:05 PM - Info nbjm(pid=6932) starting backup job (jobid=872704) for client pbad-vc, policy VMWare-vCenter, schedule Daily  
5/25/2016 1:31:05 PM - Info nbjm(pid=6932) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=872704, request id:{02BE2026-FC59-47A1-A179-F4F9CA96A5E3})  
5/25/2016 1:31:05 PM - requesting resource pbad_fsmedia_Dedup_STU
5/25/2016 1:31:05 PM - requesting resource pbad-netbackup.pbad.sbg.com.sa.NBU_CLIENT.MAXJOBS.pbad-vc
5/25/2016 1:31:05 PM - requesting resource pbad-netbackup.pbad.sbg.com.sa.NBU_POLICY.MAXJOBS.VMWare-vCenter
5/25/2016 1:31:05 PM - granted resource pbad-netbackup.pbad.sbg.com.sa.NBU_CLIENT.MAXJOBS.pbad-vc
5/25/2016 1:31:05 PM - granted resource pbad-netbackup.pbad.sbg.com.sa.NBU_POLICY.MAXJOBS.VMWare-vCenter
5/25/2016 1:31:05 PM - granted resource MediaID=@aaaay;DiskVolume=PureDiskVolume;DiskPool=pbad_fsmedia_Dedup_Pool;Path=PureDiskVolume;StorageServer=pbad-fsmedia.pbad.sbg.com.sa;MediaServer=pbad-fsmedia.pbad.sbg.com.sa
5/25/2016 1:31:05 PM - granted resource pbad_fsmedia_Dedup_STU
5/25/2016 1:31:05 PM - estimated 0 Kbytes needed
5/25/2016 1:31:05 PM - begin Parent Job
5/25/2016 1:31:05 PM - begin VMware, Start Notify Script
5/25/2016 1:31:05 PM - started
5/25/2016 1:31:06 PM - Info RUNCMD(pid=15444) started            
5/25/2016 1:31:07 PM - Info RUNCMD(pid=15444) exiting with status: 0         
Status 0
5/25/2016 1:31:07 PM - end VMware, Start Notify Script; elapsed time: 0:00:02
5/25/2016 1:31:07 PM - begin VMware, Step By Condition
Status 0
5/25/2016 1:31:07 PM - end VMware, Step By Condition; elapsed time: 0:00:00
5/25/2016 1:31:07 PM - begin VMware, Read File List
Status 0
5/25/2016 1:31:07 PM - end VMware, Read File List; elapsed time: 0:00:00
5/25/2016 1:31:07 PM - begin VMware, Create Snapshot
5/25/2016 1:31:08 PM - started process bpbrm (12023)
5/25/2016 1:33:57 PM - end writing
Status 156
5/25/2016 1:33:57 PM - end VMware, Create Snapshot; elapsed time: 0:02:50
5/25/2016 1:33:57 PM - begin VMware, Stop On Error
Status 0
5/25/2016 1:33:57 PM - end VMware, Stop On Error; elapsed time: 0:00:00
5/25/2016 1:33:57 PM - begin VMware, Delete Snapshot
5/25/2016 1:33:59 PM - started process bpbrm (13061)
5/25/2016 1:34:01 PM - end writing
Status 4207
5/25/2016 1:34:01 PM - end VMware, Delete Snapshot; elapsed time: 0:00:04
5/25/2016 1:34:01 PM - begin VMware, End Notify Script
5/25/2016 1:49:44 PM - Info bpbrm(pid=12023) pbad-vc is the host to backup data from     
5/25/2016 1:49:44 PM - Info bpbrm(pid=12023) reading file list for client        
5/25/2016 1:49:44 PM - Info bpbrm(pid=12023) start bpfis on client         
5/25/2016 1:49:46 PM - Info bpbrm(pid=12023) Starting create snapshot processing         
5/25/2016 1:49:48 PM - Info bpfis(pid=12029) Backup started           
5/25/2016 1:49:48 PM - snapshot backup of client pbad-vc using method VMware_v2
5/25/2016 1:49:52 PM - Info bpbrm(pid=12023) INF - vmwareLogger: ChangeTracking not supported       
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - VMware_freeze: VIXAPI freeze (VMware snapshot) failed with -1: Unrecognized error
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - VMware error received: The operation is not allowed in the current state.
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL -        
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 
5/25/2016 1:52:32 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL -        
5/25/2016 1:52:32 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - snapshot processing failed, status 156   
5/25/2016 1:52:32 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - snapshot creation failed, status 156   
5/25/2016 1:52:32 PM - Warning bpbrm(pid=12023) from client pbad-vc: WRN - ALL_LOCAL_DRIVES is not frozen    
5/25/2016 1:52:32 PM - Info bpfis(pid=12029) done. status: 156          
5/25/2016 1:52:32 PM - end VMware, End Notify Script; elapsed time: 0:18:31
5/25/2016 1:52:32 PM - Info bpfis(pid=12029) done. status: 156: snapshot error encountered       
5/25/2016 1:52:35 PM - Info bpbrm(pid=13061) Starting delete snapshot processing         
5/25/2016 1:52:35 PM - Info bpfis(pid=13065) Backup started           
5/25/2016 1:52:35 PM - Critical bpbrm(pid=13061) from client pbad-vc: cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.pbad-vc_1464172265.1.0       
5/25/2016 1:52:35 PM - Info bpfis(pid=13065) done. status: 4207          
5/25/2016 1:52:35 PM - end Parent Job; elapsed time: 0:21:30
5/25/2016 1:52:35 PM - Info bpfis(pid=13065) done. status: 4207: Could not fetch snapshot metadata or state files  
5/25/2016 1:34:01 PM - Info RUNCMD(pid=4756) started            
5/25/2016 1:34:02 PM - Info RUNCMD(pid=4756) exiting with status: 0         
Status 0
5/25/2016 1:34:02 PM - end Parent Job; elapsed time: 0:02:57
Status 156
5/25/2016 1:34:02 PM - end Parent Job; elapsed time: 0:02:57
5/25/2016 1:49:44 PM - Info bpbrm(pid=12023) pbad-vc is the host to backup data from     
5/25/2016 1:49:44 PM - Info bpbrm(pid=12023) reading file list for client        
5/25/2016 1:49:44 PM - Info bpbrm(pid=12023) start bpfis on client         
5/25/2016 1:49:46 PM - Info bpbrm(pid=12023) Starting create snapshot processing         
5/25/2016 1:49:48 PM - Info bpfis(pid=12029) Backup started           
5/25/2016 1:49:48 PM - snapshot backup of client pbad-vc using method VMware_v2
5/25/2016 1:49:52 PM - Info bpbrm(pid=12023) INF - vmwareLogger: ChangeTracking not supported       
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - VMware_freeze: VIXAPI freeze (VMware snapshot) failed with -1: Unrecognized error
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - VMware error received: The operation is not allowed in the current state.
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL -        
5/25/2016 1:52:31 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 
5/25/2016 1:52:32 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL -        
5/25/2016 1:52:32 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - snapshot processing failed, status 156   
5/25/2016 1:52:32 PM - Critical bpbrm(pid=12023) from client pbad-vc: FTL - snapshot creation failed, status 156   
5/25/2016 1:52:32 PM - Warning bpbrm(pid=12023) from client pbad-vc: WRN - ALL_LOCAL_DRIVES is not frozen    
5/25/2016 1:52:32 PM - Info bpfis(pid=12029) done. status: 156          
5/25/2016 1:52:32 PM - end operation
5/25/2016 1:52:32 PM - Info bpfis(pid=12029) done. status: 156: snapshot error encountered       
5/25/2016 1:52:35 PM - Info bpbrm(pid=13061) Starting delete snapshot processing         
5/25/2016 1:52:35 PM - Info bpfis(pid=13065) Backup started           
5/25/2016 1:52:35 PM - Critical bpbrm(pid=13061) from client pbad-vc: cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.pbad-vc_1464172265.1.0       
5/25/2016 1:52:35 PM - Info bpfis(pid=13065) done. status: 4207          
5/25/2016 1:52:35 PM - end operation
5/25/2016 1:52:35 PM - Info bpfis(pid=13065) done. status: 4207: Could not fetch snapshot metadata or state files  
snapshot error encountered (156)

Please help me to resolve the above issue.

[mrauf]

Hello Michal,

yea i took the snapshot from vSphere console directly and it was successfully,but while take backup from netbackup i identify that it's gives the error vmwareLogger: ChangeTracking not supported and it will processed to child job and its gives error code snapshot error encountered (156).

Regards

[Marianne]

Please ignore the references to Horizon View in this TN to understand the difference between VMware snapshot and NBU snapshot:

http://www.veritas.com/docs/000023854 

Error that we are getting from vCenter is indicating that it is failing on the Quiesce of the machine.

A snapshot taken within vCenter works just fine.

Reason we have these results is in VMware when you do a snapshot it does a "Snapshot the virtual machine's memory" and does not do a "Quiesce guest file system" unless you select that option. When coming from NetBackup it is just the opposite. It relies on the quiesce of the virtual machine to do the backup.

[mrauf]

Hello Marianne, 

as per the artrical of netbackup which you share me i tried to keep Volume Shadow Copy Service on Manul mode.  but the issue is still persists.

Regards

[Marianne · Accepted solution]

What happens when you take a snapshot in VMware and select "Quiesce guest file system" option?

Have you tried to explore the 'operation is not allowed in the current state ' error? 
Please understand that this error is coming from VMware, not NetBackup.

I am about to give up as you seem to reply to part of advice attempts, not everything.

My only other suggestion is to log a call with Veritas and VMware with high level logging enabled.

[Farissi_Oussama]

Hello everyone,

I hope you are doing well!

I have the same error with NetBackup 10.1, Backup of some VMware clients is failing with a snapshot error encountered(156). I would greatly appreciate any assistance or guidance you can provide.

you can find the script of the error. attached. 

Thank you in advance for your support.