snapshot verify failed, status 156

[ali_baniamer]

Dears

We have 5 Windows  2008 r2 hyber-v serverclients that keep getting the status code (156) snapshot error encountered error for netbackup 7.5.0.4
and master server win 2008 netbackup 7.5.0.4 

this error give 

Policy: Hyper-V_VMs5 -- client: VH_ALSHAREEF.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: VE_DARWISH.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: VE_KHWALDEH.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: VA_MASSAD.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: VH_BADWAN.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: Teller (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: VA_ABDAH.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: VG_ALNASER.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

Policy: Hyper-V_VMs5 -- client: VD_SAMARA.JDIB.JO (Hyper-V server: VDI5) --

Snapshot could not be created with Hyper-V_v2 method

7/8/2013 1:58:56 PM - Info nbjm(pid=5712) starting backup job (jobid=93804) for client vA_obale.jdib.jo, policy Hyper-V_VMs2, schedule Daily  

7/8/2013 1:58:56 PM - Info nbjm(pid=5712) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=93804, request id:{87A42EEE-6671-4890-A480-395DC236103F})  

7/8/2013 1:58:56 PM - requesting resource VD2_DeDupS

7/8/2013 1:58:56 PM - requesting resource veritas_backup.NBU_CLIENT.MAXJOBS.vA_obale.jdib.jo

7/8/2013 1:58:56 PM - requesting resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2

7/8/2013 1:58:57 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:01:28 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:01:29 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:01:32 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:01:32 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:01:39 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:01:40 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:01:43 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:01:43 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:03:44 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:03:44 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:03:45 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:03:45 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:04:12 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:04:12 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:04:18 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:04:18 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:06:25 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:06:26 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:06:36 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:06:36 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:07:48 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:07:48 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:08:24 PM - Info bpbrm(pid=9584) vA_obale.jdib.jo is the host to backup data from     

7/8/2013 2:08:27 PM - Info bpbrm(pid=9584) reading file list from client        

7/8/2013 2:08:27 PM - Info bpbrm(pid=9584) start bpfis on client         

7/8/2013 2:08:27 PM - Info bpbrm(pid=9584) Starting create snapshot processing         

7/8/2013 2:08:28 PM - Info bpfis(pid=2520) Backup started           

7/8/2013 2:09:01 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:09:02 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:09:05 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:09:05 PM - Info nbrb(pid=8720) Limit has been reached for the logical resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2    

7/8/2013 2:09:54 PM - Critical bpbrm(pid=9584) from client vA_obale.jdib.jo: FTL - snapshot creation failed, status 156   

7/8/2013 2:09:58 PM - Warning bpbrm(pid=9584) from client vA_obale.jdib.jo: WRN - ALL_LOCAL_DRIVES is not frozen    

7/8/2013 2:10:07 PM - awaiting resource VD2_DeDupS - Maximum job count has been reached for the storage unit

7/8/2013 2:10:07 PM - granted resource veritas_backup.NBU_CLIENT.MAXJOBS.vA_obale.jdib.jo

7/8/2013 2:10:07 PM - granted resource veritas_backup.NBU_POLICY.MAXJOBS.Hyper-V_VMs2

7/8/2013 2:10:07 PM - granted resource MediaID=@aaaav;DiskVolume=PureDiskVolume;DiskPool=VD2_DiskPool;Path=PureDiskVolume;StorageServer=vd2;MediaServer=vd2

7/8/2013 2:10:07 PM - granted resource VD2_DeDupS

7/8/2013 2:10:07 PM - estimated 0 Kbytes needed

7/8/2013 2:10:07 PM - begin Parent Job

7/8/2013 2:10:07 PM - begin Hyper-V, Start Notify Script

7/8/2013 2:10:07 PM - Info RUNCMD(pid=6140) started            

7/8/2013 2:10:07 PM - Info RUNCMD(pid=6140) exiting with status: 0         

Status 0

7/8/2013 2:10:07 PM - end Hyper-V, Start Notify Script; elapsed time: 00:00:00

7/8/2013 2:10:07 PM - begin Hyper-V, Step By Condition

Status 0

7/8/2013 2:10:07 PM - end Hyper-V, Step By Condition; elapsed time: 00:00:00

7/8/2013 2:10:07 PM - begin Hyper-V, Read File List

Status 0

7/8/2013 2:10:07 PM - end Hyper-V, Read File List; elapsed time: 00:00:00

7/8/2013 2:10:07 PM - begin Hyper-V, Create Snapshot

7/8/2013 2:10:07 PM - started

7/8/2013 2:10:14 PM - started process bpbrm (9584)

7/8/2013 2:10:34 PM - Info bpbrm(pid=8460) Starting delete snapshot processing         

7/8/2013 2:10:39 PM - Info bpfis(pid=0) Snapshot will not be deleted        

7/8/2013 2:10:50 PM - Info bpfis(pid=6936) Backup started           

7/8/2013 2:10:50 PM - Critical bpbrm(pid=8460) from client vA_obale.jdib.jo: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.vA_obale.jdib.jo_1373281807.1.0      

7/8/2013 2:12:04 PM - end writing

Status 156

7/8/2013 2:12:04 PM - end Hyper-V, Create Snapshot; elapsed time: 00:01:57

7/8/2013 2:12:04 PM - begin Hyper-V, Stop On Error

Status 0

7/8/2013 2:12:04 PM - end Hyper-V, Stop On Error; elapsed time: 00:00:00

7/8/2013 2:12:04 PM - begin Hyper-V, Delete Snapshot

7/8/2013 2:12:12 PM - started process bpbrm (8460)

7/8/2013 2:12:57 PM - end writing

Status 1542

7/8/2013 2:12:57 PM - end Hyper-V, Delete Snapshot; elapsed time: 00:00:53

7/8/2013 2:12:57 PM - begin Hyper-V, End Notify Script

7/8/2013 2:12:57 PM - Info RUNCMD(pid=1732) started            

7/8/2013 2:12:57 PM - Info RUNCMD(pid=1732) exiting with status: 0         

Status 0

7/8/2013 2:12:57 PM - end Hyper-V, End Notify Script; elapsed time: 00:00:00

Status 156

7/8/2013 2:12:57 PM - end Parent Job; elapsed time: 00:02:50

snapshot error encountered(156)

15:21:54.378 [10836.5316] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpfis.exe verify -nbu -pt 29 -backup_copy 0 -fim Hyper-V:nameuse=0,Virtual_machine_backup=2,prov_type=0,snap_attr=0,allow_offline_backup=1,csv_timeout=0 -vm_client VA_HAMMAD.JDIB.JO 

15:21:54.378 [10836.5316] <2> bpfis main: received FIM as [103] Hyper-V:nameuse=0,Virtual_machine_backup=2,prov_type=0,snap_attr=0,allow_offline_backup=1,csv_timeout=0

15:21:54.378 [10836.5316] <4> bpfis: Starting keep alive thread.

15:21:54.409 [10836.5316] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:21:54.409 [10836.5316] <2> logconnections: BPRD CONNECT FROM 172.20.0.92.64696 TO 192.168.101.52.1556 fd = 792

15:21:54.409 [10836.5316] <8> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:871] Ignoring VxSS authentication 2 0x2

15:21:59.089 [10836.5316] <4> bpfis: INF - BACKUP START 10836

15:21:59.089 [10836.5316] <2> bpfis main: receive filelist:<ALL_LOCAL_DRIVES> 

15:21:59.089 [10836.5316] <2> bpfis main: receive filelist:<CONTINUE> 

15:21:59.121 [10836.5316] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:21:59.121 [10836.5316] <2> logconnections: BPRD CONNECT FROM 172.20.0.92.64697 TO 192.168.101.52.1556 fd = 828

15:21:59.121 [10836.5316] <8> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:871] Ignoring VxSS authentication 2 0x2

15:22:03.769 [10836.5316] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:22:03.769 [10836.5316] <2> logconnections: BPRD CONNECT FROM 172.20.0.92.64698 TO 192.168.101.52.1556 fd = 828

15:22:03.769 [10836.5316] <8> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:871] Ignoring VxSS authentication 2 0x2

15:22:09.245 [10836.5316] <8> bpfis: WRN - VfMS error 10; see following messages:

15:22:09.245 [10836.5316] <8> bpfis: WRN - Non-fatal method error was reported

15:22:09.245 [10836.5316] <8> bpfis: WRN - vfm_configure_fi_one: method: FlashSnap, type: FIM, function: FlashSnap_init

15:22:09.245 [10836.5316] <8> bpfis: WRN - VfMS method error 3; see following message:

15:22:09.245 [10836.5316] <8> bpfis: WRN - FlashSnap_init: Veritas Volume Manager not installed.

15:22:09.245 [10836.5316] <8> bpfis: WRN - VfMS error 10; see following messages:

15:22:09.245 [10836.5316] <8> bpfis: WRN - Non-fatal method error was reported

15:22:09.245 [10836.5316] <8> bpfis: WRN - vfm_configure_fi_one: method: vxvm, type: FIM, function: vxvm_init

15:22:09.245 [10836.5316] <8> bpfis: WRN - VfMS method error 3; see following message:

15:22:09.245 [10836.5316] <8> bpfis: WRN - vxvm_init: Veritas Volume Manager not installed.

15:22:09.261 [10836.5316] <4> bpfis: INF - FIS_ID=1373286114

15:22:09.261 [10836.5316] <4> bpfis: INF - Testing freeze of ALL_LOCAL_DRIVES using snapshot method Hyper-V.

15:22:09.261 [10836.5316] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_10836_-1

15:22:09.261 [10836.5316] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:22:09.276 [10836.5316] <2> logconnections: BPRD CONNECT FROM 172.20.0.92.64699 TO 192.168.101.52.1556 fd = 844

15:22:09.276 [10836.5316] <8> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:871] Ignoring VxSS authentication 2 0x2

15:22:12.318 [10836.5316] <4> bpfis: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_10836_-1

15:22:12.318 [10836.5316] <32> bpfis: FTL - VfMS error 11; see following messages:

15:22:12.318 [10836.5316] <32> bpfis: FTL - Fatal method error was reported

15:22:12.318 [10836.5316] <32> bpfis: FTL - vfm_freeze: method: Hyper-V, type: FIM, function: Hyper-V_vfm_freeze_commit

15:22:12.318 [10836.5316] <32> bpfis: FTL - VfMS method error 6; see following message:

15:22:12.318 [10836.5316] <32> bpfis: FTL - snapshot services: snapshot target analysis failed: provider-private error.

15:22:12.318 [10836.5316] <32> bpfis: FTL - VfMS error 11; see following messages:

15:22:12.318 [10836.5316] <32> bpfis: FTL - Fatal method error was reported

15:22:12.318 [10836.5316] <32> bpfis: FTL - vfm_freeze: method: Hyper-V, type: FIM, function: Hyper-V_vfm_freeze_commit

15:22:12.318 [10836.5316] <32> bpfis: FTL - VfMS method error 6; see following message:

15:22:12.318 [10836.5316] <32> bpfis: FTL - snapshot services: snapshot target analysis failed: provider-private error.

15:22:12.318 [10836.5316] <16> bpfis: FTL - snapshot verify failed, status 156

15:22:12.318 [10836.5316] <4> bpfis: Starting keep alive thread.

15:22:12.318 [10836.5316] <4> bpfis: INF - EXIT STATUS 156: snapshot error encountered

[Mark_Solutions · Accepted solution]

I think the issue here is this:

snapshot services: snapshot target analysis failed: provider-private error.

What disk is used by your Hyper-V server?

I have seen similar issues when disk such as Hitachi SANs are used and they want to use the Hitachi snapshot method by default but this gets denied as it is not configured.

I would reccomend as an initial test to edit your Hyper-V policy .... on the Hyper-V tab click the Advanced button .. under the snapshotprovider change it from Auto to System and then re-try the backups

Hope this helps