VMWare backup failed with RC 156

[tarangdce]

All,

I have created a policy with 22 clients and 16 failed with below error and rest 6 successful at 1st attempt. 
Please share your suggession!!!

Error log as below.

10/9/2012 6:16:50 AM - Info nbjm(pid=15412) starting backup job (jobid=1420847) for client FDYTS407, policy VMBACKUP_BL465_1, schedule FULL  

10/9/2012 6:16:50 AM - Info nbjm(pid=15412) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1420847, request id:{2CE7B461-FDFA-4F34-AE08-317C9DAD68E8})  

10/9/2012 6:16:50 AM - requesting resource Findlay_Backup

10/9/2012 6:16:50 AM - requesting resource mps6645.NBU_CLIENT.MAXJOBS.FDYTS407

10/9/2012 6:16:50 AM - requesting resource mps6645.NBU_POLICY.MAXJOBS.VMBACKUP_BL465_1

10/9/2012 6:16:50 AM - granted resource mps6645.NBU_CLIENT.MAXJOBS.FDYTS407

10/9/2012 6:16:50 AM - granted resource mps6645.NBU_POLICY.MAXJOBS.VMBACKUP_BL465_1

10/9/2012 6:16:50 AM - granted resource MediaID=@aaaao;DiskVolume=fdyddsvr001-bkp-lsu;DiskPool=fdyddsvr001-bkp-lsu;Path=fdyddsvr001-bkp-lsu;StorageServer=fdyddsvr001-10g;MediaServer=MPS6647

10/9/2012 6:16:50 AM - granted resource Findlay_Backup

10/9/2012 6:16:55 AM - estimated 0 Kbytes needed

10/9/2012 6:16:55 AM - begin Parent Job

10/9/2012 6:16:55 AM - begin VMware, Start Notify Script

10/9/2012 6:16:55 AM - Info RUNCMD(pid=18744) started            

10/9/2012 6:16:55 AM - started

10/9/2012 6:16:56 AM - Info RUNCMD(pid=18744) exiting with status: 0         

Status 0

10/9/2012 6:16:56 AM - end VMware, Start Notify Script; elapsed time: 00:00:01

10/9/2012 6:16:56 AM - begin VMware, Step By Condition

Status 0

10/9/2012 6:16:56 AM - end VMware, Step By Condition; elapsed time: 00:00:00

10/9/2012 6:16:56 AM - begin VMware, Read File List

Status 0

10/9/2012 6:16:56 AM - end VMware, Read File List; elapsed time: 00:00:00

10/9/2012 6:16:56 AM - begin VMware, Create Snapshot

10/9/2012 6:16:57 AM - started process bpbrm (8892)

10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) FDYTS407 is the host to backup data from     

10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) reading file list from client        

10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) start bpfis on client         

10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) Starting create snapshot processing         

10/9/2012 6:16:59 AM - Info bpfis(pid=9856) Backup started           

10/9/2012 6:17:20 AM - snapshot backup of client FDYTS407 using method VMware_v2

10/9/2012 6:18:49 AM - end writing

Status 156

10/9/2012 6:18:49 AM - end VMware, Create Snapshot; elapsed time: 00:01:53

10/9/2012 6:18:49 AM - begin VMware, Stop On Error

Status 0

10/9/2012 6:18:49 AM - end VMware, Stop On Error; elapsed time: 00:00:00

10/9/2012 6:18:49 AM - begin VMware, Delete Snapshot

10/9/2012 6:18:50 AM - started process bpbrm (6908)

10/9/2012 6:18:51 AM - Info bpbrm(pid=6908) Starting delete snapshot processing         

10/9/2012 6:18:51 AM - Info bpfis(pid=0) Snapshot will not be deleted        

10/9/2012 6:18:53 AM - end writing

Status 1542

10/9/2012 6:18:53 AM - end VMware, Delete Snapshot; elapsed time: 00:00:04

Status 156

10/9/2012 6:18:53 AM - end Parent Job; elapsed time: 00:01:58

snapshot error encountered(156)

[Marianne]

Are you trying to backup 22 VM's simultaneously?

If you are using automatic selection of virtual machines (Query Builder) have a look a Resource Limits.

You can limit number of simultaneous backups per ESX server, per Datastore, maximum number of simultaneous snapshots, etc... See NetBackup for VMware Administrator's Guide .

The other option is to limit the number of simultaneous backups per policy. Use the Limit jobs per policy setting on the policy Attributes tab.

[tarangdce]

I am not using automatic selection. All the vms are selected from ESX cluster and writing to same vmware backup host which has 16CPU and 12GB RAM.

Please suggest.

[Anonymous]

I would look to maybe slice and dice your VM's into more than 1 policy to troubleshoot. If this is the first time run...to attempt to backup these VM's.

Status 156 is a deep chasm of possible problems/root causes.

Unless these are ALL deployed from a same template, I would doubt if all these 22 VM's are the same.

Check the hardware level compatibility. NBU 7.5 supports no greater than vmx-8. vmx-9 is available via vsphere 5.1 (not supported by NBU yet - 7.6 maybe)

Are any of the VM's using IDE controllers for their VMDK's. Need to be SCSI.

Are all the datastores these 22 VM's are stored presented to the Vmware backup host.?

Can you snapshot all 22 VM's from vsphere client without any problems.?

Are the correct Vmware credentials given with the right amount of privileges in vsphere applied from a top level hierarchy.?

I would maybe think about using automatic selection but based on a folder level to begin with.

Are VMware Tools installed. Running a recent/current version/status.

Also, update to the latest maintenance/update pack for NetBackup.

Then start to look at logs on the vmware backup host.

Lots of things to check out.  Also...

Resources:

Netbackup 7.1 for Vmware best practices
http://www.symantec.com/content/en/us/enterprise/white_papers/b-netbackup_best_practices_WP_21195951.en-us.pdf

Datasheet: NetBackup 7.5 for VMware
http://eval.symantec.com/mktginfo/enterprise/fact_sheets/b-netbackup_7-5_for_vmware_DS_21003628-3.en-us.pdf

[tarangdce]

Unless these are ALL deployed from a same template, I would doubt if all these 22 VM's are the same.

They are different Operating system.

Check the hardware level compatibility. NBU 7.5 supports no greater than vmx-8. vmx-9 is available via vsphere 5.1 (not supported by NBU yet - 7.6 maybe)

All are hardware level vmx-8

Are any of the VM's using IDE controllers for their VMDK's. Need to be SCSI.

All are SCSI controllers

Are all the datastores these 22 VM's are stored presented to the Vmware backup host.?

YES

Can you snapshot all 22 VM's from vsphere client without any problems.?

Need to check out

Are the correct Vmware credentials given with the right amount of privileges in vsphere applied from a top level hierarchy.?

YES. I can take successful backup of 6 host out of those

I would maybe think about using automatic selection but based on a folder level to begin with.

As of now i have selected "ALL LOCAL DRIVES". I can try with C:\ only.

Are VMware Tools installed. Running a recent/current version/status.

YES

NetBackup Master & Media servers are 7.5.0.3

I will check out logs and try with just C:\ and see if that make difference.

[Anonymous]

Just to clarify:

By folder level, I meant within the vsphere infrastructure. not at an OS level.

Please leave at ALL_LOCAL_DRIVES

Also lots of good reasons for 156 status discussed in this forum post

https://www-secure.symantec.com/connect/forums/status-156-vmware-vstorage-backup

[tarangdce]

Thanks for different inputs. Actual issue was selecting VM DISPLAY NAME instead of VM HOSTNAME.

Issue is solved now.