Vmware backup failed - error code 156

[Sivaranjani]

Setup:

Master server netbackup version : 7.5

Vcenter server : windows 2008 r2

Esxi version : 5.1

Proxy server : windows 2008 r2 ( NBU client 7.5)

There are 8 esxi servers and more than 30 VMs.

Backup is taken for around 20 VMs, in which backup is failing for 5 vms with the same error.

Troubleshooting steps followed till now:

1)      The backup is failing with the below error.

Netbackup job details

Critical bpbrm (pid=2361) from client XXXXX: FTL - VMware snapshot failed: SYM_VMC_TASK_REACHED_ERROR_STATE

10/07/2013 16:21:38 - Critical bpbrm (pid=2361) from client XXXXX: FTL - VMware error received: An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.

10/07/2013 16:21:38 - Critical bpbrm (pid=2361) from client XXXXX: FTL - snapshot creation failed, status 156

10/07/2013 16:21:38 - Warning bpbrm (pid=2361) from client XXXXX: WRN - ALL_LOCAL_DRIVES is not frozen

VMWARE snap shot creation failed:

Vmware error:

The guest OS has reported an error during quiescing. The error code was: 3 The error message was: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

2)      Tried to take snapshot(with quiesce option enabled) in the VM from Vsphere client GUI and it completed successfully.

3)      Verified the disk space availability more than 70 GB available.

4)      Rebooted one of the VM and retired the backup. But still backup failed with same error.

http://www.symantec.com/connect/forums/status-code-156

5)      In the above article they have faced the same issue, after reinstalling the vmware tools it worked.

6)      Reinstalled the VM tools but still same issue.

[RamNagalla]

13:56:33.352 [2512.1148] <32> onlfi_freeze_fim_fs: FTL - VfMS error 11; see following messages:
13:56:33.352 [2512.1148] <32> onlfi_freeze_fim_fs: FTL - Fatal method error was reported
13:56:33.352 [2512.1148] <32> onlfi_freeze_fim_fs: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze
13:56:33.352 [2512.1148] <32> onlfi_freeze_fim_fs: FTL - VfMS method error 7; see following message:
13:56:33.352 [2512.1148] <32> onlfi_freeze_fim_fs: FTL - VMware_freeze: VIXAPI freeze failed with 36

below is the exact T/N that is explining the error that you have..

http://www.symantec.com/business/support/index?page=content&id=TECH166561

and also another related tech note

http://www.symantec.com/business/support/index?page=content&id=TECH141518

let us know if they did help... 

[Sivaranjani]

Hi,

I have tried to disable the "quiesce" option in the policy and retried the backup. The snapshot creation completed successfully and the backup is in progress. 

Quiesce: If the <quiesce> flag is 1 or true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.

It is recommended to take quiesced backup.

The same quiesced snapshot is running properly on the affected VM when I start it from Vcenter server.

[RamNagalla]

so seems disabling the quiesce is working  though it is not highly recommmented method..

so did you try looking in 2nd T/N too and see if that helps .. 

http://www.symantec.com/business/support/index?page=content&id=TECH141518

if this helps disabling the quiesce is not required.only AppQuiescing will get disabled..

[Sivaranjani · Accepted solution]

The below steps are given by Symantec.

Uninstalled vmware vss provider .
 
1.     Uninstall VMware Tools.
2.     Allow the system to reboot.
3.     Reinstall VMware Tools. Ensure that you click Custom Install.
4.     Deselect VSS.(Volume Shadow Copy Services Support)

Once this is done install the symantec vss provider.
 

We have tried this workaround, now the VMs backups are running fine ( quiesced = enabled).