cancel
Showing results for 
Search instead for 
Did you mean: 

Status 156 on VMware vstorage backup

Bruce_Burgdoff
Level 3

 I created a list of all of our Vmware servers that were not getting backed up and setup policies for them. It was quite a few. I am receiving the following errors on a bunch of them. We are running Netbackup Enterprise Server 7.0.1.

 

   map_integer_value: virtual_machine_supported=VMware, converted value=1, status=1
09:37:54.438 [2192.3040] <2> read_vfm_conf: Found FIM: NAS_Snapshot
09:37:54.438 [2192.3040] <2> read_vfm_conf: Found library: C:\Program Files\Veritas\NetBackup\bin\libfi_NAS_Snapshot.dll
09:37:54.438 [2192.3040] <2> parse_ssm_parms: Found keyword max_snapshots
09:37:54.438 [2192.3040] <2> parse_ssm_parms: Found data type %d
09:37:54.438 [2192.3040] <2> parse_ssm_parms: Found value specification 1-:1
09:37:54.438 [2192.3040] <2> parse_value_specification: Value specification='1-:1'
09:37:54.438 [2192.3040] <2> parse_value_specification: Null maximum value; using INT_MAX (2147483647)
09:37:54.438 [2192.3040] <2> parse_value_specification: Found default 1
09:37:54.438 [2192.3040] <2> parse_ssm_parms: Found prompt Maximum Snapshots (Instant Recovery only)
09:37:54.438 [2192.3040] <2> read_vfm_conf: Found FIM: VSS_Writer
09:37:54.438 [2192.3040] <2> read_vfm_conf: Found library: C:\Program Files\Veritas\NetBackup\bin\libfi_vss_writer.dll
09:37:54.438 [2192.3040] <2> vnet_pbxConnect: pbxConnectEx Succeeded
09:37:54.438 [2192.3040] <2> logconnections: BPRD CONNECT FROM 10.230.190.148.57067 TO 10.240.192.50.1556 fd = 624
09:37:54.641 [2192.3040] <8> bpfis: WRN - VfMS error 10; see following messages:
09:37:54.641 [2192.3040] <8> bpfis: WRN - Non-fatal method error was reported
09:37:54.641 [2192.3040] <8> bpfis: WRN - vfm_configure_fi_one: method: FlashSnap, type: FIM, function: FlashSnap_init
09:37:54.641 [2192.3040] <8> bpfis: WRN - VfMS method error 3; see following message:
09:37:54.641 [2192.3040] <8> bpfis: WRN - FlashSnap_init: Veritas Volume Manager not installed.
09:37:54.641 [2192.3040] <8> bpfis: WRN - VfMS error 10; see following messages:
09:37:54.641 [2192.3040] <8> bpfis: WRN - Non-fatal method error was reported
09:37:54.641 [2192.3040] <8> bpfis: WRN - vfm_configure_fi_one: method: vxvm, type: FIM, function: vxvm_init
09:37:54.641 [2192.3040] <8> bpfis: WRN - VfMS method error 3; see following message:
09:37:54.641 [2192.3040] <8> bpfis: WRN - vxvm_init: Veritas Volume Manager not installed.
09:37:54.657 [2192.3040] <4> bpfis: INF - FIS_ID=sps-test.poweripl.com_1322577470
09:37:54.657 [2192.3040] <4> bpfis: INF - Freezing ALL_LOCAL_DRIVES using snapshot method VMware.
09:37:54.657 [2192.3040] <4> bpfis: INF - ACT=sps-test.poweripl.com
09:37:54.657 [2192.3040] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_2192
09:37:54.657 [2192.3040] <2> vnet_pbxConnect: pbxConnectEx Succeeded
09:37:54.657 [2192.3040] <2> logconnections: BPRD CONNECT FROM 10.230.190.148.57068 TO 10.240.192.50.1556 fd = 688
09:38:49.501 [2192.3040] <4> bpfis: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_2192
09:38:49.501 [2192.3040] <32> bpfis: FTL - VfMS error 11; see following messages:
09:38:49.501 [2192.3040] <32> bpfis: FTL - Fatal method error was reported
09:38:49.501 [2192.3040] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
09:38:49.501 [2192.3040] <32> bpfis: FTL - VfMS method error 7; see following message:
09:38:49.501 [2192.3040] <32> bpfis: FTL - VMware_freeze: VIXAPI freeze failed with 36
09:38:49.501 [2192.3040] <32> bpfis: FTL - VfMS error 11; see following messages:
09:38:49.501 [2192.3040] <32> bpfis: FTL - Fatal method error was reported
09:38:49.501 [2192.3040] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
09:38:49.501 [2192.3040] <32> bpfis: FTL - VfMS method error 7; see following message:
09:38:49.501 [2192.3040] <32> bpfis: FTL - VMware_freeze: VIXAPI freeze failed with 36
09:38:49.501 [2192.3040] <16> bpfis: FTL - snapshot creation failed, status 156
09:38:49.501 [2192.3040] <4> bpfis: INF - Thawing ALL_LOCAL_DRIVES using snapshot method VMware.
09:38:49.501 [2192.3040] <8> bpfis: WRN - ALL_LOCAL_DRIVES is not frozen
09:38:49.501 [2192.3040] <8> bpfis: WRN - snapshot delete returned status 20
09:38:51.501 [2192.3040] <2> vnet_pbxConnect: pbxConnectEx Succeeded
09:38:51.501 [2192.3040] <2> logconnections: BPRD CONNECT FROM 10.230.190.148.57072 TO 10.240.192.50.1556 fd = 676
09:38:52.735 [2192.3040] <4> bpfis: INF - EXIT STATUS 156: snapshot error encountered
09:38:55.751 [2816.4756] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpfis.exe delete -nbu -id sps-test.poweripl.com_1322577470 -bpstart_to 300 -bpend_to 300 -clnt viper.poweripl.com -S airboss.poweripl.com
09:38:55.766 [2816.4756] <2> vnet_pbxConnect: pbxConnectEx Succeeded
09:38:55.766 [2816.4756] <2> logconnections: BPRD CONNECT FROM 10.230.190.148.57139 TO 10.240.192.50.1556 fd = 560
09:38:57.016 [2816.4756] <4> bpfis: INF - BACKUP START 2816
09:38:57.016 [2816.4756] <32> bpfis: FTL - cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.sps-test.poweripl.com_1322577470.0
09:38:57.016 [2816.4756] <2> vnet_pbxConnect: pbxConnectEx Succeeded
09:38:57.016 [2816.4756] <2> logconnections: BPRD CONNECT FROM 10.230.190.148.57142 TO 10.240.192.50.1556 fd = 8
09:38:58.266 [2816.4756] <4> bpfis: INF - EXIT STATUS 0: the requested operation was successfully completed
09:50:34.036 [2756.1932] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpfis.exe delete -nbu -id vm-xpcl-mbs-mtr.poweripl.com_1322577155 -bpstart_to 300 -bpend_to 300 -clnt viper.poweripl.com -S airboss.poweripl.com
09:50:34.051 [2756.1932] <2> vnet_pbxConnect: pbxConnectEx Succeeded
09:50:34.051 [2756.1932] <2> logconnections: BPRD CONNECT FROM 10.230.190.148.57290 TO 10.240.192.50.1556 fd = 560
09:50:35.520 [2756.1932] <4> bpfis: INF - BACKUP START 2756
09:50:35.520 [2756.1932] <2> read_vfm_conf: options=0x2
09:50:35.520 [2756.1932] <2> read_vfm_conf: Reading file: C:\Program Files\Veritas\NetBackup\vfm.conf
09:50:35.520 [2756.1932] <2> read_vfm_conf: Found FIM: auto
09:50:35.520 [2756.1932] <2> read_vfm_conf: Found library: C:\Program Files\Veritas\NetBackup\bin\libfi_auto.dll
09:50:35.520 [2756.1932] <2> read_vfm_conf: Found FIM: Hyper-V
09:50:35.520 [2756.1932] <2> read_vfm_conf: Found library: C:\Program Files\Veritas\NetBackup\bin\libvfms.dll
09:50:35.520 [2756.1932] <2> parse_ssm_parms: Found keyword nameuse
09:50:35.520 [2756.1932] <2> parse_ssm_parms: Found data type %dd
09:50:35.520 [2756.1932] <2> parse_ssm_parms: Found value specification client_name_type_list
09:50:35.520 [2756.1932] <2> parse_value_specification: Value specification='client_name_type_list'
09:50:35.520 [2756.1932] <2> parse_value_specification: Found default client_name_type_list

 

9 REPLIES 9

CRZ
Level 6
Employee Accredited Certified

Is this a VERBOSE = 5 log?  I'm expecting to see some more entries, but I may be assuming you're verbose when you're not...

09:37:54.657 [2192.3040] <2> logconnections: BPRD CONNECT FROM 10.230.190.148.57068 TO 10.240.192.50.1556 fd = 688
09:38:49.501 [2192.3040] <4> bpfis: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_2192

I do see 45 seconds passing there - could we be failing to quiesce in time?  We've got a lot of TechNotes about that.

Vmware backup fails 'unable to quiesce filesystem'

 http://symantec.com/docs/TECH145960

status 156 - snapshot failed - vixapi freeze failed with 36 - vc error: creating quiesced snapshot failed because snapshot operation exceeded timeout limit.

 http://symantec.com/docs/TECH137001

156:Receive "snapshot creation failed, status 156" when attempting to backup a VM.

 http://symantec.com/docs/TECH154889

You could test it by disabling quiesce and retrying the backup - if it works, you've pinned down the cause, but resolving it may take some help from VMware.

DISCLAIMER: This is one of many topics where I really have no idea what I'm talking about, but I do see a lot of patterns.  :)

Bruce_Burgdoff
Level 3

I  disabled quiesce on a couple of the problem servers and they work. So what do I do now?  

CRZ
Level 6
Employee Accredited Certified

Well, you'll have to decide if you want to keep quiescing disabled - which I believe isn't recommended - or put in some work to figure out how to get your VMs into a state where they CAN be quiesced.

This is an older TechNote for 6.5.x but still valid for 7.x and might offer you some different paths to follow:

Error: Other error encountered: Snapshot creation failed: Could not quiesce file system.

 http://symantec.com/docs/TECH126993

Here's a newer one discussing quiescing which I haven't linked to yet:

Status 156 : VMware backups fail with "snapshot creation failed, status 156 WRN - ALL_LOCAL_DRIVES is not frozen " Quiescing

 http://www.symantec.com/docs/TECH160314

From there, we link to some VMware documentation as well:

Troubleshooting Volume Shadow Copy (VSS) quiesce related issues

 http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1007696

Again, this may be an issue to talk with VMware about instead of us.

OR maybe someone here with real world experience can offer a better suggestion.  :)

Good luck!

Robert_Lindenbu
Level 3
Employee Accredited Certified

Bruce,

The first thing I would verify is that I can create snapshots from vCenter. To replicate the snapshot that NetBackup does, tke a snapshot w/o memory but with quiesce enabled. If you have problems doing this, it is 100% a VMware issue and you need to talk to them.

If all else fails, you can throw a client on the troublesome VMs and back them up normally

 

Rob

schmaustech
Level 5

Are all the clients on the same datastore?  Usually you have issues when trying to snap multiple clients on the same datastore within VMWare.  If the clients are on multiple datastores, I would recommend splitting up the clients into policies based on the datastore they reside.  Then in each policy, set it to only run 1 client at a time.  You can have all the policies run at the same time, however a snapshot per datastore will only be set to one host at time thus removing the VSS snap contention.

Mark_Solutions
Level 6
Partner Accredited Certified

It may be that these failing servers have apps on them (SQL / Exchange) and it is quiecing the application that causes the failure.

As the app should be backed up using an agent you dont actually need the app quiecing.

What you can do the the affected client is to edit its tools.conf file to remove app quiece.

See this tech note:

http://www.symantec.com/docs/TECH157826

Only applies if the failing ones have apps on them but may be worth a try as you still get everything else quieced this way

Hope this helps

Bruce_Burgdoff
Level 3

I already separate the clients out per LUN. Even If I back up just that 1 server with nothing else backing up it fails.  

Mark_Solutions
Level 6
Partner Accredited Certified

Bruce

do the ones that fail have apps running on them - as per my note above?

ITXRANGER
Level 2

I've been able to resolve this error code 99% of the time by doing the following:

Login to the server

START Volume Shadow Copy Services , which is set to Manual services type

Next, STOP Volume Shadow Copy Services

The job should succeed on the next run.