NetBackUp 7.0 on Server 2003 156 error

[JohnMW]

Hi guys,

Hoping someone can help me with this, I have hunted around forum/internet and tried various things ver the last couple of days but it seems the 156 error is a bit generic and could mean many things.  The error I am getting is:-

22/04/2013 22:10:55 - requesting resource itukbackup-hcart-robot-tld-0

22/04/2013 22:10:55 - requesting resource ITUKBACKUP.NBU_CLIENT.MAXJOBS.gemini

22/04/2013 22:10:55 - requesting resource ITUKBACKUP.NBU_POLICY.MAXJOBS.INT-WIN-GEMINI-FLASH

22/04/2013 22:10:55 - granted resource ITUKBACKUP.NBU_CLIENT.MAXJOBS.gemini

22/04/2013 22:10:55 - granted resource ITUKBACKUP.NBU_POLICY.MAXJOBS.INT-WIN-GEMINI-FLASH

22/04/2013 22:10:55 - granted resource KP0039

22/04/2013 22:10:55 - granted resource HP.ULTRIUM4-SCSI.001

22/04/2013 22:10:55 - granted resource itukbackup-hcart-robot-tld-0

22/04/2013 22:10:55 - estimated 33413155 Kbytes needed

22/04/2013 22:10:55 - begin Parent Job

22/04/2013 22:10:55 - begin Flash Backup Windows, Start Notify Script

22/04/2013 22:10:55 - started process RUNCMD (14760)

22/04/2013 22:10:55 - ended process 0 (14760)

Status 0

22/04/2013 22:10:55 - end Flash Backup Windows, Start Notify Script; elapsed time: 00:00:00

22/04/2013 22:10:55 - begin Flash Backup Windows, Step By Condition

Status 0

22/04/2013 22:10:55 - end Flash Backup Windows, Step By Condition; elapsed time: 00:00:00

22/04/2013 22:10:55 - begin Flash Backup Windows, Stream Discovery

Status 0

22/04/2013 22:10:55 - end Flash Backup Windows, Stream Discovery; elapsed time: 00:00:00

22/04/2013 22:10:55 - begin Flash Backup Windows, Read File List

Status 0

22/04/2013 22:10:55 - end Flash Backup Windows, Read File List; elapsed time: 00:00:00

22/04/2013 22:10:55 - begin Flash Backup Windows, Create Snapshot

22/04/2013 22:10:55 - started

22/04/2013 22:10:56 - started process bpbrm (7200)

22/04/2013 22:11:00 - begin Create Snapshot

22/04/2013 22:11:24 - Critical bpbrm(pid=7200) from client gemini: FTL - snapshot creation failed, status 156   

22/04/2013 22:11:24 - end Create Snapshot; elapsed time: 00:00:24

22/04/2013 22:11:26 - end writing

Status 156

22/04/2013 22:11:26 - end Flash Backup Windows, Create Snapshot; elapsed time: 00:00:31

22/04/2013 22:11:26 - begin Flash Backup Windows, Stop On Error

Status 0

22/04/2013 22:11:26 - end Flash Backup Windows, Stop On Error; elapsed time: 00:00:00

22/04/2013 22:11:26 - begin Flash Backup Windows, Delete Snapshot On Exit

22/04/2013 22:11:26 - begin Flash Backup Windows, Delete Snapshot On Exit

22/04/2013 22:11:27 - started process bpbrm (13396)

22/04/2013 22:11:34 - end writing

Status 0

22/04/2013 22:11:34 - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:08

22/04/2013 22:11:34 - begin Flash Backup Windows, End Notify Script

22/04/2013 22:11:35 - started process RUNCMD (6628)

22/04/2013 22:11:35 - ended process 0 (6628)

Status 0

22/04/2013 22:11:35 - end Flash Backup Windows, End Notify Script; elapsed time: 00:00:01

Status 156

22/04/2013 22:11:35 - end Flash Backup Windows, Delete Snapshot On Exit; elapsed time: 00:00:09

snapshot error encountered(156)

Could anyone shed any light on this please?  Am new to this system so not really sure where to start!

Many Thanks

John

[Dyneshia]

1) Confrim WOFB for the client is using VSS ( instead of VSP )

2) Look in the application event viewer for any ESE or VSS errors

3) Please upload bpfis from the client

4) From a command line on the client, please run the following and reply back with the output : 

vssadmin list writers

vssadmin list providers

[JohnMW]

Hi Dyneshia,

Thanks for the reply, here goes:-

1) Yep using VSS

2) There are no errors or warnings that mention ESE or VSS

3) Sorry not sure how to do this, did a Google search and plenty of instances of people doing it but no mention of how!  I have found the bpfis.exe but do not see a log export option in parameters etc. any chance of a quick "how to" please?

4) vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001 Microsoft Corp.

Writer name: 'SqlServerWriter'

   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

   Writer Instance Id: {2a06cc94-bd0f-4b1f-a2fe-dfc9757f2a83}

   State: [1] Stable

   Last error: No error

Writer name: 'FSRM Writer'

   Writer Id: {12ce4370-5bb7-4c58-a76a-e5d5097e3674}

   Writer Instance Id: {07e3ecb2-7580-460c-9222-2e0fce020e18}

   State: [1] Stable

   Last error: No error

Writer name: 'System Writer'

   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

   Writer Instance Id: {69f68092-cb1f-4ea2-b954-f483107228d8}

   State: [1] Stable

   Last error: No error

Writer name: 'DFS Replication service writer'

   Writer Id: {2707761b-2324-473d-88eb-eb007a359533}

   Writer Instance Id: {4dbc0605-f995-4389-b775-50493346fe95}

   State: [1] Stable

   Last error: No error

Writer name: 'IIS Metabase Writer'

   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}

   Writer Instance Id: {c5f7c0f3-79f2-4e3c-bf86-d9e8e8de8c54}

   State: [1] Stable

   Last error: No error

Writer name: 'COM+ REGDB Writer'

   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

   Writer Instance Id: {8852456d-f16f-40cd-b5ae-9cbca51ce040}

   State: [1] Stable

   Last error: No error

Writer name: 'MSDEWriter'

   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}

   Writer Instance Id: {e1f74006-8029-4027-8049-73306ecef839}

   State: [1] Stable

   Last error: No error

Writer name: 'Registry Writer'

   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

   Writer Instance Id: {9db0ce48-1797-4acb-817b-6c261a72129c}

   State: [1] Stable

   Last error: No error

Writer name: 'Event Log Writer'

   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}

   Writer Instance Id: {11d5e168-7d6b-4dbf-92c8-85bba9a3ba0f}

   State: [1] Stable

   Last error: No error

Writer name: 'WMI Writer'

   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

   Writer Instance Id: {47f1a7f0-ca24-4039-aa27-da7e2df5e18c}

   State: [1] Stable

   Last error: No error

Writer name: 'BITS Writer'

   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}

   Writer Instance Id: {641ed789-0b79-463b-bb35-0ae58e0ff744}

   State: [1] Stable

   Last error: No error

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'

   Provider type: System

   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}

   Version: 1.0.0.7

[Dyneshia]

On the client create :

 <install path> veritas\netbackup\logs\bpfis

Increase verobisty : on the Client-  Open up the Backup Archive Restore GUI -> file - > client properties -> troubleshooting ->set general to 2 and verbosity to 5.

 Run through the issue and set Verbosity back to 0.

upload the bpsfis

[Dyneshia]

On more thing ( sorry .... just reread and see this is a flash backup ) do the same on the backup host .  However you can increase the verbosity through the server host properties

[Dyneshia]

Also , as a test , create snapshot manually on the client using vcentre

[JohnMW]

Beautiful, thanks will set this up to run on tonights backup and report back tomorrow.

[JohnMW]

This is a physical machine I am backing up so no vcentre I'm afraid!

[Dyneshia]

Ok, np.  Just ignore the vcneter and storage server ;)

On the client create :

 <install path> veritas\netbackup\logs\bpfis

Increase verbosity : on the Client-  Open up the Backup Archive Restore GUI -> file - > client properties -> troubleshooting ->set general to 2 and verbosity to 5.

 Run through the issue and set Verbosity back to 0.

upload the bpfis

[JohnMW]

Beautiful, thanks, one bpfis log as requested:-

[Marianne · Accepted solution]

I have not had a look at the bpfis log yet... 

What is the size of the volume that you need to backup?

There may be a problem with space on other drives to create the snapshot.

See this TN for ways to test the actual snapshot with bpfis and vssadmin: http://www.symantec.com/docs/TECH47808

[JohnMW]

Hi Marianne,

Thanks for the thoughts, I have run through that test creating and deleting the snapshots and it all seems to work successfully so guess that's all ok?

[Dyneshia]

I think Marianne might be on the right track regarding the size.

From your bpfis log :

22:00:30.333 [7056.7504] <2> onlfi_vfms_logf: snapshot services: vss:Tue Apr 23 2013 22:00:30.333000 <Thread id - 7504> VSS API ERROR:- API [IsVolumeSupported] return val = 80042308 [VSS_E_OBJECT_NOT_FOUND]
22:00:30.333 [7056.460] <2> onlfi_vfms_logf: snapshot services: fivsp:Tue Apr 23 2013 22:00:30.333000 VspFiCoordinator::interpose cant interpose due to volume [\\.\PHYSICALDRIVE2] infn not found
22:00:30.333 [7056.7504] <2> onlfi_vfms_logf: snapshot services: vss:Tue Apr 23 2013 22:00:30.333000 <Thread id - 7504> VSS API ERROR:- API [IsVolumeSupported] return val = 80042308 [VSS_E_OBJECT_NOT_FOUND]
22:00:30.333 [7056.460] <2> onlfi_vfms_logf: snapshot services: fivsp:Tue Apr 23 2013 22:00:30.333000 VspFiCoordinator::interpose cant interpose due to open volume [UDID##HP##HSV##5000-1FE1-5023-DC00##6001-4380-05DE-7859-0000-8000-0164-0000] failed


22:00:41.817 [7056.460] <16> bpfis main: FTL - snapshot creation failed, status 156
22:00:41.817 [7056.460] <2> onlfi_thaw: fim=auto
22:00:41.817 [7056.460] <2> onlfi_thaw: fim=VSS
22:00:41.817 [7056.460] <4> onlfi_thaw: Thawing \\.\D:\ using snapshot method VSS.
22:00:41.817 [7056.460] <8> onlfi_thaw: \\.\D:\ is not frozen
22:00:41.817 [7056.460] <4> onlfi_thaw: Thawing \\.\E:\ using snapshot method VSS.
22:00:41.817 [7056.460] <8> onlfi_thaw: \\.\E:\ is not frozen
22:00:41.817 [7056.460] <2> onlfi_thaw: fim=VSP
22:00:41.817 [7056.460] <2> onlfi_thaw: fim=Hyper-V
22:00:41.817 [7056.460] <2> onlfi_thaw: fim=VMware
22:00:41.817 [7056.460] <2> onlfi_thaw: fim=NAS_Snapshot
22:00:41.817 [7056.460] <2> onlfi_thaw: fim=VSS_Writer
22:00:41.817 [7056.460] <8> bpfis Exit: WRN - snapshot delete returned status 20
22:00:41.833 [7056.460] <2> ol_cleanup: removing C:\Program Files\Veritas\NetBackup\temp\INT-WIN-GEMINI-FLASH+7056+1.std_filelist
22:00:41.833 [7056.460] <2> onlfi_vfms_logf: snapshot services: improper disposition parameter.

Questions :

1) What OS is this running on ?

2) Version of NBU ?

3) Do you see ANY errors in the event viewer during the time of the backup ?

4) upload the policy output :  netbackup\bin\admincmd\bppllist <policy name> -L

This may be some help : http://www.symantec.com/docs/TECH129884

[JohnMW]

The Shadow copies are set to create on a seperate drive that is only used for them and is currently has 232Gb free space out of 255Gb.  Disk drive we are backing up is 1.4Tb used on a 2Tb drive so is that too small?  Looking at that link it suggests 1:5 ratio if I am reading it right?

[JohnMW]

Just odd as Shadow copies are working fine it is just the back ups failing

[JohnMW]

1) The server is 2003 R2

2) Do you mean the NetBackup client version if so it is 6.5.4?

3) No there is only one error since 5 pm last night (backups run at 10pm) and it is just a "Windows saved user ************ registry while an application or service was still using the registry during log off. The memory used by the user's registry has not been freed. The registry will be unloaded when it is no longer in use." so unrelated

4) attached

I will double check the space situation too!  Thanks both!

[Marianne]

Maybe just do 1 drive letter at a time? 

[JohnMW]

I could change it for tonight and see what happens, strange though as policy hasn't changed for years!  Will try anything at this point though!  

[Dyneshia]

The policy may not have changed, but your data is consantly growing/changing.

Let us know how it goes.

[JohnMW]

Came in this morning to no error message!  

So it looks like either the data size is the issue (as the second drive is an extra 270Gb which may be tipping it over the edge) I will run it tonight just on the E drive and see if it errors (incase it is something on the drive itself) but if not I think you guys may have nailed it.  I'll report back tomorrow, thank you.