Backup is failing with status 156

[john10]

HI All,

we are using w2k8 with NBU 7.0 as Master/Media/Client as same server, and coming to incremental backup is running fine, but coming to full backup D:\ is failing with 156 every week, and if we restart the backup manually  mostly its completing, for the same i'm attaching some logs, could any one help me on this ?

Total disk space:- 1.6 TB

free space 163 GB

Failed job details:-

1/6/2015 3:07:42 PM - requesting resource
1/6/2015 3:07:42 PM - requesting resource
1/6/2015 3:07:42 PM - requesting resource
1/6/2015 3:07:42 PM - granted resource 
1/6/2015 3:07:42 PM - granted resource 
1/6/2015 3:07:42 PM - granted resource 
1/6/2015 3:07:42 PM - granted resource IBM.ULTRIUM-HH5.000
1/6/2015 3:07:42 PM - granted resource 
1/6/2015 3:07:45 PM - estimated 1157183374 Kbytes needed
1/6/2015 3:07:54 PM - mounting 09I161
1/6/2015 3:08:01 PM - mounted; mount time: 00:00:07
1/6/2015 3:08:01 PM - positioning to file 28
1/6/2015 3:08:24 PM - connecting
1/6/2015 3:08:24 PM - connected; connect time: 00:00:00
1/6/2015 3:08:24 PM - positioned ; position time: 00:00:23
1/6/2015 3:08:24 PM - begin writing
1/6/2015 3:14:38 PM - Warning bpbrm(pid=28808) from client .: WRN - can't open file: D:\BACKUP OLD SERVER\Drive C\GHS\ppc424\rtserv.exe (WIN32 5: Access is denied. )
1/7/2015 5:10:28 AM - Warning bpbrm(pid=28808) from client: WRN - can't open file: D:\DeptData\HSEF\From Sharedata\SECURITY\ACCESS CARD & FP\FINGER PRINT\att_setup(3.6.6 build 9)\ZK6000\setup.exe (WIN32 5: Access is denied. )
1/7/2015 5:34:39 AM - Warning bpbrm(pid=28808) from : WRN - can't open file: D:\DeptData\IS\Public\Temporary\Runner 10.6\Program Files\Winshuttle\Winshuttle Runner\TRANSACTIONRunner\TxRunnercom.exe (WIN32 5: Access is denied. )
1/7/2015 5:37:35 AM - Warning bpbrm(pid=28808) from client : WRN - can't open file: D:\DeptData\IS\Public\Temporary\WinShuttle Installs\Runner 10.6\Program Files\Winshuttle\Winshuttle Runner\TRANSACTIONRunner\TxRunnercom.exe (WIN32 5: Access is denied. )
1/7/2015 6:01:22 AM - Warning bpbrm(pid=28808) from client : WRN - can't open file: D:\DeptData\MQ\MQ OPERATIONS\Pcsx2_0.9.1_Setup\Pcsx2_0.9.1_Setup.exe (WIN32 5: Access is denied. )
1/7/2015 8:41:50 AM - Warning bpbrm(pid=28808) from client : WRN - can't open file: D:\Drivers\Fuji Xerox Printer\Fuji Xerox 340A\EzInst\VerUp.exe (WIN32 5: Access is denied. )
1/7/2015 9:37:47 AM - Error bpbrm(pid=28808) from client ERR - failure reading file: D:\Program Files\Veritas\NetBackup\logs\nbpem\51216-116-713909151-150106-0000000000.log (WIN32 2: The system cannot find the file specified. )
1/7/2015 9:37:48 AM - Error bpbrm(pid=28808) from client .com: ERR - Snapshot Error while reading file: GLOBALROOT\Device\HarddiskVolumeShadowCopy148\Program Files\Veritas\NetBackup\logs\nbpem\51216-116-713909151-150106-0000000000.log 
1/7/2015 9:37:48 AM - Critical bpbrm(pid=28808) from client FTL - Backup operation aborted!     
1/7/2015 9:37:55 AM - end writing; write time: 18:29:31
snapshot error encountered(156)

No error found in VSS O/P:-

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>d:

D:\>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {a231cb46-acb4-4b9b-8ca8-43388549f790}
   State: [1] Stable
   Last error: No error

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {beb81556-a11d-44b6-ab1c-89b6699995e3}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {7641ecd4-fa93-43c5-a970-0535c2f07f1c}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {0f726edc-b4ae-44c3-bfbf-1bfea82c4e15}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {3ff6406f-4c87-474e-9628-040c33a0c9e4}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {c50b3e2c-9709-428c-bf1b-2b0ff5cf68ac}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {90547797-533f-4ef2-a637-1db89b16b73e}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {db8fc78b-317a-43a2-b985-0dfb1a0fac5f}
   State: [1] Stable
   Last error: No error

D:\>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

[inn_kam]

well credentials problem looking

check Netbackup Client Service as Admin priviliges 

Which Databade you are taking backup?

[inn_kam]

NetBackup VSS snapshot error encountered (156)

Article:TECH75214

|

Created: 2009-01-02

|

Updated: 2013-05-03

|

Article URL http://www.symantec.com/docs/TECH75214

Resolution: 
 

For Volume Shadow Copy (VSS) (Windows 2003 and later only) use the following steps to increase the cache size when VSS is used on Windows clients:
 

1. In Windows, double-click My Computer and select The drive being backed up (i.e. D:/)

2. Right mouse click on the drive and click Properties, select Shadow Copies.

3. Select the volume on which to make changes, and then click the Settings button

4. In the Settings dialog box, change the Maximum Size setting to either No Limit or a size large enough to suit the requirements of the installation and usage of VSS

Refer to VSS configuration procedure described in TECH150413

[john10]

Hi , 

yes i use to follow same steps for setting the vss to "no limit" while manul rerun the backup , but again after a week same problem is occuring, 

and also in "client attributes" i selected "Delete snapshot" and continue for "openfile backup's " still its failing.

please let us know further troubleshooting steps.

[john10]

Hi ,

we are not using any application backup, its just flatfiles. and client services is running as default i.e "local system" account.

D:\Program Files\Veritas\NetBackup\bin\admincmd>bppllist IB09W0001 -U
------------------------------------------------------------

Policy Name:       IB09W0001

  Policy Type:         MS-Windows
  Active:              yes
  Effective date:      02/03/2014 11:27:21
  Backup network drvs: no
  Collect TIR info:    no
  Mult. Data Streams:  yes
  Client Encrypt:      yes
  Checkpoint:          yes
       Interval:       15
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           ib09w0001-hcart2
  Volume Pool:         Ib09-Pool
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no

Granular Restore Info:  no
Ignore Client Direct:  no
  HW/OS/Client:  Windows-x64   Windows2008   ib09w0001.global.ds.honeywell.com

  Include:  ALL_LOCAL_DRIVES

  Schedule:          Full
    Type:            Full Backup
    Frequency:       every 7 days
    Maximum MPX:     7
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 5 (3 months)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Friday     21:00:00  -->  Saturday   17:10:00

  Schedule:          Incremental
    Type:            Differential Incremental Backup
    Frequency:       every 8 hours
    Maximum MPX:     9
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 5 (3 months)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Monday     21:00:00  -->  Tuesday    02:50:00
          Tuesday    21:00:00  -->  Wednesday  02:50:00
          Wednesday  21:00:00  -->  Thursday   02:50:00
          Thursday   21:00:00  -->  Friday     02:50:00

D:\Program Files\Veritas\NetBackup\bin\admincmd>

[inn_kam]

Resolution:
- Make sure the Local System Account on the client server has Full Control of the directories selected.
- Change the Logon Account for the NetBackup Client Service to a user that has Full Control of the directories selected.

STATUS CODE 0: File attributes are not backed up even thought the Policy completes with a Status 0. The All Log Entries Report shows "(WIN32 5: Access is denied. )"

Article:TECH37158

|

Created: 2005-01-01

|

Updated: 2006-01-01

|

Article URL http://www.symantec.com/docs/TECH37158

[inn_kam]

Seems there is a permission problem. 
Check logon account for NBU client service. If LocalSystem, change to a domain account with permissions to all files and folders. Select a user account with password that does not expire.

Use the bpbkar test on the C-drive as suggested above.

TAKEN From https://www-secure.symantec.com/connect/forums/backups-are-running-dead-slow#comment-9070431

[john10]

HI inn_kam,

thanks for your time and , sure i will change the client services to run with domain account,

and just want to know in which file or directory does its showing the problem in logs for credentials issue can you post the specific log ?

[Marianne]

It seems as if the backup is actually running but fails right at the end where snapshot info needs to be cleaned up. Snapshot info is written on the master server as well as the client.

You need to ensure that comms between client and master server is fine at all times - port connectivity as well as forward and reverse name lookup.
Check in client's bpfis log for comms issues with master.

Workaround:
Host Properties -> Master -> Client Attributes:
Add/select this client name
In Windows Open File Backup tab, under Snapshot error control, select 'Disable Snapshot and continue'.

[john10]

HI ,

i changed the "client services" to run on domain admin credentials, after that im not able to load the NBU master console. im getting below pop-up.

if "client services" are set to "Localsystem account" console is Launching with out any issue.

Hi Marianne, in my case Master/client is same, and i did this setting also by  "Disable Snapshot and continue'."still it failed last time.

Host Properties -> Master -> Client Attributes:
Add/select this client name
In Windows Open File Backup tab, under Snapshot error control, select 'Disable Snapshot and continue'.

FYI

1. Make sure the user "username" has priviliges on the host

2. Make sure the local host is listed in the server list of the destination host "hostname"

3. Make sure there is a valid network connection

4. Check authentication

5. Make sure all the services are running on the host.

[inn_kam]

WELL

pLEASE Note Change the Priviliges on Client Server not on Master server 

- Warning bpbrm(pid=28808) from client .: WRN - can't open file: D:\BACKUP OLD SERVER\Drive C\GHS\ppc424\rtserv.exe (WIN32 5: Access is denied. )

Resolution:
- Make sure the Local System Account on the client server has Full Control of the directories selected.
- Change the Logon Account for the NetBackup Client Service to a user that has Full Control of the directories selected.

[Marianne · Accepted solution]

Backup should NOT fail with 156 if Snapshot error control is set to 'Disable Snapshot and continue'.
Ensure the Client name in Client Attributes is exactly the same as in the policy.

Permission issues on the client should cause status 1, not 156.

Please show us bpfis and bpbkar logs on the client. Upload logs as .txt File attachments.
Also show us output of this command on the master:
bpclient  -client <client_name> -L

[john10]

Hi Inn_kem,

I'n my case Master/media/Client servers are same, so that is the reason i'm trying to change on Master itself.

Marianne,

Again Fullbackup failed with status 156, and bpbkar log is created just now, for time being im attaching bpfis log, and failed job details.

FYI

1/9/2015 9:00:36 PM - estimated 1157183374 Kbytes needed
1/9/2015 9:00:37 PM - started process bpbrm (7600)
1/9/2015 9:00:48 PM - connecting
1/9/2015 9:00:48 PM - connected; connect time: 00:00:00
1/9/2015 9:00:49 PM - mounting 09I161
1/9/2015 9:00:51 PM - mounted; mount time: 00:00:02
1/9/2015 9:00:51 PM - positioning 09I161 to file 195
1/9/2015 9:02:40 PM - positioned 09I161; position time: 00:01:49
1/9/2015 9:02:40 PM - begin writing
1/9/2015 9:09:05 PM - Warning bpbrm(pid=6988) from client .com: WRN - can't open file: D:\BACKUP OLD SERVER\Drive C\GHS\ppc424\rtserv.exe (WIN32 5: Access is denied. )
1/10/2015 12:37:16 PM - current media 09I161 complete, requesting next resource Any
1/10/2015 12:37:17 PM - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: 
     Robot Number: NONE, Robot Type: NONE, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Ib09-Pool, Storage Unit: hcart2, Drive Scan Host: N/A
    
1/10/2015 12:39:16 PM - current media -- complete, awaiting next media Any A pending request has been generated for this resource request.
     Operator action may be required. Pending Action: No action.,
     Media ID: 09I171, Barcode: --------, Density: hcart2, Access Mode: Write,
     Action Drive Name: N/A, Action Media Server: N/A, Robot Number: N/A, Robot Type: NONE,
     Volume Group: ---, Action Acs: N/A, Action Lsm: N/A
    
1/12/2015 11:13:37 AM - granted resource 09I131
1/12/2015 11:13:37 AM - granted resource IBM.ULTRIUM-HH5.000
1/12/2015 11:13:37 AM - granted resource -hcart2
1/12/2015 11:13:38 AM - mounting 09I131
1/12/2015 11:13:39 AM - Error bptm(pid=25276) FREEZING media id 09I131, Medium identifiers do not match    
1/12/2015 11:13:39 AM - Warning bptm(pid=25276) media id 09I131 load operation reported an error     
1/12/2015 11:13:39 AM - current media 09I131 complete, requesting next resource Any
1/12/2015 11:13:45 AM - current media -- complete, awaiting next media Any A pending request has been generated for this resource request.
     Operator action may be required. Pending Action: No action.,
     Media ID: 09I171, Barcode: --------, Density: hcart2, Access Mode: Write,
     Action Drive Name: N/A, Action Media Server: N/A, Robot Number: N/A, Robot Type: NONE,
     Volume Group: ---, Action Acs: N/A, Action Lsm: N/A
    
1/12/2015 1:01:52 PM - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: ib09w0001, 
     Robot Number: NONE, Robot Type: NONE, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: Ib09-Pool, Storage Unit: ib09w0001-hcart2, Drive Scan Host: N/A
    
1/12/2015 1:02:14 PM - current media -- complete, awaiting next media Any A pending request has been generated for this resource request.
     Operator action may be required. Pending Action: No action.,
     Media ID: 09I171, Barcode: --------, Density: hcart2, Access Mode: Write,
     Action Drive Name: N/A, Action Media Server: N/A, Robot Number: N/A, Robot Type: NONE,
     Volume Group: ---, Action Acs: N/A, Action Lsm: N/A
    
1/12/2015 1:03:01 PM - granted resource 09I162
1/12/2015 1:03:01 PM - granted resource IBM.ULTRIUM-HH5.000
1/12/2015 1:03:01 PM - granted resource ib09w0001-hcart2
1/12/2015 1:03:02 PM - mounting 09I162
1/12/2015 1:03:04 PM - mounted; mount time: 00:00:02
1/12/2015 1:03:04 PM - positioning 09I162 to file 1
1/12/2015 1:03:06 PM - positioned 09I162; position time: 00:00:02
1/12/2015 1:03:06 PM - begin writing
1/12/2015 1:03:11 PM - Warning bpbrm(pid=6988) from client .com: WRN - can't open directory: D:\DeptData\HSEF\Facility\Photo 2012 (WIN32 2: The system cannot find the file specified. )
1/12/2015 1:03:11 PM - Error bpbrm(pid=6988) from client .com: ERR - Snapshot Error while opening directory: GLOBALROOT\Device\HarddiskVolumeShadowCopy178\DeptData\HSEF\Facility\Photo 2012 
1/12/2015 1:03:11 PM - Critical bpbrm(pid=6988) from client .com: FTL - Backup operation aborted!     
1/12/2015 1:03:12 PM - Error bpbrm(pid=6988) could not send server status message       
1/12/2015 1:03:16 PM - end writing; write time: 00:00:10
snapshot error encountered(156)

D:\Program Files\Veritas\NetBackup\bin\admincmd>bpclient  -client ib09w0001.global.ds.x.com -L
Client Name: ib09w0001.global.ds.x.com
 Current Host:
        Hostname: ib09w0001.global.ds.x.com
        IP Address: 0.0.0.0
 Dynamic Address:       no
 Free Browse:   Allow
 List Restore:  Not Specified
 Max Jobs This Client:  Not Specified
 WOFB Enabled:  yes
 WOFB FIM:      VSS
 WOFB Usage:    Individual Drive Snapshot
 WOFB Error Control:    Disable and Continue
 Client Direct: Deduplication on the media server or
                Move data via media server
 Client Direct Restore: Move data via media server
 OST Proxy:     Off
 OST Proxy Server:      Unspecified
 Connect options:       2 2 3

[inn_kam]

STATUS CODE: 156 "snapshot error encountered" occurs when utilizing VSP or VSS for open file backups.

Article:TECH38709

|

Created: 2005-01-03

|

Updated: 2013-03-28

|

Article URL http://www.symantec.com/docs/TECH38709

[Marianne]

All I can see in bpfis log is a Windows VSS error:

13:03:35.059 [3916.27112] <2> onlfi_vfms_logf: INF - snapshot services: vss:Mon Jan 12 2015 13:03:35.059000 <Thread id - 27112> VSS API ERROR:- API [DeleteSnapshots] return val = 80042308 [VSS_E_OBJECT_NOT_FOUND]
13:03:35.059 [3916.1116] <2> onlfi_vfms_logf: INF - snapshot services: vss: "IVssBackupComponents::DeleteSnapshots" failed with error "VSS_E_OBJECT_NOT_FOUND:(error value=0x80042308)" while trying to delete snapshotID {3F57A70E-58E8-41C9-95D3-837ED34235EF}
13:03:35.059 [3916.27112] <2> onlfi_vfms_logf: INF - snapshot services: vss:Mon Jan 12 2015 13:03:35.059000 <Thread id - 27112> VSS API ERROR:- API [DeleteSnapshots] return val = 80042308 [VSS_E_OBJECT_NOT_FOUND]
13:03:35.059 [3916.1116] <2> onlfi_vfms_logf: INF - snapshot services: vss: "IVssBackupComponents::DeleteSnapshots" failed with error "VSS_E_OBJECT_NOT_FOUND:(error value=0x80042308)" while trying to delete snapshotID {1FA080A6-CBDB-447B-9E8F-EBCCC287B09C}

Have you checked Windows Event Viewer for errors?
Have you verified sufficient space for snapshots?

Please go through steps listed in this TN to test VSS snapshot functionality:

How to verify Windows Open File Backup operation using VSSADMIN and BPFIS before contacting Symantec Technical Support
http://www.symantec.com/docs/TECH47808 

H

[john10]

HI Marianne, 

i found below errors in eventmanager in Application Logs on server,

FYI

The file D:\BACKUP OLD SERVER\Drive C\GHS\ppc424\rtserv.exe contains Artemis!9FD0961253C6 Trojan.  The file was successfully deleted.

[Marianne]

This is AV error - not VSS. 

We are looking for VSS messages/errors and need to know if you have verified sufficient space for VSS snapshots.

Have you gone through TECH47808 yet?

[john10]

HI Marianne,

After changing the setting to Disable Snapshot and continue'."now backup is completing with status "0"

thanks for your time and support.

[Marianne]

Great stuff! 

My own experience with status 156 recently: 
https://vox.veritas.com/t5/NetBackup/Failing-backups-status-156/m-p/710476#M190301