Error 71 and DFSR writer disappearing

[quebecois]

Hi everyone,

I have a very strange issue on a Windows 2008 R2 Enterprise client with DFSR. For the backup configuration I followed the instructions from this link (medium to large environment): https://support.symantec.com/en_US/article.HOWTO65638.html

The backup works fine for a couple of days (sometimes weeks), but then suddenly I got error 71 on the DFSR job and 69/50 on the Shadow Copy Components (from All Local policy) job as below.

When I check the writers on the client (vssadmin list writers), DFS Replication service writer simply disappeared! Then I have to restart DFS services and after 10 minutes the writer is back and the backup works until the next occurence in a couple of days/weeks.

My environment: Netbackup standard 7.7 (RHEL), Netbackup client 7.7 running on Windows 2008 R2 (same problem occurred before upgrading, on version 7.6.0.3)

Thanks for any support!!

Error on DFSR job:

09/01/2015 20:47:16 - Info nbjm (pid=4551) starting backup job (jobid=750942) for client CLIENT, policy CLIENT01-DFSR, schedule Diff-Daily-2months
09/01/2015 20:47:16 - Info nbjm (pid=4551) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=750942, request id:{282EA100-510C-11E5-9FEE-0F666F1B2488})
09/01/2015 20:47:16 - requesting resource Mixed_mode
09/01/2015 20:47:16 - requesting resource SERVER.NBU_CLIENT.MAXJOBS.CLIENT
09/01/2015 20:47:16 - requesting resource SERVER.NBU_POLICY.MAXJOBS.CLIENT-DFSR
09/01/2015 20:47:16 - granted resource  SERVER.NBU_CLIENT.MAXJOBS.CLIENT
09/01/2015 20:47:16 - granted resource  SERVER.NBU_POLICY.MAXJOBS.CLIENT-DFSR
09/01/2015 20:47:16 - granted resource  GA3089
09/01/2015 20:47:16 - granted resource  HP.ULTRIUM6-SCSI.004
09/01/2015 20:47:16 - granted resource  SERVER-hcart3-robot-tld-1
09/01/2015 20:47:16 - estimated 49077973 kbytes needed
09/01/2015 20:47:16 - Info nbjm (pid=4551) resumed backup (backupid=CLIENT_1441152017) job for client CLIENT, policy CLIENT-DFSR, schedule Diff-Daily-2months on storage unit SERVER-hcart3-robot-tld-1
09/01/2015 20:47:17 - Info bpbrm (pid=6105) CLIENT is the host to backup data from
09/01/2015 20:47:17 - Info bpbrm (pid=6105) telling media manager to start backup on client
09/01/2015 20:47:28 - Info bptm (pid=6123) using 262144 data buffer size
09/01/2015 20:47:28 - Info bptm (pid=6123) using 32 data buffers
09/01/2015 20:47:29 - Info bpbrm (pid=6105) spawning a brm child process
09/01/2015 20:47:29 - Info bpbrm (pid=6105) child pid: 9290
09/01/2015 20:47:29 - Info bpbrm (pid=6105) sending bpsched msg: CONNECTING TO CLIENT FOR CLIENT_1441152017
09/01/2015 20:47:29 - Info bpbrm (pid=6105) start bpbkar on client
09/01/2015 20:47:29 - Info bptm (pid=9286) setting receive network buffer to 262144 bytes
09/01/2015 20:47:29 - connecting
09/01/2015 20:47:29 - connected; connect time: 0:00:00
09/01/2015 20:47:29 - begin writing
09/01/2015 20:47:32 - Info bpbkar (pid=43868) Backup started
09/01/2015 20:47:32 - Info bpbrm (pid=6105) Sending the file list to the client
09/01/2015 20:47:32 - Info bpbkar (pid=43868) change time comparison:<disabled>
09/01/2015 20:47:32 - Info bpbkar (pid=43868) archive bit processing:<disabled>
09/01/2015 20:47:53 - Error bpbrm (pid=9290) from client CLIENT: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - Estimate:-1 -1
09/01/2015 20:47:53 - Warning bpbrm (pid=9290) from client CLIENT: WRN - can't open object: Shadow Copy Components: (BEDS 0xE000FECB: A failure occurred accessing the backup component document.)
09/01/2015 20:47:53 - Error bpbrm (pid=9290) from client CLIENT: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.Edi - 0 0 0 -1 1 16676 user other 0 1441154873 1441154873 1441154873 /
09/01/2015 20:47:53 - Info bpbrm (pid=6105) sending message to media manager: STOP BACKUP CLIENT_1441152017
09/01/2015 20:47:54 - Info bpbrm (pid=6105) media manager for backup id CLIENT_1441152017 exited with status 150: termination requested by administrator
09/01/2015 20:47:54 - end writing; write time: 0:00:25
none of the files in the file list exist  (71)

Error on Shadow Copy Components backup:

09/02/2015 07:40:11 - Info nbjm (pid=4551) starting backup job (jobid=751266) for client CLIENT, policy CLIENT, schedule Diff-Daily-2months
09/02/2015 07:40:11 - Info nbjm (pid=4551) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=751266, request id:{5E4D5708-5167-11E5-9EBC-654E9B38E6C8})
09/02/2015 07:40:11 - requesting resource Mixed_mode
09/02/2015 07:40:11 - requesting resource SERVER.NBU_CLIENT.MAXJOBS.CLIENT
09/02/2015 07:40:11 - requesting resource SERVER.NBU_POLICY.MAXJOBS.CLIENT
09/02/2015 07:40:11 - granted resource  SERVER.NBU_CLIENT.MAXJOBS.CLIENT
09/02/2015 07:40:11 - granted resource  SERVER.NBU_POLICY.MAXJOBS.CLIENT
09/02/2015 07:40:11 - granted resource  GL3456
09/02/2015 07:40:11 - granted resource  HP.ULTRIUM4-SCSI.000
09/02/2015 07:40:11 - granted resource  backup01-hcart-robot-tld-0
09/02/2015 07:40:11 - estimated 16847836 kbytes needed
09/02/2015 07:40:11 - Info nbjm (pid=4551) resumed backup (backupid=CLIENT_1441185150) job for client CLIENT, policy CLIENT, schedule Diff-Daily-2months on storage unit backup01-hcart-robot-tld-0
09/02/2015 07:40:13 - Info bpbrm (pid=5577) CLIENT is the host to backup data from
09/02/2015 07:40:13 - Info bpbrm (pid=5577) telling media manager to start backup on client
09/02/2015 07:40:13 - Info bptm (pid=5610) using 262144 data buffer size
09/02/2015 07:40:13 - Info bptm (pid=5610) using 32 data buffers
09/02/2015 07:40:13 - Info bpbrm (pid=5577) spawning a brm child process
09/02/2015 07:40:13 - Info bpbrm (pid=5577) child pid: 6263
09/02/2015 07:40:14 - Info bpbrm (pid=5577) sending bpsched msg: CONNECTING TO CLIENT FOR CLIENT_1441185150
09/02/2015 07:40:14 - Info bpbrm (pid=5577) start bpbkar on client
09/02/2015 07:40:14 - Info bptm (pid=6262) setting receive network buffer to 262144 bytes
09/02/2015 07:40:14 - connecting
09/02/2015 07:40:14 - connected; connect time: 0:00:00
09/02/2015 07:40:14 - begin writing
09/02/2015 07:40:21 - Info bpbkar (pid=6408) Backup started
09/02/2015 07:40:21 - Info bpbrm (pid=5577) Sending the file list to the client
09/02/2015 07:40:21 - Info bpbkar (pid=6408) change time comparison:<disabled>
09/02/2015 07:40:21 - Info bpbkar (pid=6408) archive bit processing:<disabled>
09/02/2015 08:14:17 - Error bpbrm (pid=6263) from client CLIENT: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - Estimate:-1 -1
09/02/2015 08:14:18 - Warning bpbrm (pid=6263) from client CLIENT: WRN - can't open object: Shadow Copy Components: (BEDS 0xE000FECB: A failure occurred accessing the backup component document.)
09/02/2015 08:14:18 - Error bpbrm (pid=6263) from client CLIENT: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - EXIT STATUS 69: invalid filelist specification
09/02/2015 08:14:19 - Critical bpbrm (pid=6263) unexpected termination of client CLIENT
09/02/2015 08:14:19 - Error bpbrm (pid=6263) could not send server status message
09/02/2015 08:14:19 - Info bpbrm (pid=5577) sending message to media manager: STOP BACKUP CLIENT_1441185150
09/02/2015 08:14:25 - Info bpbrm (pid=5577) media manager for backup id CLIENT_1441185150 exited with status 90: media manager received no data for backup image
09/02/2015 08:14:25 - end writing; write time: 0:34:11
client process aborted  (50)

Vssadmin list providers on client (before restarting DFS service):

C:\Windows\system32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {858e80be-6d26-495f-96ac-b512aa85d0ed}
   State: [1] Stable
   Last error: No error

Writer name: 'FSRM Writer'
   Writer Id: {12ce4370-5bb7-4c58-a76a-e5d5097e3674}
   Writer Instance Id: {3e7735f3-b95f-4db5-bc13-e49f09056bf4}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {d43fc85e-1a5d-4b9b-ac0a-acb4e3d2d300}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {1a77fc8b-e135-422e-ba31-ae7540845d61}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {35093532-f6a4-4fee-bfef-96acde15a922}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {0b3307de-b911-431b-b631-857503339de6}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {08822103-9d76-4ee7-a2c6-ed515862fd4a}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {345052d1-b94e-4e00-80f6-33c4d464e9dd}
   State: [1] Stable
   Last error: No error

Other useful information:

- bpbkar log attached (level 2);

- DFSR backup policy setup with "Limit jobs per policy" = 1

- There are no additional providers under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\Providers

- HKLM\SOFTWARE\Veritas\NetBackup\BEDS\Engine\Shadow Copy Components\Additional Not Authorized Writers path doesn't exist

- "bpstart_notify.bat' and 'bpstop_notify.bat' within ...\NetBackup\bin\ don't exist (from TN https://support.symantec.com/en_US/article.TECH211130.html)

[GulzarShaikhAUS · Accepted solution]

Can you check the event viewer on client machine if it mentions anything about DFSR service/writer getting crashed? What is the status of DFSR service? Does it go down as well?

If the DFSR writer is crashed/hung then you will have to contact Microsoft to understand the behaviour.

This might be useful

DFS Replication service crashes when you perform a backup operation on a computer that is running Windows Server 2008 R2

https://support.microsoft.com/en-us/kb/2678241

[quebecois]

Hi SymGuy-IT, thanks for the quick response. Indeed we are getting the error message below on eventviewer whenever this issue occurs:

An unhandled exception was encountered while processing a VSS writer event callback method. The VSS writer infrastructure is in an unstable state. Restart the service or application that hosts the writer.

Writer name:            
Writer id:              {2707761b-2324-473d-88eb-eb007a359533}
Writer instance:        {96236e9a-890b-49aa-aca5-708903825bdc}
Process command line:   C:\Windows\system32\DFSRs.exe
Process ID:             45560
Writer operation:       1000
Writer state:           0
Exception code:         0xc0000005
Exception location:     02

We have just applied the hotfix from the Microsoft link you sent and will monitor in the next couple of weeks if the problem occurs again.