Forum Discussion

haboulainain's avatar
haboulainain
Level 3
8 years ago

Probleme backup Media Server Windows 2008 R2 status 69

I have a strange problem and I hope that anyone can help me.
Recently I do a migration from netbackup 7.5.0.1 to 7.6.0.4
The problem regards the bck of a Media server with nbu policy type "MS-windos" with backup of "all_local_drives" and so also shadow copy components.
The client is a win srv 2008 R2 entreprise SP1 
After the migration backup fail wih retur code 69. From job details I can find only general info about the error:
===============
11/3/2016 11:55:49 PM - Error bpbrm(pid=9776) from client xxx-nbu02: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.JBD - bpbkar waited 2334 times for empty buffer, delayed 2500 times.
11/3/2016 11:55:51 PM - end writing; write time: 1:53:51
11/3/2016 11:55:52 PM - Info sonep-nbu02(pid=2720) StorageServer=PureDisk:xxx-nbu02; Report=PDDO Stats for (xxx-nbu02): scanned: 33463880 KB, CR sent: 3121825 KB, CR sent over FC: 0 KB, dedup: 90.7%, cache disabled
11/3/2016 11:55:52 PM - Error bpbrm(pid=9776) could not send server status message       
11/3/2016 11:55:52 PM - Info bpbkar32(pid=10388) done. status: 69: invalid filelist specification       
invalid filelist specification(69)
===================
vssadmin list writers shows a bunch of writers all stable and No error on last errorvssadmin list providers shows Microsoft software shadow copy provider 1.0 version 1.0.0.7
69
      • haboulainain's avatar
        haboulainain
        Level 3
        8 years ago
         I tried many things:


        1) I checked the VSS Writers: (all good):vssadmin list writers

        vssadmin 1.1 - Volume Shadow Copy Service administrative command-

        (C) Copyright 2001-2005 Microsoft Corp.

         

        Writer name: 'Task Scheduler Writer'

           Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

           Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'VSS Metadata Store Writer'

           Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

           Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'Performance Counters Writer'

           Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}

           Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'System Writer'

           Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

           Writer Instance Id: {6f2cc347-8109-45b5-a9b5-bcb3a45a1816}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'ASR Writer'

           Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

           Writer Instance Id: {b6e524b8-04a5-406c-8153-0a0187317e5b}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'Registry Writer'

           Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

           Writer Instance Id: {8e3b348b-086f-40e7-a963-44a5fed92bf1}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'Shadow Copy Optimization Writer'

           Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

           Writer Instance Id: {7a0504c6-6770-473d-a3d7-b6f87b764db7}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'IIS Metabase Writer'

           Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}

           Writer Instance Id: {5312c4be-1c60-4162-a3d8-aaac86c70890}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'IIS Config Writer'

           Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}

           Writer Instance Id: {9ef4cbe3-3a82-4645-9e20-425cc0997a51}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'WMI Writer'

           Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

           Writer Instance Id: {f0da1a64-6376-40d5-a725-60671f9dc9ac}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'Dhcp Jet Writer'

           Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}

           Writer Instance Id: {86e1afd4-3f48-448a-981d-b239fbd8879a}

           State: [1] Stable

           Last error: No error

         

        Writer name: 'COM+ REGDB Writer'

           Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

           Writer Instance Id: {d97a0e4e-8db6-4d05-970b-b2a3e6358911}

           State: [1] Stable

           Last error: No error

        2) I checked the VSS provider (all good)

        vssadmin list providers

        vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

        (C) Copyright 2001-2005 Microsoft Corp.

         Provider name: 'Microsoft Software Shadow Copy provider 1.0'

           Provider type: System

           Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}

        3) There are volumes that exceed 62 TB, I extended the registry key value HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ BEDS \ Engine \ Misc \ BescLargeDiscBlock to 68 and I started the job but still the same thing

        https://www.veritas.com/support/en_US/article.TECH206217

        4) To registre the VSS, the OS is already Windows 2008 R2 SP1 (so Nothing to do)

        https://www.veritas.com/support/en_US/article.000015843

        5) I rebooted the server several times, but still the same problem

         

         

         

         

           Version: 1.0.0.7

         

         

    • haboulainain's avatar
      haboulainain
      Level 3
      8 years ago

      I tested this solution and I runed the job, but still the same results.
      Please find attached the log files of bpbkar: 

       01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new directory:<G:\NBU_Catalog>, forcing full backup
      01:18:12.987 [8052.10876] <2> tar_base::V_vTarMsgW: INF - Excluded: G:\NBU_Catalog
      01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new directory:<G:\System Volume Information>, forcing full backup
      01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new file:<G:\System Volume Information\tracking.log>, forcing backup
      01:18:13.003 [8052.10876] <4> dos_backup::tfs_scannext: INF - no more path list entries
      01:18:13.003 [8052.10876] <4> tar_backup::backup_endarg_state: INF - finish error count: 0
      01:18:13.003 [8052.10876] <4> tar_backup::backup_endarg_state: INF - finish fatal count: 0
      01:18:13.003 [8052.10876] <4> tar_backup_tfi::backup_arg_state: INF - remember start error count: 0
      01:18:13.003 [8052.10876] <4> tar_backup_tfi::backup_arg_state: INF - remember start fatal count: 0
      01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - starting scan for file directive:<System State:\>
      01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - starting scan using name:<System State:\>
      01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - in 'backup' mode
      01:18:13.018 [8052.10876] <2> ov_log::V_GlobalLog: ERR - BEDS_AttachToDLE():FS_AttachToDLE() DeviceName:'System?State' BackupReason:0x2 Failed! (0xE000FEC3:A failure occurred accessing the SnapShot coordinator.
      )
      01:18:13.018 [8052.10876] <2> tar_base::V_vTarMsgW: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.
      01:18:13.018 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - number of file directives not found: 3
      01:18:13.018 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - number of file directives found: 5
      01:18:13.018 [8052.7888] <4> tar_base::keepaliveThread: INF - keepalive thread terminating (reason: WAIT_OBJECT_0)
      01:18:13.018 [8052.10876] <4> tar_base::stopKeepaliveThread: INF - keepalive thread has exited. (reason: WAIT_OBJECT_0)
      01:18:13.018 [8052.10876] <4> send_msg_to_monitor: INF - in send_msg_to_monitor()...
      01:18:13.034 [8052.10876] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 69: invalid filelist specification
      01:18:13.034 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - Not waiting for server status
      01:18:14.048 [8052.10876] <4> dos_backup::ClearDFSRArchiveBits: INF - no DFSR file list was created, No bits cleared.
      01:18:14.048 [8052.10876] <4> ov_log::OVLoop: Timestamp
      01:18:14.048 [8052.10876] <4> OVStopCmd: INF - EXIT - status = 0
      01:18:14.048 [8052.10876] <2> tar_base::V_Close: closing...
      01:18:14.048 [8052.10876] <4> dos_backup::tfs_reset: INF - Snapshot deletion start
      01:18:14.064 [8052.10876] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): enter - InitFlags:0x00000101
      01:18:14.064 [8052.10876] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): ubs specifics: 0x001d0000
      01:18:14.204 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
      01:18:15.218 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
      01:18:16.232 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
      01:18:17.246 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
      01:18:17.246 [8052.10876] <4> OVShutdown: INF - Finished process
      01:18:17.246 [8052.10876] <4> WinMain: INF - Exiting D:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe

       

  • Arkya's avatar
    Arkya
    Level 4
    8 years ago

    Hello,

    As suggested earlier by Marrianee please try to backup Shadow Copy components with Windows server backup utility to make sure there is no problem with VSS.

     

    • haboulainain's avatar
      haboulainain
      Level 3
      8 years ago

      there is no problem with VSS, I perform this test successfully

      vssadmin Add ShadowStorage /For=C: /On=D: /MaxSize=900MB

       (C) Copyright 2001-2005 Microsoft Corp.

      Successfully added the shadow copy storage association

      =============================================================

      C:\Users\Admin>vssadmin list shadowstorage

      vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

      (C) Copyright 2001-2005 Microsoft Corp.

      Shadow Copy Storage association

         For volume: (\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}\)\\?\Volume{724

      0e843-914a-11e2-bb8c-806e6f6e6963}\

         Shadow Copy Storage volume: (\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}

      \)\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}\

         Used Shadow Copy Storage space: 0 B (0%)

         Allocated Shadow Copy Storage space: 0 B (0%)

         Maximum Shadow Copy Storage space: UNBOUNDED (100%)

      Shadow Copy Storage association

         For volume: (C:)\\?\Volume{7240e844-914a-11e2-bb8c-806e6f6e6963}\

         Shadow Copy Storage volume: (D:)\\?\Volume{3df823db-9166-11e2-b277-6c3be5a607

      c7}\

         Used Shadow Copy Storage space: 0 B (0%)

         Allocated Shadow Copy Storage space: 0 B (0%)

         Maximum Shadow Copy Storage space: 900 MB (0%)

      ==================================================================================

      C:\Users\Admin>vssadmin create shadow /for=C:

      vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

      (C) Copyright 2001-2005 Microsoft Corp.

      Successfully created shadow copy for 'C:\'

          Shadow Copy ID: {6e074368-ded5-4aa8-9a21-bdb48de7f497}

          Shadow Copy Volume Name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy477

      C:\Users\Admin>

      C:\Users\Admin>vssadmin list shadows

      vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

      (C) Copyright 2001-2005 Microsoft Corp.

      Contents of shadow copy set ID: {5f60055e-0248-49f5-811c-4478ed193641}

         Contained 1 shadow copies at creation time: 11/4/2016 5:06:13 AM

            Shadow Copy ID: {6e074368-ded5-4aa8-9a21-bdb48de7f497}

               Original Volume: (C:)\\?\Volume{7240e844-914a-11e2-bb8c-806e6f6e6963}\

               Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy477

               Originating Machine: SONEP-NBU02

               Service Machine: SONEP-NBU02

               Provider: 'Microsoft Software Shadow Copy provider 1.0'

               Type: ClientAccessible

               Attributes: Persistent, Client-accessible, No auto release, No writers

       Differential

      C:\Users\Admin>

       

      C:\Users\Admin>vssadmin delete shadows /for=C:

      vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

      (C) Copyright 2001-2005 Microsoft Corp.

      Do you really want to delete 1 shadow copies (Y/N): [N]? Y

      Successfully deleted 1 shadow copies.

      C:\Users\Admin>

      • Lowell_Palecek's avatar
        Lowell_Palecek
        Level 6
        8 years ago

        Your vssadmin example shows you have 900MB snapshot space for C:\.  On an active system, 900 MB is tiny. Be aware that if your copy-on-write snapshot runs out of space, the OS deletes the snapshot right out from under bpbkar. That's probably not what's happening here, but we had a pair of cases a couple of years ago where it did happen. One customer had just what you have, 900 MB snapshot space for the drive that held all their Exchange databases.

        Your bpbkar snippet is so short, all I see is G:\, not C:\. What drives are you backing up? What drives is it failing on? Can we see more of the bpbkar log, with the logging level turned up?