Highlighted

Probleme backup Media Server Windows 2008 R2 status 69

I have a strange problem and I hope that anyone can help me.
Recently I do a migration from netbackup 7.5.0.1 to 7.6.0.4
The problem regards the bck of a Media server with nbu policy type "MS-windos" with backup of "all_local_drives" and so also shadow copy components.
The client is a win srv 2008 R2 entreprise SP1 
After the migration backup fail wih retur code 69. From job details I can find only general info about the error:
===============
11/3/2016 11:55:49 PM - Error bpbrm(pid=9776) from client xxx-nbu02: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.JBD - bpbkar waited 2334 times for empty buffer, delayed 2500 times.
11/3/2016 11:55:51 PM - end writing; write time: 1:53:51
11/3/2016 11:55:52 PM - Info sonep-nbu02(pid=2720) StorageServer=PureDisk:xxx-nbu02; Report=PDDO Stats for (xxx-nbu02): scanned: 33463880 KB, CR sent: 3121825 KB, CR sent over FC: 0 KB, dedup: 90.7%, cache disabled
11/3/2016 11:55:52 PM - Error bpbrm(pid=9776) could not send server status message       
11/3/2016 11:55:52 PM - Info bpbkar32(pid=10388) done. status: 69: invalid filelist specification       
invalid filelist specification(69)
===================
vssadmin list writers shows a bunch of writers all stable and No error on last errorvssadmin list providers shows Microsoft software shadow copy provider 1.0 version 1.0.0.7
Tags (1)
1 Solution

Accepted Solutions
Highlighted
Accepted Solution!

Re: Probleme backup Media Server Windows 2008 R2 status 69

13 Replies
Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

Can you post the bpbkar log?

Also, when loading the page for your post, the sidebar showed a few posts with error 69 mentioned. One of them talks about tech note: http://www.veritas.com/docs/000017926 Check that too.

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

As your problem seems to be related to VSS/System state this blog:

https://blogs.technet.microsoft.com/askcore/2010/06/18/reasons-why-the-error-enumeration-of-the-file...

might be of help. In my experience most often the problem is a too small shadow storage area.

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue
Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

I tested this solution and I runed the job, but still the same results.
Please find attached the log files of bpbkar: 

 01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new directory:<G:\NBU_Catalog>, forcing full backup
01:18:12.987 [8052.10876] <2> tar_base::V_vTarMsgW: INF - Excluded: G:\NBU_Catalog
01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new directory:<G:\System Volume Information>, forcing full backup
01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new file:<G:\System Volume Information\tracking.log>, forcing backup
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scannext: INF - no more path list entries
01:18:13.003 [8052.10876] <4> tar_backup::backup_endarg_state: INF - finish error count: 0
01:18:13.003 [8052.10876] <4> tar_backup::backup_endarg_state: INF - finish fatal count: 0
01:18:13.003 [8052.10876] <4> tar_backup_tfi::backup_arg_state: INF - remember start error count: 0
01:18:13.003 [8052.10876] <4> tar_backup_tfi::backup_arg_state: INF - remember start fatal count: 0
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - starting scan for file directive:<System State:\>
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - starting scan using name:<System State:\>
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - in 'backup' mode
01:18:13.018 [8052.10876] <2> ov_log::V_GlobalLog: ERR - BEDS_AttachToDLE():FS_AttachToDLE() DeviceName:'System?State' BackupReason:0x2 Failed! (0xE000FEC3:A failure occurred accessing the SnapShot coordinator.
)
01:18:13.018 [8052.10876] <2> tar_base::V_vTarMsgW: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.
01:18:13.018 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - number of file directives not found: 3
01:18:13.018 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - number of file directives found: 5
01:18:13.018 [8052.7888] <4> tar_base::keepaliveThread: INF - keepalive thread terminating (reason: WAIT_OBJECT_0)
01:18:13.018 [8052.10876] <4> tar_base::stopKeepaliveThread: INF - keepalive thread has exited. (reason: WAIT_OBJECT_0)
01:18:13.018 [8052.10876] <4> send_msg_to_monitor: INF - in send_msg_to_monitor()...
01:18:13.034 [8052.10876] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 69: invalid filelist specification
01:18:13.034 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - Not waiting for server status
01:18:14.048 [8052.10876] <4> dos_backup::ClearDFSRArchiveBits: INF - no DFSR file list was created, No bits cleared.
01:18:14.048 [8052.10876] <4> ov_log:Smiley SurprisedVLoop: Timestamp
01:18:14.048 [8052.10876] <4> OVStopCmd: INF - EXIT - status = 0
01:18:14.048 [8052.10876] <2> tar_base::V_Close: closing...
01:18:14.048 [8052.10876] <4> dos_backup::tfs_reset: INF - Snapshot deletion start
01:18:14.064 [8052.10876] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): enter - InitFlags:0x00000101
01:18:14.064 [8052.10876] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): ubs specifics: 0x001d0000
01:18:14.204 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:15.218 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:16.232 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:17.246 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:17.246 [8052.10876] <4> OVShutdown: INF - Finished process
01:18:17.246 [8052.10876] <4> WinMain: INF - Exiting D:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe

 

Re: Probleme backup Media Server Windows 2008 R2 status 69

 I tried many things:


1) I checked the VSS Writers: (all good):vssadmin list writers

vssadmin 1.1 - Volume Shadow Copy Service administrative command-

(C) Copyright 2001-2005 Microsoft Corp.

 

Writer name: 'Task Scheduler Writer'

   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}

   State: [1] Stable

   Last error: No error

 

Writer name: 'VSS Metadata Store Writer'

   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

   State: [1] Stable

   Last error: No error

 

Writer name: 'Performance Counters Writer'

   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}

   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

   State: [1] Stable

   Last error: No error

 

Writer name: 'System Writer'

   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

   Writer Instance Id: {6f2cc347-8109-45b5-a9b5-bcb3a45a1816}

   State: [1] Stable

   Last error: No error

 

Writer name: 'ASR Writer'

   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

   Writer Instance Id: {b6e524b8-04a5-406c-8153-0a0187317e5b}

   State: [1] Stable

   Last error: No error

 

Writer name: 'Registry Writer'

   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

   Writer Instance Id: {8e3b348b-086f-40e7-a963-44a5fed92bf1}

   State: [1] Stable

   Last error: No error

 

Writer name: 'Shadow Copy Optimization Writer'

   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

   Writer Instance Id: {7a0504c6-6770-473d-a3d7-b6f87b764db7}

   State: [1] Stable

   Last error: No error

 

Writer name: 'IIS Metabase Writer'

   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}

   Writer Instance Id: {5312c4be-1c60-4162-a3d8-aaac86c70890}

   State: [1] Stable

   Last error: No error

 

Writer name: 'IIS Config Writer'

   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}

   Writer Instance Id: {9ef4cbe3-3a82-4645-9e20-425cc0997a51}

   State: [1] Stable

   Last error: No error

 

Writer name: 'WMI Writer'

   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

   Writer Instance Id: {f0da1a64-6376-40d5-a725-60671f9dc9ac}

   State: [1] Stable

   Last error: No error

 

Writer name: 'Dhcp Jet Writer'

   Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}

   Writer Instance Id: {86e1afd4-3f48-448a-981d-b239fbd8879a}

   State: [1] Stable

   Last error: No error

 

Writer name: 'COM+ REGDB Writer'

   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

   Writer Instance Id: {d97a0e4e-8db6-4d05-970b-b2a3e6358911}

   State: [1] Stable

   Last error: No error

2) I checked the VSS provider (all good)

vssadmin list providers

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

 Provider name: 'Microsoft Software Shadow Copy provider 1.0'

   Provider type: System

   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}

3) There are volumes that exceed 62 TB, I extended the registry key value HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ BEDS \ Engine \ Misc \ BescLargeDiscBlock to 68 and I started the job but still the same thing

https://www.veritas.com/support/en_US/article.TECH206217

4) To registre the VSS, the OS is already Windows 2008 R2 SP1 (so Nothing to do)

https://www.veritas.com/support/en_US/article.000015843

5) I rebooted the server several times, but still the same problem

 

 

 

 

   Version: 1.0.0.7

 

 

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

There seems to be something missing from the end of your last post.

One more question: have you noticed anything on the server (being backed up) at the time the backup fails in event logs?

also, check out what Michael linked above.

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

Have you tried to backup SCC using Windows Backup utility?

 

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

Is this the most verbose bpbkar log you can provide? For example, bpbkar enumerates the providers.

Michael suggested that you may not have enough space allocated to snap the G: drive. You can use vssadmin to answer check your snapshot limit.

You migrated to 7.6.0.4?  Why not 7.7.x?

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

Hello,

As suggested earlier by Marrianee please try to backup Shadow Copy components with Windows server backup utility to make sure there is no problem with VSS.

 

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

there is no problem with VSS, I perform this test successfully

vssadmin Add ShadowStorage /For=C: /On=D: /MaxSize=900MB

 (C) Copyright 2001-2005 Microsoft Corp.

Successfully added the shadow copy storage association

=============================================================

C:\Users\Admin>vssadmin list shadowstorage

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Shadow Copy Storage association

   For volume: (\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}\)\\?\Volume{724

0e843-914a-11e2-bb8c-806e6f6e6963}\

   Shadow Copy Storage volume: (\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}

\)\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}\

   Used Shadow Copy Storage space: 0 B (0%)

   Allocated Shadow Copy Storage space: 0 B (0%)

   Maximum Shadow Copy Storage space: UNBOUNDED (100%)

Shadow Copy Storage association

   For volume: (CSmiley Happy\\?\Volume{7240e844-914a-11e2-bb8c-806e6f6e6963}\

   Shadow Copy Storage volume: (DSmiley Happy\\?\Volume{3df823db-9166-11e2-b277-6c3be5a607

c7}\

   Used Shadow Copy Storage space: 0 B (0%)

   Allocated Shadow Copy Storage space: 0 B (0%)

   Maximum Shadow Copy Storage space: 900 MB (0%)

==================================================================================

C:\Users\Admin>vssadmin create shadow /for=C:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Successfully created shadow copy for 'C:\'

    Shadow Copy ID: {6e074368-ded5-4aa8-9a21-bdb48de7f497}

    Shadow Copy Volume Name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy477

C:\Users\Admin>

C:\Users\Admin>vssadmin list shadows

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Contents of shadow copy set ID: {5f60055e-0248-49f5-811c-4478ed193641}

   Contained 1 shadow copies at creation time: 11/4/2016 5:06:13 AM

      Shadow Copy ID: {6e074368-ded5-4aa8-9a21-bdb48de7f497}

         Original Volume: (CSmiley Happy\\?\Volume{7240e844-914a-11e2-bb8c-806e6f6e6963}\

         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy477

         Originating Machine: SONEP-NBU02

         Service Machine: SONEP-NBU02

         Provider: 'Microsoft Software Shadow Copy provider 1.0'

         Type: ClientAccessible

         Attributes: Persistent, Client-accessible, No auto release, No writers

 Differential

C:\Users\Admin>

 

C:\Users\Admin>vssadmin delete shadows /for=C:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Do you really want to delete 1 shadow copies (Y/N): [N]? Y

Successfully deleted 1 shadow copies.

C:\Users\Admin>

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

Hello Lowell_palecek, 

I already configured No limit in shodow copie setting for G.

We did not migrate directly to 7.7, because we tried to avoid the conversion of segments deduplication twice, it will be scheduled for a later time

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

Your vssadmin example shows you have 900MB snapshot space for C:\.  On an active system, 900 MB is tiny. Be aware that if your copy-on-write snapshot runs out of space, the OS deletes the snapshot right out from under bpbkar. That's probably not what's happening here, but we had a pair of cases a couple of years ago where it did happen. One customer had just what you have, 900 MB snapshot space for the drive that held all their Exchange databases.

Your bpbkar snippet is so short, all I see is G:\, not C:\. What drives are you backing up? What drives is it failing on? Can we see more of the bpbkar log, with the logging level turned up?

Highlighted

Re: Probleme backup Media Server Windows 2008 R2 status 69

Have you in the meantime tried Windows backup to backup SCC to local disk?

If this is working fine, have a look at the Logon Service used for NetBackup Client Service.
Sometimes this logon account is reset to LocalSystem during NBU upgrade.

Change the NBU logon account to a local or domain admin with full access to SCC and see if that makes a difference.

 

Highlighted
Accepted Solution!

Re: Probleme backup Media Server Windows 2008 R2 status 69