11-07-2016 07:42 AM
I have a strange problem and I hope that anyone can help me. Recently I do a migration from netbackup 7.5.0.1 to 7.6.0.4 The problem regards the bck of a Media server with nbu policy type "MS-windos" with backup of "all_local_drives" and so also shadow copy components. The client is a win srv 2008 R2 entreprise SP1 After the migration backup fail wih retur code 69. From job details I can find only general info about the error: =============== 11/3/2016 11:55:49 PM - Error bpbrm(pid=9776) from client xxx-nbu02: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.JBD - bpbkar waited 2334 times for empty buffer, delayed 2500 times. 11/3/2016 11:55:51 PM - end writing; write time: 1:53:51 11/3/2016 11:55:52 PM - Info sonep-nbu02(pid=2720) StorageServer=PureDisk:xxx-nbu02; Report=PDDO Stats for (xxx-nbu02): scanned: 33463880 KB, CR sent: 3121825 KB, CR sent over FC: 0 KB, dedup: 90.7%, cache disabled 11/3/2016 11:55:52 PM - Error bpbrm(pid=9776) could not send server status message 11/3/2016 11:55:52 PM - Info bpbkar32(pid=10388) done. status: 69: invalid filelist specification invalid filelist specification(69) =================== vssadmin list writers shows a bunch of writers all stable and No error on last errorvssadmin list providers shows Microsoft software shadow copy provider 1.0 version 1.0.0.7
Solved! Go to Solution.
03-09-2017 07:08 AM
11-07-2016 11:00 AM
Can you post the bpbkar log?
Also, when loading the page for your post, the sidebar showed a few posts with error 69 mentioned. One of them talks about tech note: http://www.veritas.com/docs/000017926 Check that too.
11-07-2016 11:58 PM
As your problem seems to be related to VSS/System state this blog:
might be of help. In my experience most often the problem is a too small shadow storage area.
11-08-2016 04:14 AM
I tested this solution and I runed the job, but still the same results.
Please find attached the log files of bpbkar:
01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new directory:<G:\NBU_Catalog>, forcing full backup
01:18:12.987 [8052.10876] <2> tar_base::V_vTarMsgW: INF - Excluded: G:\NBU_Catalog
01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new directory:<G:\System Volume Information>, forcing full backup
01:18:12.987 [8052.10876] <4> dos_backup::tfs_scannext: INF - detected renamed/new file:<G:\System Volume Information\tracking.log>, forcing backup
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scannext: INF - no more path list entries
01:18:13.003 [8052.10876] <4> tar_backup::backup_endarg_state: INF - finish error count: 0
01:18:13.003 [8052.10876] <4> tar_backup::backup_endarg_state: INF - finish fatal count: 0
01:18:13.003 [8052.10876] <4> tar_backup_tfi::backup_arg_state: INF - remember start error count: 0
01:18:13.003 [8052.10876] <4> tar_backup_tfi::backup_arg_state: INF - remember start fatal count: 0
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - starting scan for file directive:<System State:\>
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - starting scan using name:<System State:\>
01:18:13.003 [8052.10876] <4> dos_backup::tfs_scanstart: INF - in 'backup' mode
01:18:13.018 [8052.10876] <2> ov_log::V_GlobalLog: ERR - BEDS_AttachToDLE():FS_AttachToDLE() DeviceName:'System?State' BackupReason:0x2 Failed! (0xE000FEC3:A failure occurred accessing the SnapShot coordinator.
)
01:18:13.018 [8052.10876] <2> tar_base::V_vTarMsgW: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.
01:18:13.018 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - number of file directives not found: 3
01:18:13.018 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - number of file directives found: 5
01:18:13.018 [8052.7888] <4> tar_base::keepaliveThread: INF - keepalive thread terminating (reason: WAIT_OBJECT_0)
01:18:13.018 [8052.10876] <4> tar_base::stopKeepaliveThread: INF - keepalive thread has exited. (reason: WAIT_OBJECT_0)
01:18:13.018 [8052.10876] <4> send_msg_to_monitor: INF - in send_msg_to_monitor()...
01:18:13.034 [8052.10876] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 69: invalid filelist specification
01:18:13.034 [8052.10876] <4> tar_backup_tfi::backup_done_state: INF - Not waiting for server status
01:18:14.048 [8052.10876] <4> dos_backup::ClearDFSRArchiveBits: INF - no DFSR file list was created, No bits cleared.
01:18:14.048 [8052.10876] <4> ov_log::OVLoop: Timestamp
01:18:14.048 [8052.10876] <4> OVStopCmd: INF - EXIT - status = 0
01:18:14.048 [8052.10876] <2> tar_base::V_Close: closing...
01:18:14.048 [8052.10876] <4> dos_backup::tfs_reset: INF - Snapshot deletion start
01:18:14.064 [8052.10876] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): enter - InitFlags:0x00000101
01:18:14.064 [8052.10876] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): ubs specifics: 0x001d0000
01:18:14.204 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:15.218 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:16.232 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:17.246 [8052.10876] <16> dtcp_read: TCP - failure: recv socket (580) (TCP 10053: Software caused connection abort)
01:18:17.246 [8052.10876] <4> OVShutdown: INF - Finished process
01:18:17.246 [8052.10876] <4> WinMain: INF - Exiting D:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe
11-08-2016 04:29 AM
I tried many things:
1) I checked the VSS Writers: (all good):vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-
(C) Copyright 2001-2005 Microsoft Corp.
Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error
Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error
Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error
Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {6f2cc347-8109-45b5-a9b5-bcb3a45a1816}
State: [1] Stable
Last error: No error
Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {b6e524b8-04a5-406c-8153-0a0187317e5b}
State: [1] Stable
Last error: No error
Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {8e3b348b-086f-40e7-a963-44a5fed92bf1}
State: [1] Stable
Last error: No error
Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {7a0504c6-6770-473d-a3d7-b6f87b764db7}
State: [1] Stable
Last error: No error
Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {5312c4be-1c60-4162-a3d8-aaac86c70890}
State: [1] Stable
Last error: No error
Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {9ef4cbe3-3a82-4645-9e20-425cc0997a51}
State: [1] Stable
Last error: No error
Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {f0da1a64-6376-40d5-a725-60671f9dc9ac}
State: [1] Stable
Last error: No error
Writer name: 'Dhcp Jet Writer'
Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
Writer Instance Id: {86e1afd4-3f48-448a-981d-b239fbd8879a}
State: [1] Stable
Last error: No error
Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {d97a0e4e-8db6-4d05-970b-b2a3e6358911}
State: [1] Stable
Last error: No error
2) I checked the VSS provider (all good)
vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
3) There are volumes that exceed 62 TB, I extended the registry key value HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ BEDS \ Engine \ Misc \ BescLargeDiscBlock to 68 and I started the job but still the same thing
https://www.veritas.com/support/en_US/article.TECH206217
4) To registre the VSS, the OS is already Windows 2008 R2 SP1 (so Nothing to do)
https://www.veritas.com/support/en_US/article.000015843
5) I rebooted the server several times, but still the same problem
Version: 1.0.0.7
11-08-2016 06:41 AM
There seems to be something missing from the end of your last post.
One more question: have you noticed anything on the server (being backed up) at the time the backup fails in event logs?
also, check out what Michael linked above.
11-08-2016 06:43 AM
11-09-2016 07:06 AM
Is this the most verbose bpbkar log you can provide? For example, bpbkar enumerates the providers.
Michael suggested that you may not have enough space allocated to snap the G: drive. You can use vssadmin to answer check your snapshot limit.
You migrated to 7.6.0.4? Why not 7.7.x?
11-09-2016 01:14 PM
Hello,
As suggested earlier by Marrianee please try to backup Shadow Copy components with Windows server backup utility to make sure there is no problem with VSS.
11-09-2016 02:10 PM
there is no problem with VSS, I perform this test successfully
vssadmin Add ShadowStorage /For=C: /On=D: /MaxSize=900MB
(C) Copyright 2001-2005 Microsoft Corp.
Successfully added the shadow copy storage association
=============================================================
C:\Users\Admin>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Shadow Copy Storage association
For volume: (\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}\)\\?\Volume{724
0e843-914a-11e2-bb8c-806e6f6e6963}\
Shadow Copy Storage volume: (\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}
\)\\?\Volume{7240e843-914a-11e2-bb8c-806e6f6e6963}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: UNBOUNDED (100%)
Shadow Copy Storage association
For volume: (C:)\\?\Volume{7240e844-914a-11e2-bb8c-806e6f6e6963}\
Shadow Copy Storage volume: (D:)\\?\Volume{3df823db-9166-11e2-b277-6c3be5a607
c7}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: 900 MB (0%)
==================================================================================
C:\Users\Admin>vssadmin create shadow /for=C:
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Successfully created shadow copy for 'C:\'
Shadow Copy ID: {6e074368-ded5-4aa8-9a21-bdb48de7f497}
Shadow Copy Volume Name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy477
C:\Users\Admin>
C:\Users\Admin>vssadmin list shadows
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Contents of shadow copy set ID: {5f60055e-0248-49f5-811c-4478ed193641}
Contained 1 shadow copies at creation time: 11/4/2016 5:06:13 AM
Shadow Copy ID: {6e074368-ded5-4aa8-9a21-bdb48de7f497}
Original Volume: (C:)\\?\Volume{7240e844-914a-11e2-bb8c-806e6f6e6963}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy477
Originating Machine: SONEP-NBU02
Service Machine: SONEP-NBU02
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ClientAccessible
Attributes: Persistent, Client-accessible, No auto release, No writers
Differential
C:\Users\Admin>
C:\Users\Admin>vssadmin delete shadows /for=C:
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Do you really want to delete 1 shadow copies (Y/N): [N]? Y
Successfully deleted 1 shadow copies.
C:\Users\Admin>
11-09-2016 02:26 PM
Hello Lowell_palecek,
I already configured No limit in shodow copie setting for G.
We did not migrate directly to 7.7, because we tried to avoid the conversion of segments deduplication twice, it will be scheduled for a later time
11-10-2016 07:42 AM
Your vssadmin example shows you have 900MB snapshot space for C:\. On an active system, 900 MB is tiny. Be aware that if your copy-on-write snapshot runs out of space, the OS deletes the snapshot right out from under bpbkar. That's probably not what's happening here, but we had a pair of cases a couple of years ago where it did happen. One customer had just what you have, 900 MB snapshot space for the drive that held all their Exchange databases.
Your bpbkar snippet is so short, all I see is G:\, not C:\. What drives are you backing up? What drives is it failing on? Can we see more of the bpbkar log, with the logging level turned up?
11-11-2016 12:11 AM
Have you in the meantime tried Windows backup to backup SCC to local disk?
If this is working fine, have a look at the Logon Service used for NetBackup Client Service.
Sometimes this logon account is reset to LocalSystem during NBU upgrade.
Change the NBU logon account to a local or domain admin with full access to SCC and see if that makes a difference.
03-09-2017 07:08 AM