Backup failing with Netbackup accelerator

[Nishath]

Hello All,

I have a client running on Windows 2008(Netbackup accelerator enabled) configured on the Netbackup Master server with Linux platform. The last successful backup on this client was on 11/26/2013. However, post this backup, the other backups have been failing with the error message on the detailed job status given below:

12/19/2013 02:13:34 - Info bpbkar (pid=5100) NOT using change journal data for <D:\>: unable to validate change journal usage <reason=previous backup wasn't a successful backup>
12/19/2013 02:15:04 - Info bpbkar (pid=5100) 115000 entries sent to bpdbm
12/19/2013 02:17:22 - Info bpbkar (pid=5100) 120000 entries sent to bpdbm
12/19/2013 02:18:41 - Info bpbkar (pid=5100) 125000 entries sent to bpdbm
12/19/2013 02:19:06 - Info bpbkar (pid=5100) 130000 entries sent to bpdbm
12/19/2013 02:19:39 - Info bpbkar (pid=5100) 135000 entries sent to bpdbm
12/19/2013 02:21:59 - Info bpbkar (pid=5100) 140000 entries sent to bpdbm
12/19/2013 02:25:14 - Info bpbkar (pid=5100) 145000 entries sent to bpdbm
12/19/2013 02:26:40 - Info bpbkar (pid=5100) 150000 entries sent to bpdbm
12/19/2013 02:27:22 - Info bpbkar (pid=5100) 155000 entries sent to bpdbm
12/19/2013 02:27:41 - Info bpbkar (pid=5100) 160000 entries sent to bpdbm
12/19/2013 02:28:00 - Info bpbkar (pid=5100) 165000 entries sent to bpdbm
12/19/2013 02:28:24 - Info bpbkar (pid=5100) 170000 entries sent to bpdbm
12/19/2013 02:28:41 - Info bpbkar (pid=5100) 175000 entries sent to bpdbm
12/19/2013 02:28:57 - Info bpbkar (pid=5100) 180000 entries sent to bpdbm
12/19/2013 02:29:05 - Info bpbkar (pid=5100) 185000 entries sent to bpdbm
12/19/2013 02:29:29 - Info bpbkar (pid=5100) 190000 entries sent to bpdbm
12/19/2013 02:29:47 - Info bpbkar (pid=5100) 195000 entries sent to bpdbm
12/19/2013 02:30:42 - Info bpbkar (pid=5100) 200000 entries sent to bpdbm
12/19/2013 02:33:51 - Info bpbkar (pid=5100) 205000 entries sent to bpdbm
12/19/2013 02:36:10 - Info bpbkar (pid=5100) 210000 entries sent to bpdbm
12/19/2013 02:38:43 - Info bpbkar (pid=5100) 215000 entries sent to bpdbm
12/19/2013 02:40:20 - Info bpbkar (pid=5100) change journal enabled for <E:\>
12/19/2013 02:40:20 - Info bpbkar (pid=5100) NOT using change journal data for <E:\>: unable to validate change journal usage <reason=previous backup wasn't a successful backup>
12/19/2013 02:41:47 - Info bpbkar (pid=5100) 220000 entries sent to bpdbm
12/19/2013 06:27:44 - Info bpbkar (pid=5100) change journal enabled for <F:\>
12/19/2013 06:27:49 - Info bpbkar (pid=5100) NOT using change journal data for <F:\>: unable to validate change journal usage <reason=previous backup wasn't a successful backup>
12/19/2013 14:59:43 - Info bpbkar (pid=5100) change journal enabled for <G:\>
12/19/2013 14:59:44 - Info bpbkar (pid=5100) NOT using change journal data for <G:\>: unable to validate change journal usage <reason=previous backup wasn't a successful backup>
12/19/2013 21:03:14 - Error bpbrm (pid=17462) from client faclsna01bapi03.firstamdata.net: ERR - failure reading file: G:\MSSQL10.MSSQLSERVER\MSSQL\Backup\LoanApp_Orders.sls (WIN32 21: The device is not ready. )
12/19/2013 21:03:14 - Error bpbrm (pid=17462) from client faclsna01bapi03.firstamdata.net: ERR - Snapshot Error while reading file: GLOBALROOT\Device\HarddiskVolumeShadowCopy270\MSSQL10.MSSQLSERVER\MSSQL\Backup\LoanApp_Orders.sls
12/19/2013 21:03:14 - Critical bpbrm (pid=17462) from client faclsna01bapi03.firstamdata.net: FTL - Backup operation aborted!
12/19/2013 21:03:14 - Info bpbkar (pid=5100) accelerator sent 1596318642176 bytes out of 3760995330560 bytes to server, optimization 57.6%
12/19/2013 21:03:16 - Error bptm (pid=17580) media manager terminated by parent process
12/19/2013 21:04:17 - Info clgx-qtc-nbumed33 (pid=17580) StorageServer=PureDisk:clgx-qtc-nbumed33; Report=PDDO Stats for (clgx-qtc-nbumed33): scanned: 3672852818 KB, CR sent: 530046194 KB, CR sent over FC: 0 KB, dedup: 85.6%
12/19/2013 21:04:18 - Error bpbrm (pid=17462) could not send server status message
12/19/2013 21:04:22 - Info bpbkar (pid=5100) done. status: 156: snapshot error encountered
12/19/2013 21:04:22 - end writing; write time: 19:31:14
12/19/2013 21:04:25 - Info bpbrm (pid=8822) Starting delete snapshot processing
12/19/2013 21:04:25 - Info bpfis (pid=0) Snapshot will not be deleted
12/19/2013 21:04:26 - Error bpbrm (pid=8822) from client faclsna01bapi03.firstamdata.net: ERR - Get bpfis state from clgx-wtc-nbumst01 failed. status = 25
12/19/2013 21:04:26 - Info bpfis (pid=17556) Backup started
12/19/2013 21:04:26 - Critical bpbrm (pid=8822) from client faclsna01bapi03.firstamdata.net: FTL - cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.faclsna01bapi03.firstamdata.net_1387445533.1.0
12/19/2013 21:04:27 - Info bpfis (pid=17556) done. status: 1542
12/19/2013 21:04:27 - Info bpfis (pid=0) done. status: 1542: An existing snapshot is no longer valid and cannot be mounted for subsequent operations.

I tried to disable the "Use Change Journal" option to take a full weekly backup and then enable the "Use Change Journal" option again but no luck the backup failed while writing data after a point. It sometimes fails with error 14 and sometimes with 156.

Can someone help me to understand what's going wrong and what could be the solution to get a successful backup on it ?

Thank you,
Nishath Sultana

[RiaanBadenhorst]

Hi, This should not have anything to with Accelerator. Disable it and run the backup again. Investigate this error further by verifying your shadow copy components (vss)or checking if that file is not corrupt. 12/19/2013 21:03:14 - Error bpbrm (pid=17462) from client faclsna01bapi03.firstamdata.net: ERR - failure reading file: G:\MSSQL10.MSSQLSERVER\MSSQL\Backup\LoanApp_Orders.sls (WIN32 21: The device is not ready. ) 12/19/2013 21:03:14 - Error bpbrm (pid=17462) from client faclsna01bapi03.firstamdata.net: ERR - Snapshot Error while reading file: GLOBALROOT\Device\HarddiskVolumeShadowCopy270\MSSQL10.MSSQLSERVER\MSSQL\Backup\LoanApp_Orders.sls

[RiaanBadenhorst]

Hi, This should not have anything to with Accelerator. Disable it and run the backup again. Investigate this error further by verifying your shadow copy components (vss)or checking if that file is not corrupt. 12/19/2013 21:03:14 - Error bpbrm (pid=17462) from client faclsna01bapi03.firstamdata.net: ERR - failure reading file: G:\MSSQL10.MSSQLSERVER\MSSQL\Backup\LoanApp_Orders.sls (WIN32 21: The device is not ready. ) 12/19/2013 21:03:14 - Error bpbrm (pid=17462) from client faclsna01bapi03.firstamdata.net: ERR - Snapshot Error while reading file: GLOBALROOT\Device\HarddiskVolumeShadowCopy270\MSSQL10.MSSQLSERVER\MSSQL\Backup\LoanApp_Orders.sls

[Mark_Solutions · Accepted solution]

You need to set some exclusions by the looks of things here - open SQL objects should not be backed up in a file system backup - these also cause the job to fail.

It will help to add this client to the Master Servers host properties - Client attributes section - and select to use VSS and to disable and continue if there is a snapshot error

Get your exclusions sorted out first - once you can get a good full backup it should be OK - it may be worth doing a forced re-scan too to rebuild the track log following these errors

[Nishath]

Mark, I followed to make the changes in the Master Server host properties by disabling VSS after which it worked and there were no failures. As Riaan also said it had nothing to do with the accelerator.

Thank you folks for your help! Really appreciated

-Nishath Sultana