cancel
Showing results for 
Search instead for 
Did you mean: 

Incremental backup fail with status code 1

danguss
Level 4

Hello, I have a big problem with my system state incremental backups on two clustered servers ending with status 1, which only the full backup ending with status 0.

Master Server Version 8.1.2 - Windows server 2016

Media Server Verion  8.1.2 - Linux 3.10.0-693.17.1.el7.x86_64 

Client Version 8.1.2 - Windows server 2016

I performed the procedures for vss verification and found no failure.

can you help me?

Attached some logs

Thank you very much in advance

1 ACCEPTED SOLUTION

Accepted Solutions

sdo
Moderator
Moderator
Partner    VIP    Certified

The problem is a moving target.  It was noted by others that they could address the issue for a short while or a fair while, one to 'n' backup jobs, before the same problem began appearing again but for other so called critical files.  If you look back at the link I posted earlier, I did provide a short script to help identify the backup jobs that report a particular error code.  So I guess you could write a different script to filter out those jobs with that error code and thus boil down the list of status 1 partial backup jobs down to a list of true partial status 1 that need to be checked / addressed / fixed - but this script would be quite large and complex.

What gets me is... how are we supposed to know whether a missing critical file really is a critical or not... what this whole problem says to me is that I can never again really trust my BMR aware backups of Windows Server 2016... because I can never know whether any one given backup job is really missing something that really is critical.  IMO this simply isn't good enough, and customers should not have to put with this - no matter which vendor is the root cause (and IMO, AFAICT, the root cause is Microsoft).

At then end of the day, until Microsoft fix it, then we are all just going to have to accept partial status 1 incremental backups of Windows Server 2016 nodes that include the System_State:.  Not nice, AFAICT there's nothing else any of us can do, except keep chasing our tails.

Some options / mitigations / tasks might be :

1) filter those backup from your daily reporting / stats

2) cease taking incremental System_State backups of Windows Server 2016, and instead - for all Windows Server 2016 backup clients - take daily full backups of C: + System_State:

3) simply cease taking incremental backups of System_State: of all Windows Server 2016 nodes

4) add an entry in your IT Service "risk register" that acknowledges the problem / risk

5) regularly test BMR recovery of one or more Windows Server 2016 backup clients

.

Anyone else care to add anything ?

View solution in original post

18 REPLIES 18

Krutons
Moderator
Moderator
   VIP   

From your bpbkar logs.

So, 32 files are missing from the snapshot according to this
12:21:59.033 [2560.6892] <2> ov_log::V_GlobalLog: INF - Component Set System Files has 32 files missing on the snapshot

Those 32 files would be the following.

12:20:06.606 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:06.606 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:08.236 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:08.236 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:20.903 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
12:20:20.903 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
12:20:21.603 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:21.603 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:22.976 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:22.976 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:24.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:24.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:25.635 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:25.635 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:28.066 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:28.066 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:30.490 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:30.490 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:30.961 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:30.961 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:31.894 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:31.894 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList

12:20:37.194 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:37.194 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:37.595 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:37.595 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:40.638 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:40.638 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:50.258 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:50.258 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:52.787 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:52.787 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.379 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.379 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.384 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.384 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.041 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.041 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.046 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.046 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:10.727 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:21:10.727 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:21:13.975 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:13.975 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:17.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:17.432 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:23.588 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:21:23.588 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:21:23.823 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:23.823 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:29.406 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:21:29.406 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList

12:21:43.466 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:43.466 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:48.969 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:48.969 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:50.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:50.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:51.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:51.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:52.970 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:52.970 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:55.756 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
12:21:55.756 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList

 

So based off of what I saw while browsing your bpbkar logs, take a look at this link and figure out why those files are missing. https://www.veritas.com/support/en_US/article.100034278

 

Hello,

Thanks for the answer. About this article, step by step was performed on both nodes and then performed an incremental backup, the first one worked successfully, but when performing the incremental backup again it failed and continues to fail.

Krutons
Moderator
Moderator
   VIP   

Did you figure out why those 32 files were missing?

So the next backup you ran was an incremental after making any changes that you did, is that correct?

Could you upload the logs for a new status 1 backup after you made your changes?
Please increase the logging level above minimum.

To be honest, I don't know why these files are missing, but would that impact backups?

The logs I attached were after running the mentioned link.

I am attaching logs run before this procedure. And yes, I performed two incremental backups after the procedure was executed, the first successfully and all the next failed.

Krutons
Moderator
Moderator
   VIP   

Yea those missing files are the problem, because they remain registered in the VSS Writer config. As @sdo mentions in his link, it looks like he has had the same/similar issue and says the same thing I am.

Doesn't look like he found a permanent fix yet and Veritas hasn't addressed it yet it seems. It seems as though if you needed to perform a restore though, you wouldn't have any issues because the VSS Writer config just has files that no longer exist as well as files that do exist (the ones needed for a successful restore). Essentially it seems as though it's a "false" flag and the backup is successful (not a status 1).

Correct me if I am wrong @sdo 

sdo
Moderator
Moderator
Partner    VIP    Certified
Not a correction... just an opinion from me... in so far as I see it as a Microsoft issue, it’s not a Veritas issue - there’s nothing any backup vendor can do about it if Microsoft see fit to abandon orphaned links of so called ‘critical files’ registered in VSS.

Krutons
Moderator
Moderator
   VIP   

@danguss I glanced at those new logs, the logging level is 0 which won't tell us much.

@KrutonsCould you help me by listing these files, did a search and found how "not found, 09:38:34.264 [7932.6904] <2> ov_log::V_GlobalLog: INF - EMSDB32.dll not found"

And what action could I be performing?

@sdo The procedure was unsuccessful :(

Krutons
Moderator
Moderator
   VIP   

Totally agree sdo, didn't mean it was a Veritas problem. Just something that Microsoft hasn't helped Veritas fix. As you said and I have read, this issue doesn't affect Microsoft's own server backups which means they know how to exclude these files.

sdo
Moderator
Moderator
Partner    VIP    Certified

The problem is a moving target.  It was noted by others that they could address the issue for a short while or a fair while, one to 'n' backup jobs, before the same problem began appearing again but for other so called critical files.  If you look back at the link I posted earlier, I did provide a short script to help identify the backup jobs that report a particular error code.  So I guess you could write a different script to filter out those jobs with that error code and thus boil down the list of status 1 partial backup jobs down to a list of true partial status 1 that need to be checked / addressed / fixed - but this script would be quite large and complex.

What gets me is... how are we supposed to know whether a missing critical file really is a critical or not... what this whole problem says to me is that I can never again really trust my BMR aware backups of Windows Server 2016... because I can never know whether any one given backup job is really missing something that really is critical.  IMO this simply isn't good enough, and customers should not have to put with this - no matter which vendor is the root cause (and IMO, AFAICT, the root cause is Microsoft).

At then end of the day, until Microsoft fix it, then we are all just going to have to accept partial status 1 incremental backups of Windows Server 2016 nodes that include the System_State:.  Not nice, AFAICT there's nothing else any of us can do, except keep chasing our tails.

Some options / mitigations / tasks might be :

1) filter those backup from your daily reporting / stats

2) cease taking incremental System_State backups of Windows Server 2016, and instead - for all Windows Server 2016 backup clients - take daily full backups of C: + System_State:

3) simply cease taking incremental backups of System_State: of all Windows Server 2016 nodes

4) add an entry in your IT Service "risk register" that acknowledges the problem / risk

5) regularly test BMR recovery of one or more Windows Server 2016 backup clients

.

Anyone else care to add anything ?

I am analyzing the logs and trying to find some flaws in the operating system, still unsolved.

sdo
Moderator
Moderator
Partner    VIP    Certified

Here's a thought... if you are not using BMR then do you need to take a backup of System_State: ?

@sdo In cases of physical servers yes, we need ...

Fixed some notes on vCenter disks and booted the server, tested two incremental backups successfully, but following the schedule the incremental backups failed.

Backups are very intermittent, attached two bpbkar logs,bpbkar failed and bpbkar successfully.

We both checked the 32 missing files.

 

 

 

 

sdo
Moderator
Moderator
Partner    VIP    Certified

You, nor I, nor Veritas, can overcome the problem.  It will keep coming back, and keep hapenning to different files at different times.  You either have to ignore it, or filter it from alerting/reporting.  Microsoft expect you to treat it as a false negative, and Microsoft expect you to assume all is ok.

I agree, but do we need evidence for something to present to microsoft or not?

sdo
Moderator
Moderator
Partner    VIP    Certified

I guess you can try if you want to.  Seems to me Veritas have already tried, and hit the same wall.  Hence the Veritas tech note points (offloads) to the Microsoft tech note, and the Microsoft tech note says :

1) MS say: third party backup products may report errors

2) MS say: our own (i.e. Microsoft's) backup product does not report errors

Ask yourself why does MS own backup product not report errors ?  I can only think because they either ignore those errors or they no longer use the feature of VSS critical files or they have a new/different/undocumented/unshared mechanism for protecting System_State that other third party backup vendors do not yet know about.

sdo
Moderator
Moderator
Partner    VIP    Certified

Has anyone else in the community got any corrections, advice, suggestions, opinions, interpretations?

I'm actually hoping that I've got something wrong, or mis-construed something(s).