08-09-2019 09:00 AM
Hello, I have a big problem with my system state incremental backups on two clustered servers ending with status 1, which only the full backup ending with status 0.
Master Server Version 8.1.2 - Windows server 2016
Media Server Verion 8.1.2 - Linux 3.10.0-693.17.1.el7.x86_64
Client Version 8.1.2 - Windows server 2016
I performed the procedures for vss verification and found no failure.
can you help me?
Attached some logs
Thank you very much in advance
Solved! Go to Solution.
08-10-2019 03:44 AM - edited 08-10-2019 03:48 AM
The problem is a moving target. It was noted by others that they could address the issue for a short while or a fair while, one to 'n' backup jobs, before the same problem began appearing again but for other so called critical files. If you look back at the link I posted earlier, I did provide a short script to help identify the backup jobs that report a particular error code. So I guess you could write a different script to filter out those jobs with that error code and thus boil down the list of status 1 partial backup jobs down to a list of true partial status 1 that need to be checked / addressed / fixed - but this script would be quite large and complex.
What gets me is... how are we supposed to know whether a missing critical file really is a critical or not... what this whole problem says to me is that I can never again really trust my BMR aware backups of Windows Server 2016... because I can never know whether any one given backup job is really missing something that really is critical. IMO this simply isn't good enough, and customers should not have to put with this - no matter which vendor is the root cause (and IMO, AFAICT, the root cause is Microsoft).
At then end of the day, until Microsoft fix it, then we are all just going to have to accept partial status 1 incremental backups of Windows Server 2016 nodes that include the System_State:. Not nice, AFAICT there's nothing else any of us can do, except keep chasing our tails.
Some options / mitigations / tasks might be :
1) filter those backup from your daily reporting / stats
2) cease taking incremental System_State backups of Windows Server 2016, and instead - for all Windows Server 2016 backup clients - take daily full backups of C: + System_State:
3) simply cease taking incremental backups of System_State: of all Windows Server 2016 nodes
4) add an entry in your IT Service "risk register" that acknowledges the problem / risk
5) regularly test BMR recovery of one or more Windows Server 2016 backup clients
.
Anyone else care to add anything ?
08-09-2019 10:34 AM - edited 08-09-2019 10:48 AM
From your bpbkar logs.
So, 32 files are missing from the snapshot according to this
12:21:59.033 [2560.6892] <2> ov_log::V_GlobalLog: INF - Component Set System Files has 32 files missing on the snapshot
Those 32 files would be the following.
12:20:06.606 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:06.606 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:08.236 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:08.236 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:20.903 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
12:20:20.903 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
12:20:21.603 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:21.603 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:22.976 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:22.976 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:24.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:24.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:25.635 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:25.635 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:28.066 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:28.066 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:30.490 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:30.490 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:30.961 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:30.961 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:31.894 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:31.894 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:20:37.194 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:37.194 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:20:37.595 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:37.595 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:20:40.638 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:40.638 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:50.258 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:50.258 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:52.787 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:52.787 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.379 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.379 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.384 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:20:57.384 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.041 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.041 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.046 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:04.046 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:10.727 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:21:10.727 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
12:21:13.975 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:13.975 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:17.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:17.432 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:23.588 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:21:23.588 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:21:23.823 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:23.823 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
12:21:29.406 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:21:29.406 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
12:21:43.466 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:43.466 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
12:21:48.969 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:48.969 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:50.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:50.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:51.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:51.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
12:21:52.970 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:52.970 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
12:21:55.756 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
12:21:55.756 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
So based off of what I saw while browsing your bpbkar logs, take a look at this link and figure out why those files are missing. https://www.veritas.com/support/en_US/article.100034278
08-09-2019 10:50 AM
Hello,
Thanks for the answer. About this article, step by step was performed on both nodes and then performed an incremental backup, the first one worked successfully, but when performing the incremental backup again it failed and continues to fail.
08-09-2019 11:35 AM - edited 08-09-2019 11:38 AM
Did you figure out why those 32 files were missing?
So the next backup you ran was an incremental after making any changes that you did, is that correct?
Could you upload the logs for a new status 1 backup after you made your changes?
Please increase the logging level above minimum.
08-09-2019 11:57 AM
To be honest, I don't know why these files are missing, but would that impact backups?
The logs I attached were after running the mentioned link.
I am attaching logs run before this procedure. And yes, I performed two incremental backups after the procedure was executed, the first successfully and all the next failed.
08-09-2019 11:58 AM
08-09-2019 12:13 PM - edited 08-09-2019 12:16 PM
Yea those missing files are the problem, because they remain registered in the VSS Writer config. As @sdo mentions in his link, it looks like he has had the same/similar issue and says the same thing I am.
Doesn't look like he found a permanent fix yet and Veritas hasn't addressed it yet it seems. It seems as though if you needed to perform a restore though, you wouldn't have any issues because the VSS Writer config just has files that no longer exist as well as files that do exist (the ones needed for a successful restore). Essentially it seems as though it's a "false" flag and the backup is successful (not a status 1).
Correct me if I am wrong @sdo
08-09-2019 12:22 PM
08-09-2019 12:26 PM - edited 08-09-2019 12:27 PM
@danguss I glanced at those new logs, the logging level is 0 which won't tell us much.
08-09-2019 12:29 PM
08-09-2019 12:29 PM
Totally agree sdo, didn't mean it was a Veritas problem. Just something that Microsoft hasn't helped Veritas fix. As you said and I have read, this issue doesn't affect Microsoft's own server backups which means they know how to exclude these files.
08-10-2019 03:44 AM - edited 08-10-2019 03:48 AM
The problem is a moving target. It was noted by others that they could address the issue for a short while or a fair while, one to 'n' backup jobs, before the same problem began appearing again but for other so called critical files. If you look back at the link I posted earlier, I did provide a short script to help identify the backup jobs that report a particular error code. So I guess you could write a different script to filter out those jobs with that error code and thus boil down the list of status 1 partial backup jobs down to a list of true partial status 1 that need to be checked / addressed / fixed - but this script would be quite large and complex.
What gets me is... how are we supposed to know whether a missing critical file really is a critical or not... what this whole problem says to me is that I can never again really trust my BMR aware backups of Windows Server 2016... because I can never know whether any one given backup job is really missing something that really is critical. IMO this simply isn't good enough, and customers should not have to put with this - no matter which vendor is the root cause (and IMO, AFAICT, the root cause is Microsoft).
At then end of the day, until Microsoft fix it, then we are all just going to have to accept partial status 1 incremental backups of Windows Server 2016 nodes that include the System_State:. Not nice, AFAICT there's nothing else any of us can do, except keep chasing our tails.
Some options / mitigations / tasks might be :
1) filter those backup from your daily reporting / stats
2) cease taking incremental System_State backups of Windows Server 2016, and instead - for all Windows Server 2016 backup clients - take daily full backups of C: + System_State:
3) simply cease taking incremental backups of System_State: of all Windows Server 2016 nodes
4) add an entry in your IT Service "risk register" that acknowledges the problem / risk
5) regularly test BMR recovery of one or more Windows Server 2016 backup clients
.
Anyone else care to add anything ?
08-12-2019 12:02 PM
I am analyzing the logs and trying to find some flaws in the operating system, still unsolved.
08-12-2019 09:19 PM
Here's a thought... if you are not using BMR then do you need to take a backup of System_State: ?
08-13-2019 04:41 AM
@sdo In cases of physical servers yes, we need ...
Fixed some notes on vCenter disks and booted the server, tested two incremental backups successfully, but following the schedule the incremental backups failed.
Backups are very intermittent, attached two bpbkar logs,bpbkar failed and bpbkar successfully.
We both checked the 32 missing files.
08-13-2019 04:55 AM
You, nor I, nor Veritas, can overcome the problem. It will keep coming back, and keep hapenning to different files at different times. You either have to ignore it, or filter it from alerting/reporting. Microsoft expect you to treat it as a false negative, and Microsoft expect you to assume all is ok.
08-13-2019 05:20 AM
I agree, but do we need evidence for something to present to microsoft or not?
08-13-2019 05:27 AM
I guess you can try if you want to. Seems to me Veritas have already tried, and hit the same wall. Hence the Veritas tech note points (offloads) to the Microsoft tech note, and the Microsoft tech note says :
1) MS say: third party backup products may report errors
2) MS say: our own (i.e. Microsoft's) backup product does not report errors
Ask yourself why does MS own backup product not report errors ? I can only think because they either ignore those errors or they no longer use the feature of VSS critical files or they have a new/different/undocumented/unshared mechanism for protecting System_State that other third party backup vendors do not yet know about.
08-13-2019 05:39 AM
Has anyone else in the community got any corrections, advice, suggestions, opinions, interpretations?
I'm actually hoping that I've got something wrong, or mis-construed something(s).