Forum Discussion

danguss's avatar
danguss
Level 4
5 years ago

Incremental backup fail with status code 1

Hello, I have a big problem with my system state incremental backups on two clustered servers ending with status 1, which only the full backup ending with status 0.

Master Server Version 8.1.2 - Windows server 2016

Media Server Verion  8.1.2 - Linux 3.10.0-693.17.1.el7.x86_64 

Client Version 8.1.2 - Windows server 2016

I performed the procedures for vss verification and found no failure.

can you help me?

Attached some logs

Thank you very much in advance

  • sdo's avatar
    sdo
    5 years ago

    The problem is a moving target.  It was noted by others that they could address the issue for a short while or a fair while, one to 'n' backup jobs, before the same problem began appearing again but for other so called critical files.  If you look back at the link I posted earlier, I did provide a short script to help identify the backup jobs that report a particular error code.  So I guess you could write a different script to filter out those jobs with that error code and thus boil down the list of status 1 partial backup jobs down to a list of true partial status 1 that need to be checked / addressed / fixed - but this script would be quite large and complex.

    What gets me is... how are we supposed to know whether a missing critical file really is a critical or not... what this whole problem says to me is that I can never again really trust my BMR aware backups of Windows Server 2016... because I can never know whether any one given backup job is really missing something that really is critical.  IMO this simply isn't good enough, and customers should not have to put with this - no matter which vendor is the root cause (and IMO, AFAICT, the root cause is Microsoft).

    At then end of the day, until Microsoft fix it, then we are all just going to have to accept partial status 1 incremental backups of Windows Server 2016 nodes that include the System_State:.  Not nice, AFAICT there's nothing else any of us can do, except keep chasing our tails.

    Some options / mitigations / tasks might be :

    1) filter those backup from your daily reporting / stats

    2) cease taking incremental System_State backups of Windows Server 2016, and instead - for all Windows Server 2016 backup clients - take daily full backups of C: + System_State:

    3) simply cease taking incremental backups of System_State: of all Windows Server 2016 nodes

    4) add an entry in your IT Service "risk register" that acknowledges the problem / risk

    5) regularly test BMR recovery of one or more Windows Server 2016 backup clients

    .

    Anyone else care to add anything ?

18 Replies

  • From your bpbkar logs.

    So, 32 files are missing from the snapshot according to this
    12:21:59.033 [2560.6892] <2> ov_log::V_GlobalLog: INF - Component Set System Files has 32 files missing on the snapshot

    Those 32 files would be the following.

    12:20:06.606 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:06.606 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:08.236 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:08.236 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:20.903 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
    12:20:20.903 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
    12:20:21.603 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:21.603 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:22.976 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:22.976 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:24.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:24.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:25.635 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:25.635 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:28.066 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:28.066 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:30.490 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:30.490 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:30.961 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
    12:20:30.961 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
    12:20:31.894 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:31.894 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList

    12:20:37.194 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:37.194 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:37.595 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
    12:20:37.595 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf in brUtil::GenerateFileList
    12:20:40.638 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:40.638 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:50.258 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:50.258 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:52.787 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:52.787 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:57.379 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:57.379 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-gvlk-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:57.384 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:20:57.384 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:04.041 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:04.041 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-dm-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:04.046 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:04.046 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:10.727 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:10.727 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-slp-1-ul-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:13.975 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:13.975 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:17.431 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:17.432 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:23.588 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
    12:21:23.588 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
    12:21:23.823 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:23.823 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-store-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:29.406 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList
    12:21:29.406 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf in brUtil::GenerateFileList

    12:21:43.466 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:43.466 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-retail-tb-eval-1-pl-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:48.969 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:48.969 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:50.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:50.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:51.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:51.544 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-oob-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:52.970 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:52.970 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-nonslp-1-ul-phn-rtm.xrm-ms in brUtil::GenerateFileList
    12:21:55.756 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy78\windows\system32\spp\tokens\skus\serverweb\serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList
    12:21:55.756 [2560.6892] <2> ov_log::V_GlobalLog: INF - Status 0x00000002 for file c:\windows\system32\spp\tokens\skus\serverweb\serverweb-ppdlic.xrm-ms in brUtil::GenerateFileList

     

    So based off of what I saw while browsing your bpbkar logs, take a look at this link and figure out why those files are missing. https://www.veritas.com/support/en_US/article.100034278

     

    • danguss's avatar
      danguss
      Level 4

      Hello,

      Thanks for the answer. About this article, step by step was performed on both nodes and then performed an incremental backup, the first one worked successfully, but when performing the incremental backup again it failed and continues to fail.

      • Krutons's avatar
        Krutons
        Moderator

        Did you figure out why those 32 files were missing?

        So the next backup you ran was an incremental after making any changes that you did, is that correct?

        Could you upload the logs for a new status 1 backup after you made your changes?
        Please increase the logging level above minimum.