System State files missing from backup
NetBackup v8.1.1
Is anyone else finding that more and more Windows Server 2016 backups are suffering these symptoms...
…activity monitor shows:
18-Mar-2019 09:17:11 - Warning bpbrm (pid=278633) from client client1.internal: WRN - can't open object: System State:\System Files\System Files (WIN32 3758134305: Unknown error) 18-Mar-2019 09:17:12 - Error bpbrm (pid=278633) from client client1.internal: ERR - Error encountered while attempting to get additional files for System State:\
…a google search for “3758134305”, links to Veritas :
Incremental Backing up System State on a Windows 2016 client can intermittently produce Exit Status 1.
https://www.veritas.com/support/en_US/article.100034278
…which in turn links to Microsoft :
Third-party backup warnings after you install a servicing update in Windows Server 2016
…find backup clients suffering it:
@echo off
setlocal enabledelayedexpansion
echo ...listing...
bpdbjobs -all_columns -file a.txt
echo ...searching...
findstr "3758134305" a.txt > b.txt
echo ...extracting...
copy nul c.txt >nul
for /f "tokens=1,2,3,4,5,6,7 delims=," %%a in ('type b.txt') do (
(echo %%g)>>c.txt
)
echo ...sorting...
sort c.txt /o d.txt
echo ...reporting...
set /a z_count=0
set z_prev=
for /f "tokens=1" %%a in ('type d.txt') do (
if /i not "%%a"=="!z_prev!" (
echo %%a
set /a z_count+=1
)
set z_prev=%%a
)
echo ...found: !z_count!
exit /b…set level 5 logging on the client…
…create bpbkar log folder…
…re-run the backup…
...search the bpbkar log files:
find /i "status 0x00000002 for file" *.log > a.txt
…then use this to filter and sort the extract:
@echo off
setlocal enabledelayedexpansion
copy nul b.txt
for /f "tokens=1,2,3,4,5,6,7,8,9,10,*" %%a in ('type a.txt') do (
if not "%%k"=="" (
(echo %%k)>>b.txt
)
)
sort b.txt /o c.txt
copy nul d.txt
set z_prev=
for /f "tokens=*" %%a in ('type c.txt') do (
set z_curr=%%a
if /i not "!z_curr!"=="!z_prev!" (
(echo !z_curr!)>>d.txt
)
set z_prev=!z_curr!
)
exit /b…which shows that the client has 40 files reported as missing :
c:\windows\servicing\version\10.0.14393.0\amd64_installed c:\windows\servicing\version\10.0.14393.0\x86_installed c:\windows\servicing\version\10.0.14393.1561\amd64_installed c:\windows\servicing\version\10.0.14393.1561\x86_installed c:\windows\servicing\version\10.0.14393.2272\amd64_installed c:\windows\servicing\version\10.0.14393.2272\x86_installed c:\windows\servicing\version\10.0.14393.350\amd64_installed c:\windows\servicing\version\10.0.14393.350\x86_installed c:\windows\system32\en-us\licenses\volume\serverweb\license.rtf c:\windows\system32\licenses\neutral\volume\serverweb\license.rtf c:\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms ...circa 30 removed for brevity... c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms c:\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf c:\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf
...but...
...this problem is essentially a moving target, and the problem will never go away. The implication is that any future update from Microsoft can, and does, and will leave any files in a missing state, i.e. still listed in VSS config as required critical system files, when in fact they are not.
But, who are we to know which really are critical and which are not? It is clear that Microsoft's own server 2016 backup tool effectively ignores missings, and therefore Microsoft expect all other backup product vendors to do the same. Or put another way, Microsoft are effectively expecting all Windows 2016 sites to trust that any VSS registered files that are missing are forever and always safe to completely ignore.
Nice.
