03-25-2019 09:54 AM - edited 03-25-2019 09:56 AM
NetBackup v8.1.1
Is anyone else finding that more and more Windows Server 2016 backups are suffering these symptoms...
…activity monitor shows:
18-Mar-2019 09:17:11 - Warning bpbrm (pid=278633) from client client1.internal: WRN - can't open object: System State:\System Files\System Files (WIN32 3758134305: Unknown error) 18-Mar-2019 09:17:12 - Error bpbrm (pid=278633) from client client1.internal: ERR - Error encountered while attempting to get additional files for System State:\
…a google search for “3758134305”, links to Veritas :
Incremental Backing up System State on a Windows 2016 client can intermittently produce Exit Status 1.
https://www.veritas.com/support/en_US/article.100034278
…which in turn links to Microsoft :
Third-party backup warnings after you install a servicing update in Windows Server 2016
…find backup clients suffering it:
@echo off setlocal enabledelayedexpansion echo ...listing... bpdbjobs -all_columns -file a.txt echo ...searching... findstr "3758134305" a.txt > b.txt echo ...extracting... copy nul c.txt >nul for /f "tokens=1,2,3,4,5,6,7 delims=," %%a in ('type b.txt') do ( (echo %%g)>>c.txt ) echo ...sorting... sort c.txt /o d.txt echo ...reporting... set /a z_count=0 set z_prev= for /f "tokens=1" %%a in ('type d.txt') do ( if /i not "%%a"=="!z_prev!" ( echo %%a set /a z_count+=1 ) set z_prev=%%a ) echo ...found: !z_count! exit /b
…set level 5 logging on the client…
…create bpbkar log folder…
…re-run the backup…
...search the bpbkar log files:
find /i "status 0x00000002 for file" *.log > a.txt
…then use this to filter and sort the extract:
@echo off setlocal enabledelayedexpansion copy nul b.txt for /f "tokens=1,2,3,4,5,6,7,8,9,10,*" %%a in ('type a.txt') do ( if not "%%k"=="" ( (echo %%k)>>b.txt ) ) sort b.txt /o c.txt copy nul d.txt set z_prev= for /f "tokens=*" %%a in ('type c.txt') do ( set z_curr=%%a if /i not "!z_curr!"=="!z_prev!" ( (echo !z_curr!)>>d.txt ) set z_prev=!z_curr! ) exit /b
…which shows that the client has 40 files reported as missing :
c:\windows\servicing\version\10.0.14393.0\amd64_installed c:\windows\servicing\version\10.0.14393.0\x86_installed c:\windows\servicing\version\10.0.14393.1561\amd64_installed c:\windows\servicing\version\10.0.14393.1561\x86_installed c:\windows\servicing\version\10.0.14393.2272\amd64_installed c:\windows\servicing\version\10.0.14393.2272\x86_installed c:\windows\servicing\version\10.0.14393.350\amd64_installed c:\windows\servicing\version\10.0.14393.350\x86_installed c:\windows\system32\en-us\licenses\volume\serverweb\license.rtf c:\windows\system32\licenses\neutral\volume\serverweb\license.rtf c:\windows\system32\spp\tokens\skus\serverweb\defaultppd-serverweb-ppdlic.xrm-ms c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-pl-rtm.xrm-ms c:\windows\system32\spp\tokens\skus\serverweb\serverweb-oem-coa-1-ul-oob-rtm.xrm-ms ...circa 30 removed for brevity... c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-phn-rtm.xrm-ms c:\windows\system32\spp\tokens\skus\serverweb\serverweb-volume-mak-1-ul-store-rtm.xrm-ms c:\windows\syswow64\en-us\licenses\volume\serverweb\license.rtf c:\windows\syswow64\licenses\neutral\volume\serverweb\license.rtf
...but...
...this problem is essentially a moving target, and the problem will never go away. The implication is that any future update from Microsoft can, and does, and will leave any files in a missing state, i.e. still listed in VSS config as required critical system files, when in fact they are not.
But, who are we to know which really are critical and which are not? It is clear that Microsoft's own server 2016 backup tool effectively ignores missings, and therefore Microsoft expect all other backup product vendors to do the same. Or put another way, Microsoft are effectively expecting all Windows 2016 sites to trust that any VSS registered files that are missing are forever and always safe to completely ignore.
Nice.
03-25-2019 10:09 AM
I'm facing the same issue. Earlier I tried adding the version files manually and it worked. But the problem reappeared after a few days of that change. I have been planning to open an ticket with support for this as all Windows 2016 traditional backups are having the issue, but have not had the time yet.
The number of servers facing the issue is small though as most of them are virtual and backed up via VIP.
06-20-2019 01:05 AM
We are facing the same problem here, and as described by @sdo , we have applied the contig thing to create the missing files for 2 servers and it worked, but when we have a lot of servers with a lot of missing files, it is really not fun to create all those files on all the systems (especially after an Os upgrade..)
so , do you have guys any techntoe (Etrack number maybe) in which Veritas describe that it will be an EEB maybe to workaround this issue? I know it's a microsoft problem but we need to explain to our customers the situation.
tbh, I was expecting that the problem would be solved in the next version 8.1 or 8.1.2 (because we used to face it on 8.0), but id didn't so it is a little bit awkward..
Thanks