Backup Exec VSS Writer

Good day,

We are running Backup Exec on Windows Server 2008 R2 Standard.

The backups keep failing when running a job for the Hyper-V Host server (Windows Server 2016 Standard) with 3x VM's and generates the errors below.

The backup job will run once after the Host server has been restarted but after that it keeps failing.

When running the VSSADMIN LIST WRITER command, no errors are generated on the Hyper-V host or the clients.

Please may this get your full attention as these are critical servers that need to be backed up.

Error log found on backup Exec (Backup Server) Job Log: BEX_HEBS_08814.xml<noscript>

30 October 2018 11:20:01 AM - V-79-40960-38521 - Snapshot Technology: Initialization failure on: "VRTSRV::\\****.he.local\Hyper-V?Virtual?Machine\****". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS).
A virtual machine snapshot could not be created.  Use the Event Viewer on the Hyper-V host to identify the cause.  If the virtual machine is not properly configured, it may be necessary to use the Event Viewer in the virtual machine to identify the cause.
The following volumes are dependent on resource: "C:" .
The snapshot technology used by VSS for volume C: - Microsoft Software Shadow Copy provider 1.0 (Version 1.0.0.7).

Please see Acronis VSS Doctor report below (Errors were generated on Hyper-V Host server):

2018-10-30 11:23:23 GMT+02:00
AcronisVSSDoctor-net45, 1.0.39.0, v4.0.30319
Microsoft Windows NT 6.2.9200.0, 4.0.30319.42000

===========================================================
VSS Services Configuration and Status

Status: OK
Description: All VSS-related services are configured correctly

Services:

Name: swprv
Title: Microsoft Software Shadow Copy Provider
Status: Running
ReferenceStatus:
NeedToFixStatus: False
StartMode: Manual
ReferenceStartMode: Manual
NeedToFixStartMode: False
IsOk: True
ExpectationDescription:
Exception:

Name: VSS
Title: Volume Shadow Copy
Status: Running
ReferenceStatus:
NeedToFixStatus: False
StartMode: Manual
ReferenceStartMode: Manual
NeedToFixStartMode: False
IsOk: True
ExpectationDescription:
Exception:

Name: EventSystem
Title: COM+ Event System
Status: Running
ReferenceStatus: Running
NeedToFixStatus: False
StartMode: Auto
ReferenceStartMode: Auto
NeedToFixStartMode: False
IsOk: True
ExpectationDescription:
Exception:

===========================================================
VSS Providers Configuration

Status: OK
Description: Only known VSS provider(s) found

Providers:

Id: 89300202-3cec-4981-9171-19f59559e0f2
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{89300202-3cec-4981-9171-19f59559e0f2}
Name: Microsoft File Share Shadow Copy provider
IsDefault: True

Id: b5946137-7b9f-4925-af80-51abd60b20d5
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{b5946137-7b9f-4925-af80-51abd60b20d5}
Name: Microsoft Software Shadow Copy provider 1.0
IsDefault: True

===========================================================
VSS Access Control Configuration

Status: OK
Description: Known VSS access control entries are configured correctly

AccessControls:

UserName: NT Authority\NetworkService
EntryExists: True
UserExists: True
IsActive: True
IsOk: True
StatusDescription:

===========================================================
VSS Components Availability

Status: OK
Description: VSS components are accessible

===========================================================
VSS Writers Status

Status: OK
Description: All VSS writers are working correctly

Writers:

Name: Task Scheduler Writer
Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
State: [1] Stable
IsOk: True
Error: No error

Name: VSS Metadata Store Writer
Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
State: [1] Stable
IsOk: True
Error: No error

Name: Performance Counters Writer
Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
State: [1] Stable
IsOk: True
Error: No error

Name: System Writer
Id: {e8132975-6f93-4464-a53e-1050253ae220}
State: [1] Stable
IsOk: True
Error: No error

Name: ASR Writer
Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
State: [1] Stable
IsOk: True
Error: No error

Name: Shadow Copy Optimization Writer
Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
State: [5] Waiting for completion
IsOk: True
Error: No error

Name: FRS Writer
Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
State: [1] Stable
IsOk: True
Error: No error

Name: WMI Writer
Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
State: [1] Stable
IsOk: True
Error: No error

Name: MSSearch Service Writer
Id: {cd3f2362-8bef-46c7-9181-d62844cdc0b2}
State: [5] Waiting for completion
IsOk: True
Error: No error

Name: Registry Writer
Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
State: [1] Stable
IsOk: True
Error: No error

Name: NTDS
Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
State: [1] Stable
IsOk: True
Error: No error

===========================================================
VSS Shadow Storages Status

Status: OK
Description: All VSS shadow storages are configured correctly

Storages:

ForVolume: C:\
OnVolume: C:\
Used: 96 MB
Allocated: 50 GB
Maximum: UNBOUNDED
Minimum: 320 MB
VolumeSize: 3721 GB
Available: 3218 GB
TargetMaximum: 372 GB
MaximumGreaterThenMinimum: True
MaximumIsEnough: True
MaximumIsNotReached: True
AvailableIsEnoughForMaximum: True
IsOk: True
Description:

===========================================================
Disk Free Space Available

Status: OK
Description: All disks have enough free space

Volumes:

Name: C:\
DeviceId: \\?\Volume{1c4947d2-e53e-e711-a993-6679abe43d90}\
Size: 3721 GB
Available: 3218 GB
Minimum: 320 MB
IsOk: True
Description:
IsMounted: True

Name: E:\
DeviceId: \\?\Volume{0005f107-0000-0000-0000-100000000000}\
Size: 1862 GB
Available: 1520 GB
Minimum: 320 MB
IsOk: True
Description:
IsMounted: True

===========================================================
Disk I/O Load

Status: OK
Description: All disks are currently not overloaded

Disks:

Name: 0 C:
DiskBytesPersec: 437574
DiskReadBytesPersec: 173144
DiskReadsPersec: 9
DiskTransfersPersec: 36
DiskWriteBytesPersec: 264429
DiskWritesPersec: 27
PercentDiskReadTime: 6
PercentDiskTime: 28
PercentDiskWriteTime: 23
PercentIdleTime: 74
IsOk: True

Name: 1 E:
DiskBytesPersec: 0
DiskReadBytesPersec: 0
DiskReadsPersec: 0
DiskTransfersPersec: 0
DiskWriteBytesPersec: 0
DiskWritesPersec: 0
PercentDiskReadTime: 0
PercentDiskTime: 0
PercentDiskWriteTime: 0
PercentIdleTime: 97
IsOk: True

===========================================================
NTFS Volume Access Control Configuration

Status: OK
Description: All volumes have correct access control configuration

Volumes:

Name: C:\
SystemHasAccess: True
SystemSid: S-1-5-18
IsOk: True
Description: "SYSTEM" account has "Full Control" access

Name: E:\
SystemHasAccess: True
SystemSid: S-1-5-18
IsOk: True
Description: "SYSTEM" account has "Full Control" access

===========================================================
Event Log

Status: Warn
Description: Some VSS-related errors or warnings occurred during the previous day

Events:

Timestamp: 2018/10/30 10:31:50 AM
Type: Warning
Source: VSS
Message: A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur. . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {51b65980-2447-4878-9507-c4a25021a881} Command Line: C:\Windows\system32\vmms.exe Process ID: 4032
InstanceId: 8229
SearchLink: https://www.google.com/#newwindow=1&q=VSS+event+id+8229

Timestamp: 2018/10/30 10:18:48 AM
Type: Warning
Source: VSS
Message: A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur. . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {51b65980-2447-4878-9507-c4a25021a881} Command Line: C:\Windows\system32\vmms.exe Process ID: 4032
InstanceId: 8229
SearchLink: https://www.google.com/#newwindow=1&q=VSS+event+id+8229

Timestamp: 2018/10/29 11:04:06 PM
Type: Warning
Source: VSS
Message: A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur. . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {51b65980-2447-4878-9507-c4a25021a881} Command Line: C:\Windows\system32\vmms.exe Process ID: 4032
InstanceId: 8229
SearchLink: https://www.google.com/#newwindow=1&q=VSS+event+id+8229