What is the OS of the server which is being backed up ?
Check the status of the VSS writers on the affected server by running the following via a cmd -
vssadmin list writers
If any of the writers are not in a stable state, restart the server to bring back the writers in a stable state & rerun the backup.
Also, set the Advanced Open File option to use the system vss provider.