Vol. Shadow Copy Service Terminates Unexpectedly when using Backup Exec and NTBackup

[mvietorisz]

Hello

We have a virtual server, running Windows Server 2003 R2, that refuses to get its system state and shadow copy components backed up.

This is due to the Volume Shadow Copy service terminating unexpectedly during the backup job. Here are the specific errors:

Faulting application vssvc.exe, version 5.2.3790.4143, faulting module vssvc.exe, version 5.2.3790.4143, fault address 0x0004b452.

AND

The Volume Shadow Copy service terminated unexpectedly. It has done this 3 time(s).


We recently upgraded our Backup Exec to version 2010. Prior to this we were running 12.5 and had no problems backing up this particular virtual server.


As part of the upgrade, we reinstalled the Remote Agent for Windows Server, with the Advanced Open File Option. This is the only thing that has changed on this server since upgrading.


After talking with Symantec Support, they had me try to backup the system state via NTBackup. This also failed, with NTBackup basically just freezing up and not doing anything, while logging a bunch of the same errors as above. Symantec refuses to do anything else as they say it's a VSS/Windows/Microsoft problem and won't help any further.

It seems the VSS service is toast, or something else is conflicting with it. I have restarted the server several times, tried starting/stopping the VSS service. No matter what I do, it always crashes when trying to do a system state backup.

BExec is up to date with the latest service pack, and the remote agent has been reinstalled and updated to the most current version on the virtual server. Note that both C: and D: drives on the virtual server get backed up just fine.

Anyone have some ideas?

Thanks.

[AmolB]

Hello

  Have you installed the VSS patch from Microsoft, refer to http://support.microsoft.com/kb/940349 

[mvietorisz]

Hi Amol, yes, we've installed the VSS patch KB940349.

[sksujeet]

Please uninstall the raws and install the raws again without AOFO. Make sure from 12.5 onwards you need not to install aofo as it automatically uses vss for ws 2003 servers.
Reboot the server once done and try the backup again.

[mvietorisz]

When working with Sym support, we reinstalled the agent via a local install: copied RAWS32 folder and ran setupaa.cmd from the desktop of the server. How can I tell if it reinstalled the AOFO as well?

Nonetheless, I just uninstalled and reinstalled as you suggested, using the actual Setup.exe and not the script. It didn't give me the option to install AOFO so I guess it's not installed then.

I again tried to run an NTBackup of the System State... and it just cycles through "Preparing to backup using shadow copy..." and  "Waiting to retry shadow copy...".. each time writing a Faulting application vssvc.exe entry into the Application event viewer.

Looking at the NTBackup log:

Volume shadow copy creation: Attempt 1.
"Registry Writer" has reported an error 0x800423f2. This is part of System State. The volume shadow copy operation can be retried.
"Removable Storage Manager" has reported an error 0x800423f2. This is part of System State. The volume shadow copy operation can be retried.
"MSDEWriter" has reported an error 0x800423f2. This is part of System State. The volume shadow copy operation can be retried.
"COM+ REGDB Writer" has reported an error 0x800423f2. This is part of System State. The volume shadow copy operation can be retried.
"Event Log Writer" has reported an error 0x800423f2. This is part of System State. The volume shadow copy operation can be retried.

[sksujeet]

I am suspecting if it has to do something with your disk drive. Can you run the chkdisk on all the drives and confirm all is good and try the backup again. Also are you running any antivirus, if so please disable the antivirus and try the backup again.

[mvietorisz]

Sazz,

I ran a chkdsk on all volumes as suggested. It found errors on the C: volume, and so I ran a /fix command upon next reboot. As far as I know it fixed the errors, as after it came back up I ran a chkdsk again on C: and it found no errors.

Unfortunately this did not solve the problem, I am still unable to conduct any sort of system state backup with BExec or NTBackup.

No, this particular server is not running any antivirus.

[Fritz]

at the moment only SQL and Oracle DB-Server affected. Support from symantec didn´t solve the problem. We have w2k3 SP2 servers in use.

Event Viewer entry

Faulting application vssvc.exe, version 5.2.3790.3959, faulting module vssvc.exe, version 5.2.3790.3959, fault address 0x0004b44e

BE Log

Job ended: Donnerstag, 14. Oktober 2010 at 19:10:59 Completed status: Failed Final error: 0x8004230f - 0x8004230f (2147754767) Final error category: Other Errors For additional information regarding this error refer to link V-79-32772-8975

Backup
- AOFO: Initialization failure on: "\\SV0076\Shadow?Copy?Components". Advanced Open File Option use...
V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provi...
 - AOFO: Initialization failure on: "\\SV0076\System?State". Advanced Open File Option used: Microso...
V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provi...

Have anyone a solution?

[mvietorisz]

Hi Fritz

We did find a half-fix... after viewing this support article:

http://seer.entsupport.symantec.com/docs/337019.htm

Basically all we did was force our backup jobs to use the Microsoft Volume Shadow Copy Service (Windows 2003 and later) option underneath the Advanced Open File settings in the job properties. Then under Snapshot provider, we selected System - Use Microsoft Software Shadow Copy Provider. This is opposed to allowing BExec to automatically select the provider, which for some reason is causing problems on this particular server.

Now our jobs finish with "Completed with exceptions" message instead of outright failing. The error that we get now is:

An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.

I have a feeling something is a muck with the VSS providers on this particular server, but haven't had a chance yet to investigate further.

[DieGoettlichen]

Same here, we are also experiencing this problem on two servers now. It started to happen since either latest Windows OR latest Backup Exec updates (my personal feeling is since Backup Exec updates, because it started the day before I left for holiday, same time when I last patched the BE installation).

Now I switched the Backup Job to the "half-fix" mentioned by mvietorisz. However as he mentioned it just avoids the job being marked as failed, but does not solve the problem itself. We still get the vssvc.exe error messages in the event log on those two servers by the time the server is being backed up by BE. I also installed the above mentioned patch by Microsoft, but this didn't solve the problem either.

I made a screenshot of those errors on the servers, attached to this post.

[CVMC]

1. Install the latest service pack
Microsoft has released a number of hotfixes targeting Volume Shadow Copy.��Nearly of these hotfixes have been included in the latest service packs.

2. Reregister the Volume Shadow Copy dlls
From the command prompt type the following:

CD�C:\Windows\System32 (depending on where your copy of Windows is installed)
Net stop vss
Net stop swprv
regsvr32 ole32.dll
regsvr32 vss_ps.dll
Vssvc /Register
regsvr32 /i swprv.dll
regsvr32 /i eventcls.dll
regsvr32 es.dll
regsvr32 stdprov.dll
regsvr32 vssui.dll
regsvr32 msxml.dll
regsvr32 msxml3.dll
regsvr32 msxml4.dll

Reboot your server.
After reboot, open a command prompt and run:
vssadmin list writers
Ensure that all your writers are displayed without errors