System State & SCC gone from selection list and causing backup failures

[Stuart_MTD]

Apologies if this has been asked and answered, I searched but could not find my specific situation.  I'm running BE12.5 SP4+ on Windows 2008 Std x64.  This is a brand new build after replacing the mainboard due to power-related ASRs.  BE had been working perfectly for the first few backups, but suddenly started failing on the system state and shadow copy components of the local server running the backups.  The selection list for the jobs no longer lists these two components on the selection tab, but they're still present on the details tab.  I've re-registered VSS, and doing a vssadmin list writers returns no errors.  When the backups start each night, VSS, MS Software Shadow Copy services both start but then both end after a few minutes.  No system errors are logged.  Obviously something with VSS has gotten hosed.  The exact BE errors I'm getting are:

V-79-57344-65055 - AOFO: Initialization failure on: "Shadow?Copy?Components". Advanced Open File Option used: No. Snapshot provider error (0xE000FE1F): The device cannot be found. 

V-79-57344-65055 - AOFO: Initialization failure on: "System?State". Advanced Open File Option used: No. Snapshot provider error (0xE000FE1F): The device cannot be found. 

What is causing those components not to be listed as selections under the local server?

[CraigV]

Hi Stuart,

Sounds like a bit of corruption with BE...I'd start by doing a repair of the BEDB through BEutility. Let it run, and then check again.

If that doesn't work, then try a repair of the application itself...

Matter of interest...if you try creating a new selection list, does it show the System State then?

THanks!

[Stuart_MTD]

I've run the BEDB Repair you suggested, and it completed successfully with no mention of any problems.

System State & Shadow Copy Components are not listed when creating new selection lists.

I re-ran the VSS re-registration commands I found on a Symantec support link, and noticed this time that one of the verification commands at the end listed a file corruption.  It may have done that the first time and I just didn't notice it.  The files listed in the related log are CATSRVUT.DLL and COMSVCS.DLL, and says they couldn't be repaired.

[Stuart_MTD]

I ran the app repair, rebooted the server, and now both components are listed and selectable on new and existing selection lists.  I'll know after tonight's backup if all's truly well.  Thanks for the advice, Craig!

[Stuart_MTD]

Now it's a failure access vss writer metadata.

V-79-57344-65225 - AOFO: Initialization failure on: "System?State". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

Snapshot provider error (0xE000FEC9): A failure occurred accessing the Writer metadata.

all the vss writers show stable with no errors, and the system log shows them starting up when the backup job launchs and then stopping a few minutes later, but no vss errors.

It's not the 100MB hidden partition w/o driver letter issue, there isn't one.

Still searching...

[AmolB]

Is it 2008 or 2008 R2?

[Stuart_MTD]

2008

The box had been running R2 but I had to rebuild it when the OS got hosed due to power-related ASR issues with the previous mainboard (ironically an issue with VSS preventing backups of the server itself).  I built it with just 2008 this time because we run CA IT Client Management, and they don't fully support R2 in the current release.

[CraigV]

...is BE patched?

What happens if you take a rename the Data/Catalogs folders, create new ones, and a new BEDB? DOes this fix things?

If it does, you might want to do a comprehensive repair of the BEDB via the TN below:

http://www.symantec.com/business/support/index?page=content&id=TECH38078

THanks!

[Stuart_MTD]

Yes, BE is current on updates according to LiveUpdate.

I'll try the folder rename you suggest and reply back with results. 

Thanks!

[Stuart_MTD]

I read that tech document you  pointed to, and I may need to do the comprehensive repair on the DB after all.  When I first installed BE, and LiveUpdate first ran, it failed installing the updates.  If I remember correctly, it then installed SP4 on it's own as I saw this in the logs and the prompt to reboot, which I did.  A manual run of LiveUpdate once more failed to install post-SP4 updates but they then later installed on thier own (more log notes to this effect).

[CraigV]

Mmm...have you uninstalled the agent completely Stuart, and then done a push-install from the media server?

[Stuart_MTD]

This is the media server we're talking about.  All the remote servers backup with no issues, system state and shadow copy components included.

[CraigV]

Hi Stuart,

Not sure if you ever came right here or not, but Symantec have a repair of the VSS *.dlls that you can try doing...

http://www.symantec.com/business/support/index?page=content&id=TECH70486