Backup Exec 2012 - Failure on System State Backup

[RescueMe]

Hoping someone can provide some insight into this issue.  When I try to backup the System State on a client's SBS 2011 server, the backup fails with the following error message:

BackupV-79-57344-850 - Snapshot Technology: Initialization failure on: "System?State". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS).

Snapshot technology error (0xE0000352): The specified volume does not exist. Recreate the volume, and then try the job again.

Check the Windows Event Viewer for details.

I have looked at the Windows Event Viewer and do not see anything of significance.  I came across the following maybe related Microsoft Support article:

http://support.microsoft.com/kb/968247

But, I do not see any services with an invalid path specified.  I'm stumped.  Any ideas?

Thanks.

[Carlos_Quiroga]

Run vssadmin list writers to see if the writers are in a failed or error state and if so, reboot.

[Kiran_Bandi]

Also have a looka t this TN: http://www.symantec.com/docs/TECH125998

[RescueMe]

Carlos,

When I run "vssadmin list writers", I see that the ASR Writer returns a Last error of "Timed Out:"

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {1345b577-7a7e-4d03-8518-f60b29ee40f6}
   State: [7] Failed
   Last error: Timed out

I take it this is the source of the error.  Using this information, I came across the following article on Symantec's website that explained how to re-register the volume shadow copy service components:

http://www.symantec.com/business/support/index?page=content&id=TECH70486

After following the instructions at the link above, many of the writers that were previously output from "vssadmin list writers" were missing, but the 'ASR Writer" now had a state of "Stable."  I tried to backup the system state, and success, it worked!

However, the jubilation was short lived--after rebooting the server, the 'ASR Writer' state returned to "Failed."  Once again, I can no longer backup the system state.  Needless to say, this is an extremely frustrating situation.

What could be the cause of the ASR Writer's failed state?

Thanks.

[RescueMe]

Kiran,

I am unable to view the link specified.  It just takes me to a generic Symantec page.

[marcusolini]

Hi.  When able, please collect the following initial diagnostic information and question answers.  Please attach this information to the post for review.  Thank you.

(1)  C:\>VSSADMIN LIST VOLUMES

(2)  C:\>VSSADMIN LIST WRITERS

(3)  C:\>VSSADMIN LIST PROVIDERS

(4)  C:\>VSSADMIN LIST SHADOWS

(5)  C:\>VSSADMIN LIST SHADOWSTORAGE

(6)  C:\>BCDEDIT /enum active

(7)  C:\>MOUNTVOL

(8)  C:\>VER

(9)  C:\>REG QUERY "HKLM\SYSTEM\CurrentControlSet\control\hivelist" /s

(10)  Backup Exec backup job log for the failing System State backup.

(11)  Question: Physical Machine or Virtual Machine?

(12)  Question: Media Server backing up a Remote System?  Or Media Server backing up itself?

[Kiran_Bandi]

But it's working for me... Link

[RescueMe]

The requested output is attached.  I recently cleared the job log, but I will update this post with the exact error as soon as our client puts their tape in the tape drive this morning.

As for questions 11 and 12, this is a physical machine running Small Business Server 2011.  The backup job is backing up local files only.

As you can see from the attached output, the "ASR Writer" has a state of failed:

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {1345b577-7a7e-4d03-8518-f60b29ee40f6}
   State: [7] Failed
   Last error: Timed out

[RescueMe]

Here are the errors from the job log:

Completed status: Failed Final error: 0xe0000352 - The specified volume does not exist. Recreate the volume, and then try the job again. Final error category: Resource Errors For additional information regarding this error refer to link V-79-57344-850

V-79-57344-850 - Snapshot Technology: Initialization failure on: "System?State". Snapshot technology...Snapshot technology error (0xE0000352): The specified volume does not exist. Recreate the volume, an...Check the Windows Event Viewer for details.

[marcusolini]

Hi.  Thank you for all of the information thus far.  Unfortunately, it is still unclear at the moment as to what is causing the backup failure.  The current theory is that the VSS ASR Writer is reporting unexpected backup information.  The most effective way to proceed with the investigation is to collect Backup Exec Engine debug information for the failing backup job and to collect the VSS Writer metadata files for further review.  When able, please review and perform the procedures listed below.  Thank you.

Collect the Backup Exec Engine debug information using the Backup Exec Debug Monitor…

(13) Enable the Backup Exec Debug Monitor, by either...

(a) Executing the SGMON.EXE process from within the Backup Exec installation directory.  Example: C:\Program Files\Symantec\Backup Exec\Sgmon.exe.   -OR-

(b) Clicking the Backup Exec icon near the top left corner of application window (hotkey: Alt-a) and select ‘Technical Support’ and then click ‘Collect debug output’.

(14) Select ‘Job Engine, RAWS, Agent Browser’ from the ‘Capture’ section (Alt-j).

(15) Select ‘Capture to file’ from the ‘Capture to file:’ section (Alt-u).

(16) Run the backup job for the affected server (WIN2K11) that is failing with error:  0xE0000352 - The specified volume does not exist. Recreate the volume, and then try the job again. Check the Windows Event Viewer for details.

(17) Save the debug log output(s)  (Alt-n).

(18) Attach the zipped debug log file to this Post.

Collect the VSS Writer metadata files…

(19) The VSS Writer metadata files (*.xml) located on the affected server (WIN2K11) in Backup Exec logs directory.  For example:  ‘c:\program files\symantec\backup exec\logs\{BE000CBE-11FE-4426-9C58-531AA6355FC4} - ASR Writer.xml’.

(20) Attach the zipped files to this Post.

[_Bugs_]

I don`t think that the problem is vss related. We have the same problem with one of our servers and it is the ASR writer too that has errors. Just after a reboot are there no errors and the backup still crash with the mentioned error message.

I was able to make one backup from the machine after deleting a dynamic volume - a day after i got the same problem again. All i had changed was to assign a new drive letter to two volumes (basic ones).

On the other hand we have another server with ASR writer error which works just fine, every job quits green.

@ Kiran Bandi: The Link doesn`t work for me either, here is the text:

Problem

System State backup fails with "0xe0000352 - The specified volume does not exist. Recreate the volume, and then try the job again."

Error

Final error: 0xe0000352 - The specified volume does not exist. Recreate the volume, and then try the job again.

& following exception is seen "WARNING: "System?State\System Files\System Files" is a corrupt file.  This file cannot verify"

Cause

This error may come up if the server has restricted any access to files under System State
 

To troubleshoot the issue, try to run the backup using Windows built in NTBackup utility. 

Solution

• Run Ntbackup to backup system state for Windows 2003 . 
• For Windows server 2008, run Wbadmin from the command prompt.
  
   

Note:- If the Ntbackup / Wbadmin Fails, Contact Microsoft for further troubleshooting.

There is no solution it just says "It is not our problem, call microsoft"

Thank you but i can do without that.

[marcusolini]

Hi,  If able, let’s investigate your issue as well since it may prove to be the same or related.  When able, please provide all of the requested diagnostic information (1 to 20) for detailed review.  Keep in mind that System State backups and restore exclusively use VSS and that is why the current theory is that there is some unexpected VSS System State Writer discovery information causing the issue.  Also, as you have mentioned, since volume drive letter mount points were modified this would most likely be the cause of your issue.  Meaning, that a VSS System State Writer is still reporting the historical volume drive letter as a data path location, hence, the appropriate 0xe0000352 error.  Thank you.

[_Bugs_]

Output and Log attached

11:> Physical System

12:> Remote Server

[marcusolini]

Since the affected system is a remote server (INTEGRATOR), please ensure that all diagnostics are collected from the remote server (INTEGRATOR).  For the debug procedure, (13)(a) will need to be performed on the remote server ((INTEGRATOR).  Thank you.

[_Bugs_]

Here it is

[marcusolini]

Thank you for providing all of the diagnostic information so quickly!

For your issue, SYSVOL appears to be mis-configured.  SYSVOL is reporting two volume path locations:  ‘e:\windows\sysvol\*’ and ‘f:\windows\sysvol\*’.  SYSVOL typically only reports one volume path location and in this scenario the ‘f:’ volume is currently not valid on the server.  As suspected, this issue seems to originate from the volume drive letter assignments that were performed.

Use the MMC DFS Management utility (dfsmgmt.msc) to review the SYSVOL configuration.  If the SYSVOL Replication Group is not present, then manually add the existing group using the ‘Add Replication Groups to Display..’ from the ‘Action’ menu.  Also, review the reported volume path locations using Windows Explorer (explorer.exe).

Hope this proves helpful.  Please let us know the outcome of your review.  Thank you.

[RescueMe]

The requested files are attached.

Thanks!

[marcusolini]

Thanks for the diagnostic information.  The debug log (WIN2K11-SGMon.zip) is exactly what I need but appears to end abruptly: ‘BEREMOTE: [07/03/12 14:34:18] [26424]     2012-07-03T14:34:00.293 [fsys\shadow]…’.   Please review the original and provided debug log.  I will continue to review the information I have but ultimately a complete debug log is essential.  Please see what you can do to provide a full debug log.  Thank you.

[_Bugs_]

The Replication Group is present and works fine. There was in fact a sysvol folder under volume f:\ but that changed before we installed the BE Agent on the server. I deleted all references to the old folder in the registry, without success

[marcusolini]

I understand your predicament.  The System State FRS VSS Writer Component is still reporting the volume path of F: as a replication folder.  You will have to convince the Component otherwise by using FRS Management utilities.  By the way, I've misspoke about using DFS Management earlier because it appears that your SYSVOL replication is using the legacy FRS and not the later DFSR, which has replaced FRS.

From the perspective of BE, the FRS VSS Writer Component metadata should not report the inaccessible F: volume path.  The metadata is located in the Backup Exec logs directory; example:  ‘c:\program files\symantec\backup exec\logs\{D76F5A28-3092-4589-BA48-2958FB88CE29} - FRS Writer.xml’.  You can use this as a diagnostic to know when you have successfully removed the inaccessible F: volume path without having to perform a backup each time as confirmation.  This metadata is updated by BE whenever a new explicit browse (or backup) of System State is performed (not a browse refresh, must be a complete re-browse).

Here are solutions that can be considered and will require in-depth research...

(A) Use the FRS Management to correct the inaccessible F: volume path.

(B) Migrate FRS SYSVOL to use DFSR SYSVOL.

(C) As a temporary backup workaround, create an ‘F:’ volume and path ‘f:\windows\sysvol’ and a data file ‘temp-workaround-sysvol-data.txt’.  This is not a validated backup workaround and should be used as a temporary last resort.  The restore impacts are unknown.  Please use with extreme caution!

When able, please let us know the outcome of your research and solution.  In the meantime, I will still continue to investigate a solution for your dilemma.  Thank you.