Backup Exec 2012 VSS Failure on Windows Server 2008R2

[bret_miller]

I recently replaced our backup server running Backup Exec 2010 R3 on Windows Server 2003 standard with Backup Exec 2012 running on Windows Server 2008R2. I chose not to upgrade because of a few strange errors in the database that we've never been able to resolve. So I started from scratch, which I would have thought would lead to a cleaner and simpler setup. At the best, it's been frustrating. But we're there. Except that Backup Exec can't backup the backup server itself.

This was a new install of Windows Server 2008R2 SP1, fully patched. I then installed a few tools. Then I connected it via iSCSI to a Thecus N8900 which serves as disk storage for Backup Exec mostly. Then I installed XenCenter, which I used to backup and restore a handful of virtual machines--mostly for non-critical stuff. Critical backup is all handled by Backup Exec.

But it keeps getting VSS errors backing up itself. Other servers work fine.

This is a brand new Dell PowerEdge R320. The C: drive is on a Dell PERC S110 RAID controller using mirroring. 

I have tried re-registering the VSS components. That didn't change anything. VSSADMIN LIST WRITERS lists the writers and does not complain of errors.

I have tried using VSSADMIN ADD SHADOWSTORAGE which allowed the backup to work once, then it started failing again.

I changed the shadow storage using the volume properties in Share and Storage Management to set a limit on the shadow copy storage rather than using no limit.

vssadmin list writers:

Writer name: 'Task Scheduler Writer'

   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}

   State: [1] Stable

   Last error: No error

Writer name: 'VSS Metadata Store Writer'

   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

   State: [1] Stable

   Last error: No error

Writer name: 'Performance Counters Writer'

   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}

   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

   State: [1] Stable

   Last error: No error

Writer name: 'System Writer'

   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

   Writer Instance Id: {57a30596-1a50-4f58-adc1-4231c2c08664}

   State: [7] Failed

   Last error: Timed out

Writer name: 'SqlServerWriter'

   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

   Writer Instance Id: {c9e8f0fd-ddc4-4adf-8c03-cd5fa8f4f5e8}

   State: [1] Stable

   Last error: No error

Writer name: 'ASR Writer'

   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

   Writer Instance Id: {38547cdf-9952-45b4-b0b9-00d2d5cd8554}

   State: [7] Failed

   Last error: Timed out

Writer name: 'Shadow Copy Optimization Writer'

   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

   Writer Instance Id: {51edec9e-48c0-44c5-9bf9-41a0f1293dc6}

   State: [7] Failed

   Last error: Timed out

Writer name: 'COM+ REGDB Writer'

   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

   Writer Instance Id: {623b8096-4c1f-4e44-b30f-ff169ba80889}

   State: [7] Failed

   Last error: Timed out

Writer name: 'Registry Writer'

   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

   Writer Instance Id: {03819f02-d29d-4f78-be5c-727a94b7fabc}

   State: [7] Failed

   Last error: Timed out

Writer name: 'WMI Writer'

   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

   Writer Instance Id: {08fe356e-d0f5-4f9d-9dff-b0da33b7fc66}

   State: [7] Failed

   Last error: Timed out

vssadmin list shadowstorage:

Shadow Copy Storage association

   For volume: (E:)\\?\Volume{5f5003f2-7f61-4074-bf54-29a239c02bc9}\

   Shadow Copy Storage volume: (E:)\\?\Volume{5f5003f2-7f61-4074-bf54-29a239c02bc9}\

   Used Shadow Copy Storage space: 0 B (0%)

   Allocated Shadow Copy Storage space: 0 B (0%)

   Maximum Shadow Copy Storage space: 1.599 TB (9%)

Shadow Copy Storage association

   For volume: (C:)\\?\Volume{8009fb57-2dcc-11e2-89ce-806e6f6e6963}\

   Shadow Copy Storage volume: (C:)\\?\Volume{8009fb57-2dcc-11e2-89ce-806e6f6e6963}\

   Used Shadow Copy Storage space: 0 B (0%)

   Allocated Shadow Copy Storage space: 0 B (0%)

   Maximum Shadow Copy Storage space: 46.319 GB (9%)

Here are the errors:

Log Name:      Application

Source:        VSS

Date:          11/29/2012 9:36:59 AM

Event ID:      8193

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      BK1.hq.gci.org

Description:

Volume Shadow Copy Service error: Unexpected error calling routine Cannot find anymore diff area candidates for volume \\?\Volume{8009fb56-2dcc-11e2-89ce-806e6f6e6963}\ [1].  hr = 0x8000ffff, Catastrophic failure

. 

Operation:

   Automatically choosing a diff-area volume

   Processing EndPrepareSnapshots

Context:

   Volume Name: \\?\Volume{8009fb57-2dcc-11e2-89ce-806e6f6e6963}\

   Volume Name: \\?\Volume{8009fb56-2dcc-11e2-89ce-806e6f6e6963}\

   Execution Context: System Provider

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="VSS" />

    <EventID Qualifiers="0">8193</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2012-11-29T17:36:59.000000000Z" />

    <EventRecordID>6373</EventRecordID>

    <Channel>Application</Channel>

    <Computer>BK1.hq.gci.org</Computer>

    <Security />

  </System>

  <EventData>

    <Data>Cannot find anymore diff area candidates for volume \\?\Volume{8009fb56-2dcc-11e2-89ce-806e6f6e6963}\ [1]</Data>

    <Data>0x8000ffff, Catastrophic failure

</Data>

    <Data>

Operation:

   Automatically choosing a diff-area volume

   Processing EndPrepareSnapshots

Context:

   Volume Name: \\?\Volume{8009fb57-2dcc-11e2-89ce-806e6f6e6963}\

   Volume Name: \\?\Volume{8009fb56-2dcc-11e2-89ce-806e6f6e6963}\

   Execution Context: System Provider</Data>

    <Binary>2D20436F64653A20535052414C4C4F4330303030313137342D2043616C6C3A20535052414C4C4F4330303030303739302D205049443A202030303030333530342D205449443A202030303030333134382D20434D443A2020433A5C57696E646F77735C53797374656D33325C737663686F73742E657865202D6B2073777072762D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820</Binary>

  </EventData>

</Event>

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Log Name:      Application

Source:        VSS

Date:          11/29/2012 9:36:59 AM

Event ID:      12293

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      BK1.hq.gci.org

Description:

Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details EndPrepareSnapshots({c8836e49-2022-4af8-ac77-378fc1d114ff}) [hr = 0x80042302, A Volume Shadow Copy Service component encountered an unexpected error.

Check the Application event log for more information.

]. 

Operation:

   Executing Asynchronous Operation

Context:

   Current State: DoSnapshotSet

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="VSS" />

    <EventID Qualifiers="0">12293</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2012-11-29T17:36:59.000000000Z" />

    <EventRecordID>6374</EventRecordID>

    <Channel>Application</Channel>

    <Computer>BK1.hq.gci.org</Computer>

    <Security />

  </System>

  <EventData>

    <Data>{b5946137-7b9f-4925-af80-51abd60b20d5}</Data>

    <Data>EndPrepareSnapshots({c8836e49-2022-4af8-ac77-378fc1d114ff})</Data>

    <Data>0x80042302, A Volume Shadow Copy Service component encountered an unexpected error.

Check the Application event log for more information.

</Data>

    <Data>

Operation:

   Executing Asynchronous Operation

Context:

   Current State: DoSnapshotSet</Data>

    <Binary>2D20436F64653A20434F52534E50534330303030313632342D2043616C6C3A20434F52534E50534330303030313630352D205049443A202030303030333137322D205449443A202030303030343930302D20434D443A2020433A5C57696E646F77735C73797374656D33325C76737376632E6578652020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820</Binary>

  </EventData>

</Event>

[lmosla]

Try to run the backup using Windows built in NTBackup utility.    For Windows server 2008, run Wbadmin from the command prompt.  If it fails it will identify whether you need to bring in Microsoft for further troubleshooting.

[bret_miller]

I tried. It's not installed. I hate to install extra stuff just to debug what Backup Exec should be doing already... But if that's the only way to debug it, I guess I have to install Windows Backup...

[bret_miller]

OK... Installed Windows Server Backup. Tried this command line:

Wbadmin start systemstatebackup -backupTarget:e:

E: is the Thecus storage unit. But it gives this error:

ERROR - Command syntax incorrect. Error: -backupTarget:e:. See the command syntax below.

Same error if I change the backupTarget to c: or to the volume specification for e: \\?\Volume{5f5003f2-7f61-4074-bf54-29a239c02bc9}\.

Just to be sure I added the registry key HKLM\SYSTEM\CurrentControlSet\Services\wbengine\SystemStateBackup\AllowSSBToAnyVolume as a DWORD and set it to 1.

Then since I wasn't having any luck with the command line, I tried the GUI backup tool, backing up just the system state to E: (tried again with C:). This time I got a different error: "One of the specified source volumes has an invalid format and cannot be protected using Windows Server Backup. Only volumes formatted with NTFS can be protected."

So... going to Disk Management, I see 4 items in the list--a 39MB OEM partition with no file system, a 463.20GB NTFS partition for the C: drive, a 1.99GB OS system partition formatted FAT32 (I didn't create that), and a 16368.87GB E: drive formatted NTFS.

Comparing this to other servers (all the other 2008R2 servers are virtual), there is a 100MB "system reserved" NTFS parition, not a 2GB FAT32 "OS" partition. Those partitions don't have drive letters assigned to them... I wonder if I should run "Convert" on that crazy FAT32 volume. The other servers also do not have the OEM partition. I do have one other physical server that I started installing but have not deployed yet. It looks identical to this one except for no E: drive. I think I might wipe that disk and start over to avoid these crazy issues... 

So the question is what to do next...

[lmosla]

bret.miller,

Check your System events or any errors that could identify a third party application that may be interfering with the VSS writers, and make surethe Volume Shadow Copy service is set to manual. 

The problem with the system state backup seems more like a Microsoft or Dell issue whom should be contacted to reccomend the next steps.

also from the Backup Exec 2012 Admin guide pg 729 “If you have OEM partitions such as Dell Utility partitions on thesystem, they are considered part of a computer's critical systemcomponents and are backed up and restored as such.”

[pkh]

I don't know why you said that there are no writer errors.  In your original post, your "vssadmin list writer" listing shows that there are a couple of writers which failed, like the 'ASR Writer'.  You should re-boot the server to clear these problems and make sure that ALL the writers are o.k. before proceeding with any backups.

[bret_miller · Accepted solution]

So... the solution was to do this:

1. Use DISKPART to assign a drive letter to the OS partition
2. Use CONVERT to make the OS partition NTFS
3. Remove the drive letter with DISKPART
4. Remove the utility partition selection in BackupExec

Now since all the partitions are NTFS, the full backup completes successfully. The Dell OEM partition isn't essential to the functioning of the server anyway, so if I have to restore without it, the server will still work.