Backup Exec 2010 and Dell MD3000i VSS provider issues

Hello,

We are trying to use BE2010 to set up backups for a Hyper-V cluster on Windows Server 2008 R2. Our SAN is an iSCSI Dell MD3000i. We have installed the hardware provider on all the Hyper-V hosts, and it shows up properly in vssadmin list providers:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'SmVssProvider'

   Provider type: Hardware

   Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}

   Version: 10.60.01

Provider name: 'Microsoft Software Shadow Copy provider 1.0'

   Provider type: System

   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

If we enable Advanced Disk-Based backup in Backup Exec to move the backup snapshot to the backup server, it does not find the provider and comes up with this error:

The job failed with the following error: Backup Exec could not locate a Microsoft Volume Shadow Copy Services (VSS) software or hardware snapshot provider for this job. Select a valid VSS snapshot provider for the target computer.

If we disable the Advanced Disk-Based backup option, that error doesn't show up, however, we get an error saying The job failed

There's this additional information logged by Backup Exec:

- AOFO: Initialization failure on: "VRTSRV::\\Hyper-V.domain.int\Hyper-V?HA?VM\VM1". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

The Event Log on the target computer shows:

Volume Shadow Copy Service error: Unexpected error calling routine DeviceIoControl(IOCTL_VOLUME_GET_GPT_ATTRIBUTES). hr = 0x80070002, The system cannot find the file specified.

.

Operation:

   Processing PreFinalCommitSnapshots

   Executing Asynchronous Operation

Context:

   Provider Name: SmVssProvider

   Provider Version: 10.60.4

   Provider ID: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}

   Current State: DoSnapshotSet

Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {24e4e8a3-69cb-4370-8b7b-c5276cd49765}. Routine details PreFinalCommitSnapshots({c9faf664-b4bb-4ce7-b975-9b423a6554d0}, 2) [hr = 0x80042302, A Volume Shadow Copy Service component encountered an unexpected error.

Check the Application event log for more information.

].

Operation:

Executing Asynchronous Operation

Context:

Current State: DoSnapshotSet

We have contacted Dell assuming it was a problem with their Hardware VSS provider software, but they insisted the problem was with Backup Exec and that hardware VSS provider is working as designed.

The BE2010 documentation states that SmVssProvider for MD3000i is supported.

On a related note, we have also tried DPM 2010 with the provider and the same error occurs in the VSS provider: http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.public.dataprotectionmanager&tid=b608a39e-b0cc-4f9c-b126-98ceb723476e&mid=b608a39e-b0cc-4f9c-b126-98ceb723476e&cat=&lang=&cr=&sloc=&p=1

Again, Dell claims it's not their problem, so any help in getting this to work for us would be appreciated. We're interested in Backup Exec but can't license if this doesn't work.

If the problem turns out to be indeed in the VSS provider, please let us know so we can refer Dell to the official Symantec response.

Thank you very much.

2010

Advanced Disk-Based Backup Option

Agent for Microsoft Hyper-V

Backup and Recovery

Backup Exec

Windows

16 Replies

CraigV
Moderator
15 years ago
Hi,

So it happens with Microsoft DPM and Backup Exec? That means it isn't the software, and could possibly be the OS.
That said, here are tech articles that might address your various errors:

1. http://seer.entsupport.symantec.com/docs/326302.htm

2. http://seer.entsupport.symantec.com/docs/314418.htm

What you can also look at doing is making sure that your job is configured to automatically select the provider used, and also go to a command prompt and type the following:

vssadmin list writer

Check to see if there are any errors, and if there are, use the tech article below to troubleshoot...

http://seer.entsupport.symantec.com/docs/261993.htm

Laters!
lege
Level 3
15 years ago
Running vssadmin list writers shows:

Writer name: 'Microsoft Hyper-V VSS Writer'

   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

   Writer Instance Id: {8f6bfc43-e3cf-47c8-a600-e17895684669}

   State: [10] Failed

   Last error: Retryable error

I have read both of those tech articles, and rebooting doesn't help. I've been struggling with this for the past couple of weeks and tried a ton of stuff which made no difference.

We have also installed every hotfix for 2008 R2 we could find. The only one that made a small difference was this one:
http://support.microsoft.com/kb/977096

Prior to installing it, the Volume Shadow Copy Service would crash unexpectedly and the backup would fail with a RPC communication error in Backup Exec. After installing the hotfix, there was no unexpected crash any more and the actual error was apparent in the Event Log - because it is happening in PreFinalCommitSnapshots. which is referenced in the hotfix description. The error shown by Backup Exec also changed from RPC error (lost communication to VSS service) to just Job failed.
CraigV
Moderator
15 years ago
Post the job log please...
lege
Level 3
15 years ago
These are in the job log:

- AOFO: Initialization failure on: "VRTSRV::\\Hyper-V.xxxxxxxxxxxxx\Hyper-V?HA?VM\xxxxxxxxxxx". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

The following volumes are dependent on resource: "C:" "C:\CLUSTERSTORAGE\VOLUME1" .

---

Job ended: Saturday, May 29, 2010 at 7:58:40 AM

Completed status: Failed

Final error: 0x8004230f - 0x8004230f (2147754767)

Final error category: Other Errors

For additional information regarding this error refer to link V-79-32772-8975

---

Click an error below to locate it in the job log

Backup- botwhyperv2 - AOFO: Initialization failure on: "VRTSRV::\\Hyper-V.xxxxxxxxxxxxxx\Hyper-V?HA?VMxxxxxxxxxxxxxxxx". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.
kpalmer954
Level 2
15 years ago
Were you able to resolve this issue? I am experiencing a similar issue with DPM 2010. I have not tested BackupExec yet.

Thank You,

Kevin
teiva-boy
Level 6
15 years ago
Has a support case with Symantec been opened up?

They would be willing to do a 3-way call with Dell if needs be if the case were escalated by yourself.
lege
Level 3
14 years ago

We recently bought a new Dell MD3200i alongside our existing MD3000i to work around this backup problem, thinking the VSS provider must have been fixed in the new version of the SAN.

No luck.

We're using a trial version of Backup Exec 2010 R2 currently, so we can't get any tech support for this issue, but we'd really like to buy it if it got resolved. We already dished out $15,000 on the new SAN and can't really justify throwing out more money at this with no guarantees it will work, so any sort of help would be hugely appreciated.

Here's what's going on, again: We start the backup in Backup Exec, within 1 minute I see the CSV being put in Redirected Access mode, it stays like this for 1-2 minutes, then I can see a hardware snapshot being prepared in the Dell SAN Management software.

The CSV is put back Online, and the hardware snapshot says Optimal in the SAN control panel. Then, boom, this is logged on the physical server hosting the VM being backed up:

Volume Shadow Copy Service error: Unexpected error calling routine DeviceIoControl(IOCTL_VOLUME_GET_GPT_ATTRIBUTES). hr = 0x80070001, Incorrect function.

.

Operation:

   Processing PreFinalCommitSnapshots

   Executing Asynchronous Operation

Context:

   Provider Name: SmVssProvider

   Provider Version: 10.70.08

   Provider ID: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}

   Current State: DoSnapshotSet

Backup Exec then fails the backup with simply "Job failed" and the hardware snapshot remains there, and I have to manually go in and delete it. Needless to say, the backup is not created.

We have 4 different Hyper-V hosts with slightly different hardware and software configurations, we have tried backing up VMs from each of them and they fail with the same error.

After spending $30,000 on these two SANs, Dell support doesn't want to help us, and their not so competent (to put it kindly) technicians claim the bug is with Backup Exec and that we should call Symantec. I find that very hard to believe, but I'm getting incredibly frustrated with this issue and would appreciate any sort of help. :)

I have created a screencast of all the steps I'm going through, from setting up the backup to when the error appears in the event log, it's only a few minutes long, so if anyone wants to help and take a look at it, please let me know and I'll PM you the link.

Thanks a ton! Hope someone with some Symantec connections sees this, the amount of troubleshooting and Dell support ordeal I've been going through the past few months has literally taken years off my life.
teiva-boy
Level 6
14 years ago
I could be wrong, but I though off-host backup through VSS was ONLY supported on Exchange, SQL, and file data. It is not supported on anything else. While it may work, it's just not tested certified.
lege
Level 3
14 years ago
Offhost backups would be the next step, we can't even take regular backups right now using the SAN's hardware VSS provider.

Anyway, offhost backups are disabled, just Advanced Open File is enabled and set to Volume Shadow Copy Service - Automatic.
teiva-boy
Level 6
14 years ago
See thats the thing, by using the SAN vendor's VSS driver, you are essentially doing an off-host backup. By using the native MS VSS provider, that would be supported and should work without AOFO enabled. You'll also need the RAWS agent in the host and guests too I believe, if using the Hyper-V agent.