Backup Exec 2010 and Dell MD3000i VSS provider issues

[lege]

Hello,

We are trying to use BE2010 to set up backups for a Hyper-V cluster on Windows Server 2008 R2. Our SAN is an iSCSI Dell MD3000i. We have installed the hardware provider on all the Hyper-V hosts, and it shows up properly in vssadmin list providers:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'SmVssProvider'

   Provider type: Hardware

   Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}

   Version: 10.60.01

Provider name: 'Microsoft Software Shadow Copy provider 1.0'

   Provider type: System

   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

If we enable Advanced Disk-Based backup in Backup Exec to move the backup snapshot to the backup server, it does not find the provider and comes up with this error:

The job failed with the following error: Backup Exec could not locate a Microsoft Volume Shadow Copy Services (VSS) software or hardware snapshot provider for this job. Select a valid VSS snapshot provider for the target computer.

If we disable the Advanced Disk-Based backup option, that error doesn't show up, however, we get an error saying The job failed

There's this additional information logged by Backup Exec:

 - AOFO: Initialization failure on: "VRTSRV::\\Hyper-V.domain.int\Hyper-V?HA?VM\VM1". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

The Event Log on the target computer shows:

Volume Shadow Copy Service error: Unexpected error calling routine DeviceIoControl(IOCTL_VOLUME_GET_GPT_ATTRIBUTES).  hr = 0x80070002, The system cannot find the file specified.

. 

Operation:

   Processing PreFinalCommitSnapshots

   Executing Asynchronous Operation

Context:

   Provider Name: SmVssProvider

   Provider Version: 10.60.4

   Provider ID: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}

   Current State: DoSnapshotSet

Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {24e4e8a3-69cb-4370-8b7b-c5276cd49765}. Routine details PreFinalCommitSnapshots({c9faf664-b4bb-4ce7-b975-9b423a6554d0}, 2) [hr = 0x80042302, A Volume Shadow Copy Service component encountered an unexpected error.

Check the Application event log for more information.

]. 

Operation:

   Executing Asynchronous Operation

Context:

   Current State: DoSnapshotSet

We have contacted Dell assuming it was a problem with their Hardware VSS provider software, but they insisted the problem was with Backup Exec and that hardware VSS provider is working as designed.

The BE2010 documentation states that SmVssProvider for MD3000i is supported. 

On a related note, we have also tried DPM 2010 with the provider and the same error occurs in the VSS provider: http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.public.dataprotectio...

Again, Dell claims it's not their problem, so any help in getting this to work for us would be appreciated. We're interested in Backup Exec but can't license if this doesn't work.

If the problem turns out to be indeed in the VSS provider, please let us know so we can refer Dell to the official Symantec response.

Thank you very much.

[CraigV]

Hi,

So it happens with Microsoft DPM and Backup Exec? That means it isn't the software, and could possibly be the OS.
That said, here are tech articles that might address your various errors:

1. http://seer.entsupport.symantec.com/docs/326302.htm

2. http://seer.entsupport.symantec.com/docs/314418.htm

What you can also look at doing is making sure that your job is configured to automatically select the provider used, and also go to a command prompt and type the following:

vssadmin list writer

Check to see if there are any errors, and if there are, use the tech article below to troubleshoot...

http://seer.entsupport.symantec.com/docs/261993.htm

Laters!

[lege]

Running vssadmin list writers shows:

Writer name: 'Microsoft Hyper-V VSS Writer'

   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

   Writer Instance Id: {8f6bfc43-e3cf-47c8-a600-e17895684669}

   State: [10] Failed

   Last error: Retryable error

I have read both of those tech articles, and rebooting doesn't help. I've been struggling with this for the past couple of weeks and tried a ton of stuff which made no difference.

We have also installed every hotfix for 2008 R2 we could find. The only one that made a small difference was this one:
http://support.microsoft.com/kb/977096

Prior to installing it, the Volume Shadow Copy Service would crash unexpectedly and the backup would fail with a RPC communication error in Backup Exec. After installing the hotfix, there was no unexpected crash any more and the actual error was apparent in the Event Log - because it is happening in PreFinalCommitSnapshots. which is referenced in the hotfix description. The error shown by Backup Exec also changed from RPC error (lost communication to VSS service) to just Job failed.

[CraigV]

Post the job log please...

[lege]

These are in the job log:

 - AOFO: Initialization failure on: "VRTSRV::\\Hyper-V.xxxxxxxxxxxxx\Hyper-V?HA?VM\xxxxxxxxxxx". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

The following volumes are dependent on resource: "C:" "C:\CLUSTERSTORAGE\VOLUME1" .

---

Job ended: Saturday, May 29, 2010 at 7:58:40 AM

Completed status: Failed

Final error: 0x8004230f - 0x8004230f (2147754767)

Final error category: Other Errors

For additional information regarding this error refer to link V-79-32772-8975

---

Click an error below to locate it in the job log

Backup- botwhyperv2 - AOFO: Initialization failure on: "VRTSRV::\\Hyper-V.xxxxxxxxxxxxxx\Hyper-V?HA?VMxxxxxxxxxxxxxxxx". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.

[kpalmer954]

Were you able to resolve this issue? I am experiencing a similar issue with DPM 2010. I have not tested BackupExec yet.

Thank You,

Kevin

[teiva-boy]

Has a support case with Symantec been opened up?

They would be willing to do a 3-way call with Dell if needs be if the case were escalated by yourself.

[lege]

We recently bought a new Dell MD3200i alongside our existing MD3000i to work around this backup problem, thinking the VSS provider must have been fixed in the new version of the SAN.

No luck.

We're using a trial version of Backup Exec 2010 R2 currently, so we can't get any tech support for this issue, but we'd really like to buy it if it got resolved. We already dished out $15,000 on the new SAN and can't really justify throwing out more money at this with no guarantees it will work, so any sort of help would be hugely appreciated.

Here's what's going on, again: We start the backup in Backup Exec, within 1 minute I see the CSV being put in Redirected Access mode, it stays like this for 1-2 minutes, then I can see a hardware snapshot being prepared in the Dell SAN Management software. 

The CSV is put back Online, and the hardware snapshot says Optimal in the SAN control panel. Then, boom, this is logged on the physical server hosting the VM being backed up:

Volume Shadow Copy Service error: Unexpected error calling routine DeviceIoControl(IOCTL_VOLUME_GET_GPT_ATTRIBUTES).  hr = 0x80070001, Incorrect function.

. 

Operation:

   Processing PreFinalCommitSnapshots

   Executing Asynchronous Operation

Context:

   Provider Name: SmVssProvider

   Provider Version: 10.70.08

   Provider ID: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}

   Current State: DoSnapshotSet

Backup Exec then fails the backup with simply "Job failed" and the hardware snapshot remains there, and I have to manually go in and delete it. Needless to say, the backup is not created.

We have 4 different Hyper-V hosts with slightly different hardware and software configurations, we have tried backing up VMs from each of them and they fail with the same error.

After spending $30,000 on these two SANs, Dell support doesn't want to help us, and their not so competent (to put it kindly) technicians claim the bug is with Backup Exec and that we should call Symantec. I find that very hard to believe, but I'm getting incredibly frustrated with this issue and would appreciate any sort of help. :)

I have created a screencast of all the steps I'm going through, from setting up the backup to when the error appears in the event log, it's only a few minutes long, so if anyone wants to help and take a look at it, please let me know and I'll PM you the link.

Thanks a ton! Hope someone with some Symantec connections sees this, the amount of troubleshooting and Dell support ordeal I've been going through the past few months has literally taken years off my life.

[teiva-boy]

I could be wrong, but I though off-host backup through VSS was ONLY supported on Exchange, SQL, and file data.  It is not supported on anything else.  While it may work, it's just not tested certified.

[lege]

Offhost backups would be the next step, we can't even take regular backups right now using the SAN's hardware VSS provider. 

Anyway, offhost backups are disabled, just Advanced Open File is enabled and set to Volume Shadow Copy Service - Automatic.

[teiva-boy]

See thats the thing, by using the SAN vendor's VSS driver, you are essentially doing an off-host backup. By using the native MS VSS provider, that would be supported and should work without AOFO enabled.  You'll also need the RAWS agent in the host and guests too I believe, if using the Hyper-V agent.

[lege]

I would have to respectfully disagree on the offhost backup part. :) 

Here's Dell's official technical paper on this: http://www.dell.com/downloads/global/products/pvaul/en/powervault-md3200-series-hyper-v-protection.p...

Quote:

A more convenient way to back up VMs with Backup Exec 12.5 is to use the Microsoft VSS Service with

the MD3200 or MD3200i storage array VSS hardware provider that is included on the MD3200 / MD3200i

series of storage arrays series of storage arrays Resource CD. This method lets you create hardwarebased

snapshots on the MD3200 or MD3200i storage array and map them automatically to your

dedicated backup server without performing any scripting.

Note: Their documentation mentions Backup Exec 12.5 which is an older version and didn't support CSVs, but Backup Exec 2010 R2 does support CSVs as mentioned here: http://www.symantec.com/redirects/backup_exec/hypervfaq/ (New! Support for 2008 Server Hyper-V R2 Cluster Shared Volumes (CSVs).)

Hardware VSS is a supported scenario for Hyper-V CSV backups. Even Microsoft DPM 2010 can do it (and yet it fails with the same error)

You can even take a VSS snapshot from the command line using hardware VSS over CSVs.

A VSS provider backup is not an offhost backup unless you choose to transport the snapshot volume over to the media server, which is optional. Otherwise the snapshot volume is mounted on the host and the Backup Exec agent just streams the data over to the media server.

Also, since Microsoft DPM 2010 can also do the offhost backup part, I really doubt Backup Exec couldn't, but I'm not sure. However, this thread is not really about offhost backups.

Backup Exec is supposed to be much more mature, and it is why we're looking into it as our first option of licensing.

Again. I have created a screencast of all the steps I'm going through, from setting up the backup to when the error appears in the event log, it's only a few minutes long, so if anyone wants to help and take a look at it, please let me know and I'll PM you the link.

Thanks!

[kpalmer954]

lege,

It sounds like you have gotten as far as I have with this and it isn't working. Have you installed the latest version of the Hyper-V integration components in all of the guests you are attempting to back up?

Have you verified the following hotfix is installed correctly?

http://xtravirt.com/dpm-2010-rc-and-hyper-v-r2-vm%E2%80%99s-csv%E2%80%99s-order-patches

Hotfix uninstall on Server 2008 Core?

http://arstechnica.com/civis/viewtopic.php?f=17&t=61524

Hi

Its not a very smooth operation, but i works 

-Uninstalling a hotfix

• Open a command-prompt and execute Expand /f:* filename.msu c:\Patch

• Navigate to c:\Patch\ and open filename.xml in a text editor.

• In filename.xml, replace Install with Remove and save the file

In the command prompt, type:

Pkgmgr /n: filename.xml

Checking Integration Components version on Hyper-V RTM

http://blogs.technet.com/b/cedward/archive/2009/06/12/checking-integration-components-version-on-hyp...

Right click / properties on the c:\windows\system32\drivers\vmbus.sys file in all of the hosts. Product version should be 6.1.7600.20542 or newer.

- Reinstall both hotfixes in the correct order as described in the article.

- Install the latest integration components on all of the guests.

- Verify the integration component version is correct on all of the guests.

I have an MD3000i that I was never able to get to work with the Hardware VSS provider. I'm getting ready to give it one more try before moving to Equalogic. I worked with Microsoft Paid Product Support and the engineer was unable to get the hardware VSS provider to work with or without CSV support. I'm trying to use the MD3000i hardware VSS provider with DPM 2010. Microsoft and I both tested with the Vshadow tool from the VSS SDK.

Kevin

[diz]

Are there errors in windows application log of backup server from smprovider?

[Pascal-FR]

Hello, I found this quite interesting thread while looking at "Hyper-V off host backup".

I'm preparing the same hardware infrastructure as Lege did:

A Hyper-V cluster based on Windows Server 2008 R2 (2 nodes) attached to a DELL MD3200i iSCSI SAN.

I've asked DELL Consultant this morning about Guest VM backup stored on the CSV at the SAN. He told me this can be done with the "Enterprise Licence" for the Storage array:

This is an optional licence key you enter in your MD3200i that unlocks the Snapshot feature within the SAN firmware.

Currently I do not have this Key, and I fail to do a hardware snapshot using vssadmin from the Server where a LUN from the SAN is attached.

Two white papers describe what we all want to do:

PowerVault MD32x0 and MD32x0i Data Protection Whitepaper (explain what the features keys are for, and what VSS and snapshot)

http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32x0-md32x0i-data-protection-whi...

 and the one pointed by Lege

http://www.dell.com/downloads/global/products/pvaul/en/powervault-md3200-series-hyper-v-protection.p...

Lege, if your screenCam is still available could you PM me the URL. It seems we're facing the same issue, and i'm currently leaving BE 12.5 moving to DPM2010.

Regards,

pascal.

My current MD3200i is not at the latest Firmware issued 18 january 2011.

C:\Program Files (x86)\Dell\MD Storage Manager\util>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'SmVssProvider'
   Provider type: Hardware
   Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
   Version: 10.70.08    <----- This could be updated in the new DELL MD3200i CD

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

[lege]

We opened a support ticket with Microsoft, and the Microsoft engineer verified the obvious, and confirmed the problem is with Dell's VSS provider and we then confronted Dell about it, but Dell completely ignored us.

Apparently, they know that CSV backups are not supported (buggy) in their VSS providers, and because these SANs are not even produced by DELL, and they just resell from LSI, they don't care and won't do anything to fix it.

Needless to say, we won't be buying Dell again.

[Pascal-FR]

I will call back my DELL Rep and see what he think about this (CSV not working) anf no fix proposed.

My Rep. clearly admited the other point you mention:

The DELL PowerVault MD3200i and the HP StorageWorks P2000 are basically the same hardware; they’re both rebadged LSI Engenio enclosures. Even IBM resel it in its portfolio.

My Rep was having a Sales Tool, PDF Table compraison of the 3 offers, with features, differences and so...

It's a pitty you didn't get any fix. The main reson to puchase this Storage Array was to build an Hyper-V cluster (with CSV of course).

I will post the DELL update in this thread.

Pascal.