Shadow Copy and DFS

[MP_DM]

Media Server Win2008R2 - BE2010R2 SP1

File Server being backed up - Win2K3R2

  After a migration to our media server, we ran into issues in which the DFS data on our file server was not backing up properly.  With the help of this forum, I was instructed to backup the Shadow Copy Components as opposed to the actual volume (https://www-secure.symantec.com/connect/forums/volume-does-not-back-after-migration).  

Yesterday, I needed to restore some data for this server.  The restore seemed to hang on multiple attemtps.  Issue:  http://www.symantec.com/business/support/index?page=content&id=TECH135096

I found that there was a hotfix released for this issue.  So I installed Hotfix 150096 followed by Hotfix 144101 on the media server. This corrected the issues with restoring the data.

Now, this seems to have created a new problem.  My backup jobs on this volume are failing.  I have received several errors on several attempts.  Error  V-79-57344-33967 as well as V-79-57344-65078.  In addition to the failed jobs, I have received the following error on my file server in the Sys logs. It appears that this error has been occuring occasionally during the backup window, however it does not always result in a failed job.

Event Type: Error
Event Source: VolSnap
Event Category: None
Event ID: 25
Date:  2/24/2011
Time:  9:27:53 AM
User:  N/A
Computer: FILESERVER
Description:
The shadow copies of volume S: were deleted because the shadow copy storage could not grow in time.  Consider reducing the IO load on the system or choose a shadow copy storage volume that is not being shadow copied.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 58 00   ......X.
0008: 00 00 00 00 19 00 06 c0   .......À
0010: 03 00 00 00 00 00 00 00   ........
0018: 67 00 00 00 00 00 00 00   g.......
0020: 00 00 00 00 00 00 00 00   ........

I updated RAWS on the file server, and attempted another incremental backup (using Archive bit).  This job has been running for nearly an hour now.  Even more peculiar is that is has backed up nearly 50GB.  This can't be possible as there just aren't 50GB worth of changed files on this volume.  Historically, we may see 2 to 10GB worth of incremental data on any given day.  (example: yesterday we picked up about 5GB; no errors)

Is this a Microsoft error?  Or do I possibly have my backup job misconfigured?  Or is this a bad Hotfix?

On one hand, I can't wait 8 hours to restore 1 file... but on the other I need to backup my data successfully.  I have been suspicious that my selections are wrong for this job.  Here is what I have set up.  Please point out any issues you see with the selections.

Shadow Copy Components\User?Data\Distributed File System Replication\DfsrReplicatedFolders\S \*.* /SUBDIR
Shadow Copy Components\User?Data\Distributed File System Replication\DfsrReplicatedFolders\S \assettags\*.* /SUBDIR /EXCLUDE
Shadow Copy Components\User?Data\Distributed File System Replication\DfsrReplicatedFolders\S \DfsrPrivate\*.* /SUBDIR /EXCLUDE
Shadow Copy Components\User?Data\Distributed File System Replication\DfsrReplicatedFolders\S \europedata\*.* /SUBDIR /EXCLUDE
Shadow Copy Components\User?Data\Distributed File System Replication\DfsrReplicatedFolders\S \RECYCLER\*.* /SUBDIR /EXCLUDE
Shadow Copy Components\User?Data\Distributed File System Replication\DfsrReplicatedFolders\S \System Volume Information\*.* /SUBDIR /EXCLUDE
 

Thanks,

David

[Kenny-Knows-Stu]

Looks like the backup selections are okay, you might want to check the status of the DFS /DFSR Writer by running the VSSadmin List writers commands on the Server hosting the DFS, volsnap error may be register in case there if there are VSS related issues on the OS or if the DFS write is in unstable form.

[MP_DM]

Thank you for the reply.  I checked the writer statuses yesterday when I was having all of the trouble.  They all showed stable, however one indicated it was 'waiting' on something.  When I ran the command again today, everything is Stable and no errors.

My regularly scheduled job failed again last night, however I have run several test jobs this morning successfully.  The test incremental jobs appeared to pick up alot of data (roughly one-third of the actual volume).  This was rather unexpected, considering we generally only get about 2% of all data on an incremental.

After digging a little deeper into the DFS portion, i found an error on the other DFS member (not being backed up by backup exec) that connection to the host partner was lost, before the Hotifxes were installed to the file server.  I don't profess to be an expert on DFS, but it appears that the two were out of sync.  The hosting server was re-replicating the volume, and I believe this was causing some trouble with the VSS writer during backupexec backups. Replication takes place over a 100Mbit link, and total data on the volume is around 500GB.  So, the reseeding could have been going on all day and night when I was having the trouble.  It looks like a really crazy coincidence that DFS flaked out right before a RAWS update.

To help the cause, I have disabled Volume Shadow Copy on the file server for that volume... which should eliminate the VolSnap error.  It was set up long long ago before we were ever using DFS or Backup2Disk in symantec.  It's just an extra thing that could go wrong... and it really isn't providing enough benefit to keep it on.

I have a Full job scheduled for tonight, and I'm feeling a little better now that I've run a few test job succesfully.  At this point, I'm going to let it run tonight and see where I'm at in the morning.

Thanks again for your suggestions.

[MP_DM]

A light bulb just went off in my head... 

The error in DFS occured at about the same time (within minutes) of when I attempted to restore the data that I mentioned in my original post.  I was trying to restore the data to a redirected folder on my BE media server, and the restore kept hanging.  So, I decided to try restoring to original location on the file server.  When I did this, it kicked DFS offline... and my headaches began.

I only installed the hotfixes to eliminate the hanging restore issues.  So, the hotfixes may have nothing to do with the errors.  Moreover, DFS really didn't like BackupExec's attempt to restore data to its original location. 

[Kenny-Knows-Stu]

Disabling VSS can cause more issues, as DFS backups rely hugely on VSS , as VSS will tape a snapshot of the DFS data even if its been use by users or if the DFS data is getting replicated along the sites.

Right now fixing the VSS should be the 1st priority as disabling it wont solve the purpose.

I Would suggest you to check the space on the S : volumme, also try to set the volume`s storage are size to no limit, by right clicking on the volume and navigating to the Shadow Copies Tab\Settings button and the no limit check for in front of the Maximum size.

[CraigV]

Hi,

There is a VSS roll-up patch that you can install on the 2003 server that fixes VSS writer issues.

You can download it here, but bear in mind that it requires a restart of the server. Once restarted, run vssadmin list writers again, and make sure the VSS writers are consistent!

http://support.microsoft.com/kb/940349

Laters!

[MP_DM]

I should have been more clear in my explanation.  The volume shadow copy service hasn't been disabled.  It is still set 'Manual' in the services mmc as it always has been.  I simply turned off Shadow Copies (via shadow copies tab) for that volume to simplify and streamline future troubleshooting.  One less thing to trip me up in fixing things.

We still have 2 other volumes on the file server that create shadow copies... and are not causing us any trouble.

I agree 100% for fixing the VSS errors...  I believe MS has issued a few hotfixes for VSS, and I'm unsure if they were ever deployed to this server. That is my starting point.  Additionally, the server is due up for regular maintenance... for which we have added a disk/health check to sort out any potential disk errors that could be at the root of this.

*Edit... I just saw Craig's post about VSS update after I posted.  We were thinking the same thing.

[MP_DM]

I'm just popping into this thread to update.  The VSS hotfixes described above were installed about 2 weeks ago as part of a regular maintenance cycle.   At this point, I'm just waiting for it to throw the error again.  Based on history of this job, it usually failed every 3 to 6 weeks. 

I'll drop back in a few weeks and report success or failure.  Thanks again to all who contributed.