Backup Exec Snapshot backup of DFS

[AnKirby]

Hi All

I have a question about backing up DFS.  My research shows that it is inherently slow and that is Microsfts fault.

The customer currently either backups up the shadow copy components , extremely slow, or they stop DFS-backup- restart DFS.  The issue with this is that DFS takes too long to catch up the synchronization.

I have been asked if we can use Equallogic snapshots to back up the DFS store.  My thoughts are that if the metdata (shadow components) is on the C: which is not fromthe Equallogic then snapshots will be no good for restoring individual files/components.

Are there any recomendations out there on how best to backup the DFS stores.

Many thanks

Andy

[CraigV]

Hi Andy,

Have you checked the TNs below:

http://www.symantec.com/business/support/index?page=content&id=TECH56841

https://www-secure.symantec.com/connect/articles/dfs-and-dfsr-backing-your-distributed-file-system-b...

Thanks!

[AnKirby]

Thanks for your reply Craig.

Yes I have seen the above mentioned information above regarding backing up the shares etc.  It seems to me this is not really viable.  My install has nudreds of shares and all heavily use permissions.  These would need to be recreated manually I believe.  Not acceptable.

I was hoping for some real world experience on how to improve the performance of the backups.

We were thinking of snap shots on vmware, but I don;t believe this will enable GRT or file/object recovery.

There seems to be lots of 'questoisns' around this subject but never an answer.  Is it possible to get symantec to tell us whether or not they see this as an issue and if they care enough to work on a resolution.

[CraigV]

...if nobody answers you here, it might also be worthwhile opening up a query on the Equalogic forums...

[FischerRoman]

I'm having the same dilemma...

Backing up DFS-R via Shadow Copy Components is VERY VERY slow (slowest measured: 7 MB/Minute) with some potent hardware in background (EMC-Storage with multipathing FC-SAN, dedicated Backup-GB-LAN, fast servers, etc.).

Backing up large amounts of data takes forever, right now we only have a fraction of future data to back up (~100GB out of ~5TB) and this already takes many hours to complete.

Backing up via network shares is not an option of us, because of complex permissions which are not allowed to get lost - even this would speed up things a lot.

Plan B is using EMC ReplicationManager for storage-snapshot-based backups, these snapshots will be mounted directly on the Backup Exec 2010R3 server for local backup.

What we would lose is all DFSR configuration, but that isn't a real problem for us, because in case of some disaster there would be manual intervention anyway, dealing with the DFSR re-configuration or re-creation of replicated folders etc...

Most of these DFSR-tasks can be scripted, to reduce the chance of human error.

Stopping DFSR Service, running the backup and afterwards starting DFSR Service is not an option too - because of the higher possibility that USN-wraps occur while the backup is running.

Ned Pyle / Microsoft made some good blog posts about this issue, just search the "Ask the Directory Services Team" blog.

Restore of a whole snapshot is in fact very tricky, because you have to make sure the restored data isn't replicated in the wrong direction, and NEVER restore the DFSR-Database - it will break your replication and data loss will occur.

Restoring a whole volume is only possible by 1st remove the member to restore from the replication group(s), wait for Event 4010 in the eventlog. 2nd recreate the damaged volume. 3rd restore the snapshot, if possible without the DFSR database, if not possible stop the DFSR service, then remove the old and out-of-sync DFSR database with [RD “<drive>:\system volume information\DFSR” /s /q], restart the DFSR service. 4th re-add the removed member to the replication group(s). 5th wait for initial replication complete (Event 4104 in eventlog) - tadaa - you're done.
If you fail to 1st remove the member from the replication groups, you're in trouble.

I'm going to try out the EMC snapshot-backup/restore in the next few weeks and hopefully won't forget to update this post with the results...

Andy, please also keep this thread updated, it's hard to find good information regarding backup/restore of DFSR data, when you have to use non-standard-solutions... ;)

[AnKirby]

Fischer

I am interested to see the results you get but am thinking that...........

Snapshot backups are great for Data on a SAN.  The question is: is you C: drive (Shadow copy components) on the SAN.  If not you are only snapping the data portion, and not he DFS-R information.  I am not DFS expert but is this not needed to recover the system.

Also file/object level recovery will not be available so to restore one file you will need to restore ~5TB.  Will you have the space for this?

Either way let me know what happens.

Thanks

Andy

[FischerRoman]

Andy,

my primary backup target is for the Data on the SAN. The system running DFS-R is replacable and can be reinstalled at any time and there's a Microsoft supported way to replace a DFS-R member with a new one.
For me this isn't an issue, because I also run a DFS-R HUB with W2k8R2 Clustering - so replacing one of the hosts is like a Cluster Disaster Recovery, which is a bit tricky, but doable.

Every Volume holding one or more DFS Replicated Folders has its own DFS-R Database and therefor should be backed up using the Microsoft shadow-copy provider to go the supported way.

Backing up a snapshot does not exclude the possibility to backup VSS-resources.
Let me describe it with an example.

You have the following system:
- 1 Local volume (RAID1) C: for the operating system W2k8R2
- 1 SAN volume (whatever RAID-level on whatever storage system) D: for DFS-R data holding a replicated folder

1st we backup everything without SAN snapshots:
- backup C: (complete) - does not use VSS
- backup systemstate (complete) - uses VSS for systemstate
- backup D: using Microsoft shadow-copy provider / user data (complete) - uses VSS for DFS-R data

2nd we backup everything using SAN snapshots:
- backup C: (complete) - does not use VSS
- backup systemstate (complete) - uses VSS for systemstate
- backup D: using 3rdparty hardware based Snapshot - after snapshot creation the snapshot is mounted on the backup server and backed up locally - without VSS, because the backup server does not know anything about DFS-R and treats the data (everything including the per volume DFS-R database) as normal file data

DFS-R consists of threee parts:
1) DFS-R configuration - which is stored in AD
2) DFS-R database - which is stored once per volume under <drive:\System Volume Information\DFSR> - regardless of the volume-type, there's no difference between SAN or local attached storage
3) DFS-R User-Data

When you restore DFS-R data using Backup Exec without file/folder-redirection from a previously made VSS-backup, the DFS-R service gets notice of that operation and suspends replication until the operation is complete. What happens to the restored data depends on what type of restore you did, authorative or non-authorative (which is default).
1) non authorative: files restored to their original location are being moved to the folder <ReplicatedFolderName\DfsrPrivate\Preexisting> to prevent any inconsistence with existing data. The backup operator then has to move the file(s) manually to the original location, using the PreExisting.XML because the file- and foldernames will get renamed automatically by DFS-R.
1a) non authorative / redirected: files restored to an alternate location keep their original file/folder structure and also ACL/permissions. The backup operator then needs to move the file(s) manually to the original location - without the need to look into the PreExisting.XML!
2) authoritive: files restored to their original location will be replicated authoritively to every partner in the replication group - DFS-R triggers a COMPLETE replication of the WHOLE replicated folder - not just the restored file(s)/folder(s)! This also has the potential of unwanted replication conflicts which land in the Conflict and Deleted folder. I don't think you want this under normal circumstances...

When you restore DFS-R data using the Snapshot-Tool from your storage vendor, everything depends on the way this snapshot is presented to you.
Normally there are two possibilities:
1) the restored snapshot overwrites all previously existing data on the restored volume and is then treated as a normal volume.
2) the snapshot is provided as a new drive letter or a subfolder in the file sytem on the server of your choice, if your Snapshot-Tool supports that.

ad 1) this is VERY dangerous to your DFS-R data: the DFS-R service does not know anything about a restore and treats the older incarnation of the database as a normal database. This is where Microsoft Support comes into place and tells you that you definitely will have data loss. Because DFS-R is a multi-master database-based replication system, snapshot-restores are not supported and WILL break things. This is also the reason why you do not want to use VMware or HyperV-Snapshots on DFS-R Systems - it is unsupported and higly dangerous when used without thinking about the consequences of snapshot-restore of multi-master-databases. Also the restore of a domain controller from a snapshot will destroy your AD, which is also a multi-master-database...
ad 2) this way you can restore file- and folderdata without the chance of corruption, just copy it from the provided new driveletter or subfolder to the original destination like you would do after a redirected restore of DFS-R userdata and it will be treated like a new/modified file and will be replicated correctly. But don't think about restoring the DFS-R database itself, it will brake things...

So what you definitely need is some kind of backup/restore concept, dealing with both "normal" file/folder restore but also disaster recovery.

Which technology used is your choice, but you have to think about the consequences of each solution - this is what I'm doing right now... ;)

To answer your question about the need of the restore of a whole snapshot, i can tell you that normally this is not nescesary. I think every storage vendor already supports virtual snapshots and the presentation of these snapshots to the end-user.
Like EMC - there are some virtual LUNs used only for snapshots, when you tell the storage to show you the contents of a snapshot you don't need the place to hold the whole volume.
The snapshot itself only needs the storage capacity of the delta data created since the snapshot-creation. On our EMC storage this is some internally reserved place on a handfull 450GB FC-HDDs. I cannot tell you any more details here, because I'm not the storage specialist, I got this information from our EMC consultant which configured the whole thing...

VSS... Another Viewpoint...
Windows Server 2008 R2 does have its own software-based VSS built-in to NTFS. One additinal way to restore data is to activate VSS on every volume holding user-data with one or more snapshots created daily. You need some storage capacity extra for this, but user-data restore is really really easy using Shadow-Copies from a volume or even a folder or single file. Even users can restore their "previous versions" of a file or folder if not forbidden by group policy - without even calling 1st-level-support to (let) restore their files...

I really hope I brought some more light into these DFS-R & snapshot stuff, if anybody thinks I'm wrong please correct me.

cheers,
Roman

[rablack]

Roman,

This is very interesting. I am in a similar position, where I have a SAN volume with hardware snapshot capabilties and a number of DFS replicated folders to backup.

As you have suggested, I have already got volume shadow copies enabled to allow specific users to restore older versions of files.

What I am stuggling with is getting BE to actually backup the DFSR folders at all. It doesn't help that selections are confusing for DFSR folders in BE, with you having to select them from the shadow copy components.

So far, I have been able to get BE to backup non-DFSR data in any way I choose, using offhost backup or normal, over-theLAN backup. However, when I have tried to backup DFSR folders, I get errors I have asociated with the fact that the volume presented to BE by the hardware snapshot process does not contain DFSR information, but BE knows the files and folders in that location are replicated and so refuses to back them up as normal files and folders. My backups are hundreds of bytes in size for a 500GB volume!

In terms of restoring data, I think we would be happy with an alternate location only solution, to avoid the DFSR database isues mentioned, and to allow for user confirmation that the restored files are the right ones. On a separate media server with BE 11d installed, I can see the DFS service stopping with event ID 1102 during the backup and restarting again afterwards. We have no issues with resyncing.

With that in mind, how sensible would it be to have BE (on the server with the SAN volume) stop the DFS service with a pre-command, backup the files as though they were normal, and restart the service with a post command? Would that actually work?

Thanks,

Richard