New File Server - New Backup Solution - Need advise

[Steppan]

Hello!

Have spent some time reading the cummunity forum and collecting information regarding file servers environments and netbackup but I would really like some more focused help on my particular environment here at the company. I will appreciate any help.

Today we have a VMware Cluster with Dell Compellent LUNs as datastores and some as RDM disks too.

Our file server is a single 2012 R2 VM and has a 12Tb RDM LUN mapped to it (physical mode), where the main folders are shared to users (we have many quotas and file screening policies configured too).

Today we have 11Tb deduped (by windows feature) into something near 7Tb of data in that disk. We have no redundancy (I mean failover), no storage replication, just HA from VMware cluster.

We recently bought a solution with Netbackup (30Tb dedup license), Dell DR4100 backup appliance (+30Tb of capacity), 2-drive LTO7 tapelibrary (24 slots) and licenses to backup VMware Environment, SQL (physical failover cluster), Exchange (physical DAG), Sharepoint and everything else.

The main goal is to take advantage from Netbackup features like Synthetic Full, Dedup, Granular Recovery, OST integration, and to make less use of tape than we have today - this is the reason we bought a backup appliance - and aside that, we still need to deploy a new file server infrastructure.

We have Microsoft EA and SA on our Windows Server Datacenter Licenses and we're planning to use the new Windows 2016 Server Stretched Cluster Feature.

This is a brief of the cluster:

https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-replica/stretch-cluster-repl...

We have 2 datacenters (LAN and SAN connected between them), one with HP Blade and HP 3PAR Storage and another one with Dell Blade and Dell Compellent Storage. Using Windows 2016 Stretched Cluster we plan to use these 2 blade/storage kits in separate datacenters, using VMware clusters on both, hosting two fileserver VM nodes on HP side, two fileserver VM nodes on Dell side.

My questions are regarding the Netbackup features and best practices and the alignment with a good performance and redundant capable fileserver.

1. Will Netbackup Full synthetic feature be compatible with that storage replication windows feature (the hardware is arriving and the software is still in order processing, so let's assume that the installed version will be Netbackup 8)?

2. What would be the best practice in LUN presentation to file server? Considering the size of data (lets assume 15Tb), is it better to present 4 or more LUNs with smaller sizes or one big 15Tb LUN? How will it affect the backup job performance?

3. What is the best practice for backing up the fileserver? Using disk agent into it's OS or using VMware integration from Netbackup and it's CBT functions? What about the restore?

4. We have today windows dedup activated, the Dell DR4100 backup appliance has it's dedup and the netbackup has it's dedup features too. Is it better to disable windows dedup to avoid dedup rehydration? What would be the best dedup feature choice? DR4100 or Netbackup? Source or destination?

Thank you

Steppan

[Michael_G_Ander]

You will need to put a client on the VM, you cannot snapshot RDMs from hypervisors.

Netbackup will probably not be aware of any storage replication, as it is happening at lower level.

Normally my suggestion would be to use accelerator, but not sure it is supported with anything but Netbackup's own deduplication feature.

Usually better to present more LUNs as every "physical disk" gets a queue in the OS, of course there can be other limits.

You should probably consider backup for file/folder restores and backup for disaster recovery of the full volume(s)

Being a RDM snapshot/clone might be worth to look into.

[sdo]

See page 66 of the HCL for NetBackup v7.7.x here:

https://www.veritas.com/support/en_US/article.000025228

...which looks like Dell DRxxxx firmware version 3.0 and above supports Accelerator into their dedupe stack.

And (oddly enough) page 66 (also) of the SCL for NetBackup v7.7.x here:

https://www.veritas.com/support/en_US/article.000025229

...indicates that backups of Windows 2012 R2 NTFS native dedupe volumes is supported, but you cannot perform an "NTFS dedupe aware" backup, only something like a normal backup, but you can use Accelerator (which is a different form of optimisation) and this will greatly reduce the folder walking of a full backup when that full backup is sent directly to the Dell DR4100.

Page 21 of the NetBackup for Virtual environments compatility doc , here:

https://www.veritas.com/support/en_US/article.000006177

...says that NetBackup does not support VM style backups of VMs containing NTFS native dedupe volumes, plus those volumes are on RDMs anyway and so couldn't be backed-up by VM style backups anyway.

.

So, for your big file server with NTFS native dedupe voluems on RDM LUNs... the backup style will be... plain client inside the VM, using an MS-Windows backup policy type, making sure that "Enable optimized backup of Windows deduplicated volumes" is *not* selected on the backup policy, configure an SLP with two stages, a backup stage with say 2/3/4 weeks retention and a duplication stage with say 5/6/7/8 or months or years retention, and finally use this SLP as the storage unit backup target in the policy header and/or schedule definition.

.

As ever, when you're deploying something new... who did the sizing?  RPO/RTO considered?  Did you realize that any restore of 7TB of NTFS dedupe data (i.e. 11TB of actual data) will require a restore target of 11TB... so you may need to two stage this particular restore in a DR scenario.

HTH.

[Steppan]

You're right, I forgot the fact that RDM won't be able to do snapshot.

Great info on the number of LUNs. I've never worked with Netbackup. I came from the HP world, always worked with Data Protector, but had some contact with other platforms like backup exec and arcserve brightstor. Based on that, I don't know what to expect from Netbackup. The objective of this topic is to align my company's infrastructure with the new coming backup solution.

Thanks!

[Steppan]

SDO,

This fact about the restore really annoys me. There was a guy from a third part IT services provider playing a POC for backup exec here before we decided to go with Netbackup and he told me that even if I choose to make a granular restore, the entire content of data (lets say 11Tb not-deduped) needs to be "mounted" on another LUN just to collect a 1Mb granular file. During BE POC we backep up a clone VM from the file server containing the same data of it but using VMDK instead of RDM, using VMware integration. When I tried to do a restore, it says that I would have to choose a "temporary mount disk" or something like that as BE would need to extract the entire data, even if I need to restore one single granular file. Is it true for Netbackup too?

We have many granular restores for this file server (I would say 50 per week) nowadays with data protector. The RTO is not a concern, we have 1 full backup every weekend and 5 enhanced incrementals during the week at 8pm. The concern is being able to restore a 1Mb excel sheet in a matter of minutes without waiting for a long extraction/mapping/mounting time for such big bunch of data when the only thing I want is a tiny file.

Considering the fact that RDM won't be able to snapshot (so we can't use VMware integration) and Michael's approach on presenting more than one unique huge LUN, the best way would be converting the actual one RDM LUN model to a multi disk VMDK scenario, at different datastores (different storage boxes for sure), disable the windows dedup service and use the VMware integration for backing up this VM?

The RPO and RTO will be considered of course. The strategy for file server is to backup it up to DR4100 with synthetic full during the year and 1 or 2 times per year we are going to copy the data from DR4100 to LTO7 tape for archiving. We need to safeguard these tapes for 5 years. We need faster restore and lower dependency on tape handle (our safe where the tapes are stored is located 20km from here in another plant).

Today we have:

Yearly jobs (Permanent)

Monthly jobs (1 year protection)

Weekly jobs (1 month protection)

Daily jobs (7 days protection)

The backup is made first to a Software D2D appliance for faster backup and restore (very tight backup window). Only Yearly and Monthly jobs are copied to tape and deleted from disk. 

Thank you!

[sdo]

1) [Steppan] This fact about the restore really annoys me. There was a guy from a third part IT services provider playing a POC for backup exec here before we decided to go with Netbackup and he told me that even if I choose to make a granular restore, the entire content of data (lets say 11Tb not-deduped) needs to be "mounted" on another LUN just to collect a 1Mb granular file. During BE POC we backep up a clone VM from the file server containing the same data of it but using VMDK instead of RDM, using VMware integration. When I tried to do a restore, it says that I would have to choose a "temporary mount disk" or something like that as BE would need to extract the entire data, even if I need to restore one single granular file. Is it true for Netbackup too?
[sdo] I think that BE might be considered as having the edge here because BE does appear to support "file" level recovery from a VM style backup of a VM containing an NTFS native dedupe volume. But we already know that NetBackup doesn't even support VM style backups of a VM containing an NTFS native dedupe volume, so to answer your question, then "no", NetBackup doesn't do that, because NetBackup can't do that. I think that the real issue here is any file copy out of an NTFS native dedupe volume requires random access and an understanding of the meta-data... so maybe one day... when MS and Veritas get it together on this... then we may see a feature where file level restore out of a backup image containing NTFS native dedupe is possible from a backup image residing on disk (e.g. BasicDisk, AdvancedDisk, MSDP, OST)... but there's several complex layers of abstraction to pick through. Personally I think we should consider ourselves lucky that we can even take a backup in the first place. ;)

2) [Steppan] We have many granular restores for this file server (I would say 50 per week) nowadays with data protector. The RTO is not a concern, we have 1 full backup every weekend and 5 enhanced incrementals during the week at 8pm. The concern is being able to restore a 1Mb excel sheet in a matter of minutes without waiting for a long extraction/mapping/mounting time for such big bunch of data when the only thing I want is a tiny file.
[sdo] Have you considered enabling VSS on a daily schedule, as this may offer you the ability to recover changed/deleted files without having to restore from a backup product. But this very much depends upon the rate of change... and so sometimes it is useful - especially with such large file systems, to provision an additional disk/vmdk/LUN to act as VSS delta space, so that the original base source volume cannot fill up even if the VSS delta space fills up.

3) [Steppan] Considering the fact that RDM won't be able to snapshot (so we can't use VMware integration) and Michael's approach on presenting more than one unique huge LUN, the best way would be converting the actual one RDM LUN model to a multi disk VMDK scenario, at different datastores (different storage boxes for sure), disable the windows dedup service and use the VMware integration for backing up this VM?
[sdo] That sounds like it would fly. And your Dell DRxxxx appliance appears to support the "Accel_VMware" feature, so you should be able to perform accelerated VMstyle BLIB/CBT backups if you used VMDKs instead of hard RDM LUNs inside your big file server.

4) [Steppan] The RPO and RTO will be considered of course. The strategy for file server is to backup it up to DR4100 with synthetic full during the year and 1 or 2 times per year we are going to copy the data from DR4100 to LTO7 tape for archiving. We need to safeguard these tapes for 5 years. We need faster restore and lower dependency on tape handle (our safe where the tapes are stored is located 20km from here in another plant).
[sdo] IMO, I wouldn't consider "NetBackup synthetic full" because this involves post processing to re-munge to re-create a new full backup. I think you would be better off with NetBackup Accelerator - either as plain client inside the VM, or using Accelerator on the VM style backups themselves. What is not clear to me is whether the Dell DRxxxx can leverage FC or iSCSI based shared LUN access to the VMFS datastores on your FC/iSCSI/SAN attached storage array - or whether all VMFS data-store block level backup traffic is going to have to be passed out through the NIC of the ESXi host, to the NetBackup Media Server (element of your master) and then in turn passed on again via LAN to your Dell DRxxxx.  You can think of Accelerator as a bit like a forever incremental.  Only the changed blocks are ever moved, even when doing a "full", and so the volume of LAN traffic from "VMDK/datastore -> ESXi -> NetBackup -> Backup Appliance" should only ever be as high as your actual rate of data change on disk.

[sdo]

Some useful links related to this topic:

https://vox.veritas.com/t5/NetBackup/difference-between-synthetic-backup-optimised-synthetic-backup/...

And a nice doc from Andrew McC, here:

https://vox.veritas.com/t5/Backup-Recovery-Community-Blog/Presentations-from-User-Group-Meeting-24th...

[Steppan]

SDO,

Great info!

About the windows dedup, got the message! We will surely disable it for the sake of backup! We got advise about doing this when that guy was doing the BE POC, he stated that windows dedup would make things difficult.

We've been considering using VSS to support better/faster restore but the concern was that disk grow you talked about. Our file server grows like a monster, ~100Gb a week. There were weeks that it growed almost 200Gb. It is already hard to manage a storage box being eaten alive by just one crazy growing LUN, could you imagine managing two crazy growing LUN? :)

Great approach on the synthetic full myth. I will study that Accelerator stuff. Clearing things for you about DR4100, no, it can't leverage traffic direct from FC LUNs because it just don't communicate with SAN. It only has 10Gbit LAN ports (don't understand why). Our master server will be a VM hosted @ the VMware Cluster within the Dell M1000e blade enclosure. The media server will be a dedicated Dell blade in the adjacent bay of the vmware blades in the same enclosure. The enclosure has two Force-10 switches with 40Gbit LAN throughput at enclosure midplane and the DR4100 will be connected in that Force-10 switches (10Gbit LAN). Dell Compellent storage that presents all datastore LUNs today is in that rack too, and it's connected to the enclosure brocade SAN switches (8Gbit SAN). We hope it can fly.

Thank you all!

[sdo]

5) Steppan - you can almost discard what I said about not-considering "Synthetic Full" because your Dell DRxxxx backup appliance supports both Opt_Synth and Opt_Dup.  It's been so long since I did synthetic backup that I'd forgotten about Opt_Synth.

6) AFAIK - Opt_Synth is only for client style backups, and not for VM VADP style backups.  But I'm sure someone will correct me.

7) Now then, I think the questions re "Accel_VMware" versus "Opt_Synth" are:

8) Which of these two results in less source disk read - i.e. Opt_Synth casuing client to read VMDK, or Accel_VMware causing VADP (in ESXi) to read VMFS datastore LUN (and therefore less sent to the NetBackup Media Server)?   And why?

9) Which of these two results in less CPU usage on the NetBackup Media Server?  And why?

10) Which of these two results in less LAN traffic from NetBackup Media Server to Dell DRxxxx?  And why?

11) AFAIK "client side dedupe" cannot be leveraged with "Opt_Synth" or old schoold Synthetics.  If so, then which of these is better:

11a) Client side dedupe within the client for Accelerator on client style backup...

11b) Client side dedupe with the NetBackup Media Server (acting a VM backup proxy) for Accelerator based VM VADP style backups.

11c) And why?

[sdo]

@Steppan - some hupefully useful links:

12) Accelerator vs. Opt_Synth:

http://www.mrvray.com/2014/01/netbackup-accelerator-for-vmware-vs-simpana-dash-full-the-gory-details...

13) Not quite hitting the mark (and I think factually incorrect in on eplace), but a bit of history:

http://www.mrvray.com/2014/01/netbackup-accelerator-for-vmware-vs-simpana-dash-full-the-gory-details...

14) If you work out that you're dead keen on Opt_Synth for technical reasons, then this may help re configuring for Opt_Synth to a Dell DRxxxx backup appliance:

https://documents.software.dell.com/dr-series-software/3.2.6/administrator-guide/configuring-and-usi...

15) Veritas' own brief description of Opt_Synth:

https://www.veritas.com/support/en_US/article.000092299

16) Veritas' own TN regarding configuring, on the NetBackup side of things the object which points to / describes an OST "Storage Server":

https://www.veritas.com/support/en_US/article.TECH77767

[sdo]

17) This description from Adbul R (I've heard/seen he left Veritas - and you guys should get him back!), provides his insight as to Accelerator versus Opt_Synth (and seems I did remember) and that Opt_Synth is for client style backups and not for VM VADP style backups:

https://vox.veritas.com/t5/Netting-Out-NetBackup-Blog/Frequently-Asked-Questions-on-NetBackup-Accele...

.

So, for me, it's Accelerator all the way... so I'd take Michael's advice and go with several VMDKs and not one big RDM LUN in the VM, do not use NTFS native dedupe, and leverage Accel_VMware for VADP VM style backups with file level restore capabilities.  Fast backup, fast full VM restore, fast file restore, dedupes.  All you need in one place.

[sdo]

It seems that MS' own backup product, SCDPM, also cannot perform ILR (Item Level Recovery) of files from backups of VM which contain NTFS native dedupe volumes:

https://technet.microsoft.com/en-us/library/dn554221(v=sc.12).aspx

...says:

Dedup file system can’t be protected to a secondary DPM server
Issue: After a dedup file system is protected by a primary DPM server, it can’t be protected to a secondary DPM server.
Workaround: None.
Item-level recovery not supported
Issue: If you’re protecting virtual machines that contain deduped volumes, you can’t perform Item level recovery (ILR) from those VHD/VHDX files.
Workaround: Restore the .VHD/VHDX file that contains the deduped volume to a Windows 2012 R2 server that has the Data Deduplication role installed. Mount the .VHD file in disk management, and copy out the desired files.

[sdo · Accepted solution]

If the Dell blades for the ESXi hosts are FC or CNA (i.e. leading to NPIV) connected and the Dell Compellent SAN LUNs really do appear (to native Windows OS) as FC HBA based multi-path LUNs, then you should be able to leverage SAN off-host backups via your bare metal (on a Dell blade) (i.e. non-virtualised) Windows based NetBackup Media Server - and your backups... should... be rapid!   (assuming your lowly storage array can keep up with NetBackup) - but it may transpire that the LAN ingest at the Dell DRxxxx is the bottleneck... but not if you can also leverage client-side dedupe on the bare metal NetBackup Media Server, and use the oodles of CPU that we tend to have on blades so that NetBackup Media Server (acting as client) can fingerprint all those blocks coming up so rapidly from the SAN.

Sound like you have a really nice project on your hands.  Hope it goes well for you.

[sdo]

Steppan - I've given you several kudos for:  initially providing a good level of background info and for taking the time to phrase your question so well... and also for: sticking with that trend and further expanding as we went along.

There's a lesson in here for a great many other posters... which is... if one takes the time to ask a great question... then several of us on here will/may do our best to help you along - we can't promise great answers (obvious really) but those of us that do answer can only ever try to match the original quality of the question.

Clearly, if the question is poor, then the response can probably/usually only ever be nearly as equally poor.

FYI - I've asked some of the others if they can help you with my questions 8) through 11)... because those questions were reaching the edge of my experience and ability to reason out.

And finally... Welcome to NetBackup!   I sincerely hope that you acquire the experience and impression that NetBackup really is a great backup product *and* a great restore product, and that it delivers everything (well almost everything) that you need to help you in your job/role of... protecting your employer's data.

[Steppan]

@sdoThank you for all the effort in helping.

Surely I'll take a time to read all those articles. We received the media server blade and once we receive the other  blade (that will be a VMware host that will expand our cluster and make possible hosting that huge Master Server VM), I'll start the tests with Netbackup (I'm assuming that there's a trial like BE) until we wait for the official license information. We already have a piece of DR4100 (15.9Tb) in production. We ordered another expansion with more 15.9 so I can start the tests without having to wait the expansion (that comes from outside the country and customs will delay everything, same for tape library).

I'll keep you guys aware of the news in the project.

Thanks!

[Steppan]

So, the project is starting to get shape. Hardware is almost all here (only DR4100 D2D appliance expansion is on it's way). All licenses are in hand too.

Tons of other questions about Exchange, Sharepoint, VMware, GRT, Acceleration, Dedupe, Backup to disk, backup to tape (LTO7), SAN or LAN backup, and so on.

Do I create a new topic or stay in this one and change the topic subject?

Regards

[sdo]

Exciting times ahead   :)   Good luck.

New topic/post for each question please Steppan.  Makes it easier to know what we are all talking about.  Thanks.

[Marianne]

@Steppan please close off this discussion by selecting one or more helpful posts and mark as Solution.

As @sdo said - one discussion per topic, please.

In the meantime, herewith links to all relevant NBU documentation:

NBU 7.7.x Documentation: http://www.veritas.com/docs/000094423

NBU 8.0 Documentation: http://www.veritas.com/docs/000116412