NetBackup and 3Par VSS

[Dan_Giberson]

A couple of months ago I posted about backups not crossing mount points to get data and now I can shed a bit more information on the troubleshooting but I am still a bit stumped and would appreciate your advice.

We are migrating a bunch of our VM's over to a 3PAR 7400 which also includes VM's that have RDM's attached. Since the migration, what we have noticed is that the data inside the RDM does not get backed up by Netbackup when using Snapshots. The backup completes with a 0, so I always assume the data is sitting in my tapes or Datadomains...until I try to restore. As soon as I turn Snapshot off, and use Cross Mount Points, the data gets backed up with a problem, however it seems to run 2-3 times slower than a snapshot backup. SInce we are talking 10-15 Tb of data, I need a quicker solution.

I have talked with Symantec about this and they suggested just turning off the Snapshot, however my backups for 7 servers takes approximately 6 days to complete when I do that.  Symantec's other suggestion is the 3PAR VSS writer, however that requires a license we don't have on the 3PAR.

I'm open to any suggestions as the one's i've received from my VAR when we set this up 3 years ago, appear to not be the greatest set of best practices.

Master : 7.6.0.1

Client: 7.6.0.1

[sdo]

I would suggest cease using RDMs... but then maybe you have to use RDMs because you are using some kind of synchronous storage array replication controlled from within the guest OS, and/or are using the RDMs in a cluster config (maybe stretched even too?) ?

A solution I saw once was to integrate storage array (HDS VSP) synchronous replication (TrueCopy) at the VMFS LUN layer, i.e. have the storage array replicate the VMFS datastore LUNs - but that required "horcm" to be integrated with VMware SRM, and of course VMware SRM licensing too.

I'm not sure that I can suggest anything else.

Can you get the 3PAR VSS writer license on a trial basis, and lab test it?

[RiaanBadenhorst]

Hi Dan,

Please ellaborate on how your VM environment/clients/policies are configured.

[Marianne]

I cannot see how you can solve this while using RDMs and not have HP/3PAR VSS for offhost snapshots.

See Appendix D Backup of VMware raw devices (RDM) in NetBackup for VMware Administrator's Guide 

Another possibility may be FlashBackup with client installed in the VM and the path to mount point specified in Backup Selection. See NetBackup Snapshot Client Administrator's Guide  

[sdo]

Are there databases on the RDM 3PAR LUNs?  Or plain Windows NTFS volumes?  Again, why RDMs?

[Dan_Giberson]

Ok....a bit of an update for everyone...right now if I turn Snapshots off in NBU, I am able to get the data, and that's my work around for now.

As for RDM vs. VMDK, the snapshot backup doesn't work on either solution. I have presented a test VMDK to a server and ran the same backup wiht the same result. So the issue is a bit more general in that its all Snapshots on 3PAR.

Sdo...these are just NTFS volumes that house our departmental data user Home drives. Nothing special.

Marianne, the odd thing is that the snapshots have always worked on other storage arrays, it's just the 3PAR that has changed...that's why we are stumped.

As always, any suggestions are appreciated.

[sdo]

1) When you ran the test VM style backup of a VM now using VMDKs, did this test VM server still have RDMs attached?

2) Could you post this from within the OS of the VM which has RDMs on 3PAR...

vssadmin list providers

vssadmin list writers

3) Can you list the backup policy which does a VM backup of a VM which has RDMs which does not work?

bppllist policyname -U

4) And same list for a policy for a VM backup for a VM which has RDMs which does work?

bppllist policyname -U

.

We need to remember that it's not a NetBackup limitation that VMs with RDMs cannot have snapshot based off-host backups - it's a limitation in the VDDK for ESXi by VMware.

I must admit that I still don't understand what it is that you think you got to work when you said "As soon as I turn Snapshot off, and use Cross Mount Points, the data gets backed up with a problem,".  Can you expand upon this in more detail, please?  I really am curious about this topic.  Because, VM VADP level backups of the data on RDM volumes used by VMs is not supposed to work - ever - so I suspect that anything which may have appeared to work, is probably un-restorable.  Do you see what I mean?  I can't see how a VM level NetBackup triggered VADP off-host backup (whether using HotAdd, or NBD, or SAN transport) of the data on the RDM based volumes of the guest VM can ever have worked - unless I'm just out of date and it's all possible now.

And, when you indicate that it worked, did you perform a test restore?  If you diddn't perform a test restore then you can't be sure that it did acutually work at doing what it did, whatever that was - which, like I said, I'm hoping that you can explain in more detail.