Netbackup media server virtualisation, multi-tenancy etc - best practices.

[ArisFC]

Hi,

I have a number of netbackup questions and as you will notice the reccurring "theme" in those questions is around virtulisation of media servers as well as multi-tenancy/security.

1.What is Symantec’s recommendation on Virtual vs. Physical media servers in relation to scalability and performance. Comparing a media server VM and a physical media server with similar CPU/mem etc should we expect similar performance? Note: I am fully aware that for tape backups we need physical.

2. We have customers with mixed physical/virtual (VMware) environments that require backup, and use VAPD to backup the virtual part. Assuming we make the choice to use media VM's rather than physical media servers to backup those virtual environments, can/should we use the same media VM’s to backup the physical environments of those customers too (note: bare metal customer servers to be backed up in our case are typically RHEL 5/6 and Windows 2008R2/2012).

3. Which of the two is preferable: to increase the number of media VM’s rather than increase the resource in each media VM when the number of clients we backup grows.

4. Can a physical media server (either RHEL6 or 2008R2/2012) be used to backup to tape via FC, and the same media server to also connect via Ethernet and backup via IP OST to a different disk-based target (EMC Data Domain for that matter).

5. Media server and multi-tenancy in relation to performance/scalability and security: If I look after a multi-tenant environment requiring backup where customers need to be kept separate, is it preferable to use a single media server (OST-based) for all customers, or would I be better off virtualising the media server and dedicating a media VM per tenant/customer? Note: the master has to be shared amongst all those customers.

Many thanks in advance.

[Nicolai · Accepted solution]

Lots of question - I expect you will get a lot of answers from different users with different views in this thread :D

1: Symantec/Veritas don't have a recommendation. In my view using VM's or physical is a question of performance, price and usage. 

2: You can if performance allow. But do some math on the number of times traffic passes forth and back. Also beware that debugging VM performance issues can be very hard because of the virtual layer.

3: In my view it don't make sense increasing number of VM when the underlying hardware is maxed out.

4: yes, no problem

5: I use physical server for multiple customers. I do not have a security problem with that, but please spend some time on the number of VM you may need when scaling. Also do you have secuirty staff in house with diffrent views. Firewall where traffic pass ?

My own comment: I use physical servers for everything, but gain I work large scale (last I checked the counnter it said 700TB/24h). I do have master media servers VM in small remote location. But not in the primary data centers. The bottom line in this is - fine to use VM when workload permits - but dont force VM just becuase the strategy say "VM". A mixed match may be what you need.

[ArisFC]

Thanks Nicolai much appreciated. I guess Q3 assumes that the hardware is not maxed out and the number of clients/data to backup is the same in both cases. I was looking for recommendation if I had to choose between multiple smaller media VM's vs. fewer bigger ones, i.e. which is preferable. Theoretical example (don't worry too much about actual figures, this is just conceptual): 4 media VM's of 32GB vRAM and 2vCPU each, or, 2 media VM's of 64GB vRAM and 4vCPU each?

Thanks.

[sdo · Accepted solution]

I'm not aware of any guidelines.  My own gut feel is to go with with fewer but more gruntier virtualised NetBackup Media Servers - with more vCPUs there should be less process context switching, and more chance that a process will continue with CPU 'quantum time slicing' for longer (i.e. stay 'current on CPU' and not be moved off to a 'waiting for compute' resource wait state).

.

Will these 'Media Server' VMs also be doing de-dupe?  I can't quite put my finger on it, but it just doesn't sound right to me.  I'm left with a disquieting sense of unease about supposedly big'n'beefy virtualised Media Servers.  IMO de-dupe needs a really really fast 'bus' between CPU and RAM - in de-dupe land there is just so much data moving around between RAM and CPU (for hash fingerprinting (and encryption at rest)) that, IMO, only a physcal server can cut the mustard.  Here's a question... Why do Symantec not 'sell' a 'VM appliance'?  I bet they've checked it out... easy package... easy sell... easy money... but poor performance... unhappy customer... bad press.

And - why is there no 'word on the street' (i.e. on here, in this forum) regarding success stories of large installations of virtualised 'backup data movers'.  In fact, why do the other backup vendors also push tin and not pre-packaged VM appliances?

The more I think about this, the less I like it.  And, I've never seen any whitepapers extolling the virtues of virtualised media servers.

[Nicolai]

Valid point SDO !

Also ESX servers do Vmotion to optimize ressources - this will impact MSDP performance.

In general MSDP like faster CPU over multiple kernels.

[ArisFC]

Thanks guys.

sdo - To answer your question: no dedupe on Media server, we dedupe on the Data Domain only. But yes, I do hear your concerns about virtualising the media server. Very interesting points. The truth is we are struggling with anything physical that ends up as "cost of sales" for the customer... On the other hand, we don't want unhappy customers and bad press either! A catch 22. It will be interesting to hear experiences from other people, good or bad.

Many thanks again!

[Nicolai]

Sell backup as a service - this way customer do not have to worry about license and hardware cost. 

[sdo]

As I'm sure you are aware, it is usually quite (very?) difficult to scale a real, valid, and appropriate test environment.  And you don't want to make a commitment to customers without knowing whether something is achievable and profitable, and you don't want to spend unnecessarily, nor end-up having to soak-up expenditure (loss) from later finding out that you have to implement in physical servers the functionality that a virtual server might not be able to cope with.

What we really need is... a few forum users to come forward and provide a little bit (but not too much) of detail re their virtualised media servers - and their sustained data transfer rates.  And then any scenarios which look similar to your plans - you might be able to get some traction for deeper detail via private messaging on this forum.

I'm sure many of us would dearly like to hear more about virtualised media servers - to get a feeling on what is, and what is not, possible.

.

So, your virtualised media servers will not be storage servers, and will simply act as data concentrator/movers?  i.e. no OST, no de-dupe, and simply acting as a conduit along this lines of:

Data Storage:      Spindles -> SAS bus -> SAS ASIC -> Parity Groups -> Storage Array bus(es) -> FC ASIC -> front-end SAN ports...

Transport:            -> FC cable -> SAN switching -> (trunked ISLs? multi-hop switching? zoning...) -> FC cable...

Server:                 -> HBA -> VMFS driver -> ESXi hypervisor -> vHBA -> vSCSI -> vRAM -> bpbkar -> VMDK file system stream handler (for NTFS, EXT4, etc) -> vCPU -> bptm -> vRAM -> vTCP -> vNIC -> ESXi hypervisor -> pNIC...

Transport:            -> LAN cable -> LAN switching (bonding? VLAN routing? multi-hop edge to core to edge? bonding?) -> LAN cable...

Backup Storage:  -> NIC -> RAM -> CPU -> SCSI -> RAID -> Parity groups -> spindles (i.e. the landing zone   (and then later de-duped?)).

...plus all the catalog/index comms (of the contents of the snapshotted VMDKs) all out to the master server.

...plus multiple instances, some overlapping, some bottlenecking... of all of the above...

.

I think you could do some calculations... if... you can find out what the actual achievable sustained data transfer rates are within a VM within ESXi - i.e. for all points in the "server" section above.  There has to somewhere in the path be some loss/detriment due to virtualisation?  But where?  And how much?  And how many times?

The 'points that I list' within the the 'server' section are not meant to be exhaustive.  True detailed capacity planning will go down to the level of clock frequency and bus width on the PCIe lanes within the server hardware itself, and data rates for all levels and contact points and protocol conversions and cables and carriers and stacks and drivers and components, etc..etc...  Whether you can get to that level, I can't comment on, because it may all be hidden/abstracted by the 'hypervisor' (i.e. vSphere ESXi).

[sdo]

Good point - we're doing that too, selling BaaS internally - helps smooth out costs and scaling - and should leverage economies of scale - but, from the little bits that I've seen, requires very careful fiscal management.

[Nicolai]

There are also information to get from Abduls VMware blogs:

https://www-secure.symantec.com/connect/blogs/nuts-and-bolts-netbackup-vmware

[ArisFC]

This has gone quiet.... :(

Anybody any real-life experiences with virtual media servers?

Thanks.

[Nicolai]

Do yore own lab testing. Even if I told you I could 500MB/sec, it does not necessary mean you could do it too in your environment.

[ArisFC]

Yes definitely. Planning to do that.

[Jaime_Vazquez · Accepted solution]

I would suggest a look at page 5 of tis document:

Statement of Support for NetBackup 7.x in a Virtual Environment (Virtualization Technologies)   
http://www.symantec.com/docs/TECH127089
 

For all Virtualized environments, there is the inherent overhead of the hypervisor layer between the guest OS and the VM installed software. When using phyusical hardware, the OS and all applications are running directly on the physical hardware itself.  In a virtualized envieonment, the OS running on the hardware is the virtualization OS, pplus hypervisor which translates the normal OS API calls from what the guest OS thinks it is using to what the hardware itself is using,. The code path length is longer to address the API calls.

Running on a physical server also means it is not sharing the hardware resources amonst other applications or guest OS instances.