01-24-2017 01:26 PM
Hi Experts,
I'm facing performance issues in my backup env. I tried to go through with tuning guides but it didn't help much.
my env. netbackup master/media server version 7.7.3
1 master server [virtual machine] OS :windows 2008 standard R2
3 media servers [Dell R510, Dell 720] including 8 drives [Dell tl400] OS windows 2012
I've seen that VM backups are really slow and takes a long time to complete which always delayed the duplication [destaging to tape drive and DR site].
all clients are behind firewall, which means data pass through firewall -ofc it slow down the whole process.
I am looking for some suggestions if i can improve the performance , might to implement backup vlan's or any other idea's.
Thanks,
dpx
01-24-2017 01:47 PM
I would imagine that all the traffic going through the firewall may be killing your performance.
Especially if you are not doing SAN-based VMware backups.
01-24-2017 02:09 PM
The firewall is undoubtedly crushing your performance.
If you can, move to SAN-based VADP backups and bypass the firewall (go under it essentially). You'll still take some performance hit sending control data through the firewall, but it should be relatively small and not generally noticeable.
Another thing to check out is the performance of the master VM, just to be sure you're not suffering any choke points in disk or network on it.
01-25-2017 12:50 AM
As a mitigation - have you enabled the accelerator option in the VMware policy ?
To use accelerator, you must have either MSDP or a accelerator capable OST device.
01-25-2017 01:03 AM
01-25-2017 03:26 AM
How many streams do you have from each ESX host ? In earlier versions more than 4 really affected performance badly.
Besides the other suggestions, using the Resource Limit on the Master could help you.
Have you looked at what your network infrastructure can deliver ? with Iperf, ntttcp or the like ?
Is the backups distributed over all 3 media servers ? Don't know the Dell TL400 drives, but one server can rarely stream 8 modern enterprise tape drives.
01-25-2017 04:01 AM
AFAIK, NBD application protocol is carried over NFC transport protocol, and NFC transport protocol is limited to VMkernel ports, and VMkernel port sessions are throttled to 100MB/s, and decreasing gradients of throughput as the number of VMkernel sessions increase.
So, you might see improved speeds using HotAdd or FC/SAN.
VADP methods via VDDK here:
...and more detail here, with nice examples of how the more that one tries to push via NBD, the slower the combined averages get:
01-25-2017 06:12 AM
One ting to one that's not related to VMware or network at all.
For a tape only tape infrastructure, have you configured SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS. If not, do it right away - you won't regret, I promise
SIZE_DATA_BUFFERS = 262144
NUMBER_DATA_BUFFERS = 256
Documented in the performance and tuning guide 123 : http://www.veritas.com/docs/000004178
For windows create the following files as below. Important - no file extension. Be aware Windows explorer may hide know file extensions. Using notepad (not wordpad) write the value 256 in both of the files.
install_path\NetBackup\db\config\NUMBER_DATA_BUFFERS install_path\NetBackup\db\config\NUMBER_DATA_BUFFERS_RESTORE
No need to re-restart Netbackup. Activity monitors detailed text should tell if the new number of buffers are in effect.
e.g:
01/24/2017 18:44:06 - Info bptm (pid=18016) using 262144 data buffer size
01/24/2017 18:44:06 - Info bptm (pid=18016) using 512 data buffers
SIZE_DATA_BUFFERS should have the same value across all media servers, but not necessary NUMBER_DATA_BUFFERS.
01-26-2017 06:32 AM
Many Thanks Nicolai,
Quick question : are these files only be created on media servers ? or also should be on master server [VM] ?
Thanks,
Aman
01-26-2017 06:58 AM - edited 01-26-2017 07:00 AM
The SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS is for Media Servers or Master/Media Servers - if they have tape storage units (FC, SAS, SCSI, iSCSI).
If your Master Server does not have any "storage units" for itself, then your Master Server is a pure Master Server.
Plus, seeing as your Master Server is a virtual server, then AFAIK it cannot have FC/SAN tape provisioned/presented to it (unless maybe you've done something with tape via iSCSI) - but anyway, assuming that your Master Server does not have a tape storage unit, then those two tuning files will not be relevant to your Master Server.
Does your Master Server have any "disk" based storage units which receive the slow backups?
02-01-2017 03:47 AM
yes, accelerator option is enabled in the vmware policy
02-01-2017 04:09 AM
There have been many suggestions over the last week.
If you don't mind - PLEASE take a minute to read through all of them and then reply to all the suggestions?
@elanmbx and @vtas_chas suggested SAN-based backup
I suggested a media server in the DMZ
@Michael_G_Ander asked about streams and suggested Resource Limits and suggested network infrastructure testing, media server utilization, etc
@sdo suggested alternate transport methods and asked about devices on the master server
@Nicolai suggested buffer settings and asked about Accellerator...
02-01-2017 04:12 AM
I've already implemented SAN based backup and backup is transferred over 4G FC.
02-01-2017 04:15 AM
Good suggestion but i don't have many servers in DMZ to backup.
02-01-2017 04:31 AM
Hi Michael
I agree on point network infrastructure. this is one of issue which i need to resolve by implementing dedicated vlans.
02-02-2017 12:08 AM
I am confused, are these backups running through the network or the SAN ?
vlans will never be faster than the underlying physical LAN infratructure, rather the oppesite as you can have multiple vlans on the physical hardware. But if the backups running through SAN, vlans is not at all in play.
Really think you need to get to know this infrastructure better, and how the backup is flowing through it.. Suggest you talk to the relevant administrators and do a drawing.
Probably been mentioned before, but you need to know:
1. How fast can you read the data from datastore(s)
2. How fast can you transfer data from client(s) (esx host/backup host) to media server(s)
3. How fast can you write data to the storage unit(s).
Remember theoretical throughput values most often is above what you will get in practice, usually you will get somewhere between 50-75% of the theoretical values in practice.
02-03-2017 01:22 AM
Let me clear the confusion:
application , filesystem backups are running on LAN [behind firewall] and VM backup is running over SAN.
implementing new vlan's and configure the same vlan on media servers could be one of solution to avoid firewall.
Quick question, If i would implement Hotadd backup [VM backup host media server] do i need a license for this ?
02-06-2017 12:48 AM
Hi
I couldn't find config directory. do i need to create a new directory and files?
C:\Program Files\Veritas\NetBackup\db>dir /a
Volume in drive C has no label.
Volume Serial Number is D002-BDEF
Directory of C:\Program Files\Veritas\NetBackup\db
2016-11-12 15:27 <DIR> .
2016-11-12 15:27 <DIR> ..
2016-11-12 15:27 <DIR> cloud
2014-03-17 13:52 <DIR> discovery
2016-11-12 15:27 <DIR> images
2013-03-10 05:08 <DIR> media
0 File(s) 0 bytes
6 Dir(s) 103 764 455 424 bytes free
C:\Program Files\Veritas\NetBackup\db>
02-06-2017 01:06 AM
config folder does not exist by default on a media server (only on a master).
You need to create the folder, then create the ...BUFFER... text files in config folder (without any extention).
02-06-2017 12:35 PM
Many Thanks Marianne