cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup performance issues

dpx_99
Level 4

Hi Experts,

I'm facing performance issues in my backup env. I tried to go through with tuning guides but it didn't help much.

my env. netbackup master/media server version 7.7.3

1 master server [virtual machine]  OS :windows 2008 standard R2

3 media servers [Dell R510, Dell 720] including 8 drives [Dell tl400]  OS windows 2012

I've seen that VM backups are really slow and takes a long time to complete which always delayed the duplication [destaging to tape drive and DR site].

all clients are behind firewall, which means data pass through firewall -ofc it slow down the whole process.

I am looking for some suggestions if i can improve the performance , might to implement backup vlan's or any other idea's.

Thanks,

dpx

 

 

20 REPLIES 20

elanmbx
Level 6

I would imagine that all the traffic going through the firewall may be killing your performance.

Especially if you are not doing SAN-based VMware backups.

The firewall is undoubtedly crushing your performance.

If you can, move to SAN-based VADP backups and bypass the firewall (go under it essentially).  You'll still take some performance hit sending control data through the firewall, but it should be relatively small and not generally noticeable.

Another thing to check out is the performance of the master VM, just to be sure you're not suffering any choke points in disk or network on it.  

Charles
VCS, NBU & Appliances

Nicolai
Moderator
Moderator
Partner    VIP   

As a mitigation - have you enabled the accelerator option in the VMware policy ?

To use accelerator, you must have either MSDP or a accelerator capable OST device.

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

How about a media server inside the DMZ?

 

How many streams do you have from each ESX host ? In earlier versions more than 4 really affected performance badly.

Besides the other suggestions, using the Resource Limit on the Master could help you.

Have you looked at what your network infrastructure can deliver ?  with Iperf, ntttcp or the like ?

Is the backups distributed over all 3 media servers ? Don't know the Dell TL400 drives, but one server can rarely stream 8 modern enterprise tape drives.

 

 

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

sdo
Moderator
Moderator
Partner    VIP    Certified

AFAIK, NBD application protocol is carried over NFC transport protocol, and NFC transport protocol is limited to VMkernel ports, and VMkernel port sessions are throttled to 100MB/s, and decreasing gradients of throughput as the number of VMkernel sessions increase.

So, you might see improved speeds using HotAdd or FC/SAN.

VADP methods via VDDK here:

https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkDataStruct.5.5.ht...

...and more detail here, with nice examples of how the more that one tries to push via NBD, the slower the combined averages get:

http://www.mrvray.com/2015/06/getting-to-know-the-network-block-device-transport-in-vmware-vstroage-...

Nicolai
Moderator
Moderator
Partner    VIP   

One ting to one that's not related to VMware or network at all. 

For a tape only tape infrastructure, have you configured SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS. If not, do it right away - you won't regret, I promise Smiley Happy

SIZE_DATA_BUFFERS = 262144 

NUMBER_DATA_BUFFERS = 256

Documented in the performance and tuning guide 123 : http://www.veritas.com/docs/000004178

For windows create the following files as below. Important - no file extension. Be aware Windows explorer may hide know file extensions. Using notepad (not wordpad) write the value 256 in both of the files.

install_path\NetBackup\db\config\NUMBER_DATA_BUFFERS install_path\NetBackup\db\config\NUMBER_DATA_BUFFERS_RESTORE

No need to re-restart Netbackup. Activity monitors detailed text should tell if the new number of buffers are in effect.

e.g:

01/24/2017 18:44:06 - Info bptm (pid=18016) using 262144 data buffer size
01/24/2017 18:44:06 - Info bptm (pid=18016) using 512 data buffers

SIZE_DATA_BUFFERS should have the same value across all media servers, but not necessary NUMBER_DATA_BUFFERS.

Many Thanks Nicolai,

Quick question : are these files only be created on media servers ? or also should be on master server [VM] ?

Thanks,

Aman

 

 

sdo
Moderator
Moderator
Partner    VIP    Certified

The SIZE_DATA_BUFFERS and NUMBER_DATA_BUFFERS is for Media Servers or Master/Media Servers - if they have tape storage units (FC, SAS, SCSI, iSCSI).

If your Master Server does not have any "storage units" for itself, then your Master Server is a pure Master Server.

Plus, seeing as your Master Server is a virtual server, then AFAIK it cannot have FC/SAN tape provisioned/presented to it (unless maybe you've done something with tape via iSCSI) - but anyway, assuming that your Master Server does not have a tape storage unit, then those two tuning files will not be relevant to your Master Server.

Does your Master Server have any "disk" based storage units which receive the slow backups?

yes, accelerator option is enabled in the vmware policy

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

There have been many suggestions over the last week.

If you don't mind - PLEASE take a minute to read through all of them and then reply to all the suggestions?

@elanmbx and @vtas_chas suggested SAN-based backup

I suggested a media server in the DMZ

@Michael_G_Ander asked about streams and suggested Resource Limits and suggested network infrastructure testing, media server utilization, etc

@sdo suggested alternate transport methods and asked about devices on the master server

@Nicolai suggested buffer settings and asked about Accellerator...

 

I've already implemented SAN based backup and backup is transferred over 4G FC.

Good suggestion but i don't have many servers in DMZ to backup.

Hi Michael

I agree on point network infrastructure. this is one of issue which i need to resolve by implementing dedicated vlans.

I am confused, are these backups running through the network or the SAN ?

vlans will never be faster than the underlying physical LAN infratructure, rather the oppesite as you can have multiple vlans on the physical hardware. But if the backups running through SAN, vlans is not at all in play.

Really think you need to get to know this infrastructure better, and how the backup is flowing through it.. Suggest you talk to the relevant administrators and do a drawing.

Probably been mentioned before, but you need to know:

1. How fast can you read the data from datastore(s)

2. How fast can you transfer data from client(s) (esx host/backup host) to media server(s)

3. How fast can you write data to the storage unit(s).

Remember theoretical throughput values most often is above what you will get in practice, usually you will get somewhere between 50-75% of the theoretical values in practice.

 

 

 

 

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue

Let me clear the confusion:

application , filesystem backups are running on LAN [behind firewall] and VM backup is running over SAN.

implementing new vlan's and configure the same vlan on media servers could be one of solution to avoid firewall.

Quick question, If i would implement Hotadd backup [VM backup host media server] do i need a license for this ?

Hi

I couldn't find config directory. do i need to create a new directory and files?

 

C:\Program Files\Veritas\NetBackup\db>dir /a
 Volume in drive C has no label.
 Volume Serial Number is D002-BDEF

 Directory of C:\Program Files\Veritas\NetBackup\db

2016-11-12  15:27    <DIR>          .
2016-11-12  15:27    <DIR>          ..
2016-11-12  15:27    <DIR>          cloud
2014-03-17  13:52    <DIR>          discovery
2016-11-12  15:27    <DIR>          images
2013-03-10  05:08    <DIR>          media
               0 File(s)              0 bytes
               6 Dir(s)  103 764 455 424 bytes free

C:\Program Files\Veritas\NetBackup\db>

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

config folder does not exist by default on a media server (only on a master).
You need to create the folder, then create the ...BUFFER... text files in config folder (without any extention). 

Many Thanks Marianne