Taking too long time to finish backup

[cynthiapinto]

Hi,

I have installed Master & Media server (1single server) with VNB 7.0 & the server is having RHEL OS . Also installed  VNB 7.0 client in the other RHEL server

I initiated 300Gb of backup & took for me almost 7hrs to complete. I use the tape library with LTO4 tapes.

Need a help to bring down the backup window. 

[Marianne]

You need to give us more info...

Firstly, why NBU 7.0?

Symantec recommends 7.1 with latest patches or 7.5 with latest patches.

Tell us more about your setup:

Have you tested disk read speed on the client? How fast can the client read its own data?

What kind of network link between server and client? Have you tested network speed by ftp'ing some data from client to server?

What are you backing up on the client?
Have you enabled multiple data streams for data to be backed up?
Have you increased max jobs per client? (The default is 1)

Is Multiplexing enabled in Storage Unit and Policy schedules?

As you can see, the entire data path from client disk right down to media server tape drive needs to be checked. There are lots of ways to increase backup performance, but you first of all need to examine each component of the data path.

See NetBackup™ 7.0 - 7.1 Backup Planning and Performance Tuning Guide for detailed info.

[cynthiapinto]

Hi Marianne,

We have lincese to download onlt NBU7.0 . I cannot download anyother version.  I have not installed any patches gor NBU.

Client performance/Bandwitdth  is good.The connectivity between the servers & drive is LAN. 

We are firing a nornal Standard backup i.e. a directories.

multiple data streams is not enabled

Max jobs per unit is 1 I have not increased

No multiplexing is not enabled.

[Marianne · Accepted solution]

Please take time and read through the Perf Tuning guide.
Go through the exercise to test disk read speed with bpbkar followed by a network copy, such as ftp. Also see buffer tuning section, but only change buffers once you have analysed default settings in bptm log (details in the Tuning guide).

A quick win is normally to increase max jobs per client to +- 4, allow multiple data streams in policy attribute (assuming that Backup Selection is ALL_LOCAL_DRIVES or multiple filesystems), increase MPX value in STU to 4 as well as MPX of 4 in Policy schedule.

Hope this helps.

PS: Updated license keys are free if NBU maintenance is up to date.

[Mark_Solutions]

In addition to Mariannes excellent points.....

If you are using an LTO4 tape library and backing up straight to tape and you have not done any tuning then as a first step do the following on the Media Server:

Add a file named SIZE_DATA_BUFFERS (no file extension) to the netbackup\db\config directory on the media sever (directory does not exist by default)

Open the file and put the value 262144 inside it

Retry the backup to a new tape (used tapes will maintain their default 64kb block size and will need to be relabelled once they have expired in order to use th enew block size)

See if this helps - With a 1gb network you should be able to back up in about 1 hour

[cynthiapinto]

Thanks Marianne & Mark,

I have done the changes as you said.  I will initiate a backup & come back to you soon

[Marianne]

Please create bpbkar log folder on client and bptm folder on media server before you start.

[cynthiapinto]

Can you tell me how to enable logs & the path to create the log directories

[Marianne]

Create directories under /usr/openv/netbackup/logs.

e.g. on Master/media server:
mkdir /usr/openv/netbackup/logs/bptm

On client:
mkdir /usr/openv/netbackup/logs/bpbkar

No restart is needed for these logs - they just need to exist before backups are started.

[cynthiapinto]

Thanks a lot I will update once the backup is completed.

[cynthiapinto]

Hi,

I had initaited the backup last weekend. My backup window is not reduced.  I have done all the above mentioned changes. 

Please help to over come this. Attached the logs

[mph999]

This could be the issue :

bptm 

12:04:32.323 [4979] <2> write_data: waited for full buffer 1097037 times, delayed 1577467 times

18:05:54.470 [20292] <2> write_backup_completion_stats: waited for full buffer 922531 times, delayed 1329217 times

11:55:58.273 [8893] <2> write_data: waited for full buffer 1065630 times, delayed 1545769 times

12:04:07.965 [20355] <2> write_data: waited for full buffer 1092553 times, delayed 1575131 times

17:50:14.602 [3458] <2> write_backup_completion_stats: waited for full buffer 900244 times, delayed 1287709 times

On the media servers, what values do you have in these files :

/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS

/usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS

I would suggest in size data buffers you put 262144 and in nymber data buffers 64 and see if that makes a diffrence.

If the tuning values are already set, then it would most likely be an issue getting the data to the media server, that is an issue either with the read speed of the disk, or the network between the client and the media server.

Martin

[Marianne]

Please perform the 'bpbkar' test on the client to test read speed from disk.

Extract from Performance tuning guide:

Measuring disk performance with bpbkar

Use this procedure to measure disk I/O using bpbkar

1. Turn on the legacy bpbkar log by ensuring that the bpbkar directory exists.

/usr/openv/netbackup/logs/bpbkar

2.  Set logging level to 1.

3. Enter the following:

/usr/openv/netbackup/bin/bpbkar -nocont -dt 0 -nofileinfo -nokeepalives /u01/app/oracle/Backup > /dev/null  

4. Check how long it took NetBackup to move the data from the client disk:

UNIX: The start time is the first PrintFile entry in the bpbkar log. The end

time is the entry "Client completed sending data for backup." The amount of

data is given in the entry "Total Size."

The bpbkar command will probably run for quite a while. You can check client's process table with 'ps -ef |grep bpbkar'. When bpbkar process has finished, rename the log to bpbkar.txt and post here as file attachment.

[cynthiapinto]

Marianne,

I have done the above said. I have attached the bpbkar logs.

Now the client,master,media servers are in same VLAN & connected to the same switch. Still backup window is not reduced.

[Marianne]

We can see that you started the bpbkar test at 18:00:25.
It finished at 19:02:30.

So, about an hour to read ± 263.8GB.
We now know that read speed from disk is good.

Next step is to test network speed. 

Find a file under /u01/app/oracle/Backup/ that is at least 1GB in size.
Ftp this file to the master/media server.

Let us know what the result is.

[cynthiapinto]

Thanks Marianne for all the inputs you have given to me.

I have move the Master/Media/Client to the same VLAN & to same switch.  now the backup s going on really good. Could finish 250Gb of data withing 30 Mins. 

Thank you all for your support.