Netbackup SPEED problem

Have you tested read speed from disk?

If you have, you would probably figure out that the backup speed is firstly depended on how fast the client can read data from disk. All other factors like network, NetBackup, backup devices, etc can only transfer data as fast as the client can read and produce the data.

The 1st attempt was probably the slowest because none of the data was in cache on client memory or disk cache.
The more you repeated the backup test of the same data, the more data was loaded into cache making each attempt faster.
Probably a bigger gap between attempt 2 and 3?

Doing read tests with bpbkar or 3rd-party tools will probably give you the same results with multiple subsequent reads.

So, getting irritated is not going to help or change anything...

On your backup selection run:

time bpbkar -nocont /<filesystem> > /dev/null 2>&1

See how long it runs for, then find out how big the filesystem is.

Work out how many seconds the bpbkar process was running for, then divide the size of the filesystem (in Mb) by the total number of seconds to get your Mb/second speed.

And, take that same content and put it on your master/media server, then back it up 5 times. I'll bet you the time / speed is consistent for all 5 attempts (unless there is something seriously wrong with your storage devices).

When you ran your tests, were there any other backups happening?

What are you using to measure the throughput?

Also what are you backing up to? Tape, Disk, MSDP?

Hello

Many times it has been observerd that the backup speed can be substantially increased by tuning some NetBackup parameters but most of the times the issue may be related to other parameters like hardware, software and networking component

NetBackup software offers plenty of resources to help isolate performance problems and assess the impact of configuration changes. However it is essential to thoroughly test both backupand restore processes after making any configuraton changes

Please refer to the following performance tuning guide that ay give you some insight into the problems you are facing

http://www.symantec.com/docs/DOC4483

Hope it helps

Sushma

@ revaroo

1. time bpbkar -nocont /<filesystem> > /dev/null 2>&1 couldn't run on this client.

I got error message :

ERR - CONTINUE BACKUP message not received
INF - EXIT STATUS 66: client backup failed to receive the CONTINUE BACKUP message

From  bpbkar log:

9:00:26.007 [8690] <8> bpbkar: WRN - Could not set TCP_NODELAY on fd 1. Errno = 88: Socket operation on non-socket
19:00:26.007 [8690] <8> bpbkar: WRN - Could not set TCP_NODELAY on fd 2. Errno = 88: Socket operation on non-socket
19:00:26.007 [8690] <4> bpbkar: INF - setenv KEYWORD=NONE
19:00:26.007 [8690] <4> bpbkar: INF - setenv STREAM_PID=8690
19:00:26.007 [8690] <4> bpbkar: INF - setenv STREAM_NUMBER=0
19:00:26.007 [8690] <4> bpbkar: INF - setenv STREAM_COUNT=0
19:00:26.007 [8690] <4> bpbkar: INF - setenv STREAMS=0
19:00:26.007 [8690] <4> bpbkar: INF - setenv BPSTART_TIMEOUT=0
19:00:26.007 [8690] <4> bpbkar: INF - setenv BPEND_TIMEOUT=0
19:00:26.007 [8690] <4> bpbkar: INF - setenv RESTARTED=0
19:00:26.008 [8690] <4> bpbkar: INF - BACKUP START 8690
19:00:26.008 [8690] <4> bpbkar: INF - Estimate:-1 -1
19:00:26.631 [8690] <16> bpbkar: ERR - CONTINUE BACKUP message not received
19:00:26.631 [8690] <16> bpbkar:    read:

2. RMAN script.. Backing up to VTL (fujitsu cs800).

------------------------

@Marianne

There is no big gap between 2nd and 3rd try.

We detected  the backup from another system (Media Server/client: HPUX 11.31 ,Netbackup 7.1) .20TB data. At friday we had  started the  backup and speed was ~200-2350MB/s. When it finished at Monday we 've restarted that policy and speed was ~400-500MB/s.(PS we didnt change any parameters).

So I think it not because of cache.

Disk read performance is as follow:

#sudo hdparm -tT /dev/sdb

/dev/sdb:
 Timing cached reads:   13802 MB in  2.00 seconds = 6908.68 MB/sec
 Timing buffered disk reads:  194 MB in  3.01 seconds =  64.47 MB/sec
#
#sudo hdparm -tT /dev/sdc

/dev/sdc:
 Timing cached reads:   13194 MB in  2.00 seconds = 6606.00 MB/sec
 Timing buffered disk reads:  292 MB in  3.10 seconds =  94.11 MB/sec
#
#sudo hdparm -tT /dev/sdd

/dev/sdd:
 Timing cached reads:   13202 MB in  2.00 seconds = 6609.13 MB/sec
 Timing buffered disk reads:  400 MB in  3.05 seconds = 131.27 MB/sec
#sudo hdparm -tT /dev/sde

/dev/sde:
 Timing cached reads:   11660 MB in  2.00 seconds = 5835.60 MB/sec
 Timing buffered disk reads:  
246 MB in  3.45 seconds =  71.31 MB/sec
#
#sudo hdparm -tT /dev/sdf

/dev/sdf:
 Timing cached reads:   9384 MB in  2.00 seconds = 4695.78 MB/sec
 Timing buffered disk reads:   60 MB in  3.08 seconds =  19.47 MB/sec

_________

@ Riaan.Badenhorst

  When we're trying this backup, another jobs performing with normal speed. So storage can be excluded.

______________

It is up to measure what.

Storage speed from it's management.

Netbackup GUI.

disk read speed from command line.

try sudo time bpbkar -nocont /<filesystem> >/dev/null 2>&1 if you havn't already

I interpret the hdparm results as you will get varying speeds depending on the ratio between cached and buffered reads for each run

That said, I do think the network speed is low for a 10 Gb connection, pure network throughput should be about 700-800 MB/s.

Sorry I forgot to use -nocont.

So result is as follow:

# time /usr/openv/netbackup/bin/bpbkar -nocont /data1/dir/test /dev/null 2>&1 > /test/f.txt
INF - Estimate:-1 -1
Edi - 0 0 0 64512 1 16749 root root 0 1421263887 1421263881 1421263881 /
Edi - 0 0 3 64516 10 16877 oracle oinstall 0 1421245331 1383029624 1383029624 /data1/
Edi - 0 0 4 64516 16 16877 oracle oinstall 0 1421261405 1421261396 1421261396 /data1/dir/
Dir - 0 1 7 64516 21 16877 root root 0 1421264095 1421264198 1421264198 /data1/dir/test/

Lff - 1073741824 9 0 9 2 10 64516 26 33188 root root 1073741824 1421264198 1421264198 1421264199 /data1/dir/test/fil16
Edi - 0 0 20971534 5 5 16877 root root 0 1421192356 1408960045 1408960045 /dev/
WRN - /dev/null is a character special file. Backing up the raw partition.
Raw - 0 3 20971537 5 9 8630 root root 0 1399645613 1399645613 1399645613 /dev/null
Dfi - 0 0 20971541 -1 23 292 root root 0 1421264235 1421264235 1421264235 /VRTS_IMAGE_SIZE_RECORD
INF - Total Size:1073741824 + 9GB
INF - EXIT STATUS 0: the requested operation was successfully completed

real    0m32.476s
user    0m0.073s
sys     0m23.554s
# pwd
/data1/dir/test
# ls -lh
total 11G
-rw-r--r--. 1 root root 10G Jan 14 23:36 fil16

10G*1024= 10240MB

10240MB/32.4s =316MB/s

And you missed off the > before /dev/null, but anyway...

If you have logging that will also give you some stats.

Even so this shows us something interesting....this is oracle data am I right? So is it running a live database? On the disk you ran the test against?

Even if the disk is used solely for the backup test but the server is loaded then  you would expect the speed to vary : better yet just stop everything else on the server and repeat the test a few times.

I would expect that to be fairly consistant..but it may not be be possible to perform that specific test.

As an aside, if it is hot ora data then you might be better off running with the Ora agent, but anyways thats somewhat out of scope.

Jim