cancel
Showing results for 
Search instead for 
Did you mean: 

So-so LTO5 read performance

D_Thomas
Level 4

We are restoring a 2.5TB RMAN backup to a linux oracle server from tape (its a copy of a copy of a copy), and so far its going pretty well. We have only done this from MSDP systems in the past.

However, we have two LTO5 drives in the library, and they only seem to read at 500mbps. That's pretty good, and it will be fine for this excercise, but I know the LTO-5 drives can read data faster than that.

I don't know a ton about Oracle or RMAN, but I am wondering if anyone has any insight into why the read speed for these would be stuck at this rate. I tried added a restore data buffers file with 512 in it, but the restore speed is no different.

Thanks
DT

12 REPLIES 12

Nicolai
Moderator
Moderator
Partner    VIP   

Did you also configure NUMBER_DATA_BUFFERS_RESTORE ?

Consider to do a NBU GEN_DATA performance test from the Oracle server. This will establish a baseline for what the infrastructure is capable of. GEN_DATA files directive generate data in memory. Restores travel the path but never to any file system. 

http://www.symantec.com/docs/TECH75213

LTO5 do 140MB/sec uncompressed, you are writing 500mps - it that mega bits ?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Your restore can only be as fast as the slowest component in the data path.

So if the path is something like this:

tape drive -> media server -> network -> client disk

you can understand that slow network or NIC settings on media server or client or slow client disk will cause poor transfer rates.

Slow read/write speed is hardly ever a tape drive or even buffer issue.
If you have bptm log enabled on the media server, you will probably find 'waited for empty buffers .....'.
This means that the media server was quick to read data from tape and fill its own bptm buffers, but that it had to wait for the network and client to empty these buffers before more buffers could be filled (read from tape).

Nicolai
Moderator
Moderator
Partner    VIP   

Multiplexed backup also slow down restore because data from other backup images is discarded. If that the case the tape drive will run with full speed but the restore look like it's slow.

D_Thomas
Level 4

500mbps is 500 megabits per second. 50MB/s ish give or take on either side.

I did configure NUMBER_DATA_BUFFERS_RESTORE to be 512 per your suggestion on another thread, but that made no difference.

The backups are not multiplexed. They are part of an SLP that goes from MSDP to LTO5, so its just one stream. Tape Drives are SAN attached through a 4gbps connection.

The Network and Disk are able to sustain 1 Gigabit per second when two tapes are running simultaneously, so its not the network or client disk as a bottleneck, its definitely the tape drive that is stopping at 50 MB/s.

Now that the RMAN restore is done, I am going to try some Windows File Restores and see if the speed is any better, just to take RMAN/Oracle out of the equation.

Thanks for your replies.
DT

mph999
Level 6
Employee Accredited

Any idea of the block size used for the backup ?

(Look in /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS)

For LTO drives 262144 should give good performance, certainly above 50MB/s for backup speed.

(Depending on number buffers, network speeds etc ...)

 

D_Thomas
Level 4

So I found something interesting, it doesn't look like the tuning parameters are being picked up by the media server. the jobs are showing 30 buffers of 65536:

 

1/8/2014 10:35:10 AM - Info bptm(pid=1656) start            
1/8/2014 10:35:10 AM - Info bptm(pid=1656) using 65536 data buffer size        
1/8/2014 10:35:10 AM - Info bptm(pid=1656) setting receive network buffer to 263168 bytes      
1/8/2014 10:35:10 AM - Info bptm(pid=1656) using 30 data buffer

I have checked to make sure I don't have any trailing spaces in the names or anything of the like. Everything I have read (and remembered) says that restarting services isn't requitred for these settings to go into effect. Any ideas?

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Check that filenames do not have .txt extentions.

Use Folder Options in Windows Explorer to de-select 'Hide extentions for known file types'.

 

Use cmd and 'dir' command to check/verify. Please copy text output of dir command and post here.

D_Thomas
Level 4

Thanks for the suggestion, but that's never the case. I always have hidden files disabled and extensions always visable. When I create the files I see the .txt extention and delete it as part of the process.

But since you asked so nicely:

 Directory of D:\Program Files\Veritas\NetBackup\db\config

01/08/2014  10:38 AM    <DIR>          .
01/08/2014  10:38 AM    <DIR>          ..
01/28/2012  05:16 AM                 0 .tir_bug_fixed
11/24/2013  01:33 PM               364 behavior
09/09/2011  01:17 PM               272 dc
11/25/2013  12:39 PM               151 LIFECYCLE_PARAMETERS
01/08/2014  10:34 AM                 6 NET_BUFFER_SIZE.old
01/08/2014  09:41 AM                 3 NUMBER_DATA_BUFFERS
01/07/2014  09:34 PM                 3 NUMBER_DATA_BUFFERS_RESTORE
01/08/2014  10:32 AM                 6 SIZE_DATA_ BUFFERS
06/29/2012  03:22 PM               714 user_retention
06/29/2012  03:22 PM               719 user_retention.bak
              10 File(s)          2,238 bytes
               2 Dir(s)  61,166,526,464 bytes free

D_Thomas
Level 4

I opened a ticket with support. We'll see what happens. I am doing this tuning in two other datacenters, and its working as it should. Just not this one.

Thanks for all of your input. I will post any info I get back if it gets resolved.

revarooo
Level 6
Employee

Delete the buffer files and re-create them.

D_Thomas
Level 4

So the issue was that I only had this set on the master server, and not on the media server. I am getting speeds of 150MB/s now.

I never remembered putting this on all the media servers indivdually in the past, and the other servers with tape drives in my environments are the Masters, so I guess that's how I missed it.

Thanks again for all the suggestions.
DT

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

So the issue was that I only had this set on the master server, and not on the media server. I am getting speeds of 150MB/s now.

So.... the dir output that you posted was from the master and not the media server?

All documentation about buffer sizes says that the files must be created on the media servers... All tuning is local to each media server. We all assumed that you knew this! Especially since it seems that you have been working with NBU for some time... 

As Nicolai wisely says: 
Assumption is the mother of all mess ups!