cancel
Showing results for 
Search instead for 
Did you mean: 

NDMP restore EXTREMELY slow........in the KB/s rate.

Toddman214
Level 6

Netbackup 7.6.0.4, Windows 2008r2 Master and media servers, DataDomian dd4200 DDOS 5.4.4.1

 

 

Hello all.

 

Because I've been researching this issue, I can see that it has already been beaten to death, but I may be having a different issue. We have an nfs share on our Isilon that data is sent to, to the tune of about 500,000 to 700,000 transactions per day. These are small files, several KB in size. I back this share up to DataDomain each day.  But, the data they need is spread across 3 months, and I cant even restore a day's worth with any performace. Here is a sample in which one of my *faster* attempts took 113 hours to restore 15gb of data. This was 1 days worth of data! I've read about others having issues with slow NDMP restore performance, but they typically mention 5 to 20MB/s restore rates. I'm maxing out at 42KB/s!  They want me to restore the needed folders by month, and then they will use a script to extract the files they need. I've even tried selecting each file individually, but once I click on one file in the BAR, either Netbackup takes 3 to 5 minutes to respond, OR the BAR gui just freezes and greys out. Plus, of the millions of files I'm needing to restore, they only need about 1500 specific ones, so selecting them individually isnt really an option either. I thought maybe it was DataDomain causing the issue, so I recalled a copy of the same data from tape, and the performance appears to be about the same. Any ideas as to what this might be? I've not seen anyone having restore issues this slow, and my data is essentially trapped. I regularly perform restores of data that lives on the same Isilon unit, without issue, so I'm sure the issue relates to the sheer number of files, so I guess I'm looking for a way to deal with that and get my data.

 

NDMP_restore_0.PNG

 

Thanks all!

 

 

Todd

11 REPLIES 11

Nicolai
Moderator
Moderator
Partner    VIP   

I notice the time between "begin reading" and "file transfer" is very high - almost 6 hours.

Well worth digging into why. 

Some questions:

Netbackup fragment size ?

The Isilon connected to the Data Domain direct via ??

Please also enable the NDMP debug log in Netbackup.

Jim-90
Level 6

Tradtionally dedupe engines dislike tiny files - it renders them inefficient.   What sort of dedupe are you getting in DDR for the files.  No mention of tape technology.  A good comparison with tape would be to have no multplexing on tape.   Another method would to test it with a directly mapped/mounted exported share from the DDR.
 

Toddman214
Level 6

Thanks for the info. I'll dig into that.

 

Nicolai, the time between "begin reading" and "file transfer" wasnt actually 6 hours. It began reading on Sept 2 at 3:59pm, and started transferring at 9:47am........on Sept 7.  :(  6 hours would be awesome!

mnolan
Level 6
Employee Accredited Certified

What is the NIC on the media server that is used for incoming datat from the Data Domain?

 

I've seen certain nics with large receive offload enabled cause really slow restores.

Toddman214
Level 6

Nicolai,

 

Fragment size is set at 524288 mb

The Data Domain is fully IP based in our environment. It is not directly connected to the Isilon.

There are ndmp agent logs on the media servers. These?

 

Jim - 90,

 

My dedupe accross the board is about 36x in our environment.

Tape device I also attempted a restore from is a Dell ML6000 series, LTO3.

 

mnolan,

The nics on the media server are Broadcom BCM57098 NetXtreme II

 

 

 

 

Nicolai
Moderator
Moderator
Partner    VIP   

Try running Iperf between master/media server and Data Domain. This will tell you if the connection between those two are healthy or not. You may also consider to open a support ticket with EMC as well.

I have written a small blog on how to setup the Iperf daemon on data domain

http://www.mass.dk/netbackup-guides/iperf-data-domain/

You will need a Iperf application on the backup servers as well. Please note , data domain support Iperf2 but not iperf3.

One hint: if you restore data from the data domain to the backup servers, what the speed then ?

WVT
Level 4
Partner Accredited Certified
If they can give you the specific names of the files in a text file, try using bprestore with a filelist just to get it done. Then you can troubleshoot at leisure.

WVT
Level 4
Partner Accredited Certified
Restore only to like hardware..has to go back to isilon.

WVT
Level 4
Partner Accredited Certified
Okay, so you restore a big blob and they cherry pick files based on some criteria. Perhaps you could give them a bplist output in a text file and they could churn and burn on it externally, provide you with a list, and then you could create a filelist for bprestore. Should be much faster.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Toddman has not been back in 4 weeks....

WVT
Level 4
Partner Accredited Certified
That's okay