09-10-2015 12:03 PM
Netbackup 7.6.0.4, Windows 2008r2 Master and media servers, DataDomian dd4200 DDOS 5.4.4.1
Hello all.
Because I've been researching this issue, I can see that it has already been beaten to death, but I may be having a different issue. We have an nfs share on our Isilon that data is sent to, to the tune of about 500,000 to 700,000 transactions per day. These are small files, several KB in size. I back this share up to DataDomain each day. But, the data they need is spread across 3 months, and I cant even restore a day's worth with any performace. Here is a sample in which one of my *faster* attempts took 113 hours to restore 15gb of data. This was 1 days worth of data! I've read about others having issues with slow NDMP restore performance, but they typically mention 5 to 20MB/s restore rates. I'm maxing out at 42KB/s! They want me to restore the needed folders by month, and then they will use a script to extract the files they need. I've even tried selecting each file individually, but once I click on one file in the BAR, either Netbackup takes 3 to 5 minutes to respond, OR the BAR gui just freezes and greys out. Plus, of the millions of files I'm needing to restore, they only need about 1500 specific ones, so selecting them individually isnt really an option either. I thought maybe it was DataDomain causing the issue, so I recalled a copy of the same data from tape, and the performance appears to be about the same. Any ideas as to what this might be? I've not seen anyone having restore issues this slow, and my data is essentially trapped. I regularly perform restores of data that lives on the same Isilon unit, without issue, so I'm sure the issue relates to the sheer number of files, so I guess I'm looking for a way to deal with that and get my data.
Thanks all!
Todd
09-10-2015 01:57 PM
I notice the time between "begin reading" and "file transfer" is very high - almost 6 hours.
Well worth digging into why.
Some questions:
Netbackup fragment size ?
The Isilon connected to the Data Domain direct via ??
Please also enable the NDMP debug log in Netbackup.
09-10-2015 04:06 PM
Tradtionally dedupe engines dislike tiny files - it renders them inefficient. What sort of dedupe are you getting in DDR for the files. No mention of tape technology. A good comparison with tape would be to have no multplexing on tape. Another method would to test it with a directly mapped/mounted exported share from the DDR.
09-11-2015 10:51 AM
Thanks for the info. I'll dig into that.
Nicolai, the time between "begin reading" and "file transfer" wasnt actually 6 hours. It began reading on Sept 2 at 3:59pm, and started transferring at 9:47am........on Sept 7. :( 6 hours would be awesome!
09-11-2015 02:00 PM
What is the NIC on the media server that is used for incoming datat from the Data Domain?
I've seen certain nics with large receive offload enabled cause really slow restores.
09-14-2015 09:41 AM
Nicolai,
Fragment size is set at 524288 mb
The Data Domain is fully IP based in our environment. It is not directly connected to the Isilon.
There are ndmp agent logs on the media servers. These?
Jim - 90,
My dedupe accross the board is about 36x in our environment.
Tape device I also attempted a restore from is a Dell ML6000 series, LTO3.
mnolan,
The nics on the media server are Broadcom BCM57098 NetXtreme II
10-09-2015 12:08 AM
Try running Iperf between master/media server and Data Domain. This will tell you if the connection between those two are healthy or not. You may also consider to open a support ticket with EMC as well.
I have written a small blog on how to setup the Iperf daemon on data domain
http://www.mass.dk/netbackup-guides/iperf-data-domain/
You will need a Iperf application on the backup servers as well. Please note , data domain support Iperf2 but not iperf3.
One hint: if you restore data from the data domain to the backup servers, what the speed then ?
10-09-2015 06:52 AM
10-09-2015 08:03 AM
10-09-2015 08:08 AM
10-09-2015 08:15 AM
10-09-2015 08:20 AM