cancel
Showing results for 
Search instead for 
Did you mean: 

Slow duplication from DSSU to tape

alazanowski
Level 5

Infrastructure:

Master/Media Server (the one having the issue)
OS: Windows 2008 R2 standard x64
RAM: 16GB
Fiber: 4gb/s, on an 8gb/s switch
Tape: LTO-5 (8gb/s) , no compression setting set
Network: 2gb/s
DSSU Storage: iscsi NAS via a 2gb/s dedicated conection

Backups direct to the DSSU write around 60-70 MB/s.
Backups direct to the tape write around 55-60 MB/s
Backing up the DSSU to tape (file level) writes around 55-60 MB/s
Duplicating from DSSU to tape writes around 17 MB/s.

Obviously the issue im having here is with duplication.

I have SIZE_DATA_BUFFERS set to 524288 (which is the maximum of the tape drive allowed)
with NUMBER_DATA_BUFFERS set to 32

With default settings the duplication was writing around 1-2 MB/s. With NUMBER_DATA_BUFFERS set to 32 with default size set, the duplications hit 5MB/s. with SIZE_DATA_BUFFERS set, i've finally gotten it to 17 MB/s. However, i can't seem to push it anymore than that.

Anyone have any ideas?
1 ACCEPTED SOLUTION

Accepted Solutions

alazanowski
Level 5

I ended up adjusting the TCP buffers on the nics themselves on both the iscsi NAS and the server having the issue- maxed them out (we've got gobs of un-used memory). This bumped me up to over a 100MB/sec - so obviously that was the issue at hand. So much for having symantec try to adjust tape buffers. I'll write up a more clear doc later. Thanks everyone!

View solution in original post

6 REPLIES 6

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I found this very entertaining article on NBU performance tuning:
http://recoverymonkey.net/wordpress/?p=23

In the 'Responses' section, look for # 6 where a question is asked about Disk Staging performance.
"The other issue we are seeing is during Disk staging. We are using a 1TB store on the EMC (connected to Media1 via SAN), and a 1.5TB store on the iSCSI (connected to Media2 via iSCSI LAN). We are seeing write speeds of about 20-30MBps on BOTH the EMC and the iSCSI disks. Slow. I realize the issue is likely more to do with the network speed than the disk, but this could be part of the issue too? Thoughts?"

See Dimitri's recommendations in # 7.

Ed_Wilts
Level 6

I had a case open with Symantec Engineering for about a year and they were unable to find the problem.  They couldn't identify any issues in our environment - and they came onsite to investigate since we're just down the road from them - and they couldn't identify issues in the code.

It seems to work for some people but many people have the same issues you do - I've seen a bunch of postings to the veritas-bu mailing list.  My environment is very different from yours with SAN-attached disk on Solaris servers but the end result is the same - destaging performance is well below what it should be.

who5e
Not applicable
That recoverymonkey article is amusing but also useful too.

This is such a common complaint and yet there are so many different scenarios and causes. From my experience (my environment is FC 2gb going to LTO4), I bit the bullet and partitioned my DSSU's up a little more rather than having huge luns. All my tiny database backups were mixed in with my huge file server backups and in the end disk fragmentation was terrible. Slow disk read speed also contributed to poor duplication performance. I'm now getting 30-40mb/s duplication speeds. Not fantastic since I can write 100mb/s straight from the DSSU to tape with a file system backup but better than before.

alazanowski
Level 5

I ended up adjusting the TCP buffers on the nics themselves on both the iscsi NAS and the server having the issue- maxed them out (we've got gobs of un-used memory). This bumped me up to over a 100MB/sec - so obviously that was the issue at hand. So much for having symantec try to adjust tape buffers. I'll write up a more clear doc later. Thanks everyone!

teiva-boy
Level 6
Please dont forget to update your post with your solution, I'm intrigued with what you did.

Was it Jumbo Frames that you enabled?

alazanowski
Level 5

I didn't enable Jumbo Frames, although I plan to go talk to my network team in discussing the setup of trunking on the switches with LACP and jumbo frames.

I'm not positive that its a cure-all for everyone, but in each nic on the server itself, there should be options to adjust the buffer size on outgoing/incoming receipts. Ironically though, the solution for one server didn't fix the other server, although the other server is using a similar setup with a deduplication pool instead of a DSSU.

Honestly I feel there is some other process occuring within the software that is causing a delay or not treating an iscsi device as appropriately as it should. When i copied the files using file transfer on windows, it worked extremely fast, but duplication still doesn't work as fast. Could be a windows thing (honestly havent hooked up an iscsi to our linux media servers) but there are a ton of people on the forums who are seeing duplication issues with all sorts of disks, devices, etc.