Duplication from disk to tape slower than expected

[Mnietek]

I have a multi-stage backups set up. And all is OK except for the performance of the duplication to tape.
My setup consists of a W2k3 server with BE2010 server. I have an FC array on which I hold the B2D devices. And from them I'm duplicating to a TS3100 tape library with SAS connected LTO4 drive.
I'm aware that I should not expect filling whole bandwidth and the jobs running at theoretical peak speeds but 400-500MB/min is kind slow for such setup.
I had hardware compression enabled on the drive but currently I turned it off. I'm not sure whether it applies to any jobs from now on or just new tapes.
I have no idea where to start troubleshooting.
Straight-to-tape backup jobs seem to run with the same speed.
Logs show that I used to have duplicate jobs that ran faster than those 500MB/min that I mentioned, but they were all "small" jobs (not more than few Gigs). Bigger jobs (more than 10 Gigs of data) tend to not exceed that boundary.
Or maybe I'm expecting too much? What throughput do you achieve on LTO4/SAS?

[ZeRoC00L · Accepted solution]

I have seen a setup last week with LTO4 (2Gb fibre connected), and it was running around 2000MB/min.
Once I also had a setup with backup2disk2tape where the backup to tape was running slower than expected, and I did a full defrag of the disks (that took some time) and then the backup was running much quicker. So it can be a starting point to start with.

[sksujeet]

As you said the data when goes directly to tape it writes fast but when duplication is done from disk to tape it is kind of slow. The FC is faster communication as compare to SAS still the backup to SAS with LTO4 should be 1000 mb/min as I have seen in many cases. But it depends upon the network utilization as well as how devices are setup.

If you duplicate the same data from disk to disk how much speed you get or may be from tape to disk.

One very important thing is that hardware compression will make it slow so please try without that option to get a better idea 

[JoaoMatos]

Hi,
how is performed the connection between disk and tape. As I understand, it is a perforrmance problem between disk backup device and tape.

Regards,
JoaoMatos

[Mnietek]

According to your suggestions I checked some things and found out that:
- The B2D files were indeed fragmented. Even copying from one device to another was quite slow so I'll erase old B2D files, defragment and see if it helps.
- I didn't have the "allocate all space for B2D fragment" options checked. Thus the fragmentation. I checked it. We'll see if it helps
- Since I do the backup via 100Mbps network, the direct-to-tape backup speed seems to be restricted by that speed, so the measurement is indeed not very reliable as means of testing the SAS throughput limit.

When I will have performed the defrag (which will be slow, I suppose since it's over 2TiB of data), I'll let you know if it helped.

[teiva-boy]

Since you're on Win2k3...  You HAVE TO partition align the volume that holds your B2D data.  Doing so, will up your throughput more than 10%, in some cases closer to 20%.

Add to that, your RAID card (if equipped) could use larger block sizes, and possibly even your NTFS formatting could use some tweaking, perhaps try a larger formatting sector size?  But partition aligning is key. This is not an issue on Win2k8

[ZeRoC00L]

Nice, let us know the outcome of the results after the defrag.

What i have done is scheduled a weekly defrag on the backup2disk folder in the weekend. Now the backups keep on running fast.

[Colin_Weaver]

If you backup a reasonable amount of data (a small amount will not be representative) directly from a local harddisk on your media server  to the tape device and compare that speed with the speed seen in the duplicate job what sort of speed differences do you see?

[Mnietek]

I'm not sure what you mean by "partition align the volume".
And yes, I'm thinking about reformatting the volumes with larger block size.
(defragmentation seemed to be taking forever so I decided to move the files elsewhere and then move them back; it should be faster than waiting for the defragmentation process)

[Ken_Putnam]

If you decide to do this,  be sure to defragment before you move them back.  With much less data, the Defrag should be much faster, and you will have a whole lot more contiguous disk space to prevent future fragmentation

[Mnietek]

While doing my moves/formats/defrags I noticed two strange things:
1. I had many B2D*.bkf files that were not seen in either device nor media manager in BE. When I chose to inventory the B2D device they got imported as BE/NT Backup media. I decided to delete the media and remove the files from disk. It seems than deleting media in BE is not enough if you want to free disk space. You have to manually remove the files.
2. Even though I have the "Allocate maximum size for B2D files" option enabled in my B2D folders definitions, the *.bkf files are not getting allocated with maximum size. And they're getting fragmented even though I have more than 50% of volume space free (and contiguous).

Anyway, so far I only had one duplicate job from the freshly organized device so I can't yet tell what was the impact of the aforementioned operations (the job was quite small - some 15G, so I'm not yet sure how the big ones will behave).
Of course when I'll have completed my reorganization, I'll post the results here.

[Mnietek]

Yes. Good point, but I moved all the data from the volume and formatted the volume to have bigger block size, so in my case defrag wasn't needed here.

[Mnietek]

Ok. I finally completed my formats and defrags. I ran a big backup to disk and then duplicate. And, surprise, surprise, the duplication speed was 10 times what it used to be. It jumped from around 500MiB/s to a whooping 5500MiB/s. I\m yet to see whether it stays this way but it seems that the B2D device files fragmentation was indeed the cause of the duplication slowness.
So I now set up two scheduled jobs to defrag my B2D disks and hope the performance stays high.
Thanks for all the hints.

[teiva-boy]

I hope you did a disk alignment..  It could further bump up the speed.  This does not apply to Win2k8 or higher.  Only Win2k3 or lower.
http://technet.microsoft.com/en-us/library/aa995867(EXCHG.65).aspx

This is also strongly recommended for ALL Exch and SQL servers too, let alone any volume you can get your hands on.  Boot volumes take a bit more work though....

[ZeRoC00L]

Glad to hear that it is running faster now.

What you can do is schedule a weekly defragmentation of the B2D volume.
Probably on Sunday your server is quite idle, so I suggest to schedule it to run then.
If it is running on a weekly basis it will not take that long as it did now.