on 05-07-2010 02:00 AM
Recently, I embarked on an exercise to see if changing the properties of the tape drive has any effect on the speed of my backups.
I am using BE 12.5. I back up my data to a SAS-attached HP tape library with a LTO4 tape drive and I write to both LTO3 and LTO4 tapes.
First, I unchecked the single block mode parameters to enable buffering. This has no effect on the backup time.
Next, I increased the high water count which is the number of buffers that is filled before the data is written out to the tape drive. This sped up the backup a little, but not significantly. This indicates that my system is unable to stream enough data to tax the tape drive.
Since I have 4GB of memory on my media server, the next parameter to change is the buffer size and/or the block size. Before changing these parameters, I checked with HP as to whether my tape drive supports buffer and block sizes greater than the default 64k and whether bigger sizes will speed up things. As expected, HP is rather non-committal. They just said that it support bigger block and buffer sizes, but they have no specific recommendations as to which size to use. It is all up to the backup software.
When I tried to change the block size, I get dire warnings from BE that bigger block size may not be support and that I should test that I can restore the data. So I changed the buffer size first to 256k. What a dramatic effect this has: The job rate for my job which writes to LTO4 tapes jumped from 2 GB/m to 2.7 GB/m, saving an hour! Strangely, this change had minimal effect on my job which writes to LTO3 tapes.
I thought everything is fine until a few days later when my job failed with this error
An inconsistency was encountered on the storage media in HP 3.
V-79-57344-33994- The data being read from the media is inconsistent.
I thought this was a one-off failure due to dirt on the tape, etc. A couple of similar failures convinced me otherwise. Looking closer at the job logs for the failed jobs, I noticed that all of them were spanning tapes. For the jobs that did not span tapes, there were no failures. I set the buffer size back to the default 64k and the problem disappeared.
However, I am not about to give up on the improvement in backup time. I changed both the block size and buffer size to 256k. After the change, I run a small backup job with verify to check that the tape drive can handle the new block size. It ran successfully. The backup time for my LTO4 tapes with these new settings was the same as when the buffer size alone is 256k, i.e. 2.7 GB/m. A few days later, my job spanned to a second tape and there was no error.
Next, I changed the block and buffer sizes to 512k. This has no effect on the backup timings, so I reverted back to 256k. There is no point in tying up extra memory for nothing.
These are the final settings that I am using are shown in the figure above.
Before I use these settings permanently, I checked that I am able to read the tapes that were previously written with 64k blocks. I ran a small restore job and it ran fine. The tape drive was able to read the 64k blocks without me setting the block size to 64k first.
I also check with Symantec Technical Support as to whether there is any report of the problem that I encountered when my job spanned tapes. They said that there is no report of such a problem. I leave it at that. This kind of problem occurs at a fairly low level between the backup software and the tape device. Talking to the technical support staff at either Symantec or HP would be futile. This problem can only be traced at the development lab where it is possible to see the actual data passing between BE and the tape device.
For my tape drive,
If you are using a SAS-attached LTO4 tape drive, my settings can probably be used as it is.
For other types of tape drives or connection, you would need to test what parameter changes work for you. If you do want to tune your tape drive parameters, I suggest that you take baby steps and observe the results. If anything adverse happens, revert back to the old settings. The sequence of changes that I went through will be a good sequence, i.e.,
It might pay to check whether you can restore from tapes that were written with a larger block size than current. My understanding you cannot restore from tapes that were written with a larger block size, you can only restore from tapes that were written a smaller block size.
This was resolved in Backup Exec 2010 R2 in the DDI20100915. As you indicated, this only occurred with Hardware Encyption + spanned tapes.
If you upgrade to Backup Exec 2010 R2 as well as the 0915DDI, you should no longer see this issue.
For more information see:
I am setting up LTO4 now for test lab and will be backing up small files primarily. Find this useful reading for prep.
Here's some more information on the "dire warning" that PKH points out:
The reason we try to strike that type of fear when altering these settings is that it is very important to remember what you're changing the settings to. If you uninstall Backup Exec or move Backup Exec to another server, you need to recall exactly what these settings were in order to run inventory and catalog jobs on the tape. It's not possible for the tape drive to read the tapes unless these settings are the same as they were when the tape was written, and there’s no real mechanism available to determine the settings if all you have is a tape. It is highly recommended to keep these settings written down somewhere safe if you decide to go with non-default settings.
The reason that vendors such as HP as well Symantec do not have a ‘best performance setting’ for all LTO4 drives, is there are a lot of factors that can be at play in regards to backup performance. What makes PKH’s backups run fast may have no impact at all for you, and you may actually find better performance with lower settings. The HBA, the data that you are backing up, as well as the speed at which your data arrives to the tape drive can all significantly impact (good or bad) your backup speed. Not all HBA’s support these higher block and buffer sizes either. It’s not always black and white as to what settings are the best.
Small files can be a nightmare for some LTO4 drives. If data does not arrive to tape drive fast enough, the drive will have to stop and reposition itself if it has to wait for more data to arrive, an issue known as the shoe-shine effect. Finding the best performance setting helps decrease the frequency of this, thus making the tape drive write faster. This also allows the drive to last longer, as the mechanics inside the drive do not stop and start as much, decreasing the wear and tear from daily use.
Please feel free to share your results from your analysis, and good luck with your testing!
'Small files can be a nightmare for some LTO4 drives '
Are there any particular manufacturers or models under some?
It really affects most LTO4 drives. LTO4 is rated to accept 120 MB/s (or 432 GB/hr). If you're not feeding that much data to the tape drive, it's possible to see some degree of shoe-shining.
There are some disk arrays that aren't capable of feeding data to the tape drive that fast. Smaller files always tend to backup slower than larger files as well.
Almost every drive has technology built into it to handle buffering of data so that shoe-shining has as minimal impact as possible. HP uses a technology called Data Rate Matching, which is designed at limiting this. IBM has something similar in their drives. Almost every LTO4 has some type of techology built in to limit this behavior.
That's making things clear. I'm using HP drives mostly. Second vendor is... yes, IBM. So, I'll be fine :)
Info in Data Rate Matching for HP StorageWorks Ultrium Tape Drives can be found here.
What happened with restores? Have you run into any issues like files that wouldn't restore correctly?
No. There is no problem with any restore.
Very good write up and thanks for referring it to me from:
Looking at the HP quickspecs for your drive, the buffer size should be listed there. The cache size exists on your controller, and the controller will just flush data to the tape drive... I assume HP takes a vague stance because "it's safe to say" that the throughput of the tape drive will hit a bottleneck at the throughput of the bus (cable).
I use an HP SC11Xe adapter, which delivers 320 MB/second bandwidth over the Ultra320 bus (source: quickspecs of SC11Xe) in a PCI-e 4x slot which is about 2000 MB/second (source: quickspecs of server). This board has a LSI Logic 53C1020A A1 chip/dedicated cpu on board (I identified this using lsiutil.exe). The SC11Xe is the LSI MegaRAID SCSI 320-1 and has "Integrated 64 MB ECC SDRAM cache memory," expandable to 128MB.
64 MB is the buffer for my tape drive, the ultrium 920 (according to the quickspecs).
I'll work on trying to get the max throughput, but right now, the speed is reported pretty high assumption of total available bandwidth. Maybe I'll have to go back to my storage subsystem read speed.