Poor performance when backing up compressed Veeam file to tape using BE2010R3

[ExpressISTeam]

Hi All

I am investigating some performance issues we are having backing up to tape.

THe way our process work is we have a backup server where we have Veeam install which we use to backup to disk. We then use BE 2010 R3 to backup the veeam file created (one large 1tb .vbk file). This veeam file is a file which Veeam creates when creating backups of our VMs in VMware.

This file has already been compressed within Veeam. We then backup this 1 file to tape onto our IBM TS3200 tape library directly connected via 6GB SAS on the backup server. We recently upgraded the unit in here to LT0-5. We were using LT0-4 before but the performance we are getting is the same as LT0-4.

Firmware upgrades on the tape library unit, server have all been done. BE is fully patched up on the server. Its running Windows 2008 32-bit with 4gb ram.

We have tried numerous tests with buffer settings but still the same performance. We have turned encryption off and compression within BE only off.  

I am running out of ideas and am now thinking could it be that because this veeam file is already compressed. Is symantec just slowing down because its already compressed?

Is there any advice on backing to tape changing settings within BE? I have tried numerous settings within BE. Currently the device is set as the following:

Compression enabled

block size (per device) - 64kb

buffer size (per device) - 1mb

buffer count - 10

high water count - 0

read single block mode - off

write single block mode - off

read scsi passthrough mode - off

write scsi passthrough mode - on

[pkh]

Try increasing the block size.  See my article on how to tune your tape drive.

https://www-secure.symantec.com/connect/articles/tuning-my-lto4-tape-drive

Although it is for LTO4, it is equally applicable for LTO5

[CraigV]

Be VERY careful how you tune your tape drive in this regard, and I am speaking from experience. We've started this as well on a number of sites (for whatever reason, Veeam was chosen to back up the VM infrastructures without investigating BE's AVVI option...long story around Veeam too!). However, I had to get tapes from Asia sent back to my country and catalog them here.

Because I ran the backups with increased buffer and block size settings, I couldn't catalog the tapes correctly. I could see the data on each spanned tape, but trying to restore it saw the process fail with an incorrect block size setting, along with catalogging the tape failing for the same reason.

Whatever you do, test a restore immediately otherwise you might find yourself in a position where you cannot restore the *.vbk file.

LTO5 isn't necessarily faster than LTO4 (although in theory it is). It just transfers the bottleneck elsewhere, so a lot of guys who have gone the LTO5 route thinking they'll get much faster backups are disappointed when they don't. LTO5 is great for the amount of data it can hold.

I'd also suggest turning off compression within BE...the *.vbk file is already compressed.

[pkh]

Testing that you can read the tape is definitely part of the tuning process.  Some tape drives/tapes cannot handle big blocksizes, but would not fail when you are writing them.  You only find out the problem when you try to read the tapes.

Having said that, a 64kb blocksize is definitely too small. See this discussion.

https://www-secure.symantec.com/connect/forums/very-disappointing-lto5-job-rate#comment-6114791

[teiva-boy · Accepted solution]

Your backup is only as fast as its slowest link.  It could be read speeds, disk fragmentation, NTFS tuning, Gb links, etc...  You need to verify that each component is running optimally.  I've seen way too many people upgrade to LTO4 and LTO5, and not see an imprvement because of other issues upstream.

[Lesta_G]

I would look at trailing the AVVI agent for backups to see if this problem goes away....

Where is the 1TB file on the media server the tape is attached to or somewhere else?

I think you have already tried this but....Do not enable comprerssion in the BE job (set it to none, not hardware only)

[CraigV]

Lesta: Not sure if you have worked with Veeam, but it seems like you haven't. Compared to BE 2010 R3 (and I am a big fan), Veeam compresses the backups considerably...basically deduping the backup into a much smaller file.

This isn't about trialling AVVI...the OP is asking how his backups could be as slow as they are. The problem will probably still be around. You should be focussing on that...

[Lesta_G]

i have used Veem  a bit. I just look at other ways to resolve the issue and to compare performance.

If fibre connected tape drive to a fibre switch, i would be checking the switch statistics to see if there is a significant error rate on the tape port. If so , I would get the fibre connectors cleaned

OP what is your backup rate? I run a TS3200, with the settings you describe but a slight difference

Best backup speed achieved 8.2GB/Min

(Drive is per setup recommendations from a IBM RedBook)

buffer size (per device) - 64kb

write single block mode - on

[pkh]

When "write single block" is on, you are not using your buffers.

[CraigV]

...Veeam doesn't support tape drives, and the issue is writing the flat-file to tape! You need to compare apples with apples here.