Slow backups using Backup Exec 2012

[heathramos]

I am having extremely slow backup times with my new set up for backups.

I have an open ticket with tech support but no luck fixing it so far.

What kind of throughput are others getting? 

Our set up:

-          Windows 2008 R2 Server (production file server)

-          Source files located on Directly Attached Storage device that is connected to the same server that is running backup software

-          SpectraLogic T50e Tape Library with one LTO-6 tape drive is connected directly to same server that is running backup software (connected using fibre to HBA card on server)

-          Backup software is Backup Exec 2012 (new install)

-          Trying to back up over 2TB of data located on DAS device

We ran the following testing scenarios:

Note: G drive is the DAS device

#1 Backed up Public and User folders located on G drive to tape.

-          This backup was approximately 2.3TB and took 38 hours 19 min to complete.

-          According to Backup Exec, the throughput was 1,015 MB/min

#2 Used ITDT program to test the tape drive

-          Ran a write test to a tape and was getting 149 MB/sec write speeds

#3 Ran a test backup with Anti-Virus installed on the server

-          Backed up G:\Image to Tape

-          172 GB backed up in 1hr 32min 32 sec

-          Throughput was 1902 MB/min

#4 Ran a test backup without Anti-Virus installed on the server

-          Backed up G:\Image to Tape

-          172 GB backed up in 1 hr 17min 45 sec

-          Throughput was 2263 MB/min

#5 Ran a test backup without Anti-Virus installed on server from different source

-          Backed up D:\Test to Tape (same files as G:\Image)

-          172 GB backed up in 1 hr 14 min 50 sec

-          Throughput was 2352 MB/min

Set up a Disc Storage device using Backup Exec on the D drive of that same server for testing

#6 Ran a test backup to Disc Storage

-          Backed up G:\Image to Disc

-          172 GB backed up in 1 hr 7 min 50 sec

-          Throughput was 2595 MB/min

#7 Test copying speed from OS (not using backup software)

-          Set up a folder D:\Test copy

-          Copied G:\Image to D:\Test copy

-          172GB copied in 11 min 13 sec

I am running out of ideas.

[CraigV]

Hi,

Backing up an image and backing up data are 2 things...the image is going to be a single large file which means your tape drive spins up to the fastest speed and continues until the backup completes.

Individual files, and lots of them will mean the drive can't maintain a constant speed meaning it does a lot of stop/start all the time. You've proven though that your drive is capable of a faster speed, so it isn't hardware.

So 2.3TB of data can conceivably mean you're not going to get much faster unless you look at INCR backups as a strategy for instance.

Easiest way to check is to backup the same data to disk and see what speeds you get. If it is still slow, then your data and the amount of files is causing the slowness.

As an aside, make sure you're using the SYmantec drivers for the tape drive which can be installed using tapeinst.exe (do a search for it).

Thanks!

[heathramos]

I understand that a few large files would backup faster than a bunch of small files and that explains the rate difference between backing up the 2.3TB and the image folder.

I still believe that it should be backing up much faster than it is.

LTO-6 drives can backup 9,600MB/min. I was getting 8940MB/min when running the tape drive write test. Both results are not close to real world results but I was getting 1/9 of that speed.

The number and size of the files will affect the speed of the backup, which is why the image folder backups up faster than a typical folder with small files but the throughput of that folder was still only 2263MB/min. Did I expect to get 8940MB/min on that folder? no but expected better than that.

I also realize that the bottle neck could be where the source files are located and the speed of that interface. It is on a DAS device connected using a SAS connection. As far as I know, this interface is slower than the fibre interface I'm using with the tape library. 

But what I can't understand is why copying 172gb of data from that same source took 11 minutes (which is a pretty good rate) but backing up to tape (which doesn't seem to be the bottleneck in this case) or to disc on that same raid array takes over an hour. I'm sure the backup software has some overhead (index, writing to a database, etc) but that seems excessive to me.

[CraigV]

...for sure, you should.

However, disk and tape are 2 different things. Copy 172GB of files of varying sizes to disk and it just copies & writes. Do the same to tape, and it literally is: write...stop!...wriiiiiiiiite..stop! write...stop...etc. It stops between files and never gets a chance to spin up to a constant speed.

DAS should also be OK...6GB interface would be fine for most things!

You can also check and make sure your AV isn't scanning the BE services, and if so, put in an exclusion for them. See if it makes any sort of difference!

Thanks!

[heathramos]

I uninstalled the AV completely, just in case and it only made a small improvement.

[heathramos]

tech support gave this as a response:

"While LTO6 transfer rate is 160MB/s (uncompressed), which is about 9.6GB/m, the limitation is still the network that the data is being transferred over. Backup Exec uses the network for all backups, even of the media server."

Is he saying that Backup Exec is limited by the network speed even with local backups?

if so, that seems really strange.

[pkh]

You can try increasing the block and buffer sizes.  See my article below

https://www-secure.symantec.com/connect/articles/tuning-my-lto4-tape-drive

Although it is written for LTO4, the principle is valid for other LTO tape drives.

[CraigV]

...yes it would be the case...if you were backing up data across your LAN. Your LAN, slow disk, amount and type of data, type of drive backing up too, time of backups etc. all make an impact.

HOWEVER...call them back and ask the guy if he understands your environment. Backing up locally attached DAS disks to a SAN means no LAN is involved and therefore can't be the issue.

Thanks!

[heathramos]

I did a lot more testing on Friday with Backup Exec tech support.

Backing up to disc is fast if you uncheck Enable Checkpoint Restart. I was getting 20,000MB/min throughput backing up to disc after doing that.

Backing up to tape still is too slow, however.

I backed up 2,636GB in 22.5 hours with a throughput of 1,995MB/min when backing up local files to tape.

Not sure what I should do next.

I could install a Microsoft hotfix that enables native LTO-6 support.

Or...I could uninstall and re-install Backup Exec, just in case the original install got hosed in some way.

Or...I could install a trial version of some other 3rd party backup solution to see if backups to tape are faster with a different product (proving its a Backup Exec issue and not an issue with the tape drive or source files).

What sucks is that this is our production file server so people get really unhappy when I bring the server down for any reason.

[CraigV]

1. Make sure there are no known issues with installing that patch; if not, install it and check.

2. Doing the complete reinstall is also an option (done this before!), but make sure it is fully patched again!

3. I've done this the other way around...installed Backup Exec 2010 R3 on a SAP BW production server to prove that BE was faster than CA ARCserve r15...it was by 3x. Doing it your way is going to show that it is or isn't a BE issue...not a bad idea either if all else fails!

Unhappy people is something you will never get around...the business doesn't understand downtime to troubleshoot, but they DO understand data that can't be restored and will raise the red flag big time!

Thanks!

[heathramos]

I don't see much info on this microsoft hotfix besides what's on their website

http://support.microsoft.com/kb/2619836

not sure how to uninstall it if I do run into an issue.

[CraigV]

...nothing in that TN stating you can't uninstall it at all. You'd simply go to Add/Remove Programs and choose the option to show Windows Updates before uninstalling this particular HF.

Thanks!

[heathramos]

well...here is the latest when it comes to my troubleshooting...

- I installed the Windows hotfix and it didn't seem to make any difference, although it is functional.

- I uninstalled Backup Exec and installed ArcServe.

I ran a test backup (172GB) using ArcServe and took 29 minutes (6gb/min). That same backup in Backup Exec took 1 hour 17 minutes. At this point I thought I might be done and ArcServe would work for me.

I ran a backup (over 2TB) and the ArcServe backup completed in 16 hours with 2760mb/min throughput. The same backup in Backup Exec took 22.5 hours and had 1995mb/min throughput.

ArcServe does seem to run faster than Backup Exec but it is still too slow. I would love for the backups to take around 7 hours to run but could live with it completing in 13 hours (so it can run overnight and complete before the next work day).

How can run file level backups and increase the speed? Not sure what the bottle neck is and if I change something, how would I know just how much it would shorten my backup window?

I assume I could upgrade the source file storage by getting faster drives (currently 7,200rpm SATA drives) and/or getting new drive array (currently 3GB/s SAS. newer version is 6GB/s). Either would be exensive and would cause some significant downtime. How would I know just how much improvement that kind of upgrade would give me? It would not be good if I spent a bunch of money, caused downtime and it only shortened it by 1/2 hour, for instance.

I could try moving the database the backup software writes to. Maybe install an esata card in the server and buy an external SSD drive. Place the database on it. Would that make a difference?

The CPU isn't geting much usage during the backup process but would a newer type of processor with more cores make any difference at all?

I just don't know what I can do about this.