BE 2010 - Suggestions to complete weekly backup jobs

[Matt34]

I am trying to backup a large amount of data on the weekend.  In the past these jobs have been completing withing 55 hrs or less.  Now the jobs are not completing on time and I have to cancel them because it slows the entire environment down.

The data I am backing up is 27 Million (27,000,000) small files (.tif and .doc).  It is around 5.5 TB worth of data.  All of this data is housed on a SAN that is presenting LUNs to a Server 2003 Standard box that is virtual.  

Jobs use to run around 66.00 MB/min and are now running around 34.00 MB/min. There are 25 different jobs just to back up this data.  24 of them are going B2D the 1 other job is going to tape.  

I believe I am running into an issue with our SAN and the drives.  Most of these documents are on SATA drives and the B2D targets are on the same SATA drives.  So I am sure I am crushing I/O on those.  I have tried to reduce some of the stress by having some of the B2D going to non SAN storage, but is not working any longer.  By me moving some off the SAN it use to improve the speed a good amount to around 100 MB/min for those couple of jobs.

Another thing to note is that the B2T job use to run at a speed around 300 MB/min but is currently running at 28 MB/min.

I am going to be rebooting the server where all of these documents are housed (Server 2003 Standard) to see if that helps any.  The server is virtual, but it has been virtual for years.  I cannot think of anything that has changed in the last 3 weeks that would have caused everything to slow down to 30 MB/min.

Backups do use AOFO, but have been using this for years. 

Thanks for the help!

[Simon_B_ · Accepted solution]

27.000.000 files is a whole lot. Almost any backup software will struggle to read these even at remotly satisfying rates because for every single file a read process has to take place. This comes with seek times for the HDD heads on the storage etc etc. Over time fragmentation will also increase. Not to mention that reading from and writing to the same disks is a bad idea in general as you already pointed out. 

One way to significantly improve performance might be to partly switch to a snapshot/image-backup for this volume. Symantec System recovery (http://www.symantec.com/de/de/business/system-recovery-server-edition) for example works on block level and as such should have lesser problems with backing up huge masses of single files. You could try something like that: Backup the Volume with SSR to a Staged area (not hosted in the same SAN!) and then grab the *.s2i/*.svi files generated by SSR with BE to put them onto the B2D folder or directly to tape. Although I have no experience with SSR installation which so many files I'd give it a try.

[Matt34]

I will try to take an SR of the small data store but I would like to get back to using BE.  I guess the question I have is, am I going about backing up the data incorrectly?  Do I need more jobs running?  Do I need less?  Am I always going to have issues backing up this data because of how many documents there are?  Are there any options I can change that might help the speeds?

[CraigV]

Hi Matt,

SATA disks aren't the fastest, but are good enough for D2D backups.

However, if you're running full backups, chances are very good you're backing up redundant data all the time. Even splitting the job into multiple separate jobs doesn't help matters, as after a while, the throughput to your SAN is going to get saturated.

Why not consider the following:

1. Incremental/Differential backups - your full backup will take the longest, but your INCR/DIFF jobs are going to save you a lot of time if you run them over week days.

2. Run a trial of BE 2010's deduplication option. This way you're cutting down on the amount of data being backed up, the more often the dedupe job runs. BE 2010 R3 would be the latest version.

Other than that, check and make sure you have tuned your SAN as best as you can for performance, and update your media server, and push out any updates to any remote servers you might have.

Thanks!

[teiva-boy]

Symantec Storage Foundation Flashsnap which integrates with BackupExec as a snapshot option was made for this kind of dataset.  This will solve all your problems, but is not a cheap nor easy to setup option.

27M files is just too much for BackupExec alone.  You have outgrown the product IMO, if you continue to use it the way you do. 

[Simon_B_]

Well as described you can pick up the SSR files lateron with BE. I know it is not an elegant solution and the feasability depends on how often you get restore requests.

But yes - with that many small files you will always have a performance issue when backing up conventionally (=> accessing every single file). You might be able to tune it a bit (defragmentation/separate data and b2d physically on the disks etc) but you will not be able to get a really performant backup with that setup. The largest issue is the seek time that occurs for every file access which prevents that sequential reads take place - and you wont be able to improve that significantly (apart from switching to SSD storage maybe..).

One thing to note regarding Craig's post: He suggested the deduplication option which I suppose will not increase throughput rate a lot (because of the seek times) but one thing could be interesting: The Dedup engine actually understands the *.s2i image format of SSR as long as you do not compress them. So you would achieve high dedup rates on those