Backing up millions of files from SAN

[AussieDaz]

Hi,

I may be fighting a loosing battle here but we're having issues backing up large ammounts of small image files (10+ million!) from our HP LeftHand SAN.

A quick run down of our set up:

- 5x SATA based SAN nodes (each node has 12x 1TB drives) connected to the network via 2x 1gb NICs per node

- IBM Server running Backup Exec 2010 R2 connected to the SAN via iSCSI over dual 10gb links (overkill!)

- The server is attached to a IBM dual drive LTO4 tape library via 2x SAS connectors.

- The SAN takes nightly snapshots from another SAS based SAN, we then mount those snapshots on the Backup Exec server.

Our issue is trying to back up approximately 4.1 terrabytes of data from the SAN, mainly consisting of small image files of which there are about 10 million, is extremely slow. Obviously this is due to the extreme ammounts of I/O happening on all the small files. We are seeing transfer rates of 400MB/min to as low as 25MB/min. Which means backups are running for days on end.

Ideally we would like to get a full backup to tape once a week for offsite DR. Differential backups during the week would be ideal, but perhaps a pipedream with this current setup.

We are hoping to get another tape library so we can run parralel jobs and double our throughput to tape.

Is there anything else we can do to speed up backups? Can anyone see any fundamental flaws in the way we are trying to backup?

Personally I think we are running as fast as we can, but I am very open to new ideas, thoughts and suggestions.

Thanks.

[CraigV]

Hi,

That number of files will affect throughput as the tape drive isn't able to spin up to any sort of constant speed.

Differential backups will actually help in this situation. If your data only changes 50GB per night, your data will be backed up to tape during your normal backup window, leaving full backups for the weekend.

You can look at increasing the buffer on the tape drive as 1 way to get a speed increase, but don't expect massive changes.

Alternatively, why not investigate BE 2010's dedupe option...this cuts down backup times and sizes tremendously. Just bear in mind that when duplicating to tape it will rehydrate the data.

Thanks!

[pkh]

You might be better off using SSR which does backup disk images.  This would probably be faster.  However, it only backs up to disk.