slow Exchange GRT restore

[sanhun]

dear community,

i found many topics already about the problem with slow GRT restore, and i fear there is no solution to the problem, but as every case different, i share my problem - maybe there is something what i missed.

The used system:

• Symantec BackupExec 2010 R3 SP1 (13.0 rev. 5204)
• Microsoft Windows 2008R2 Standard SP1
• Microsoft Exchange 2010 SP2
• HP MSL2024 LTO3 (latest firmware used)
• McAfee antivirus, but was completely turned off during the tries

I create a backup every day directly to tape, some sql database, some files, and an GRT exchange backup. My entire exchange store backup size is around 200GB.

I tried to restore a single email message from the tape, and i got the long staging time - the speed was around 200 MB\min. I was reading about the problem, and decided to try to duplicate the exchange backup to disk first, and restore the email message from the disk, so i started a duplicate backup set job to the disk (of the local server), but the speed was same: 200 MB\min.

After this things, i tried to restore some single file, not an exchange thing, just to check the speed: i had a 70GB large database dump, i restored it with 6182 MB\min, in 17 minutes. I think the system don't have speed problems.

So even if i accept the fact that even if i want to restore only 1 email message need to "cache" the entire exchange backup - it would finish in 1-2 hours. But at this moment it would last long 12 hours or something like (never waited till end, because next backup session would begin before this one finish).

Only a side note: we using this config since 3 months ago. The change was needed because we upgraded to exchange 2010, so was needed to upgrade to backupexec 2010, and not only this, but to 64-bit OS, so was needed to buy a new server too (the old one support 32-bit OS only). As far as i remember with the old system (Backupexec 12.5, windows 2003, exchange 2003, same tape) i had no problem. I restored a single email message once, and it was finished in normal timing, around 1 hour - but my memory can be wrong... But sure was not 8-12 hours, (and store size was not significant lesser) because it would not fit in worktime.

So after all, my question is:

- can i improve this 200MB speed somehow, did i miss some settings or something?

- if the restoring time is unacceptable like this, what is the recommended non-disaster backing up - restoring method? The most common case if we need 1-1 deleted email from the past. Creating a recovery database on the exchange-server? To be honest i would skip the message-hunting in the powershell. Or should i backup with GRT to disk, then write the image to tape? And at restoring i copy the GRT image back to disk, and backup from the disk image? Can i do this? Would be same effect i think (i would able to select single items from the store), but would be faster?

Thank you for any help,

Mátyás

[sanhun]

A minor update for the topic:

today i tried a GRT-enabled backup from the Exchange to disk, and i got same slow process: the writing to the disk was same 200MB \ min.

Is looks like for me that anytime when i doin' this GRT process (direct to disk \ to disk from tape \ direct from tape), its became to damn slow. However the server itself is not busy (95% free CPU time), so i have no idea what happening.

I looking for other exchange backup methods, because this one would be very painful at restore.

[CraigV]

Hi,

What type of disk are you duplicating too? Have you checked the file fragmentation on the disks in question as well?

And have you moved the temporary staging area from C:\Temp to another drive with enough disk space?

Thanks!

[sanhun]

Hello Craig,

I started the duplicate from the tape to the disk of the local server. This is a pretty new server, bought few months ago. I did the steps as is in your guide :p , and yes, the destination point is moved from the C:\temp, the new destination is on another - and totally free partition. So i didn't check the fragmentation, because its a new server's new partition what never had any data before.

I restored a single 70GB file to same destination from same tape\backup session, and its completed with 6182 MB\min, so i would say the connection between the tape and disks are ok.

[CraigV]

Good to hear you checked out my article

OK...so create a new stand-alone job and back up Exchange only, making sure that AOFO isn't running in the job. Once done, try the duplicate to disk again (or a direct restore from tape) and report back here.

Are you using AOFO in your current job, in which case if yes, consider splitting your jobs apart (Exchange/DBs separately from data):

https://www-secure.symantec.com/connect/articles/backing-databases-and-files-same-tape-separate-jobs-using-backup-exec

[sanhun]

Thank You Craig,

i made the following tests:

- backup job: Exchange --> tape with GRT without AOFO = 2488MB / min (see tape.pdf)

- backup job: Exchange --> disk with GRT without AOFO = 200MB / min (see disk.pdf)

- restore job: tape with GRT --> Exchange = 200MB / min

- duplicate job: tape with GRT --> disk = 200MB / min

some tries without GRT:

- backup job: Exchange --> tape without GRT = 2600MB / min

- backup job: Exchange --> disk without GRT without AOFO = 4400MB / min (over the network to the backup server)

So after all, there is no any difference if i turn on\off the AOFO, but turning off GRT dramatically fasten the job.

I attach here two documents about the settings of the backup jobs, maybe you can point to something what i doin' wrong.

So if my numbers are confusing, my problem is:

- if i backup to tape with GRT, the backup is fast, but restore\duplicate is damn slow

- if i backup to disk with GRT, the backup is already damn slow

Okay i am dumb for the exchange \ backupexec in this deep, but as i understood the method of the GRT, there is a step what making "GRT" from a single backup process: if i backup to tape, this step made only at the restore, and if i backup to disk, this step is made during the backup process. As far as i got it, its because some dynamic image making, what is not possible to directly to tape, so is created at exchange-disk; tape-restore; tape-disk times. This things are slow for me. Looks like some process is slow what the backupexec software doing with the GRT.

I cant recognize what is this step, because i cannot follow the "Understanding GRT Backup and Restore functionality in Symantec Backup Exec" article step-to-step.

[sanhun]

meanwhile i updated to newest version of the backupexec 2010 R3: SP2 and the hotfix 180429, but no changes in action

[sanhun]

a little update about this thing,

in the last days i tried to do a duplicate from tape to disk, to different destinations. We have an HP EVA 4400  storage, few server is connected to it, even the Exchange itself. The numbers are confusing however. I was not able to do a full duplicate to every server, because i dont have 200GB free space everywhere, and the duplicate process would be too long with slow speed.

duplicate job from tape, or backup to disk			Job rate in the first 5GB	job rate in the entire 200GB
backupexec server	Windows 2008 R2	own disk	220 MB/min	220 MB/min
exchange server	Windows 2008	EVA	477 MB/min
server1	Windows 2003 Standard	own disk	344 MB/min
server2	Windows 2003 Standard	EVA	1177 MB/min
server3	Windows 2003 storage server	EVA	2700 MB/min	701 MB/min
direct restore from tape to exchange
exchange server	Windows 2008	EVA	220 MB/min

again, the numbers are for GRT restoring \ duplicating only, with the normal file-restoring i have no problem, i can reach very high speed with them - only the exchange GRT is slow.

So, i am sure there is some settings, running service, or some OS version, or anything what affect the speed, but i cannot figure out what is.

Any idea, or a higher reference speed with given configuration, so i could made some tests?

thank you

[CraigV · Accepted solution]

Your speeds would be a lot better if you had a full SAN environment with SAN SSO in place for instance. Your servers being backed up would have access to the SAN and the library (assuming it was SAN-attached), giving you SAN speed for restores/backups.

That said, you need to take into account network speed, type of disk restoring/duplicating too, the amount of data as well as the type of data, and whether or not you have an AV that might be scanning your details.

Typically I am getting anything from 1.5GB/m to 4-5GB/min when duplicating backups.

So start by checking to see whether or not your AV is scanning the BE services on your media server and the target server, and if so, exclude them.

Thanks!