Backing up Exchange Mailboxes slow as molasses

[Dave_Z]

* Symantec Backup Exec 10d (10.1.5629)
* HP Ultrium 3-SCSI LTO 64K (firmware: G2AD)
* Windows Server 2003 SP2 (all updates current)
* Exchange Server 2003 (6.5.7638.1)

I have an 86GB mailbox store on my Exchange Server.  I back up a total of 670GB of data and half of the backup job time is spent backing up the individual mailboxes.  In an attempt to speed things up I actually have Backup Exec running ON the Exchange server so the job won't have to go across the wire.  But I'm still seeing backup jobs run as long as 96 hours.

Has anyone else seen this behavior? Any advice is much appreciated.

Thanks,
- Dave Z

[Don_S]

This is typical of doing individual mailbox backup.  That's why poeple are getting away from doing that.   They are doing either just a store backup and restoring it to a recovery group or they are doing granular backup and restore.  there just isn't enough time in the day to do mailbox by mailbox with a store that big - no matter how fast the exchange server or backup server is.  And, from my experience, it's actually faster to backup over the wire becuase the exchange server is usually the bottleneck - not the wire.

[Ken_Putnam]

As Don says, this is pretty much what you can expect from a Brick Level backup in v10d and earlier.  This is why  Veritas developed and released GRT in v11.  back when v10 was the latest and greatest, we saw reports of brick level backups running at 4-5MB/min when flat file backups were running at 400-500MB/min

it is because of the way that pre-v11 brick level backups are done.

There is no API from Microsoft to do it, so what Backup Exec does is open an Outlook session to the Exchange server (this is why you need a mailbox for the BackupExec service account).   then it attaches to each selected mailbox , then  "opens" and "reads" each item in those mailboxes using standard MAPI calls.  Very inefficient.

This is the content of a good explaination that used to be on the net, but has since been taken down

Intro
This is my rant against individual mailbox backup. I was reading/posting in the exchange newsgroups, but I'm not doing much admin anymore, so if you need something, find something wrong, or find a dead link please e-mail me. This will probably be the last update (2001-10-24) unless this becomes seriously out of date.
Important Exchange Links
.
Before explaining in more detail, here are some resources you should look at if you are responsible for administering an Exchange Server. You should especially look at the Microsoft documents, they goes over a number of details that I don't mention in this message. The last link points to Ed Crowley 's (THE Exchange Guru) 'Never Restore Method', the source of the procedures I mention at the end. I also need to thank Ken Putnam for providing the majority of these links.
http://www.microsoft.com/exchange/techinfo/administration/55/BackupRestore.asp (required reading)
http://www.microsoft.com/exchange (MS's Exchange page, the source of the 2 above links)
news:microsoft.public.exchange.admin (The Exchange newsgroup)
news:microsoft.public.exchange2000.admin (The Exchange 2000 newsgroup)
http://www.swinc.com/resource/exchange.htm (Exchange FAQ)
http://www.swinc.com/resource/exch_faq_appxb.htm (Ed Crowley's Never Restore Method)

Definitions
In the text I use 'individual mailbox backup', 'mailbox backup' and 'brick-level backup' interchangeably (depending on what day I wrote a particular section). They all mean the same thing, using a MAPI client to do a server backup.
IS = Information Store (the database containing all the mail contents), DS = Directory Store (the list of mailboxes, owners, etc.). Those two items make up the Exchange database. If you are running Exchange 2000, you will only see the IS. The DS has been incorporated into the Active Directory.
Why some people want to use brick-level backup
It is easy to get pulled in by the siren song of brick-level backup. To do any kind of a restore from an IS backup, you need to restore the entire IS. If you only need a few items, you need to do the full restore and then export the items you need. The documentation says you can restore individual mail folders from a brick-level backup without having to restore the entire IS. If everything worked as well as the documentation stated, it would be great.

Some people argue that brick-level backup is a useful addition to an IS/DS backup. You have an IS/DS backup for disaster recovery and a brick-level backup for individual item restore. I can't recommend it due to the number of newsgroup posts of people having problems with brick-level backup. If it was just an issue of being big and slow, I wouldn't be as strongly against it.
To be fair, there are some people who truly understand the issues and make some use of brick-level backup. I'm glad it works for them, but working in a small number of cases isn't enough to recommend it.

The detailed rant against brick-level backup

The problem is that it doesn't work as well as the documentation states. Just about everyone who uses it gets the corrupt message warning. Some people set their backup software to ignore corrupt files (since it is usually a false warning). Still, that removes a potentially important warning from the backup software.
Also, it may lull unsuspecting users into the belief that a brick-level backup is sufficient for disaster recovery. This is absolutely wrong. Most of the points against brick-level backup are from the aspect of disaster recovery.

The problem with an IS backup is that you must restore the entire IS, not just an individual mailbox or message item. While this is true, using the methods described reduce the need to restore at all (the best way to go), and allow you to recover the contents of an individual mailbox or individual mail items if necessary.
There are three problems we are trying to fix:

1) Users accidentally deleting message items.
2) Admins accidentally deleting mailboxes.
3) Users damaging mail items

Problem #1 is best handled by delete retention. This way you don't have to restore at all. The users get the items back by themselves.

Problem #2 should be rare enough that it isn't an issue when deciding which backup method to use.

Problem #3 is a mix. Delete retention doesn't help, so restoring from the IS backup will be a pain. My view is that if there is something damaging the data often enough to make me want brick-level backups, I will try to fix the root cause of the damage instead of band-aiding over it with frequent restores. If the damage is limited to a very small number of mailboxes, then I might use individual mailbox backup for the few.

The Real Problems of using brick level backups:

1) Not the Microsoft suggested way. Microsoft's Exchange server documentation explicitly says to backup the database files either online via IS and DS services or offline in a flat file backup. The only mention of brick-level backup is not very flattering. Who would know better then the company that wrote the software?

2) Reliability. If you look at the previous posts from people using brick level backups, you will find posts describing problems with items being marked as corrupted. As far as I can tell, the root cause of the corruption has not been determined. Veritas' solution is to delete and recreate the mailbox. That is a lot of effort, and there is no telling when it will strike again.
** New for 2001-01-19 ** I read on microsoft.public.exchange.admin that someone using Arcserve was having problems with a corrupted IS possibly due to brick-level backup. See the thread by Chris Hunter on 2001-01-18 13:41.

3) The wrong type of interface. Brick-level backup uses the MAPI interface, which is the mail client interface. This interface has methods for sending mail, retrieving mail, etc. for an individual mailbox. This interface was not designed to do server level operations such as backup. That's why there is a separate IS/DS interface for backup. This issue is similar to why you should use SMTP instead of POP on your Exchange server to fetch your domain mail.

4) Granularity. This sounds strange when talking about individual mailbox backups, but is true. When restoring from a brick-level backup, you must restore an entire folder. You cannot select individual items. If you need to restore just a few items, you restore the entire folder then manually remove the duplicates. Obviously, this isn't an issue if you are restoring an entire folder.

5) Increased size. In the IS, Exchange uses Single Instance Storage to save space. Messages that are sent to multiple recipients are stored only once. The brick-level backup will store a separate copy for each recipient. This increases the time and space used for the backup. When you do a restore from the brick level backup, the messages that were backed-up as many duplicates get restored as many duplicates. The IS that fit nicely on your hard drive before may have problems now due to size bloat.

6) Increased time. When backing-up the IS, the IS is opened once and the entire contents are streamed to the backup device. Brick-level has to open and close each mail item individually. This overhead can be enormous, multiplying the backup time many times.

7) Doesn't use transaction logs. If you are using the IS backup, and if your transaction logs are intact (since
they should be on a different drive), you may be able to recover to the very minute you had the problem. Restore
the IS and replay the transaction logs. Obviously, this doesn't work if you restore from brick-level.

How to do everything without using brick-level backups

0) The point is to reduce the need for recovering items from tape. To be honest, recovering items from the IS backup is a pain. But you only pay that cost when you restore. Since we upgraded to 5.5 and been using delete retention, I have only once needed to restore an item from tape (because 6 months had transpired). The cost of brick-level backups is paid on every backup: the extra space, the extra time and the hiding of corrupt warnings.

1) Use deleted item retention (this cures problem #1, users deleting things). For me, this is the best thing about Exchange 5.5. Set the retention time to something long, say 30 days (you can use even more if you have the disk space). Show your users how to recover deleted items (it shows up as a menu item in Outlook, when viewing the 'deleted items' folder). This is the best, since you don't have to do anything. The users do for themselves. If they come to you after the retention time has expired, you get to give them a dope-slap and charge their cost center for your time. Read item 3 for recovering after the retention period. See http://support.microsoft.com/support/kb/articles/Q178/6/30.ASP for recovering deleted items from all folders (thanks to Glen Morangie).

2) When someone leaves, don't immediately delete their mailbox. Remove it from any distribution lists, take away any SMTP addresses, set the alternate delivery to their boss (or someone else responsible), put the leaving date in the notes field and mark it hidden. After some time has gone by (say a month) and you are sure so one needs the mailbox items, you can delete the mailbox.
3) This leaves admins who delete too early, users who wait too long, and accidents that corrupt data. To recover a individual items, you have to set up a recovery server (a machine created with a new site and org, that happens to be the same site name and org name as the original). You restore the entire IS to the recovery server and then export the desired items as a .PST file. Import the items from the .PST file into the production server. This method gives you exact control over what you restore. You can export any mix of individual items you choose.

[teiva-boy]

Hey I remember that newsgroup article!  

If you upgrade to 11d or better, ideally 12.5 with the latest and greatest SP2 applied, it will solve your issues.