Slow backup rate (30MB/min) while backuping Exchange 2003 SP2 using remote agent.

[trep]

Hi,

I'm having an issue here where Backup Exec is very slow to backup Exchange remotely using Backup Exec remote agent. We use the same tape drive to backup another server remotely using Backup Exec remote agent to backup files and directories and the rate is about 400MB/min. I'm not sure why Exchange slows thing down to 25-30MB/min. Could anyone help ?

Best regards,

trep

[Ken_Putnam]

Are you doing stores backups, or are you doing Brick-Level (mailbox) backups.  The way that BackupExec does brick level backups is horrendously inefficient.  In v9, there were also lots of problems with individual item restores.

Also, do to a Disaster Recovery you will need Stores level backups, Brick Level backups cannot be used

used to be a real good explaination by Charles Villa, but the link has been dead for several years now

Edit:

found it on Google's cache

Intro

This is my rant against individual mailbox backup.  I was reading/posting in the exchange newsgroups, but I'm not doing much admin anymore, so if you need something,  find something wrong, or find a dead link please e-mail me.  This will probably be the last update (2001-10-24) unless this becomes seriously out of date.

Important Exchange Links

Before explaining in more detail, here are some resources you should look at if you are responsible for administering an Exchange Server. You should especially look at the Microsoft documents, they goes over a number of details that I don't mention in this message.  The last link points to Ed Crowley 's (THE Exchange Guru) 'Never Restore Method', the source of the procedures I mention at the end.  I also need to thank Ken Putnam for providing the majority of these links.
http://www.microsoft.com/exchange/techinfo/administration/55/BackupRestore.asp (required reading)
www.microsoft.com/exchange (MS's Exchange page, the source of the 2 above links)
news:microsoft.public.exchange.admin (The Exchange newsgroup)
news:microsoft.public.exchange2000.admin (The Exchange 2000 newsgroup)
http://www.swinc.com/resource/exchange.htm (Exchange FAQ)
http://www.swinc.com/resource/exch_faq_appxb.htm (Ed Crowley's Never Restore Method)

Definitions

In the text I use 'individual mailbox backup', 'mailbox backup' and 'brick-level backup' interchangeably (depending on what day I wrote a particular section).  They all mean the same thing, using a MAPI client to do a server backup.

IS = Information Store (the database containing all the mail contents), DS = Directory Store (the list of mailboxes, owners, etc.).  Those two items make up the Exchange database.  If you are running Exchange 2000, you will only see the IS.  The DS has been incorporated into the Active Directory.

Why some people want to use brick-level backup

It is easy to get pulled in by the siren song of brick-level backup.  To do any kind of a restore from an IS backup, you need to restore the entire IS.  If you only need a few items, you need to do the full restore and then export the items you need.  The documentation says you can restore individual mail folders from a brick-level backup without having to restore the entire IS.  If everything worked as well as the documentation stated, it would be great.

Some people argue that brick-level backup is a useful addition to an IS/DS backup.  You have an IS/DS backup for disaster recovery and a brick-level backup for individual item restore.  I can't recommend it due to the number of newsgroup posts of people having problems with brick-level backup.  If it was just an issue of being big and slow, I wouldn't be as strongly against it.

To be fair, there are some people who truly understand the issues and make some use of brick-level backup.  I'm glad it works for them, but working in a small number of cases isn't enough to recommend it.

The detailed rant against brick-level backup

The problem is that it doesn't work as well as the documentation states.  Just about everyone who uses it gets the corrupt message warning.  Some people set their backup software to ignore corrupt files (since it is usually a false warning).  Still, that removes a potentially important warning from the backup software.

Also, it may lull unsuspecting users into the belief that a brick-level backup is sufficient for disaster recovery.  This is absolutely wrong. Most of the points against brick-level backup are from the aspect of disaster recovery. 

The problem with an IS backup is that you must restore the entire IS, not just an individual mailbox or message item. While this is true, using the methods described reduce the need to restore at all (the best way to go), and allow you to recover the contents of an individual mailbox or individual mail items if necessary.

There are three problems we are trying to fix:

1) Users accidentally deleting message items.
2) Admins accidentally deleting mailboxes.
3) Users damaging mail items

Problem #1 is best handled by delete retention.  This way you don't have to restore at all.  The users get the items back by themselves.

Problem #2 should be rare enough that it isn't an issue when deciding which backup method to use.

Problem #3 is a mix.  Delete retention doesn't help, so restoring from the IS backup will be a pain.  My view is that if there is something damaging the data often enough to make me want brick-level backups, I will try to fix the root cause of the damage instead of band-aiding over it with frequent restores.  If the damage is limited to a very small number of mailboxes, then I might use individual mailbox backup for the few.

The Real Problems of using brick level backups:

1) Not the Microsoft suggested way.  Microsoft's Exchange server documentation explicitly says to backup the database files either online via IS and DS services or offline in a flat file backup.  The only mention of brick-level backup is not very flattering.  Who would know better then the company that wrote the software?

2) Reliability. If you look at the previous posts from people using brick level backups, you will find posts describing problems with items being marked as corrupted. As far as I can tell, the root cause of the corruption has not been determined.  Veritas' solution is to delete and recreate the mailbox.  That is a lot of effort, and there is no telling when it will strike again.
** New for 2001-01-19 ** I read on microsoft.public.exchange.admin that someone using Arcserve was having problems with a corrupted IS possibly due to brick-level backup.  See the thread by Chris Hunter on 2001-01-18 13:41.

3) The wrong type of interface.  Brick-level backup uses the MAPI interface, which is the mail client interface.  This interface has methods for sending mail, retrieving mail, etc. for an individual mailbox.  This interface was not designed to do server level operations such as backup.  That's why there is a separate IS/DS interface for backup.  This issue is similar to why you should use SMTP instead of POP on your Exchange server to fetch your domain mail.

4) Granularity.  This sounds strange when talking about individual mailbox backups, but is true.  When restoring from a brick-level backup, you must restore an entire folder.  You cannot select individual items.  If you need to restore just a few items, you restore the entire folder then manually remove the duplicates.  Obviously, this isn't an issue if you are restoring an entire folder.

5) Increased size. In the IS, Exchange uses Single Instance Storage to save space.  Messages that are sent to multiple recipients are stored only once. The brick-level backup will store a separate copy for each recipient. This increases the time and space used for the backup.  When you do a restore from the brick level backup, the messages that were backed-up as many duplicates get restored as many duplicates. The IS that fit nicely on your hard drive before may have problems now due to size bloat.

6) Increased time. When backing-up the IS, the IS is opened once and the entire contents are streamed to the backup device. Brick-level has to open and close each mail item individually. This overhead can be enormous, multiplying the backup time many times.

7) Doesn't use transaction logs. If you are using the IS backup, and if your transaction logs are intact (since they should be on a different drive), you may be able to recover to the very minute you had the problem. Restore the IS and replay the transaction logs. Obviously, this doesn't work if you restore from brick-level.

How to do everything without using brick-level backups

0) The point is to reduce the need for recovering items from tape. To be honest, recovering items from the IS backup is a pain. But you only pay that cost when you restore.  Since we upgraded to 5.5 and been using delete retention, I have only once needed to restore an item from tape (because 6 months had transpired).  The cost of brick-level backups is paid on every backup: the extra space, the extra time and the hiding of corrupt warnings.

1) Use deleted item retention (this cures problem #1, users deleting things). For me, this is the best thing about Exchange 5.5. Set the retention time to something long, say 30 days (you can use even more if you have the disk space). Show your users how to recover deleted items (it shows up as a menu item in Outlook, when viewing the 'deleted items' folder). This is the best, since you don't have to do anything. The users do for themselves. If they come to you after the retention time has expired, you get to give them a dope-slap and charge their cost center for your time. Read item 3 for recovering after the retention period.  See http://support.microsoft.com/support/kb/articles/Q178/6/30.ASP for recovering deleted items from all folders (thanks to Glen Morangie).

2) When someone leaves, don't immediately delete their mailbox. Remove it from any distribution lists, take away any SMTP addresses, set the alternate delivery to their boss (or someone else responsible), put the leaving date in the notes field and mark it hidden. After some time has gone by (say a month) and you are sure so one needs the mailbox items, you can delete the mailbox.

3) This leaves admins who delete too early, users who wait too long, and accidents that corrupt data. To recover a individual items, you have to set up a recovery server (a machine created with a new site and org, that happens to be the same site name and org name as the original). You restore the entire IS to the recovery server and then export the desired items as a .PST file.  Import the items from the .PST file into the production server.  This method gives you exact control over what you restore.  You can export any mix of individual items you choose.

Charles Villa (CVilla at tekscan dot com)

2001-10-24 09:55

Message Edited by Ken Putnam on 02-24-2009 02:31 PM

Message Edited by Ken Putnam on 02-24-2009 02:33 PM

[trep]

Hi,

I can only do brick level backup for some reason, i can't select the Information Store in the selection. I've installed the Windows Remote Agent on the Exchange server but i can't install the Exchange agent remotely (or haven't found how to do so). So i'm left with the choice of selection mailboxes and public folders.

What version of Backup Exec is good to do Exchange brick level backup ? We have another server with backup Exec 10D and we're experiencing the same issue.

Best regards,

trep

[Ken_Putnam]

Beginning with v11d, Veritas/Symantec offer a technology called GRT (Granular Restore Technology) that will let you restore individual items from a Stores level backup.  It has some quirks, but totaly bypasses the slow Bricklevel approach

The Exchange "Agent" is only a serial that installs on the media server to unlock the capability.  The RemoteAgent is the only code that actually installs on remote machines. 

Is the BackupExec service account an ExchangeAdmin?

[trep]

Yep, i've followed the KB found on symantec's website to configure the BESA account to backup Exchange 2003. In the selection screen i have 2 choices to backup Exchange

- Microsoft Exchange Mailboxes

- Microsoft Exchange Public Folders

I tought i would also see Microsoft Exchange Information Store but there's nothing there besides what i said above.

Best regards,

trep

[Ken_Putnam]

Hmmm

Used to be (in the v5 & v8 days, that you had to have the Exchange Admin Tools installed on the Media Server to see the Stores, but I thought that went away with v9

Sorry, I'm out of ideas at this point

Message Edited by Ken Putnam on 02-25-2009 11:25 AM