cancel
Showing results for 
Search instead for 
Did you mean: 

Backup Exec 10d Ver 10.1 Rev 5629 Taking 14+ Hours To Backup

jlalonde
Level 3

We're currently running Backup Exec 10d and it's progressively getting slower and slower. We've got service pack 4 installed and that did not solve the issue. Currently, we're backing up approximately 180 gigs of data to an HP RDX system. I've been told this should be a matter of hours, not 14+. Any suggestions?

1 ACCEPTED SOLUTION

Accepted Solutions

Ken_Putnam
Level 6
v10 + Echange +  slow backups    DING! DING! DING!

you are not by any chance doing Brick level backups (also know as mailbox backups) in addition to or instead of Stores level backups are you?


The method used in v10d and prior versions to backup individual mailboxes was a kludge and VERY slow.  speeds of 10-20 MBytes/min  were reported here on the forum on decent hardware  
 

View solution in original post

17 REPLIES 17

CraigV
Moderator
Moderator
Partner    VIP    Accredited
Hi there,

When did this start occurring? After installing SP4? Do you have remote servers, and did you push out the RAWS agents again after installing SP4?
How much data are you backing up, and how much as well?
I have seen a reboot of the backup environment (backup server + backup device) showing an improvement. IN this case, it would probably be related to WIndows, so give that a bash first.
Lastly, I have included 2 Symantec tech articles for you to read over...

http://seer.entsupport.symantec.com/docs/231488.htm

http://seer.entsupport.symantec.com/docs/285756.htm

Laters!

jlalonde
Level 3
Craig,

It's been occuring for quite some time. It's been getting longer and longer, prior to installing SP4. I was thinking maybe that would help, but it did not. We have one remote server and I did not push out the RAWS agent(not sure what that is).

I've restarted the server and the device(we've actually swapped out devices). No improvement from that.

Thanks for the two articles. Tonight I'm going to try breaking down the backup into sections for each drive that we need backed up, as stated in one of the articles. We'll see if one area is taking an extreme amount of time.

CraigV
Moderator
Moderator
Partner    VIP    Accredited
Rad...let me know if it worked at all or not.

jlalonde
Level 3
Will do, thanks again. Also, forgot to state in my last reply that we're backing up around 180 gigs of data.

jlalonde
Level 3
Craig, looks like it may be our exchange portion that's taking a long time. It's currently at 9 hours, 32,000,000,000 bytes, and only 66% done.

Now to figure out what's going on with that. Any ideas?

CraigV
Moderator
Moderator
Partner    VIP    Accredited
Mmm...

remotely or locally? Do you have the Exchange agent installed (silly question, but valid)?
What is your set up for backing up Exchange? And have you considered splitting Exchange off from your data backups?

jlalonde
Level 3

Exchange server is local on our primary server. I believe the exchange agent is installed. Is there a way to check to make sure? **edit** we do have the exchange agent installed.

We have it backing up all of our Exchange data.

After seeing this, we may have to figure out some solution regarding the Exchange server. So we may split it off or stop doing it(really don't want to do that as we would lose lots of valuable company emails if anything happened) if we can't find a solution.

Ken_Putnam
Level 6
v10 + Echange +  slow backups    DING! DING! DING!

you are not by any chance doing Brick level backups (also know as mailbox backups) in addition to or instead of Stores level backups are you?


The method used in v10d and prior versions to backup individual mailboxes was a kludge and VERY slow.  speeds of 10-20 MBytes/min  were reported here on the forum on decent hardware  
 

Khue
Level 4
I usually separate functional backups. I have 3 individual backup schedules: one for file based, one for exchange database based, and one for SQL backups. Just to give you a ball park, we have approximately 82 gigs in our exchange database and it usually takes us about an hour and a half to go to disk and then 45 minutes to go to tape from disk. How much data are you backing up in exchange? 44 Gigs? Do you have separate mailstores setup?

jlalonde
Level 3

Ken,

I think we may be doing Brick level backups. Under selections we have "Microsoft Exchange Boxes" and "Microsoft Exchange Public Folders" checked. I'm not seeing any options for Store level backups. Is there a place to check to see if Store level backups are happening?

jlalonde
Level 3

Khue,

We're backing up approximately 47 gigs for Exchange. We've got a Mailbox Store and a Public Folder Store

Ken_Putnam
Level 6

IIRC Microsoft Exchange Boxes is brick level (they change the descriptions from version to version

As for Stores Level.  When creating the selections list if you choose Microsoft Exchange after browsing to the Exchange Serverr, you are getting the Stores


In addition to being very slow, Brick Level restores were problematic at best.  And they should NOT be used for DR purposes.  The consensus seemed to be:  Get your stores backup and then do a limited number of VIP brick levels.  For worker-bee restores, use either a Stand-by server or the RSG



Charles Villa had a great rant posted on his server against the kludge Veritas came up with for Brick level backups, but I haven't seen it in a while



Found a copy on mh HD  |8^)






<begin Charles Villa's Rant>
 

Intro

This is my rant against individual mailbox backup. I was reading/posting in the exchange newsgroups, but I'm not doing much admin anymore, so if you need something, find something wrong, or find a dead link please e-mail me. This will probably be the last update (2001-10-24) unless this becomes seriously out of date.

Important Exchange Links

.
Before explaining in more detail, here are some resources you should look at if you are responsible for administering an Exchange Server. You should especially look at the Microsoft documents, they goes over a number of details that I don't mention in this message. The last link points to Ed Crowley 's (THE Exchange Guru) 'Never Restore Method', the source of the procedures I mention at the end. I also need to thank Ken Putnam for providing the majority of these links.
http://www.microsoft.com/exchange/techinfo/adminis... (required reading)
http://www.microsoft.com/exchange (MS's Exchange page, the source of the 2 above links)
news:microsoft.public.exchange.admin (The Exchange newsgroup)
news:microsoft.public.exchange2000.admin (The Exchange 2000 newsgroup)
http://www.swinc.com/resource/exchange.htm (Exchange FAQ)
http://www.swinc.com/resource/exch_faq_appxb.htm (Ed Crowley's Never Restore Method)

 

Definitions

In the text I use 'individual mailbox backup', 'mailbox backup' and 'brick-level backup' interchangeably (depending on what day I wrote a particular section). They all mean the same thing, using a MAPI client to do a server backup.

IS = Information Store (the database containing all the mail contents), DS = Directory Store (the list of mailboxes, owners, etc.). Those two items make up the Exchange database. If you are running Exchange 2000, you will only see the IS. The DS has been incorporated into the Active Directory.

Why some people want to use brick-level backup

It is easy to get pulled in by the siren song of brick-level backup. To do any kind of a restore from an IS backup, you need to restore the entire IS. If you only need a few items, you need to do the full restore and then export the items you need. The documentation says you can restore individual mail folders from a brick-level backup without having to restore the entire IS. If everything worked as well as the documentation stated, it would be great.

 

Some people argue that brick-level backup is a useful addition to an IS/DS backup. You have an IS/DS backup for disaster recovery and a brick-level backup for individual item restore. I can't recommend it due to the number of newsgroup posts of people having problems with brick-level backup. If it was just an issue of being big and slow, I wouldn't be as strongly against it.

To be fair, there are some people who truly understand the issues and make some use of brick-level backup. I'm glad it works for them, but working in a small number of cases isn't enough to recommend it.

 

The detailed rant against brick-level backup

 

The problem is that it doesn't work as well as the documentation states. Just about everyone who uses it gets the corrupt message warning. Some people set their backup software to ignore corrupt files (since it is usually a false warning). Still, that removes a potentially important warning from the backup software.

Also, it may lull unsuspecting users into the belief that a brick-level backup is sufficient for disaster recovery. This is absolutely wrong. Most of the points against brick-level backup are from the aspect of disaster recovery.

 

The problem with an IS backup is that you must restore the entire IS, not just an individual mailbox or message item. While this is true, using the methods described reduce the need to restore at all (the best way to go), and allow you to recover the contents of an individual mailbox or individual mail items if necessary.

There are three problems we are trying to fix:

 

1) Users accidentally deleting message items.
2) Admins accidentally deleting mailboxes.
3) Users damaging mail items

 

Problem #1 is best handled by delete retention. This way you don't have to restore at all. The users get the items back by themselves.


Problem #2 should be rare enough that it isn't an issue when deciding which backup method to use.


Problem #3 is a mix. Delete retention doesn't help, so restoring from the IS backup will be a pain. My view is that if there is something damaging the data often enough to make me want brick-level backups, I will try to fix the root cause of the damage instead of band-aiding over it with frequent restores. If the damage is limited to a very small number of mailboxes, then I might use individual mailbox backup for the few.

 

The Real Problems of using brick level backups:

 

1) Not the Microsoft suggested way. Microsoft's Exchange server documentation explicitly says to backup the database files either online via IS and DS services or offline in a flat file backup. The only mention of brick-level backup is not very flattering. Who would know better then the company that wrote the software?

 

2) Reliability. If you look at the previous posts from people using brick level backups, you will find posts describing problems with items being marked as corrupted. As far as I can tell, the root cause of the corruption has not been determined. Veritas' solution is to delete and recreate the mailbox. That is a lot of effort, and there is no telling when it will strike again.
** New for 2001-01-19 ** I read on microsoft.public.exchange.admin that someone using Arcserve was having problems with a corrupted IS possibly due to brick-level backup. See the thread by Chris Hunter on 2001-01-18 13:41.

 

3) The wrong type of interface. Brick-level backup uses the MAPI interface, which is the mail client interface. This interface has methods for sending mail, retrieving mail, etc. for an individual mailbox. This interface was not designed to do server level operations such as backup. That's why there is a separate IS/DS interface for backup. This issue is similar to why you should use SMTP instead of POP on your Exchange server to fetch your domain mail.

 

4) Granularity. This sounds strange when talking about individual mailbox backups, but is true. When restoring from a brick-level backup, you must restore an entire folder. You cannot select individual items. If you need to restore just a few items, you restore the entire folder then manually remove the duplicates. Obviously, this isn't an issue if you are restoring an entire folder.

 

5) Increased size. In the IS, Exchange uses Single Instance Storage to save space. Messages that are sent to multiple recipients are stored only once. The brick-level backup will store a separate copy for each recipient. This increases the time and space used for the backup. When you do a restore from the brick level backup, the messages that were backed-up as many duplicates get restored as many duplicates. The IS that fit nicely on your hard drive before may have problems now due to size bloat.

 

6) Increased time. When backing-up the IS, the IS is opened once and the entire contents are streamed to the backup device. Brick-level has to open and close each mail item individually. This overhead can be enormous, multiplying the backup time many times.

 

7) Doesn't use transaction logs. If you are using the IS backup, and if your transaction logs are intact (since

they should be on a different drive), you may be able to recover to the very minute you had the problem. Restore

 the IS and replay the transaction logs. Obviously, this doesn't work if you restore from brick-level.

 

How to do everything without using brick-level backups

 

0) The point is to reduce the need for recovering items from tape. To be honest, recovering items from the IS backup is a pain. But you only pay that cost when you restore. Since we upgraded to 5.5 and been using delete retention, I have only once needed to restore an item from tape (because 6 months had transpired). The cost of brick-level backups is paid on every backup: the extra space, the extra time and the hiding of corrupt warnings.

 

1) Use deleted item retention (this cures problem #1, users deleting things). For me, this is the best thing about Exchange 5.5. Set the retention time to something long, say 30 days (you can use even more if you have the disk space). Show your users how to recover deleted items (it shows up as a menu item in Outlook, when viewing the 'deleted items' folder). This is the best, since you don't have to do anything. The users do for themselves. If they come to you after the retention time has expired, you get to give them a dope-slap and charge their cost center for your time. Read item 3 for recovering after the retention period. See http://support.microsoft.com/support/kb/articles/Q... for recovering deleted items from all folders (thanks to Glen Morangie).

 

2) When someone leaves, don't immediately delete their mailbox. Remove it from any distribution lists, take away any SMTP addresses, set the alternate delivery to their boss (or someone else responsible), put the leaving date in the notes field and mark it hidden. After some time has gone by (say a month) and you are sure so one needs the mailbox items, you can delete the mailbox.

3) This leaves admins who delete too early, users who wait too long, and accidents that corrupt data. To recover a individual items, you have to set up a recovery server (a machine created with a new site and org, that happens to be the same site name and org name as the original). You restore the entire IS to the recovery server and then export the desired items as a .PST file. Import the items from the .PST file into the production server. This method gives you exact control over what you restore. You can export any mix of individual items you choose.

 

Khue
Level 4
I would attempt to do a back up apart from, and outside of the backup window for the rest of your backups. Isolate all sources of I/O for your exchange backups and see what happens.

jlalonde
Level 3

Turns out it was the exchange taking a long time to backup. Doing our data and then exchange stores knocked our time down to just over 3 hours. Huge difference from the normal 14+ hours. Thanks for all the help guys.

CraigV
Moderator
Moderator
Partner    VIP    Accredited
Ken: You are a ninja

Ken_Putnam
Level 6
and that is a good thing, I take it?

CraigV
Moderator
Moderator
Partner    VIP    Accredited
Very...kind of a slang term for someone who really, REALLY knows their stuff. hehehe!