cancel
Showing results for 
Search instead for 
Did you mean: 

Mailbox(es) Backup VERY Slow

ross_aseron
Level 4
I've already spent HOURS troubleshooting this issue, have logged one support call and spent time with a 'technician' on the phone, but still no resolution. here's the deal, it seems to be a common one.

running bews 10d, sav 10, smse 4.6

when backing up the mailboxes (not the information store), the throughput throttles down to less than 1mb/min or hangs. oftentimes i can backup individual mailboxes successfully, but if i try to do them all (20), it invariably hangs.

i've already been referred to (or tried myself) the following tech docs:

symantec 2001010813441454
veritas 264851
veritas 238618
veritas 267437
veritas 275013

in fact, this last one seems to be the best fit. except that it suggests that i disable virus scanning via the registry, and that's just plain ridiculous.

i've searched and read several threads here, but they only seem to offer the following suggestions:
- turn off anti-virus scanning (kind of defeats the purpose of having AV)
- make sure all occurrences of outlook are closed (this will never happen)
- ask veritas/symantec for this 'enhancement' (having products of the same company effectively work together hardly seems to me an 'enhancement', but rather something that should be fundamental)

my question for you veritas/symantec tech support is this: is there a solution to this problem, a problem that others have as well, that doesn't involve disabling anti-virus scanning?

thank you,

ross
85 REPLIES 85

Michael_Pietrza
Level 3
Ross,

I am sorry, I am only now taking a look at this thread and scanned the various posts. I think you are actually seeing normal performance. When performing brick level backups, that veritas account has to open each mailbox, read the contents, then write it to disk. EXTREMELY inefficient. And if users have huge mailboxes, versus a bunch of small ones, it will be even worse.

I can't believe you think the process to restore an individual mail item to be difficult. I can perform the whole operation, using a 15 gig mail store, in about an hour.

Utilize the recovery storage group and the free tool exmerge and you will have yourself a nice pst that you can freely pull the contents from.

It's pretty darn simple and if you practiced it, you might find it to be not that hard.

Michael

Robert_Duker
Level 3
I beg to differ with your... Analysis. Take your hour and mulitply it by 5 and you will realize how much of a problem and inconvenience it is.

We need the brick level to work because we are talking about a process that takes, at brick level, a few minutes. Compare taking a 60 Gig database and retrieving a mailbox in under 5 minutes because I did a brick level backup to 5 hours the other way.

Most admins are in a similar position and cannot spend half or more of their day bringing back a single mailbox. There are multitudes of tasks one needs to attend to during the day that preclude wasting that amount of time. Most of us would much rather spend the off hour time on an outomated process that takes Hours of time out of the restore process.

Michael_Pietrza
Level 3
Most admins are not in the same position as most admins know how futile brick level backups are.

If it's that business critical and you can't get backup exec to work for you, try another path. How about someone else's more Exchange specific backup software. Here are two excellent options...

http://www.quest.com/recovery_manager_for_exchange/

http://www.ontrack.com/

Good luck,

Michael

Robert_Duker
Level 3
The point of this thread is that Backup Exec IS working for me. That's why I have posted my settings and specifics in order to help some of those here who are having trouble with it. I have yet to encounter any "problematic" symptoms with brick level backups that haven't been resolved with common sense solutions or a little help from the forum community here (BE techs conspicuously absent most of the time)
Every time I try it, brick level works for me and I will never go back to the old method.

ross_aseron
Level 4
first off, it's nice to know that brick level backups work for some people. robert and brandon, may yours never change.

brandon, i upgraded from 9.1 to 10d in the hopes that it would take care of the slow backup. it didn't. except that one time that i've alluded to in the past posts. i hate to say it, but it sure would make me nervous to do the upgrade after reading how others are having trouble. especially if mine was working.

michael, i've been researching and talking with others about brick level backups and have a pretty decent understanding of how they work and the impact such a 'process' has on throughput. i realize that it won't be anything like the 500mb/min i get when backing up other areas. however, i don't believe that a rate of less than 1mb/min could possibly be the 'expected' throughput. additionally, on one occasion (for reasons that have eluded me) i got the brick level to work at 82mb/min. others have reported similar scenarios.

there's also little doubt in my situation that if i could get the brick level backups running at an acceptable speed, i would do them. i've had to restore individual messages in the past - when my brick level backup worked - and it was very quick and easy. less than 10 minutes. though i've been led to believe the recovery storage groups process is fairly straightforward, i don't think it can compare.

all this is, in a way, beside the point. it seems to be generally accepted that the brick level backups do not function 'as advertised' and that help in getting them to do so is, in effect, nonexistent.

thanks to all who are contributing.

ross

Michael_Pietrza
Level 3
Ross,

"t seems to be generally accepted that the brick level backups do not function 'as advertised' and that help in getting them to do so is, in effect, nonexistent."

I completely agree. It seems that Veritas\Symantec has ALOT of work to do on this product series. It is frustrating to read that BackExec had AD integration when in fact it doesn't and these brick level backups are so inconsistent.

Seems to me things have only gotten worse since Symantec has taken over. Tech support is now outsourced and American techs are few and far between. I had a good relationship with a NetBackup engineer who assured me nothing would change. Too bad that was not true.

Best of luck with your efforts. Sorry I couldn't offer more to the discussion.

Michael

ross_aseron
Level 4
thanks, michael. yeah, i agree that the move to symantec hasn't been a good one for veritas.

ross

Brandon_Johnson
Level 3
Thanks for the info Ross. I took the plunge. It did not fix my problem, so I am back to the drawing board.

Brandon

Victor_Wong
Not applicable
Brandon,

Are you saying the upgrade to SAVCE 10.1 didn't fix the slow backup issue? I am only getting 16 MB/min on the brick level backup as the report says. Yikes! There are only about 90 mailboxes with less than 12 GB of messages. It takes almost 12 hours to finish. I am running BE 10.0. I am going to try to enable to single instance backup for attachments to see how it works. With this rate, we can hardly afford to run the brick level backup even on the weekend.

ross_aseron
Level 4
victor,

i hate to say it, but join the club. brick level backups are no longer viable for me too because mine go even slower than yours. if you look back over the thread you'll see a few predominant themes:

- people with similar configurations getting radically different throughputs
- the general consensus that the brick level backups don't work as touted by symantec
- the total lack of contribution on the part of symantec/veritas techs

as has been sited in this thread, you can check out MS aricle ID 824126 on recovery storage groups which allows you to mount a second copy of an exchange mailbox store on the same computer as the original mailbox store. for those of us with worthless brick level, that appears to be our only option.

ross

Eli_Spaans
Level 4
This is pretty much becoming a long post. So I could say things mentioned before. Sorry for any inconvenience there.

1. Uninstall the Remote Agent on the Exchange Server, reboot, make sure every item of RAWS is gone and reinstall the Agent.
2. Make sure single-instance backup for Exchange is turned off and see what happens.
3. You could just make a store backup and set the Deleted Items retention time (in the Exchange System Manager) to a pretty long time. This way you can restore most mail and still have disaster recovery.
4. Backing up mailboxes has improved a lot from 8.x to 10.x. But still, I haven't seen any BE doing more than 300 MB/min. But that still is a lot more than you're getting so far.
5. Try B2D. Not to a network drive, but to a local or iSCSI drive. I'm pushing about 4000MB/min. for the Exchange store to B2D. Theoretical (real world) max. on our network is 5100MB/min. BTW. Get these B2D files on tape afterwards.
6. I've seen Exchange to Tape Backups drop to less than 60 MB/min. This depends on what kind of tapedrive you're using. Backing up mailboxes to tape directly is not a best practice if you ask me. Don't get the tapedrive in streaming mode and it can die on you within a year. And since most tapedrives do much more than 60 MB/min. (or even 300 MB/min.) nowadays, that could be a problem.
7. Stop the Mail Receive Connector on the Exchange Server. This way you can disable the AV for a while without running too much risk. But I already read you didn't win too much performance there.
8. The Exchange Server. Is it running any apps besides Exchange?
9. Check if either the BE or the Exchange Server is running low on resources during backup of the mailboxes.
10. It could even be a long shot as upgrading the firmware of the tapedrive. This due to possible compatibility issues.
11. Uninstall all tapedrivers, get the newest tapedriver pack from Veritas and install that.

Well, so far for my 2 cents. I don't get 'm anyway, but hopefully this wil help you out.

EDIT: Adding this article which might be usefull:
http://www.msexchange.org/tutorials/Exchange-Backup-Strategies.html

Robert_Duker
Level 3
OK, an update on my notes:
I have lately completed an offline defrag /compression of the mailbox database and am now enjoying a rate of:

Backed up 698542 mail messages in 6327 folders in 112 mailbox(es)
Processed 56,590,354,214 bytes in 5 hours, 41 minutes, and 46 seconds.
Throughput rate: 158 MB/min

I believe I mentioned before that we had pushed our users to archive old email data and reduced our database from over 65Gb to its present size. Defragging, although it took almost the entire night, was worth every minute of my time as now I have plenty of breathing room on my backup process. On the whole, the steps I outlined in my previous posts have netted me over 3 hours time saved on my backups each night.

I hope these notes help out, since I Veritas support has never stuck with anyone to the end of the issue even with me, and my paid service support plan.

ross_aseron
Level 4
robert,

i've been meaning to get to eseutil for a while now. i hope to perform it over the next couple weeks and then i'll compare with my previous results.

back when veritas was only veritas, i found their support quite good. now that they've been assimilated by symantec, i haven't found their support at all.

ross

Craig_Srednick
Level 3
Ross:

I haven't written in a while but I read every post concerning our snail problems. I am currently dealing with Symantec on another issue, and you are so right in saying that Veritas support was so much better than the current Symantec support.

We have an inconsistency error in backing up a UNIX server, and they cannot figure it out.

Wish me luck.


Craig

Russ_Perry
Level 6
Employee
Mailbox backups, in my experience, are usually about 10% the throughput you get with IS backups...in most cases between 20 and 70 M/Min. If you are under 10 M/Min then something is interfering with the backup. The first suspects are database fragmentation and virus scanning/spam filtering. Defragment your IS if possible. Since Backup Exec uses MAPI calls and backs up mailboxes/messages similar to a user accessing via Outlook, the same rules apply. If virus scanning or any other mail monitoring/managing tool is set to scan every inbound/outbound message it can slow things way down. Stop or change scanning methods at your own risk.

In some situations with huge mailbox stores, there just isn't enough time in a day to perform backups even with decent throughput. In those cases I recommend setting deleted item retention period for a week or more and doing less often mailbox backups. The other option is to only back up the high-importance-user mailboxes (find someone else to determine whose mailboxes to back up :-). Finally, you can use recovery storage groups and exmerge to recover mailboxes from IS backups if you want to eliminate mailbox backups altogether...Hope this helps.
Russ

sammyc53
Level 5
I couldn't read all this thread, it's so long.

I ask this one question: With all the time messign with mailbox backups, you could have restored a storage group the Recovery Storage group in Exchange, and ExMerge out the messages or mailboxes that you want.

I have done it multiple times with a 100gb storage group. It never fails.

It's not worth the time to do mailbox backups. If you have a few users that you are worried about, then separate them into their own, smaller store.



PS: In order to use the Recovery Storage Group, be sure to have enough room on your drive to mount another copy of the store, and then some.

But, you can get around this by using a USB drive or network drive. I use a USB drive when I don't have enough space, and even then the 100gb restore is lightning fast. (can;t remember specific numbers...)

Robert_Duker
Level 3
The point is this: I spent the time messing with it because the payoff made it all worth while. Not only am I now much more fmailiar with the whole process, but my backups went from 13.5 hours each to just under 7 using the information contained in this thread. My backup speed went from 68mb to 158mb
The restore time without inerrupting a single user's access to his/her email is a maximum of 45 minutes with a 3Gb mailbox. No muss, no fuss, and I always have a store backup to fall back on if it doesn't work.

I must say I actually spent more time responding to this forum than it took me to impliment all but one of the suggestions made here. So, time well spent? Definately.

Jeffrey_Owen
Level 3
I see I'm in good company with this issue.

Here's some interesting info from the Symantec tech I'm dealing with for my open support call on this. For the most part the technician has not been very helpful because he maintains the performance I'm getting is fairly normal. I find it laughable that "normal" for them is completely unacceptable in a real world situation. They'd be better off taking this feature out of the product until they can remove it's reliability on MAPI.

Below is quoted text from the technician describing what he says causes the further performance issue beyond the inherent issues with using MAPI:

"When the mailbox backup runs into an issue whether its with a mailbox or mail message, the agent lowers the throughput rate to where it is safe to transfer over the files it needs. Since Backups Exec backs up files by resource, the "safe" throughput rate will be maintained until the resource is done being backed up. The resource in this case being the mailboxes. Once the errors during the job are encountered, then the throughput rate will maintain."

He's referring to corrupt email messages in user's mailboxes. So to try and test this I've started going through mailboxes and deleting email messages that BEWS says are corrupt. So far, this hasn't improved performance, but that's not entirely unexpected because one corrupt email at the beginning of the mailbox backup could slow down the entire job if he's right.

So his suggestion is to split the job into multiple jobs and see if performance improves, at least for some of the separate jobs. And also to continue to delete corrupt emails from mailboxes.

Since my mailbox stores are so large I can only test this on the weekends. I have one final test to run this weekend to verify a few things that people have suggested in this thread. If that doesn't improve the backup I'll try this splitting technique and report back with the results in a few weeks.

Stephen_Staffor
Not applicable
Has anyone mentioned the badmail folder under exchsrvr/mailroot/vsi 1. It could be way too big and needs to be cleaned out. Just rename it and create a new one and then delete all the files in it and delete the renamed folder. I have seen slow down due to this folder becoming too large.

Scott_Sinclair
Not applicable
Hi all,

I have just finished reading this thread with great interest and even greater horror. One response from symantec and that's it....what a joke.

Anyway, one of my clients is having pretty much the same issue. thanking over 14.5 hours to run a backup of exchange. I have found the first problem which is a throughput rate of 0.045mb/min when backing up the exchange edb files. The 2 things I have changed is to change the schedule of the exchange server database maintainence from nightly between 2am and 5am (must be a default setting) to only run on the weekends and also defrag the drive where these files are located. Since the public file was over 55gb and the private was just over 10gb it took the night to run. I have scheduled the backup to run tonight and I will let you know if this increases it greatly. I say greatly but an increase to 0.1 would be a big improvment.

SYMANTEC...TAKE YOUR FINGER OUT OF YOUR A** AND HELP!!!