cancel
Showing results for 
Search instead for 
Did you mean: 

Mailbox(es) Backup VERY Slow

ross_aseron
Level 4
I've already spent HOURS troubleshooting this issue, have logged one support call and spent time with a 'technician' on the phone, but still no resolution. here's the deal, it seems to be a common one.

running bews 10d, sav 10, smse 4.6

when backing up the mailboxes (not the information store), the throughput throttles down to less than 1mb/min or hangs. oftentimes i can backup individual mailboxes successfully, but if i try to do them all (20), it invariably hangs.

i've already been referred to (or tried myself) the following tech docs:

symantec 2001010813441454
veritas 264851
veritas 238618
veritas 267437
veritas 275013

in fact, this last one seems to be the best fit. except that it suggests that i disable virus scanning via the registry, and that's just plain ridiculous.

i've searched and read several threads here, but they only seem to offer the following suggestions:
- turn off anti-virus scanning (kind of defeats the purpose of having AV)
- make sure all occurrences of outlook are closed (this will never happen)
- ask veritas/symantec for this 'enhancement' (having products of the same company effectively work together hardly seems to me an 'enhancement', but rather something that should be fundamental)

my question for you veritas/symantec tech support is this: is there a solution to this problem, a problem that others have as well, that doesn't involve disabling anti-virus scanning?

thank you,

ross
85 REPLIES 85

Craig_Srednick
Level 3
Robert:

85 throughput would make my day. You are fortunate.

I have backup still running form yesterday and its running at 20MB /min and its at 30Gbs, and its almost 24 hours since it started, the total size of the backup will be close to 53 Gbs, so it still has a lot to go.

And I still need to run one this weekend.....


We prefer to do these types of backups on tapes due to the nature of our business and the amount of user's mailboxes.

Thanks,

Craig

Douglas_Dahlber
Level 4
> briefly, for about half an hour. and throughput
> wasn't noticeably better.
>
> i've a thought. im going to get very detailed about
> how i've configured SAV and SMSE and see if we've got
> them doing the same thing. i'll post when i've
> gathered that info.
>
> you up for it?

Forgive me if I get too detailed. Just trying to help out. I'm posting as requested. My results are far from impressive. We store more info under our public folders than in the individual mailboxes. So it's not quite apples to apples as the rest of the posters following this thread.

Exchange 2003 server SP1
Dell PE 2650 - 2 Xexon 2.8 procs with HT enabled.
4gb Ram - Windows Server 2003 SP1
- OS drive 12 GB mirrored partition Perc 3/DI channel 0 (10k drives)
- Logs drive 20 GB mirrored partition Perc 3/DI channel 0 (same physical drives as above)
- Database partition 136 GB Raid 5 on Perc 3/DI channel 1 (15k drives )
- 1 intel gig nic
System runs using on 1.88 GB of ram on average.


SMSMSE Version: 4.5.2.736 (updating next week)
Autoprotec Enabled
Exchange background scanning enabled
Scan message bodies enabled
Scan SMTP messages leaving server not enabled.



SAV 10.0.1.1007 (autoprotect excludes \SMSMSE\Downloads\, \SMSMSE\Quarantine\, \SMSMSE\spamprevention\ , \SMSMSE\Temp\, all drives - \Backup Exec AOFO Store\, data base drive \Mdbdata\*.* , \logs\ exchange trans logs, \exchsrvr\mailroot\, \exchsrvr\mtadata, \exchsrvr\mdbdata\

Note: I have an SMTP gate with much tighter security in a DMZ that handles inbound and outbound email traffic. This email server does not connect directly to the internet.



Backup Exec server:

Dell 2850 2 Xexon 2.8 procs with HT enabled.
2gb Ram
Windows 2003 Server SP1
No SAV
Backupexec 10.1 rev 5629 HFs 4,6,7,8,9,10
CPS server SP1 runs from 6:00 AM to 6:00 PM. Does not run during backup windows.

1 200 GB raid 5 partition for B2D of Mail server backups

In the Mail server backup job properties:
Full - Database & logs (flush committed logs)
Full - Backup messages - Reset Archive bit

Enable single instance is checked
Perform consistency check is checked
Continue with backup if consistency is checked

Job Duplicates to tape when done. Exabyte 430 w/ 2 vxa-2 drives.


Individual Mailbox job results from log:

Backed up 217880 mail messages in 2830 folders in 89 mailbox(es)
Processed 11,644,466,015 bytes in 1 hour, 31 minutes, and 59 seconds.
Throughput rate: 121 MB/min
Public Folders job results from log:

Backed up 88535 mail messages in 2590 folders in 0 mailbox(es)
Processed 40,038,146,715 bytes in 1 hour, 9 minutes, and 31 seconds.
Throughput rate: 549 MB/min

Is that what you were looking for? If I have left out something you wanted to know please let me know.

Robert_Duker
Level 3
OK, if it helps, here's what I changed to make what is now a throughput of 94 for 64 GB taking 10 1/2 hours (Over 850k messages for 108 mailboxes) of data on a single mailbox store running Exchange 2003 SP1 on Poweredge 2600 with SAV:

General:
Full backup selected
Hardware compression
Advanced:
Single Instance enabled
backup following junction points enabled
Nomral priority for remote agent.
Open file backup with a lock

AOFO: off
Exchange:
Full Exchange server backup for database & logs
Full Mailbox backup
Enable single instance for message attachments disabled
I dont use Volume shadow copy, so that consistency check option is disabled as well

I run full IS & Mailbox backups to a Dell LTO2 tape drive. for my purposes, I do not do a full system backup of the the hard drives during the week, only on Fridays and month end, so during the week I'm only backing up Exchange via the Agent and nothing else. (I don't use this server for any other functions or services)

Oh, and the person who installed this (before my time) kept the SQL based Dell open management suite installed & running. I turned that off and SQL is no longer running on this server.

ross_aseron
Level 4
just wanted to let you all know that i'm gathering my info, comparing to yours, and will post later. hopefully today.

thanks to everyone posting here. we may come to some conclusions yet.

ross

Robert_Duker
Level 3
Oh, and one other thing I just completed that seems to have helped bring my speed on the mailbox store up to 98, I bugged all of my users to archive or delete their oldest data and they got rid of or archived 100,000 emails (Leaving me still with 740,000).

I read in the knowledge base that a lot of small emails take much longer to backup than larger attachements and such, so I began a campaign of showing the users how to delete and archive their oldest emails onto a network share which is also backed up regularly. Now I really don't care what they do with the archives and it is incredibly simple to restore them, more so than the Exchange mailboxes, so for me, this seems to have been the best alternative. Even the most die-hard email collectors thought it was great that I would allow them to keep their emails forever as long as they archived them. Of course, you could always take the unilateral approach and institute some exchange policies that would cull the emails anyway, but that's just asking for trouble in most cases.

In the end, it seems that the more I convince the users to take responsibility for cleaning out their emails, the faster my backups become. Even if they are not getting rid of the bulky emails, the stuff they tend to delete are the little emails with very little substance to them, which as it turns out is what does the most good for the backup progress.

Hope that helps...

Craig_Srednick
Level 3
We provide unlimited archive storage for all of our users. We also set a mailbox size limit that once they receive the administrative warning about the size of their mailbox, they call us and we tell them to clean up your mailbox and move older items to your archive folders. If they do not, then they max out the mailbox and then cannot send or receive emails. They do learn very quick.

Even with this procedure, our backups run very slow.

Thanks

tony_adduci
Level 2
Hello,
I concur that 10.d of BENT takes way to long backing Exchange 2003 mailboxes. It appears to be fine on the store but I am seeing results like this:
18.6 gig of mail is taking 28min to backup via the store
and 4h50min to backup via the mailboxes.

WOW!
I have 97,808 messages in total in all my mailboxes with 1109 folders and 29 mailboxes.

I have a LTO-2 drive.

Alvin_Lee
Level 2
Well, here are my details:

Exchange Server:
-------------------------
Dell PowerEdge 2600 - Dual Pentium IV 3.0 Ghz, HT Enabled, 4 GB RAM
OS on RAID 1 (2 x 36GB Ultra320 SCSI Drives)
Database on RAID 5 - (4 x 72 GB Ultra320 SCSI Drives)
Log Files on RAID 1 - (Fibre Channel SAN enclosure)

Running Exchange 2003 SP1 with two storage groups

BackupExec Server:
-----------------------------
Dell PowerEdge 2500 - Dual Pentium III 1.0 Ghz, 4.5 GB RAM
OS on RAID 1 (2 x 18GB Ultra320 SCSI Drives)

Running BackupExec v10.1 Rev 5629

Exchange Backup:
---------------------------

With both storage groups:

Backed up 781805 message in 7385 folders in 247 mailboxes
Processed 26,326,443,491 bytes in 9 hours, 58 minutes and 53 seconds
Throughput rate: 41.9 MB/min

Public folders:

Backed up 340 mail messages in 4117 folders in 0 mailbox(es)
Processed 29,541,399 bytes in 2 minutes and 52 seconds
Throughput rate: 9.83 MB/min

I then tried to backup only one storage group, here are the details of that:

Backed up 413268 messages in 4250 folders in 146 mailbox(es)
Processed 16,266,686,987 in 7 hours, 12 minutes, and 25 seconds
Throughput rate: 35.9 MB/min

Brian_Long
Level 3
Stats:
Backed up 4 Exchange Server stores
Backed up 1 Exchange Server logs
Processed 30,286,496,570 bytes in 1 hour, 29 minutes, and 23 seconds.
Throughput rate: 323 MB/min

The Mailbox only portion is:
Byte count: 26,630,782,226 bytes
Rate: 93.00 MB/Min

Other:
We do not, and have not used Outlook on the media server, so I am not sure where this fits in as a requirement.
We are running TrendMicro on the Exchange server, during backups, without incident.

Question:
You said you do not see anthing in BE other than the job failure from time-out issues, do you see anything on the exchange server event logs?

Then of course the other 'yeah did that' questions:
checked DNS?
checked AD?
Anyone "changed" the name of the storage group(s) since you started backing up Exchange?
Is the backup account named such that it has no conflicts e.g. an account of Robert would conflict with an account of Rob?
Is the IO on the Exchange box rocketing during backups?
What about the IO on the BE box?

My personal opinion, blame Veritas first and everything else later. But ya never know. We have had two instances in the last few years that were only resolved with reinstalling BE.

ross_aseron
Level 4
everyone,

sorry, i've been out of commission for a couple days and am just getting back to things. which means i have other things on my immediate plate that i must see to first.

thank you all for posting and i look forward to trying to figure this out. i still have to collate my data and compare it to that given here.

and i'd like to thank the symantec/veritas techs for ... oh, wait ... what am i thinking ... how silly of me. thanking them presupposes their existence.

ross

brad_flom
Level 2
My company does outsourced management for various businesses and I have ran into this same issue with a couple of our customers. The issue I have found is with Symantec Mail Security. I schedule a pre job to stop the SMSMSE service on the remote Exchange server and then schedule a post job to start the SMSMSE service after the backup is finished. This helped out tremendously. The backups are now back to normal, however I am having problems restarting the SMSMSE service using the following syntax in my batch file: netsvc smsmse \\server name /start. Does anyone know of a better way to do this?

ross_aseron
Level 4
brad,

i'm looking forward to giving that a try. though i have to reiterate what's been said here before: any fix that turns off various protections in order to succeed isn't an ideal fix. especially when the products all come from the same company.

thanks for the post,

ross

brad_flom
Level 2
Ross,

I agree with you. I would like to find a better solution as well, but until that happens I feel this is the best one. At this point, pick your poison. Slow backups or turning off SMS for a few hours and rely on NAV client for protection.

ross_aseron
Level 4
brad,

sorry for the delayed response. i'm only in the office tues - thurs, so if there are fires to put out, i've got to do those first. i'm sure you know what i mean.

i hope to try the backup with SMSME turned off within the next week. i'll post my results.

as it is now, the ms tech doc on using recovery storage groups seems like a good failsafe.

ross

TO SYMANTEC/VERITAS TECHS:
I find your lack of involvement and contribution appalling.

Brandon_Johnson
Level 3
Wow, I have a headache after reading all this. My first question, where is the Symantec moderators in all this? This is probably going to make you sick, but we backup about 15GB in mailboxes in less than 2 hours. We are using BE 10.0 Rev 5484 for our Exchange 2003 Environment. Our media server is Windows 2K3 Server as well as our Exchange Server. For Virus protection, we are using McAfee Groupshield. We have all of our servers on a gigabit ethernet segment of our network, and have had little to no problems with our backups.

Our Exchange settings from BE are:

-Exchange server backup method: Full - Database & Logs (flush committed logs)
-Mailbox backup method: Full - Backup messages - Reset archive bit
Checked Enable instance backup for message attachments

I have been dealing with an issue myself with BE 10. Out of the blue, the software flags my tapes as "Not Appendable (End Marker Unreadable)". I use Sony's tape utility to check the tapes and they are all good. I just now got a response from Symantec asking me some basic questions...here we go again.

I hope that at least some of this might help you.

Craig_Srednick
Level 3
Brandon:

Thanks for the info.

We are running very similar setups except we are BE 10.0 Rev5629, as suggested by Veritas....wait Symantec tech's....
Our Exchange settings are the same..
But we still experience the same snail pace backups as we are all experiencing.

Now as far as the Moderators.......good question...

We all ask the same, where is the help that we pay for and need??

now take 2 tylenol for your headache and call us in the morning.

Craig

ross_aseron
Level 4
brandon,

you get that kind of throughput on individual mailbox backups? not the information store, right?

that's amazing.

how nice for you. said with only a little sarcasm. :)

ross

Brandon_Johnson
Level 3
Craig,

I bet right now, they are sleeping.

Have you considered implementing a parallel network with it's sole purpose being for your backup's? Again, that is probably a work around instead of a fix. Have you put a sniffer on your network monitoring traffic between the servers to see how chatty they are? How many servers are you backing up? We are backing up 15 servers in a Windows from 5 PM to 8 AM. Granted, a lot of that is Incremental.

I was going to install that Rev, but maybe I will stay away now that I see you having your problems.

Thanks, Brandon

Brandon_Johnson
Level 3
Ross,

No, I backup the Information Store and the Individual Mailboxes. I guess I will never complain about my backup being slow again!

Brandon_Johnson
Level 3
Ross,

I just got information from Symantec to install 10.1, and that would fix my problems. Now, did you have the slow Mailbox backups before 10.1...or only after? Thank you.

Brandon