SLOW EXCHANGE BACKUP ISSUE for B2D

[Abhijit_Soman]

Hi

I have following configuration for my backup environment

1 Media server (12.5), 5 Exchange server (2 Exchange 2003 & 3 Exchange 1007), 15 File servers and so on

I am facing a problem with Exchange database backup on all exchange servers. I have configured B2D for all exchange databases, all servers are having local hard drive attached on which databases are located.

All backups are running in LAN environment. B2D is configured on media server local drive. Exchange database backup job rate is 150 MBPS max. for B2D. File level backup of exchange servers is above 1500 MBPS for B2D

I have tested all possibilities to improve job rate but no use.

Please suggest the solution ASAP

Regards

Abhijit

[CraigV]

Hi Abhijit,

Are you sending all Exchange servers to that B2D folder at the same time?
You can also read the following:

http://seer.entsupport.symantec.com/docs/231488.htm

http://seer.entsupport.symantec.com/docs/285756.htm

B2D is going to be slower than tape when backing up unfortunately, but your speeds should be quicker.
WHat happens when you direct your Exchange backups to a tape?

[Lively]

I ran into an issue when creating a new policy that I didn't have the compression set for software.  After correcting that issue my speeds doubled what they were (so they were back to normal).

Are you using any compression?

[Abhijit_Soman]

I have created seperate backup jobs for each exchange sever with different schedule
Regarding compression... the default setting i have kept
A per my understandings, the speed for B2D should be above 1 gig but here the case is different, on tape I am getting speed arround 1 / 1.2 gig for exchange database backup (if backup is fired directly on tape)
As I thought there could be issue with RAID configuration so I have re-configured my RAID with level - 0 but still it is not resolving.
I have also tried by changing my backup server...result is same.

The fact is I have logged the case in Symantec and the case was running for 3 - 4 months and finally Symantec engineering team have raised their hands and answered me that its enironmental issue and we can not work more on it.

[CraigV]

Oh no Abhijit...if you want your server to kill all your information when it tanks, choose RAID0. THere is no parity and therefore no protection.
Your best bet for RAID  levels would be RAID1 which has a better write value than RAID5.
What is the speed of your NICs on your backup server? Have you hard-coded your switch ports and server NICs to the maximum speed they can take?
Again though, are you streaming those jobs to the B2D at the same time?

[Abhijit_Soman]

I know Craig its very dangerous but this is what Symantec engg. team have suggested me to test with raid 0

previously b2d was configured with raid 5, but engg. team thought there could be some performance impact with raid 5 so they have suggested me to go for raid 0

Regarding NIC speed - I have 1 gig network everywhere

Backup server - 2 cards with fault tollerent

exchange servers - 2 cards with fault tollerent

Switches are not hard coaded

no jobs are not streaming (1 job at a time)

[CraigV]

OK well, they're wrong and they shouldn't be recommending RAID0 either. If it worked, would they have told you to stay on that I wonder!
This slow speed is at any time right? Not when you're doing a virus scan, maintenance on Exchange etc?

[NTGuru]

I dissagree with you on the RAID 0 issue if the following is true. There is a duplicate job from the disk to tape. with modern server hardware prefailure detection (assuming you are using an actual server not a white box or pc) RAID0 should not be an issue, also te intent of the symantec support is to eliminate a disk write bottleneck. RAID5, 4, 6 ADG should never be used in a B2D environment due to the excessive write overhead. RAIDs 0,1,10,0+1 depending on you vendor will reduce or eliminate (in the case of RAID0) any write penalty. Once again raid 0 only if you are dumping to tape immediately after the B2D. comments about B2D being slower are in fact not entirely true. in an exchange backup scenario, you are limited based on disk file fragmentation, number of spindles, raid level, network config and Server load. in fact i have found that B2D will shave almost 10-20 percent off a comperable B2Tape time of the same data (I have logs to prove it).

My Exchange Backup Best Practices:
1) Limit your mail stores to 1-2 per raid set Mirror drives reccomended Hot Spare Optional (notice i did not say drive)
2) max out your memory in the server (4GB for 32bit and as much as you can afford or server can handle on 64bit)
3) Use real server hardware. no High performance pc's, ata/sata drives, software raid.
4) Use high quality network switches (Cisco, HP, nortel, extreme...) that support LACP for multiple NIC's to provide high Bandwidth.
5) Set all NIC's and Switchports manually Gig/Full and test throughput.
6) Make sure ALL your drivers and updates are applied to your servers, Os, network cards, and your switch firmware is also up to date.
7)  also check all your SCSI settings or SAS settings. if you are using USB for your tape drive or to an external disk, Trade up.

Example B2D config:

Exchange server:
HP DL380 g5 2x QC 2.13 proc, 16GB ram, 8 15k 146GB drives (4 - R1+0) SAS, windows 2008 x64

Backup server:
HP DL380 g4 2 x DC 1.8 proc, 4GB ram, 2 x 15k 72GB drives (Boot), 14 x 146g 15k RAID0) + HP MSL4048 LTO3 SCSI.

job dumps to disk and then duplicates immediately to tape.

500g B2D 19 hours from exchange, 500G File backup 4.2hr.

[CraigV]

It assumes you have some sort of software (like HP Systems Insight Manager) that notifies you of drives that are in prefailure status. If not, when your disk dies, that's your data gone.
RAID0 would be fine if you didn't mind losing the data on there, but not for anything else on backups. Certainly not to keep multiple days-worth of backups on disk.
150MB/sec to disk is a lot slower than what he should be getting, especially since it's a solid file. That would, or should always, be faster than multiple files that are smaller in size.

[Abhijit_Soman]

Logically ntGURU is right...

RAID 0 - Data Striping, high performance for data writing but high risk (1 disk fail data is lost)
RAID 1 - Data is mirrored, low performance in the writing (data is written twice) but low disk (sustain 1 disk fail)

That means RAID 1, RAID 3 or RAID 5 will not give me write performance

About switches, nics I am using branded switches only (CISCO) all servers are server class machines

Another thing is on tape the backup is happening with good speed, on disk file level backup of same exchange server is very nice speed (above 1.5 gig)

Regarding exchange server performance, more than 4,000 clients are connecting world wide.

issue is something different

[Abhijit_Soman]

Any updates on this??

[CraigV]

Have you looked into uninstalling the RAWS on that server manually, and then reinstalling it via a push-install from BEWS?
Is that agent publishing itself correctly to your backup server?

[teiva-boy]

 Yes, patches, and reinstalling the remote agent after running the patches via the BE console.

Trunk your NIC on the BE server, do not use fault tolerance.  You want the bandwidth for multiple jobs running simultaneously.

Then, reconfigure your B2D location.

A RAID 5 array is fine, or 0, or 1.  It doesn't really matter unless you are doing multiple streams, then you size at roughly one spindle per stream/job.  I like RAID 5 myself.

IMPORTANT, when creating an array, you want the largest block sizes your RAID card can make, use the diskpart command" create part pri align=64" and format using larger cluster sizes.  If it's WIn2k8 as your BE server, you can skip the diskpart commands.


Also HP makes available tools to test disk write and read performance, and I think even network performance.  You should check out where there may be a bottleneck in your servers and/or network.  A good backup app, is a great network tester.

[Abhijit_Soman]

I have already did this

Also tried with BEWS 12.5 with latest updates

[teiva-boy]

 Then verify with the HP tools the read/write performance of your Exchange servers, as well as your BE servers, specifically targeting the source and destination drives.

There is PAT, HPReadData, and HPCreateData  All free and made for troubleshooting performance with HP servers and storage (including tapes)