cancel
Showing results for 
Search instead for 
Did you mean: 

Backup Exec 15 Performance Slow

rreed2
Level 2

Hi, we're another struggler w/ Backup Exec's performance. 

Brand new fresh install of Windows 2012 R2, Backup Exec 15, Dell PowerEdge R430, internal RAID5 (or 6 I don't recall what the Perc on it had available), dual port SAS 6Gbps card to Dell TL2100 w/ two drives, (4) 1G NIC's LACP'ed to one team - and we tried just one single un-teamed NIC by itself, all current firmwares, Windows and BE updates/patches, trying to backup VMware 6.0u1 in the same rack; VMware environment is Dell M1000 blade stack w/ EQL 6110's, all 10Gb in blade stack and storage infrastructure, connected at 10Gb uplink to 1Gb Cisco 3750 switches, the 1Gb to our BE server.  Jumbo frames enabled throughout entire infrastructure.

Same as other folks who have this issue, we can single-stream file copy at full 1Gbps, or multi-stream file copy up to nearly 4Gbps of our team.  We've also tried stripping out LACP team entirely, just one single NIC connected at all, same thing, can easily drive up full 1Gbps throughput at the OS level file copy (either drag-drop in Windows or use mutliple instances of robocopy).  Backup Exec will only pull data at ~300-500Mbps no matter what either B2D or B2T using VMware agent backing up VMDK (not Windows agent).  That's watching network utilization of 300-500Mbps, and BE job rate of 2.5-3.5GBpm which seems to jive.

Our argument is the same as others, network speed is proven to be good since we can move files at speed so BE should be able to do the same.  Everything runs very fast except BE.  Trouble shooting steps taken:

-Repair, despite it being a fresh install; no change

-Local disk defrag, despite being fresh install and OS reporting 0% fragmentation - also wouldn't seem to matter when B2T runs same slow speed; no change

-Trend Micro excluding install and BEData folder, unloaded; no change

-NIC tuning, including enabling/disabling flow control, confirmed 1Gbps full-duplex on each NIC, as mentioned can push up full bandwidth w/ file copies, enabling/disabling off-loading, NIC buffers, flow control on switch; no change

-File size and buffer tuning from w/in BE; no change

-Registry hack buffer tuning via article #85311; no change

-Numerous reboots, including reboot every trouble shooting attempt listed above; no change

-Wireshark ran while running backups; no trouble found, good clean network connectivity, jumbo frames confirmed, good window size, etc.

Backing up a couple of small 20GB VM's runs at about 2.5GBpm/300Mbps average.  Backing up single 460GB VM runs at about 3.5GBpm/500Mbps average.  Even the large VM at the faster speed is unusable if we have 13.7TB worth of critical data - 30TB overall data, that needs to be backed up in a 29 hour window (calculated 90-95 hours for 13.7TB!!).  Please double-check my math just in case.

Opened support case #20190763; overseas technician indicated our speed was "already better than average" or "better than most people get."  Did include notes for above registy hack buffer tuning as well as article #90966, disabling buffer reads.  I'll try this next.

Searching this forum, I did find several similar issues but the threads were closed w/o resolution.  I don't know why.  I also found similar posts on other forums of our same issue w/ no resolution.  We've engaged our VAR about the backup system they've sold us.  Does anyone here have any ideas please?

1 ACCEPTED SOLUTION

Accepted Solutions

teiva-boy
Level 6

This sounds pretty normal for NBD.  Is that what transport you're using, over SAN?

Why do you also not have 10gbE on the Backup server?  The limitation to backupexec among many is performance and lack of many key enterprise features like multiplex and multistreams.  Thus ive found 10gbe almost a requirement especially at your data totals.

Ultimately for any customer with over around ~5TB+ of data i require them to have more than one BE server, or move to a real enterprise server.  It just cannt move that kind of data in a 12hr window, or even 24 in many cases.

View solution in original post

9 REPLIES 9

rreed2
Level 2

Almost forgot, our Dell R430 is 12CPU, 8BG RAM.  CPU usage during backup hovers around 2-3%, overall system RAM usage is about 2GB.

NIC's are onboard Broadcom Netxtreme II w/ latest firmware.  We've also tried tuning the B2D block sizes, concurrent writes, etc. w/ no change.  Same on the tape drive, nothing seems to make any difference.  BE throughput remains unusably slow.  Transport mode list in order is SAN, Hotadd, and NBD.  I presume it's using Hotadd because I see snapshots getting made, but I don't know the product well enough to know for sure.  This is a physical server, seperate from our VMware enviroment.  Even in NBD mode it should be able to push close to 1Gbps throughput if our VMware environment is 10Gbps and BE server is 1Gbps single NIC's/4Gbps LACP team (if, respectfully, BE development hasn't yet become aware of mutliple data streams as of 11/06/2015).

Applied article #90966, no change.  Still ~300Mbps/2.5GBpm.  Takes about 21-22 minutes to backup a couple of 20GB VM's, nearly 2 hours to backup a single 460GB VM.

pkh
Moderator
Moderator
   VIP    Certified

You can do some additional tuning.  See my article below

https://www-secure.symantec.com/connect/articles/tuning-backup-exec

teiva-boy
Level 6

This sounds pretty normal for NBD.  Is that what transport you're using, over SAN?

Why do you also not have 10gbE on the Backup server?  The limitation to backupexec among many is performance and lack of many key enterprise features like multiplex and multistreams.  Thus ive found 10gbe almost a requirement especially at your data totals.

Ultimately for any customer with over around ~5TB+ of data i require them to have more than one BE server, or move to a real enterprise server.  It just cannt move that kind of data in a 12hr window, or even 24 in many cases.

Dawidb
Level 2
Certified

Hello,

I have very similar situations as yours rreed2, In the begining we were using Backup Exec 2012 + MCAfee and I didn`t saw any performance problems. Right now after when licence of MCAfee and BE 2012 has expired we bought Trend Micro and Backup Exec 15. Right now simple incremental backup which should take no longer than 1 hour, takes few hours without any reason...
I can be totaly wrong but I think that root cause is inside of Trend Micro engine, somehow it is impacting performance of backup even when you add exclusions. I don`t have any option to make test with other AV solution but maybe you will be. Please let us know if you will find something. I totaly agree with you that it is not normal situation.

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

In the past versions of ESX did have limitations on performance of the NBD port (which I think used the Service Console port at the time) - I am not sure if there are still limitations but you might want to research this problem from the VMware side against optimizing NBD performance.

I would recommend that if you can, you configure your environment for SAN Transport by presenting the LUN's that hold the datastores to the Backup Exec Server (this assumes your Backup Exec Server is physical, use HotAdd Transport if your Backup Exec Server is virtualized)

Dawidb
Level 2
Certified

In my env. we`re using Vsphere 5.1 (ESXi hosts) with NAS, all LUNs are presented to Backup Exec Server, any comments regarding this ?

rreed2
Level 2

Once I was able to talk to an experienced tech person state-side, it was explained to me that "the ESX kernel will throttle network badwidth and that's why we're only seeing about 300-500Mbps (or 2.5-3GBpm)."  Our BE server being physical we would need to set it up for SAN transport (enabling the Microsoft iSCSI intitator, add it to all our LUNs, etc.) to see usable speeds.  Outside of that, he also indicated our backup size is probably better suited for NetBackup (which would be "an investment") rather than BE.  We found all of this to be unreasonable to have to go through while feeling our expecations were not unreasonable for software out of the box since Veeam can do this easily even when it fails over to NBD.  I'm not trying to sell Veeam or get anyone to switch, they've just set the bar.  And our Veeam envrionment is writing to 1Gbps connected repositories.

Our TL2100 having about 1.2Gbps throughput per tape drive, we didn't see any need to add a 10G card since we could team up the (4) 1G onboard NICs to get well more than the tape is capable of, even w/ both of them running simultaneously (two drives in the library).  Unfortunately we've also learned, among the many things that BE simply hasn't kept up w/ is multi-streaming.  Along w/ what we felt was the unreasonable effort and risk of adding a Windows server to our iSCSI fabric and connect it to VMware-formatted drives, was the administrative overhead of setting up multiple jobs each to point to a different physical tape drive and run at the same time to simulate multiple data streams.  When we tried backing up straight to local disk, which really should be using multiple streams and run at full speed of our NIC team, throughput was the same.  Didn't matter if it B2D or B2T.  BE worked great the last time I used it nearly 10 years ago, but that was backing up a few 40-120GB physical servers w/ the local client.  It doesn't look like it has kept up w/ the times of higher capacities, VMware, multi-streaming, etc.

We've tried every tuning registry hack BE and the internet could throw at us, none of which made any difference.  Having to hack the registry to attempt to squeeze any performance out of this thing was just another reason for us.

Many thanks for the input everyone, it sounds like BE just isn't going to do it for us.  We also mentioned internally this software appears geared more toward mom and pop shop SMB's w/ probably no more than 125-250GB worth of data to backup w/ a full weekend window rather than a viable enterprise solution.  Having to buy this software 2-3 times and setup a fleet of BE servers to further simulate reasonable througput was also out of the question.  Hopefully others will find this before making an uninformed purchase, and our experience (as well as others' it sounds like) will be useful.

teiva-boy
Level 6

Glad youve thought things through methodically and came up with a solution.  Dont forget there is also the free vdp product that comes with vmware.  Better dedupe and replication than veeam, but out of the box not as scalable without a datadomain behind it

virat8586
Level 3

By the answer has been given, but I would also like to mention something.
About the causes and reason and on how to do it in a easy way.
For that you may check out this Veritas blog:
How To Improve Slow & poor Perfirmance