Backup Exec 2014 performance very slow

[JohnGWalker]

Hi,

I have been battling this for a couple of days and can't seem to make any headway.  I recently installed Backup Exec 2014 on a client site and it is working perfectly, backups running at over 2Gb/min throughput which seems to be acceptable to me.

I have been put on a different client site now and have been tasked with (amongst other things) getting the Backup Exec solution working.  Backup jobs take a really long time to complete (for instance, I currnetly have a file sever backup running that has done 112gb in just over a day, exchange backups which are about 1.5TB take well over 24 hours to complete, system state backup of the 2 AD controllers with GRT turned on took 5 hours to do 20 GB)

At first I thought this might be network speed so I patched in the other three NIC's and teamed them (OS 2012R2) but this made no difference to the throughput.

So my next logical thing was to take the SAN (B2D location) out of the equation to see if that helped.  I created a local storage on the Backup Exec server and set a backup to run to that.  No change.

I tried copying a file over the network to the BE server and got around a 40 - 50M/sec through put which if my maths is correct would equate to about 2400 MB/min which is the kind of speed I would expect form BE

I then revisited the network and tried backing up the backup exec server to the local storage.  While it was faster, it was still no way near the kind of througput I would expect (last test averaged out at 400MB/min)

Finally I told the backup job to disable Sophos before running and re-enable post job. I am not happy with this as it leaves us vulnerable during the backup window but again this seemed to make no difference to throuput.

So, I think I have something wrong with the Backup Exec server but can't for the life of me figure out what.  It's been a very long time since I last used Backup Exec (it was still a veritas product last time I used it) so I might be missing something obvious. Any Ideas?

Server - HP DL360p G8 (brand new purchased for this application with nothing else running on it)

CPU - 2*6 core 2.2Ghz (average about 10% use)

Memory - 16Gb (40% utalised)

OS - Windows 2012 R2 fully patched

NIC - 4*Gbit NIC's in one teamed interface (currently running around 500Mbit/sec)

Backup Exec - 2014 fully patched

SAN - HP Store Once 4500 with 12*2TB 7.2k SAS disks connected over LAN not iSCSI

[VJware]

What is the throughput with GRT disabled ?

And there is no need to competely disable Sophos (or any other AV for that matter) as long as you have excluded from scanning the BE processes. This KB may help - https://support.symantec.com/en_US/article.TECH210790.html

Is BE 2014 completely patched ? (With atleast SP2) ?

This KB may help as well - https://support.symantec.com/en_US/article.TECH49521.html

Disable "Checkpoint Restart" from the backup job and do ensure the option to prescan is disabled from BE settings - preferences.

[JohnGWalker]

Disabling GRT for the exchange backups initialy gave over 4GB/min but this soon dropped away.  However it is still much faster than with GRT

Disabling Sophos made no difference so I don't think that is the problem anyway.  However, excluding proccesses from Sophos is not an easy proccess (registry editing and Sophos state that it should only be done if absolutley neccessary)

Pre Scan is disabled (always thought this was a pointless option anyway except for when you are initially setting up the backups.  the theory being that backup should complete overnight so you will never need to know how long it has left to run) and checkpoint restart is unticked

BE is already at SP2 with no patches reported by live update

Current test stats (all these jobs are currently running) -

Exchange Mail Stores only - No GRT

145GB in 1.5 hours (1.6Gb/min) 

187GB in 2 hours (1.56Gb/min) 

Total when finished >1.5TB - Estimate 17 hours to complete

Active Directory - System state of 2 DC's with GRT

5GB in 1.5 hours (56Mb/min)

9.7GB in 2 hours (83Mb/min)

total when finished 22GB - Estimate 4.5 hours to complete

Files - Data drives of 3 file servers

30Gb in 1.5 hours (341Mb/min)

31.5Gb in 2 hours (269Mb/min)

total when finished >1.2TB - Estimate 3.2 days

SQL server - files and SQL DB's

38GB no GRT in 1.5 hours (432MB/min)

45.7GB no GRT in 2 hours (490MB/min)

Total when finished >900GB - Estimate 34 hours to complete

Same SQL server DB's incremental only (seperate to the DB's above)

43.2Gb in just under 2 hours (377Mb/min)

For info, Copying 10GB of SQL genereated backup files from the SQL server to the BE server using windows file copy completed in just over 6 minutes which is about what I would expect from a Gbit network, especially with two backup jobs running at the same time.

The Exchange, AD and file servers are on the same 5 host vSphere cluster all on fast disks in their SAN so I think network speed can be rulled out.  The SQL server is a physical box in the same rack and switch as the BE server

[JohnGWalker]

Follow up stats -

Exchange Mail Stores only - No GRT

257GB in 3 hours (1.43Gb/min) 

Total when finished >1.5TB - Estimate 18 hours to complete

Active Directory - System state of 2 DC's with GRT

13GB in 3 hours (74Mb/min)

total when finished 22GB - Estimate 5 hours to complete

Files - Data drives of 3 file servers

34.3Gb in 3 hours (195Mb/min)

total when finished >1.2TB - Estimate 4.5 days

SQL server - files and SQL DB's

66.9GB no GRT in 3 hours (381MB/min)

Total when finished >900GB - Estimate 40 hours to complete

[JohnGWalker]

OK, I may have found the problem and introduced another.

It appears that the HP StoreOnce device has been set up as a NAS device not as a StoreOnce Catalyst which from the information I have found so far is not the right way to set it up.  SO consequently the BE server is pulling the backup data from the clients and then passing it to the StoreOnce device over one of the 4 NIC's

Using it in catalyst mode appears to make the remote agents backup directly to the StoreOnce so utalising the full bandwidth.

So, Is there an idiots guide on how to set up BE 2014 with an HP StoreOnce device?

[pkh]

Check the HCL below to make sure that the model of your device is supported as an OST appliance.

http://www.symantec.com/docs/TECH214803

To use the device as an OST appliance, you would need to buy and install the licence for the dedup option.

To configure the device, read the dedup chapter in the Admin Guide.