Forum Discussion

blueknigh7's avatar
blueknigh7
Level 2
11 years ago
Solved

Slow Backup Exec Performance after migrating File Servers

So I just migrated our file server from 2008 R2 to 2012 R2, and our backup speeds dropped from over 2 gigabytes/minute to just over 85megabytes/minute.  As you can imagine, this reduced our backup times of several terrabytes from just over a day, to an estimated 24 days.  sad

Some general info

  • Running Backup Exec 2014
  • Server 2012 has deduplication enabled.  2008 did not.
  • Both servers had file shares on iSCSI attached volumes.
  • I used DFSR to help with the replication/migration
  • I tested the old file server, and speeds have not changed
  • I've tested the network speeds, and Server 2012 doesn't have any issues.  I've resorted to using the Windows Server Backup in this situation as my fallback option.  Speeds using that tool are comparable to our 2008 R2 backup job.
  • I've tested backup to disk and backup to tape (LTO5).  Speeds on 2012 are the same either way, while our 2008 machine is just fine.

Obviously, I need this corrected.  But given that BE 2014 and Server 2012 are relatively new, the techs and documentation don't have any answers.

I actually tested the 2012 setup prior to migration, and it seemed to work fine - I was getting the same backup speeds as 2008.  However, something happened when I went into production.

The only lead I have, is that there are other old forum posts that say if DFS is enabled, then BE uses a DFS reader, which is notoriously slow?  Is this still possible?  I was under the impression that this was fixed or patched out.  I have since removed the DFS replication and namespace roles from the server, as they're no longer needed, but my speeds haven't improved.  Is there a way I can tell if this is the case?

I have a tech ticket open, but the techs are just pointing me to "this is how dedup works" articles.  Any thoughts or suggestions would be appreciated!

  • I found the cause of our issue.  It turned out to be anti-virus, Symantec Endpoint Protection irconically enough.

    Even though we had plenty of spare performance headroom, there were still disk queues present, even with very little activity.

    I went through the anti-virus steps as recommended, (don't scan while backup, add beremote.exe to the exception list), but this didn't make a difference.  I've uninstalled the scanner for now, and saw an immediate 27X increase in perfomance.  (No joke,  83mb/min to 2300mb/min is a big jump.)

    Anyway, i might be anti-virus back on at some point and play with the settings again - seems to be some debate if it's a good idea on multi TB file servers.

  • ...it might be possible that the data on the dedupe volume is being rehydrated during the backup, which might explain why your speeds have crashed. Support must be able to give youmore detail on this though as I am sure it would've come up in testing...

    Thanks!

  • I thought this too, but the slow speeds are even when I only backup the C:\ Drive - which isn't (and can't be) deduped.   There's got to be something else happening.

    Plus rehydration speeds are much faster doing copy operations to other file shares - so I don't think this is the main issue.  

  • ...did you upgrade the remote agent on that server? Have you made sure that any antivirus installed isn't perhaps scanning the BE services, and if so, put in exclusions for this...

    Thanks!

  • I found the cause of our issue.  It turned out to be anti-virus, Symantec Endpoint Protection irconically enough.

    Even though we had plenty of spare performance headroom, there were still disk queues present, even with very little activity.

    I went through the anti-virus steps as recommended, (don't scan while backup, add beremote.exe to the exception list), but this didn't make a difference.  I've uninstalled the scanner for now, and saw an immediate 27X increase in perfomance.  (No joke,  83mb/min to 2300mb/min is a big jump.)

    Anyway, i might be anti-virus back on at some point and play with the settings again - seems to be some debate if it's a good idea on multi TB file servers.