Agent Slow with New Backup Server

[bschmidt25]

I have been experiencing consistent slowness with one of our agents since we implemented a new backup server a few weeks ago.  The old backup server was a HP DL380 G5 on Windows Server 2008 R2 Enterprise (x64).  Our new server is a DL370 G6 on Windows Server 2008 R2 Standard (x64).  Both systems are running BE 2010 R3 with the latest updates.  After implementing the new server, I also upgraded all of our remote agents to the latest version.  The biggest issue we are having is slowness while backing up our main file server cluster - almost 4TB of data.  It is a two node cluster on Windows Server 2003 (Unified Data Storage Server).  We went from about 2400MB/min on the old server to between 500-600MB/min on the new one.  We are not hitting our weekend backup window because of this.  We have two tape libraries attached to the system.  Our primary is an Overland Neo 2000e with two LTO5 fiber channel drives.  The other is a HP MSL2024 with one LTO4 SCSI drive.  I ruled the libraries out by running a backup to disk operation from the file server to the backup server and observing the exact same speeds.  I tested the libraries by backing up local data to each unit.  The speeds were 7.5GB/min on the Overland and 2.8GB/min on the HP.  Other things we have already done include moving the file server and backup server to the same switch, hard coding speed and duplex on the NIC and switch, and changing the active node on the cluster.  I have also made sure we are using the latest drivers for the NIC on the new server.  Nothing has made a difference.  This weekend I decided to go back to the old server because I needed to make sure we had a good backup.  The speed was back up to 2400MB/min on the old server.  I opened a case with Symantec, but they say they will not assist with performance issues.  Not sure where else to go at this point.  Any input is welcome.  Thanks in advance!

[CraigV]

Hi there,

What sort of speeds are you getting with a network copy between the file server and the media server, and vice versa...this will rule out the network as being a bottleneck along with the servers...

Post back with an update.

THanks!

[Sriram_MCT]

Hi

 Could you check the virutal memory?

Regards

Sriram

[bschmidt25]

Sorry for the delay.  I actually ended up rebuilding the machine today because I wanted to make sure I had a clean slate again.  The NIC has been replaced (same model - HP NC375i, just a different card).  I also replaced the network cable.  I'm still getting the same speeds - between 400-500MB/min on a backup to disk job.  If I copy the same folder by browsing to it and copying it to the same local disk, I get around 5MB/sec, so roughly 3GB per minute.  Virtual memory settings are set to system managed with 24GB currently allocated.  The system has 24GB of physical memory.  Hope this helps.  Thanks again.

[bschmidt25]

Hey guys.  Thanks for the quick follow up.  I'll be back in tomorrow and post the answers to your questions.  I appreciate your help!

[Simon_B_]

The general speed of file backups mainly depends on the file count. Backing up a large number of files will always be a lot slower than backing up one single large file. Can you observe the disk queue length on the server which is backed up while the backup is running?
I guess that the remote agent was somehow changed to read data in another way than the new agent, which is obviously less perormant in your environment. What you also could try is to run the server at BE2010R3 and still use the 12.5 Agent on the server. This will produce a warning, however the resulting speed would be interesting.
If the speed with the 12.5 agent is back to normal please also try the R2 agent. In the R3 remote agent encryption was added between the agent and the be media server, maybe this is responsible for the performance you are experiencing.

Also: do you have software compression enabled for this job? If yes, disable it.

[bschmidt25]

Thanks Simon.  I was actually debating going back to the old agent.  I have heard that the new agent communicates differently as well.  I can't run a backup on the system at the moment to check the queue length (we have hundreds of users on it during the day), but I will post that information when I can.  I still have a case open with Symantec but they haven't really done much with it.  Does anyone know how I can download R2?  Our code for File Connect only gives us R3.  Thanks again.

[bschmidt25]

I have downgraded the agent to R2 on the secondary (passive) node on the file server cluster and will fail over to it tonight so our full backup scheduled for the weekend runs on it.  I'm hoping we're on to something here.  I'll post my findings on Monday morning.  Thanks again to all!