Backup Performance Issues, only some servers affected

[CCWW]

We have a large virtual environment and our Backup Exec server is a physical server. Symantec server is connected to our tape library via 8Gb fiber connection. Network is a flat 1Gb LAN.

We know the slow performance is not caused by the library or drives (LTO5) because the backup exec server can backup at 8 GB / min. I assume that is because it's going straight to the library via the fiber connection.

The next fastest server runs at 1.5GB / min to tape. That is acceptable performance but still seems kinda slow. The worst performers are backing up at 60MB / min, which is about as bad as it could get. That is slower than USB2.

I thought maybe it was a network transfer issue but all servers can copy data to the Backup Exec server much faster than the backup is going.

All servers are on the same network, same blade enclosure, same SAN storage. There is nothing different about any of them other than their backup speeds.

I've read all of the performance tuning documents but in our situation it's not all servers that are affected, only about 1/2 so we know that the environment is in good shape.

What else can we look at?

[CraigV]

Hi,

Make sure that any AV installed on the servers giving performance issues aren't scanning beremote.exe, or that backups aren't being done during maintenance periods, ie. Exchange maintenance on a nightly basis.

You can also check that the NICs on these servers are hard-coded to the fastest speed they can run at, ie 1GB FULL.

Thanks!

[CCWW]

All the NICs and switch ports have been confirmed to be running at 100Gb full duplex. There is no maintenance running during these jobs. We have strict schedules for when things are performed so it's documented when something is going on.

I can check the AV possibility but all installations of Symantec End Point are configured identically on all services and I've watched the performance montiors during backup jobs and End Point protection is idle.

[CraigV]

...what data are you backing up on those servers?

Thanks!

[CCWW]

All of the servers are 2008R2. It's a dev environment so databases and IIS/web. It doesn't appear to be related to the type of data because we have identical servers with the same applications and data and some are much faster than the others.

[lmosla]

Have you checked Windows Event logs for errors on the slow performing servers?  

Try recreating one of the jobs to see if that changes anything.

[CCWW]

Checked the windows event logs for the servers and found no errors/warnings that would affect backups. Did a LAN speed test from the slow server to backup exec and from the fast servers to backup exec and found both servers copy data to backup exec server at 25MB-30MB/s which should be enough to keep the tape heads going and should equate to a much higher backup rate.

I also tried creating a new job for the slow server and backups are still running at 60 MB/minute.

[CCWW · Accepted solution]

Finally was able to resolve this issue. It was the Symantec Endpoint Protection Firewall. I disabled the firewall through the enpoint policies for all of the computer groups and performance jumped dramatically.

Backup jobs that used to take 4 hrs are now taking 40 minutes.