Help find Backup Exec bottleneck

[jmayniac]

I'm having a performance issue with Backup Exec 12.5.  My backups are running slow, too slow.  I'm currently getting about 2500MB/min backup up a 5GB test file of highly compressible text from our storage cluster.  Verify of the file runs great at about 11000MB/min though, which is what I would expect.  The server is a dual Opteron 2.6GHz IBM eSeries Server.  It has 4GB RAM, 70GB U320 15k SCSI hard drive, only one and dual gigabit NICs, only one is connected right now.  The server is running Windows 2003 Standard Server, although I've tried XP also with the same results.  There are two HP Ultrium 1860 Drives in an ArcVault 48 library.  This is connected to an LSI 22320 SCSI card and both drives are on a different channel.  With the high verify speeds, I wouldn't expect this is a problem with the drives, library or SCSI connection.  Backing up from local disk is the same speed of 2550MB/min, this is with the agent running.  Backing up from a pretty speed Core 2 Duo workstation with a fast SATA drive and Gigabit connection is still around 2600MB/min.  On the same machine, I get 3800MB/min with the agent installed, unfortunately this is not an option on our storage clusters as it runs a proprietary OS.

I've tried different NIC options of teaming and aggregating the two gigabit links, but that didn't help or alleviate the problem.  I've tried connecting directly to the switch the storage cluster is connected to, but that didn't help either, the speeds stll hovered around 2550MB/min.  I've also made sure the ports are set to Gigabit and not auto.

System resources aren't being taxed at all on the server while the backup is running either.  CPU utilization usually runs about 10% or less and memory utilization is more than 2GB free.  The NIC get about 40% utilization while the backup is running.

Here is what I've tried:
Installed Trial of Backup Exec 9.1 - 2500MB/min speed
Installed Trial of Backup Exec 10d - 2400MB/min speed
Installed purchased copy of Backup exec 12.5 - 2550MB/min speed

Using HP LTO 1.0.5.2 drivers - 2550MB/min speed
Using Veritas/Symantec drivers - 2500MB/min speed

I'm not sure what else to do, or what to look at next.  Any ideas?

[CraigV]

2550MB/min is really a good speed for backing up LAN servers.
You can check your switch ports, and make sure they're hardcoded to the maximum speed of your NICs.
Other than that, you can try increase the buffer of your drives in Backup Exec too. I set mine to 1024MB generally. It helps a bit, but doesn't offer massive increases in speed.

[MitchR]

Those are all excellent throughput numbers.  The only times you'll see much better is when you're backing up very large, highly compressable files.

You may want to think about running several jobs in parallel (if you have enough drive heads) - it shouldn't effect the individual job speeds much until you have 4+ jobs running at the same time.

[cam_owen]

I spent months trying to solve this same problem, in my case with v11d. I opened multiple cases with Symantec, and also with HP for server and tape library support. I upgraded server processors, changed to 64bit OS, and more. My numbers match yours, and in my opinion they are too low. I think that the bottleneck is on the server being backed up - the BE remote agent, and backup at the file system level rather than file system block level. I have been frustrated to see that resource utilization of the BE agent is relatively low - under 30% CPU, and I/O resources are well below threshold - both IOPS and queue length. Running parallel jobs increases total throughput, and so I am in agreement with a previous comment that you might run two concurrent jobs. This frustrated me, because if the remote agent can pipe data at twice the rate to two concurrent jobs, it seems it should be able to do as well with one job given that tape and backup server are nowhere near their limits. In the end, however, I was never able to meet my required backup window.