NetBackup master/media server slow performance issue?

[BackupsVA]

Our environment consists of one W2k3 x64-bit Standard Master/Media server (8GB RAM) with 4 SAN DSSUs  and one internal disk DSSU. We have a 2nd W2k3 x64-bit Standard Media server (6GB RAM) with 10 tape drives.

I have noticed backups going to either the SAN or internal disk on the master/media server are slower when there are more backups running even if the other jobs are going to tape on the 2nd media server. As soon as the tape backups finish the disk backups speed up. I have never seen this behavior before. The only connection between them is the master server so I am assuming the problem is there but I am at a loss. The memory available on the master during backups is not consumed as there is at least 3-4GB available for NetBackup to use.
 
Any ideas on how to find the cause of this issue?

[Omar_Villa]

Is the HBA and PCI bus channel the same where you have your drives and disks, like a mix of a 2 port HBA with one port seeing tape drives and the other the disks? if it is this can explain why it gets faster after the tape backups are done, because they are sharing that resource. Other option is on the switch, do you have any IOA anabled on your switch? this normaly gives priority to Tape backups and that can slow down your disk jobs.

[David_McMullin]

Don't share HBA between disk and tape. If you have both tape and disks on the same HBA there are known performance issues.
Try adding another HBA and moving one over.

[BackupsVA]

The RAID controller and the QLogic boards are not on the same bus. Our tapes are on a different switch to the disk.

[Will_Restore]

Your master server only has DSSU?
Your media server only has tape?

To get from one to another via SAN or Ethernet?

[marekkedzierski]

What's with SAN switch.. Did you check for errors on FC switches ?

[Giroevolver]

Are you backing up from the same clients?

If you have a disk job running on server A to your master server and a tape job on server a to the media server you will have contention as the server is streaming 2 jobs at once.

Possible contention on the network switch/switches if you have alot of jobs running through them.

Master server could be under to much load as it running high I/O and CPU load at the same time.

How many jobs are you running at once on your master and media server? Are the DSU's on different SAN disks or the same set and how many are there?

Are you doing any disk staging at the same time as reading and writing to the same set of disks will have an impact. Is it Fibre channel disk or a FATA type disk?

What tape drives are they? LTO3/4

Performance troubleshooting requires alot of infomation about the environment and is not always easy to solve!