Backup speeds randomly dropped - Backup Exec 2010 SP4

[SmartiesRock5]

Hi,

I'm having a big issue with our backups all of a sudden. We installed Service Pack 4 a few weeks back; on a Tuesday morning (not sure if this is related). Backups from Tuesday night to Thursday night were all fine. On Friday night's backup later that week multiple issues started coming up with a few of our servers. (We have ~26 servers only 3 showed issues with slower job rates).

Notes:

These are all types of data being backed up - Full disks, to only C drives, to individual VMs on a disk (all 3 servers have different functions).

All on 2008 R2

Servers are all in LAN all connected by one switch

What was tested:

We tried on different tapes, on two different tape drives (LTO5s) even onto local disk; same results.

Installed updates to the agents on the three servers; no changes to the job rates

Rebooted all three servers that night - seemed to resolved issue for the next night's backup

Issue came up again right after the good backup

We've disabled our SEP (12.1.403) during the duration of a backup with no changes (both on the backup server and the remote server(s))

Fragmentation does not seem to be the issue as one of the disks is 0% fragmented and still having a problem

Tested copying a folder from the remote server to the backup server and got rates of 20mbs-50mbs (so the network does not seem to be the problem)

Last thing I found that really didn't make sense is that I could have 3 simultaneous jobs going, all to the same server with the slow backup rate to different media and that wouldn't affect the job rate - so it doesn't seem like it's an issue with resources since this server was pushing 1800mb/min on one drive alone. For example, I could back up:

Backup C drive - 400mb/min

Backup D drive - 200mb/min

Backup E drive - 500mb/min

All at the same time, or individually, it would still not change the speed to the full capacity it use to do.

Any suggestions/comments would be very helpful..

Thank you

[lmosla]

Hello,

Please check your Windows Event Application and System logs for any errors and/or applications that might be interferring with the jobs.

Also, are your writers stable?  run from a cmd prompt vssadmin list writers

[SmartiesRock5]

All three servers do not show anything in the event logs (nothing of error, critcial, or warning) many of the usual informational events are there and all writers are stable.

[SmartiesRock5]

Is there anyone out there that has also experienced this issue? Or have any suggestions in fixing the issue?

[lmosla]

For troubleshooting see if breaking one of the jobs into smaller jobs helps. If you have a database that you are backing up seperate this into its own job. 

If you have antivirus running make sure that Backup Exec processes are excluded from being scanned.

also check to make sure you are using Symantec drivers for the tape drive. If you arent use tapeinst.exe to install them.

[CraigV]

...unless there is an issue with your network switch (copy a large amount of different types of files between the media server and the server/s giving you an issue), and there isn't some sort of application maintenance happening, I'd recommend opening a case with Symantec about this so that they can investigate further.

Thanks!

[SmartiesRock5]

We've tried only backing up the C drive of one of the servers (only 4GB vs the usual 300GB data job) and still go the same errors.

We've also disabled Symantec Endpoint Protection during the backup window and did not see changes

It can't be the switch since all other servers are fine.

Also when monitoring the servers, there's almost no resources being strained (cpu and memory load is low)

I do have an open case with Symantec that's still under investigation - will update this thread with the solution, if there is one.

Thanks for all the help and I would still appreciate any suggestions anyone has for me to test