BE2010R3/MSServer2008R2 SlowBackup/HighJobrate (snapshot processing)

[Judh]

I have an issue with one backup. Which appears to be a very slow system state/vss backup on the backup server itself.

A few weeks ago, almost a month now, the backup of my Backup Exec Server started taking 14 hours, when it should be taking an hour.

Found the culprit, what appeared to be a hang up on the vss/system state.  At first I took it for an error, but no errors were being put in event logs, or being thrown by backup exec.  The backups would finish, albeit very slowly, but with the normal job rates.

VSS list writers at first returned some [5] waiting for completion, no error...messages

I found the 'repair vss' routine, performed that, rebooted, no effect on job completion time.  I created a new backup, tried selecting system state and it would sit with an hourglass...I decided to let it sit, and see if it would eventually allow me to select the system state contents.  it did, after 10 minutes. same with vss.  Checked list writers again, everything came up [1] stable.

I am looking at the backup now, it is snapshot processing of shadow copy components, the Job Rate keeps going  up, and currently stands at 7244 mb/min, previously it was snapshot processing of D: at 2900 Mb/min, after already backing up 134G of data from the server. 

Essentially System state/VSS appear to have a huge performance issue which I can't seem to figure is tied to anything because it isn't throwing any errors (eventlogs or be2010), just being extremely slow to access.  All other backups being done from this server of other servers are normal.

[CraigV]

Hi,

Are you on BE 2010 R3? If not, it might be worth your while to upgrade to this as it is free and you're able to use your current licenses. R3 fixed a number of issues with previous versions of 2010.

If you are, then make sure your media server is fully patched, and that these patches have been push-installed to any remote servers you might have.

Thanks!

[Judh]

Yes, I am fully patched, 2010 R3 for at least a year (patched up), Windows Server 2008 R2, Server is up to date on everything that can be.  All other servers backed up normally and have no performance issues.

I am not sure if this makes any difference, but the BE server is also being used as a HYPERV server.

[CraigV]

...in terms of using a Hyper-V host as a media server, it's listed on the SCL so you're OK there.

Are there any other backups/maintenance happening at the time of the backup of the Exchange server?

Thanks!

[Judh]

There are no other backups or maintenance running on this server when this backup occurs.

The performance of this backup is constant and duplicateable, there is no point at which it will complete successfully in the normal timeframe. 

The original backup had 6 or 7 servers being backed up in it, but when the slow issue cropped up, I split the backup into two parts, the first 6 servers in thier own backup, and the media server self backup on its own.  The 6 servers backup normally, and in a normal timeframe of 6 hours +-, but the self backup of the media server is taking between 11-14 hours to complete. 

[Judh]

Each volume (C, D, E etc) will backup in a reasonable amount of time(10-30 minutes, 1700mb/min-2500mb/min etc), but there is a gap of time between each volume of 1-2 hours where no backup or verify occurs. 

Shadow Copy Components used to take 240-275mb/min, now 27m/min but still 'Completes' in 2 minutes

Snapshot processing appears to be taking 1-2 hours between each volume.

The verification process proceeds normally after all backups are completed.

VSSAdmin List writers are all stable, but take 10 minutes to run the command