Found solution at
gnawgnu.blogspot.com/2009/08/windows-2008-r2-backup-exec-and-failure.html
-----------------------------------
I got pretty familiar with the VSSADMIN command while working with Hyper-V and backups so I knew that it could be used to redirect VSS snapshots to larger partitions. The problem I ran into is that it didn't like the fact that the System Reserved partition didn't have a drive letter. So I did the quick fix and used Disk Management to assign it a random drive letter - in this case P:
Then a quick drop to a command prompt and run
vssadmin list volumes
C:\Users\Administrator>vssadmin list volumes
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Volume path: P:\
Volume name: \\?\Volume{a2b716d3-8c1f-11de-a5ed-826d6f6e6973}\
Volume path: C:\
Volume name: \\?\Volume{a2b716d4-8c1f-11de-a5ed-826d6f6e6973}\
Volume path: D:\
Volume name: \\?\Volume{75c2418c-8c0e-11de-ae3c-001143dd2544}\
You'll note there's an entry for all your partitions. Now we set up a ShadowStorage for P:\ (100MB partition). ShadowStorage basically sets aside space on a volume to store snapshots of a volume. In this case I'm going to store snapshots of P: on D:
vssadmin add shadowstorage /For=P: /On=D: /MaxSize=1GB
And you have to put a MaxSize so I picked 1GB.
Now run
vssadmin list shadowstorage to confirm the link has been set up.
C:\Users\Administrator>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Shadow Copy Storage association
For volume: (P:)\\?\Volume{a2b716d3-8c1f-11de-a5ed-826d6f6e6973}\
Shadow Copy Storage volume: (D:)\\?\Volume{75b2419c-8c5e-11de-af3b-001143dd23
44}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: 1 GB (4%)
If you have any other volumes configured for Shadow Copies you'll also see them listed there. (i.e. If you enabled "Previous Versions" for a file share, etc)
At this point you're done. I was able to do a successful backup of the server with the AOFO (Advanced open file option) enabled after making this change. My backup seemed a bit slow but it is an older server so I can't be sure if speed was a machine issue or an R2/Symantec issue.