I would not focus on the VSS option as a possible issue. Look elsewhere.
Logs:
On the proxy server make the
bpfis ,
bpbkar ,
bpbkar ,
bpvmutil directories. Up the logging to level 5 on the proxy.
bpfis should give you the most help in what is going wrong.
Does the Full backup work successfully.? If so then there must be something to do with mapping or change journals.
Also,
from the Admin Guide confirm the following
Incremental backups of virtual machines must be based on time stamps, not on archive bit. Youmust configure theNetBackup client on theVMware backup proxy server as described in this section.
To configure the client on the proxy server for incremental backups
1 Start the Backup, Archive, and Restore interface from the NetBackup client that is installed on theVMware backup proxy server. For example: Click Start
> Programs>VeritasNetBackup > Backup,Archive, and Restore.
2 Click File >NetBackup Client Properties.
3 On the General tab, make sure
Perform incrementals based on archive bit is clear (unchecked).
• Make sure also your policy type is
FlashBackup-Windows as only this works for 3-FullVM and file for incrementals
• VCB proxy Staging area specified in the Policy should not be encrypted or compressed.
• VMware Tools installed and up to date in guest OS VM.
http://seer.entsupport.symantec.com/docs/320865.htm
• All Incremental VMware Consolidated Backups (VCB) are failing with error 156 and bpfis reports the error "Cannot open disk file: Error : Disk library failed (Unspecified error)"
http://seer.entsupport.symantec.com/docs/320310.htm
• If you have performed Storage Motion of your VM to a newer datastore, then backups fail. And as in the last technote, you might have to restart the vmount service on the proxy. This has occurred to me several times recently. Have expanded non-boot disks in the OS. Full VM backups work. Incrementals fail. Restart vmount service and Incrementals start to work.