There is unfortunately no 'crossgrade' license option between BE and NetBackup. You will need to get hold of your local Veritas office to negotiate discounts.
About the backup-to-tape requirement - this would've been easy if this was a physical server - it would be possible to configure the Sybase server as SAN Media server to backup itself directly to tape.
Because tape is not supported by VMware, it is not supported by Veritas either (NBU and BE).
Snapshot Client does not seem to be an option either, as Sybase is not listed as supported database/application under 'Supported Functionality' in 'Snapshot Solutions' section of NBU 8 HCL.
So, with the current config (Sybase on VMware), direct-to-tape backup will not be possible.
I see 2 possible options:
1. Deploy NBU Dedupe Media server and implement Client-side dedupe backups (over decent network infrasctructure between VM and Media server).
2. Migrate the Sybase server to physical server and configure as NBU SAN Media Server.
Please reach out to a knowledgeable Veritas Partner to assist with best possible backup design for the entire environment.
If you are a Veritas Partner, please contact your local Veritas SE.