I would like to know if it is possible to have both VADP and NBU AGENT based backup on same windows machine
I have a windows 2016 server with SQL cluster i want to backup
C drive, System State etc ---> Using VADP backup
Data drive (RDM disks) (D, E, Q etc..) ---> Using NBU agent Backup
Can this be achieved on the windows VM machine ?
Yes, you can configure your vADP to exclude the Data Disk and will just backup the boot disk.
Or in 8.1 the exclude feature is more indepth and you can exclude the VMDK of your choice.
Then use the NBU agent to backup everything else.
If your RDM volume is SHARED between VMs (I.e., bus-sharing enabled), you cannot VADP backup those VMs sharing it.
If your RDM volume is not shared (I.e., just one single VM mounting it), then yes, you can VADP backup that VM. NetBackup will automatically exclude the RDM volume.
There has been so much confusion on the topic about RDM VADP snapshot backups - not just with NetBackup since this applies to all VADP aware backup products - and it is all because VMware actually allows you to snapshot VMs with RDMs if the RDMs are NOT shared. (I.e., bus-sharing not enabled).
You can easily verify this by mounting an RDM to a VM, right click the VM, then create a snapshot manually. It will succeed.
But if you share that RDM to a second VM, you will find that when you right click the VM again, the snapshot options will be greyed out. Here is the VMware KB that explains it.
Since VADP relies on the ability to create snapshots, if a VM contains shared RDMs, then VADP backup products such as NetBackup will not be able to VADP snapshot it for backup.
The reason why there are so many contradicting answers online, is because when people answer "Oh yes you can snapshot a VM that has an RDM attached!", they are not wrong, they are just not giving you the full answer.
Alot of times when people use RDM, they use it because they want the volumes shared between VMs.
So a better answer would have been "VMs with Non-shared RDMs can be snapshoted, but those with Shared-RDMs cannot."
To directly answer your question:
Your proposed plan will work if those RDMs are not shared with another VM. (I.e., bus-sharing is not enabled).
So you can VADP snapshot backup C:\, then Nbu agent backup the RDM drives.
If the RDMs are shared, then the only option is to Nbu agent backup everything.