NetBackup 7.5 V-Ray Technology on SQL VMs and Transaction Logs

The following is from a non-public document.
As a Symantec Partner, you should have access to it.
I tried but I couldn't find any public references to the points made by the document.
Hopefully they will put this important information in the publicly available Nbu VMware / SQL admin guides. But for now...

NetBackup 7.5 Feature Briefing - Application Protection for VMware Virtual Machines:
Under the VM MS SQL Server section, regarding the use of the VMware policy to backup entire VMs together with the SQL server inside:

  ...
SQL logs may optionally be truncated using a policy setting. This requires installation
of the Symantec VSS provider as noted above.
If this is selected, logs are backed up to the OS temp directory and deleted. The process occurs for each eligible database.
  ...
Backup restrictions:
- No differential (BLIB) backup support
- No clustering support (I.e., if your sql servers are VMs, they must run as standalone servers)
- No mirroring support (Same as above)

Restore Restrictions:
- Full database restore/recovery only
- Consistency check is disabled
- Page verification is disabled
- Mandatory restore with replace

VM Database backups cannot be mixed with legacy agent backups:
 - Transaction Logs
 - Differentials
 - Filegroups
 - Filegroup differentials

If your VM SQL servers are not running in cluster/mirror configurations, then you can back them up the way you are suggesting, and have the Sym VSS provider truncate the T.logs. However, do not backup the T.logs using a separate MS-SQL-Server policy-type policy because it is not supported.
This makes sense because the policies will break each other's T.log chain, which breaks the restore when T.Logs are used in the process.
Since the above would imply that you'd have to backup the entire VM hourly (according to your T.log backup requirements), I would suggest to instead do SQL backups separately using a MS-SQL-Server policy type policy.

If your VM SQL servers are running in cluster/mirror configurations, then you have to use the MS-SQL-Server policy type to back them up, which does not support having the backup traffic going through the SAN in anyway, and must go through the LAN. (I.e., no SAN transfer mode, and no SAN client because SQL is in VM)
You can still separately backup the individual SQL cluster/mirror nodes using the VMware policy type, but you have to uncheck the check box that says to also backup the SQL app.
In this case, for entire cluster node restores you can first restore the entire VM of the node, then either let the application's own cluster mechanisms handle the rest, or proceed further and restore the SQL DB backup on top of it, which is a two-step restore: OS first, then application.

1. The customer takes V-Ray backup during the night (and truncate the logs)

Only supported if the SQL server is not running in clustered or mirrored configurations.

2. When the day starts, they start taking backups of transaction logs hourly, then truncate the logs

Not supported. The backup will complete successfully, but the two policies will break each other's transaction log chain. This will cause problems during restores.

3. After the end of the day they take another V-Ray backup and so on...

Same problem as the above.

Is this setup correct and can it provide us an RPO of 1 hour during the day?

You'd have to backup the entire VM hourly to provide a SQL RPO of 1.
That is why I recommend treating the OS and the SQL server as unrelate entities and backing them up separately. That way, you will have an OS backup every night, and a SQL RPO of 1 hour.
There are currently too many limitations when you try to protect both the OS and the SQL app in one swift VM level backup.