Throttle bandwidth on VMware backups

Any ideas how you would throttle the bandwidth of a media server that only does VMware VADP backups i.e. none of the clients have an agent installed.

In the master server properties you can throttle bandwidth of the clients of a designated media server. My understanding is the throttling is done on the client-side via the NBU agent.  

The idea here is to restrict the bandwidth leaving a media server that goes through a physical firewall port.  Current throttling is by limiting job count on policies.