A possible exception to this- if your tape library/drives are fiber channel connected, you can see them from more than 1 server at a time. Now, if you're talking virtual servers, you'd have to make sure that a fibre card driver works inside the virtual server. Dunno if that's possible.
Also, it's not what you're looking for, but there's another way to tackle this problem. Install BE on both the virtual & physical server. Make the server with the tape drive a Media Server, and the virtual server the CASO (Central Admin something-something) server. All of the scheduling & logs would be on that one, while the actual backups would run on the Media Server that's got the tape hardware.
This option has the down site of requiring more licenses, and doesn't get you totally into a virtual server.