The odds of you getting 800MB/s out of that device are pretty slim, and probably in very rare or specific use cases and configs. Best to consult with NetApp on recommended configs. I know they have some technical specs that claim xxxMB/s as well as a slew of other numbers, but under what conditions and hardware? It also seems that with VTLOS 6 over v5, it got slower in performance! Perhaps due to integrated Dedupe, or perhaps just bloated code?
http://www.netapp.com/us/products/storage-systems/netapp-vtl/netapp-vtl-techspecs.html
My best recommendations would be to start with at least a 2U server like a Dell R710 or older 2950, or an HP DL380 G5 or better. The 2U as they have more slots avail for things like 10GbE or multiple 4Gb FC cards. Not to mention, most of them can support dual quad core CPU's, and more RAM than is needed for a media server.
From there, you'll need at least 2 upwards of 4 of the above servers to reach that throughput of 800MB/s. Which means the Enterprise Disk license will be needed not only for load balancing of media servers, but also since you are sending data to a VTL, it's a requirement from a licensing standpoint. Windows is a cheaper platform, and can perform every bit as well as RHEL IMO. Just needs to be tuned right both on the OS, and in NBU. 2k8 R2 should be the fastest, but you need to be at NBU7 to make that happen.
Sounds like to me, you have a very fun project, with lots of testing time you'll have to put in. I for one, would love to here some of the progress you've made with any performance numbers you can provide, as well as the tweaks you performed.