Backup exec 12.5 can only be installed on Windows platform as MEDIA SERVER
Hence answer to question 1 :
Q : "Can we define the Unix server as the Media server??"
A : No .
But you can define unix only as 'unix as media server AGENT' ( it just keeps major info of catalogs -IT IS NOT MEDIA SERVER)
Qeustion 2 :
Yes you can protect your whole ESX server (provided agent is installed for same)
Refer the software compatiblity link BE 12.5 : Page 33 of 69
You can also have a Windows Media Server, but have the tape drive attached to the Linux/Unix server by using the Remote Media Agent for Linux (RMAL) - please check the SCL for compatibility as this may not be possible in an ESX environment.