Hi there,
Here are the answers to your concerns:
Server 2008 R2 64bit with SQL Server 2005 on the same server. Should the SQL Dbase be on a separate system?
Ans: First, you will need to move the SQL Server to seperate server before upgrading the EV. EV 10 requires atleast SQL 2005 SP3 or SP4. So make sure, when you prepare new Server for SQL, install SQL 2005 with SP3 or SP4. The Seperate SQL server is recommended for several performance reasons.
I should be able to do an in place upgrade by running the deployment scanner and applying the 10.0.3. cumulative hotfix?
Ans: If I understand you correct, you have EV 9.0.2 running on Windows Server 2008 R2 x64 then you can perform in-place upgrade to EV 10.0.3.
You can refer following technotes for your assistance while performing above steps:
How to move the Enterprise Vault (EV) SQL databases
http://www.symantec.com/docs/TECH35744
Symantec Enterprise Vault 10.0 Performance Guide
http://www.symantec.com/docs/DOC4553
Symantec Enterprise Vault 9.0 Performance Guide
http://www.symantec.com/docs/DOC2817
I hope this helps..