Boot server needs to be of same type as the client you are protecting.
It has nothing to do with the NetBackup master server type.
If you have NetBackup master server on Winx64, and you have Linux, Solaris, Winx86, Winx64 clients then you need to have 4 boot servers, each running Linux, Solaris, Winx86, Winx64.
The compat list does say Win2003 (Enterprise, not Datacenter) x64 is a supported platform to be a BMR server (from 6.5.2), and it also mentions CDs as a special consideration, so check out if your CD is the right one, or if you need a newer one from Symantec Customer Care team.
Given that it's only supported from 6.5.2, it is also possible that the maintenance pack distribution has the install pkg for BMR server - try this 6.5.2A link,
http://support.veritas.com/docs/305654.
Also see Page 25 of the compat list.
ftp://exftpp.symantec.com/pub/support/products/NetBackup_Enterprise_Server/325328.pdf
Cheers,
Abe