Yes, the 11d version of Backup Exec Continuous Protection Server will install on a 64-bit version of Windows (Windows 2003 SP1). You need to make sure it's updated to the latest service pack.
If that's the case, then depending on the actual errors you are seeing, you could try the following:
Having the CMS or CPA on a domain controller; or simply backing up the domain controller should not be a problem. Basically there are not issues we've seen with that. We do have technotes online on how to restore a domain controller properly through CPS, but I would recommend Backup Exec as opposed to CPS for actually doing a disaster recovery of it.