Probably something in the security setting on 2005 that is preventing the install detecting the new SQL 2005 intance. Again if you called for support there not a chance you will supported with that config.
No - you can only password protect the media which NTBACKUP can get around. Netback support encryption or you can get third party hardware that sits between the backup and the tape device
Licensing...the server probably has the license called "Exchange Remote Agent option" and Not Exchange 2000 agent option. The first can only backup exchange 5.5 and the 2000 license can backup all versions of exchange.
