Shut down BackupExec on the old server
Copy the entire BackupExec install tree to the new server (includeing the \Catalogs directory if you have re-directed the catalogs)
Install on top of the copied code.
You will have the tape device from the old server still defined and "offline", and will need to re-run the hardware wizard to detect the backup devices on the new server. You'll then need to change all your jobs to point to the new backup device(s)