You cannot simply copy the entire netbackup and volmgr folders, as the binaries will be different for Solaris 8 and Solaris 10. Also, /usr/openv is just a symbolic link on Solaris with the actual installation somewhere else - default is /opt.
Do a catalog backup do disk. Copy/ftp the disk images to the new server and recover catalog from disk backup.
You also need to check device compatibility matrix - I don't think LTO4 is supported with NBU 5.1. You might only be able to use them after upgrading to 6.5.
Replace point 3 with:
Install
and patch NBU 5.1 (Solaris 10 support was added with a patch - if I remember correctly, it was MP5. Look for documentation regarding Solaris 10 support on NBU 5.1).
After restoring catalog, you need to copy vm.conf and bp.conf from old server.
The chances are very slim that O/S device names will match exactly on the new server. For this reason, I would do the following after point 8:
Run sgscan. If all devices are not displayed, run sg.build and sg.install.
Delete all existing devices in NetBackup config.
Run device wizard to re-create devices.
Inventory robot.