First of all Windows 2008 is not supported in NBU 5.x
Second, NBU 5.x itself is not supported anymore.
The only way to have supported configuration is to perform the upgrade on current server to 6.x branch and if target server is windows 2008 you have to upgrade current system to minimum patch level 6.5.4.
After that you can backup/restore the catalog database on a new server with new OS.
In theory it is possible to migrate current 5.x to windows 2003, it will work, but as I mentioned above 5.x branch is not supported.