The safe way is to perform platform migration to RHEL 7.x first using https://www.veritas.com/support/en_US/article.000041077 and then upgrade to 8.x by reading and following https://www.veritas.com/content/support/en_US/doc/125240132-130536378-0/v125240137-130536378
The migration procedure between RHELs will take some time and I'd let the dust settle before performing the upgrade, so I'd refrain from doing it in one shot.
Thank you @Mouse
From support document:
The type and version of O/S on the new master server does not matter provided it supports the version of NetBackup used on the old master server. For example this procedure may be used to replace a Solaris 9 Sparc master server with a Solaris 10 X64 master server. However, use caution if any filesystem extended attributes were in use on the original master server. In such a case, it will likely be necessary that the new server be partitioned using the same filesystem type that existed on the old master server.
I want to install RHEL7 on the new master server (old master server is RHEL5), it means I can't use EXT4 or XFS for file system? old master server fulesystem is EXT3.
The filesystem type or version is not really an issue.
The OS compatibility guide simply requires 'POSIX compliant' filesystem.
I have migrated Solaris master to Linux - cannot remember what filesystem (whatever the default was...)
It was certainly not UFS as on the old Solaris master.
All that you need to ensure that NBU is installed in the same folder structure as old master.
If you had NBU images folder on a separate partition, create same partition on new server.
Or, if you want a different filesystem and you are an experienced system administrator, create symbolic links before installation and catalog recovery.
Marianne has already provided a comprehensive response, I'd just add that unless you have used extended attributes on the file system, you won't have issues, NBU uses simple tar which will not save them. For Linux it does not matter at all because NBU does not use extended attributes and will not save them during catalog backup and restore, you can move between filesystems, as long as they are supported by source NBU (for backup) and destination NBU (for restore).