What is the best way to upgrade OS?
The appliance version is 5.3.
The master server OS reads Red Hat Enterprise Linux release 8.6 (Ootpa) and also Oracle Linux Server release 8.6.
The Veritas NetBackup application version is 10.3.0.1.
According to the first table (NetBackup Appliance) on this page, I am supposed to upgrade to RHEL 8.8.
What is the best way I can perform this job? And will the Veritas NetBackup application get affected?
"According to the first table (NetBackup Appliance) on this page, I am supposed to upgrade to RHEL 8.8."
You don't have to upgrade your master server OS version as its version is irrelevant with the OS version used by appliance.
It's always good practice to keep your OS up-to-date wrt security concerns then just run "yum update" on your master server. NetBackup 10.3.0.1 is fully compatible with RHEL 8.x and even 9.x.