recover catalog of NBU instance into BYO server

Technically you can although you should check if the resulting environment would be a supportable one...  (I don't recommend using that method for migrations, but I am biased here as my company has its own product for catalog migrations...)... 

If you did go down that route,   you may have to  build the BYO to look like the Flex, such that the paths where data is held for the catalog, and relational database would be the same...   This is because you can't change where the catalog recovery restores data to, and it will go back to the same paths that it backed up from.

In addition to the paths, you also need to ensure that :-
The new server MUST have the Same name as the Flex Master Instance
The new server MUST also be a Linux system (but that can be physical or virtual)

You will need to engage Veritas for this, speak to your TAM(Technical Account Manager). They have a special team to migrate from different hardware platforms. 

I never read about it do restore appliance in BYO, good question, agree with sugestion about open a ticket in Veritas.