Did you do this install on the Master Server? I run into the same problem while trying to set up NOM, which requires Vxss. I removed the authentication and authorization but that didnt help either. It seems that the SQL database or a .dll within the EMM became corrupt and would not let me reconnect to the EMM. I worked with Veritas Support on this and was able to get it resolved.
The first thing we tried was to replace the corrupt dll that was mentioned in event viewer with a copy of one from the EMM on his installation. After none of that didnt work, the tech had me to remove Netbackup completely from the Master Server. We re-installed and setup the devices again. We did a bprecover and recovered the catalog from the previous night. After the re-installation and the catalog recovery, all the backups worked fine.
The tech said that NOM should not be setup on the Master Server. The documentation said it can be installed on the Master Server as long as it wasn't in a cluster. When I go to setup NOM again, I plan on setting it up on one of the clients. NOM only needs the netbackup client. Not sure if this is what you were trying to do too but that's where I ran into my problem.
You may be able to do a bprecover without un-installing everything to become functional again. This was only a temp fix for me because it became corrupt again the next day. The uninstallation and re-install and bpecover was the solution for me.