NBAC wont start after upgrade to 7.6 in clustered master server

[Germano]

Hey guys, issue with NBAC after uprading NBU 7.5.0.4 to 7.6.1 on a Windows 2008 R2 Cluster.

We have a Master Server-only clustered with Microsoft Cluster. NBAC was working normally before the upgrade.

Before upgrading, we set the USE_VXSS to PROHIBITED, and the upgrade went fine. Installation logs show the install going thru the AZ files normally. 

After the upgrade, when changing USE_VXSS back to AUTOMATIC, the Authorization and other services won't start.

A few troubleshooting we've made:

1) Running admincmd\bpnbaz.exe -upgrade and admincmd\bpnbaz.exe -setupmaster. Both fail with "Cannot contact the Authorization Service".

2) As I have no problems in loosing the NBAC configs (they are very simple and I could rebuild it from scratch), tried replacing the NBAZDB.db with the NBAZDB.template but the services wouldn't start after that, so I rolled it back.

3) Putting hostnames in server.conf, vxdbms.conf and EngineName key with the same value

    http://www.symantec.com/connect/forums/netbackup-71-75-upgrade-issue-when-nbac-enabled

    Service still doesnt start

4) Compared \sec\az\bin\VRTSaz.conf with a working 7.6.1 NBAC installation, as suggested by another KB.

    We noticed the build numbers didn't match: problematic cluster has it as 5.0.27.0 and our working lab has it as 5.0.54.0. Changing this config didn't help, so we rolled it back too.

5) Started the AZ deamon in debug mode, as suggested by the debugging NBAC KB. Returned looks like database related:

sec\az\bin>nbazd.exe -eazs "C:\Program Files\Veritas\netbackup\sec\az\bin\eazs.loc" -fg 9

...

Entered "db_odbc_cache_handle () "

Left "db_odbc_cache_handle () ": 0 ticks

txn.c(511) Calling SQLError...

txn.c(522) SQLState = [08001] SQLErrMsg = [[Sybase][ODBC Driver][SQL Anywhere]Database server not found]

txn.c(528) Unable to connect to database, db_rc = (0), try_count = 0 

Left "db_odbc_check_db_state () ": 10063 ticks

28/04/2015 15:44:06 db_odbc_check_db_state () failed, rc = 3004 

Left "db_check_db_state () ": 10063 ticks

28/04/2015 15:44:06 External DB engine hasn't been started

28/04/2015 15:44:06 Fatal error encountered: main.c(1358) 3004 3004

28/04/2015 15:44:06 VxSS-vxazd ERROR V-18-3078 Fatal error encountered. (main.c:1358) 

28/04/2015 15:44:06 VxSS-vxazd LOG V-18-4204 Server is stopped.

Relational database is obviously up.

We'd like to bring the NBAC up or at least drop it so we can reconfigure it. But the unsintalling NBAC KB (https://support.symantec.com/en_US/article.TECH189775.html) does't offer the steps for MSCS server.

Any advice?

Thanks a lot.

[RiaanBadenhorst]

Just follow that note and do it on both nodes (where possible).

[Marianne]

Best to log a call with Symantec Support.

Each time I had anything to do with NBAC on a clustered master server, I had to log a Support call.
I find the NBAC documentation extremely inadequate.

[Germano]

Guys,

Tried removing everything (and using the steps that support sent me).

Now dependent services go online with the template database in place, but still getting the same database error when trying to setup NBAC. AZ service also does not go up.

Any toughts? Support is lost, obviously.

FYI, I've attached the procedure support sent me.

[watsons]

This is probably a long shot.. I'd try:

1) to verify if NBDB & NBAZDB has the same version. Also will run a nbdb_admin to validate both database to ensure they're error-free.

2) The NBAZDB template you're trying to use, I assume it is of version 7.6.1? 

3) Double check the VRTSaz.conf  to ensure it's pointing to the correct path of vxdbms.conf (shared drive?)

Since you're fine with removing NBAC, how about uninstalling it first instead of using the "replacing with NBAZDB template"?  Refer to: http://www.symantec.com/docs/TECH190214

[Germano · Accepted solution]

Hey guys,

Sorry the delay, but we finally could work it out: looks like it was all about minor bugs in the upgrade process in the cluster environment. Actually, it seems the upgrade process in a few steps upgrades the files as if it wasn't a cluster.

After running the reset NBAC process for MSCS (support sent us the steps, see attached), we noticed AT and AZ services wouldn't start.

So after going back and forward dozens of times, we noticed 3 problems:

1-After running the setup with REPAIR option, the Authentication service couldn't stop. Looking at the service startup command and paramaters, we've noticed it was pointing to the .EXE file in the C: disk. We set it to the folder in the shared disk (manually changed at HKLM/System/Current Control Set/Services), and the service went up.

2-As the Authorization Service still didn't worked and nbbpaz -setupmaster couldn't run, we started the debug again, and noticed the database log (NBDB folder at the logging directory) hadn't increased a single line since the first upgrade pass. So, after further investigation, we noticed the server.conf file present in the shared disk had only a few parameters, and the server.conf file in the C: folder had it all (including the log path). So copying the server.conf from C: to the cluster shared disk did the trick. (NOTE the server.conf file in the local disk should be only a backup file. All the configuration should be at the shared disk. We neved moved it, so we assumed the upgrade process made "reset" the conf file at the shared disk for some reason)

3-After the server.conf change, bpnbaz.exe -setupmaster stopped returning databse connection error, but was failing with an error saying it could not remove an account from a security group.

Doing the entire process again (disabling services, cleaning all the data files and databases, repair installation, etc) made it work.

Hope this helps other unlucky guys who are given the chalenge to upgrade and NBAC cluster.

Thanks a lot for your help,

Luis