After upgrading to NetBackup 8.0, the Java Console fails to connect, certificate error

[POGY]

We had NetBackup 7.6.1 on Solaris 11.2 we upgraded to NetBackup 8.0 during upgrade we had an errors

touch: cannot create /usr/openv/var/global/vxss/tomcatcreds/openssl.cfg: No such file or directory
WARNING: can't open config file: /usr/openv/var/global/vxss/tomcatcreds/openssl.cfg
touch: cannot create /usr/openv/var/global/vxss/tomcatcreds/openssl.cfg: No such file or directory
/usr/openv/var/global/vxss/tomcatcreds/nbwebsvc/.VRTSat/profile/certstore/trusted/*: No such file or directory
/usr/openv/var/global/vxss/tomcatcreds/nbwebsvc/.VRTSat/profile/certstore/*-backup*: No such file or directory

ERROR: /usr/openv/wmc/bin/install/configureCerts failed. Aborting...
ERROR: NetBackup Web Service configuration failed. Rerun
/usr/openv/netbackup/bin/install_bp when the problem has been resolved.
NetBackup server installation complete.

when we attempts to launch the NetBackup 8.0 Administration Console result in the following message:

I tried to generate certificates manually but I get this error
C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -getCACertificate
nbcertcmd: The -getCACertificate operation failed for server netbackup.
EXIT STATUS 26: client/server handshaking failed

I tried to do all steps on this article but without luck
https://www.veritas.com/support/en_US/article.000125372

https://www.veritas.com/support/en_US/article.000125069

[Nicolai]

You got a basic operating system error to deal with:

touch: cannot create /usr/openv/var/global/vxss/tomcatcreds/openssl.cfg: No such file or directory

Seems that /usr/openv is either mounted in read only mode or not mounted at all. Alternative it may be protected by Access Control Lists. 

Certificates is not yore problem - yet.

[POGY]

I checked and found the directory /usr/openv/var/global/vxss/tomcatcreds exists 

but the file openssl.cfg does not exist

[Mouse]

The message 

touch: cannot create /usr/openv/var/global/vxss/tomcatcreds/openssl.cfg: No such file or directory

says a process was unable to create it so it looks like the directory exists but not writeable

[Marianne]

@POGY

Please work with Veritas Support to troubleshoot and fix upgrade issues.

Every effort should be made to complete the upgrade rather than rolling back to 7.6.x.
Have you checked and verified all requirements as per Installion and Upgrade Guides? 
And double-checked upgrade readiness on SORT?

About Java login - please use hostname, not IP address. Even if you have to add hosts entry for the master on your desktop. 

[POGY]

@Marianne

I already made a case and still collecting logs but it takes much time. They already check the prerequisites and verifiy that the certificates didn't generated and can not do it manually

We stuck now and cannot take any backup since yesterday

What I think now is to uninstall the NetBackup and reinstall the old version to recover the catalog

[Nicolai]

Unless you find the cause why /usr/openv seems to be in read only mode, a roll back will likley also fail.

If you have a unix admin in the house, it's now to call him/her.

[POGY]

@Nicolai

I have the Linux admin with me and we work with root user and also we added "nbwebsvc" to root as he sade it hase the same privilages now

we also add full permissions on openv to "others" group

[Nicolai]

Hi @POGY

One hint: if the file system has encountered a error, it may have switched to RO mode. Had the same issues on XFS some weeks ago on a test system.

Once root can "touch" (create empty files) in file systems under /usr/openv (there may be more than one file system)  proceed with repairing Netbackup installation.

[POGY]

@Nicolai thank you

i already run a repair installation after the full permissions to others and nbwebsvc 

I created test touch files on /usr/openv and it's created normally

but still no luck it failes to generate the certificates required

:/usr/openv/netbackup/bin# nbcertcmd -getCACertificate
nbcertcmd: The -getCACertificate operation failed for server backup.
EXIT STATUS 26: client/server handshaking failed

[Marianne]

As per https://www.veritas.com/support/en_US/article.000081350

Linux and non-AIX UNIX:

1. To create the local group, enter the following:

    # groupadd nbwebgrp 
   

2. To create the local user account, enter the following:

    # useradd -g nbwebgrp -c 'NetBackup Web Services account' -d /usr/openv/wmc nbwebsvc 
   

Status 26 seems to be an internal connectivity issue.
Can you please post the contents of bp.conf?
And entries in /etc/hosts for localhost and server itself?
What does 'hostname' command return?
Is there any kind of OS security/hardening in place?

I am still wondering why you are using IP address instead of hostnames. 

Have you checked logs such as bpcd and admin?
And nbwebservice (using vxlogview)?

[Amol_Nair]

nbcercmd reporting a status 26 usually means that eithr the nbwmc service is not running on the master server or not responding correctly..

Could you check “bpps -x | grep -i nbwmc” on the master server directly.. if its running on the master server does the “nbcertcmd -ping” command report any error or?

If itz not running that is the main problem.. do share the install logs and the wmc configuration logs as well to identify wt other errors were reported during the upgrade.. probably you could also attempt to execute the configurecerts command manually

[POGY]

@Marianne

This is how I created the nbwebgrp and nbwebsvc before I run the upgrade

After 6 hours with Veritas support they confirmed that current installation was not able to setup bpnbaz and asked to roll back and do the upgrade in two steps. First upgrade from 7.6.1 to 7.7.3 then to 8.0

I did upgrade from 7.6.1 to 8.0 befor and every thing ends fine I'm not sure why this time these issues happen

[POGY · Accepted solution]

Problem solved after uninstalling NetBackup 8.0 and rollback to NetBackup 7.6.1 and then upgrade to 7.7.3 then 8.0

Thank you all