Cannot login to admin console using jnbSA

Hi,

Needed help with connection to the admin console.

Netbackup version 8.0

Master server: RHEL 6.2

I am attempting to open the admin console by directly connecting a display to the master server.

On attempting to open Netbackup admin console using /usr/open/java/jnbSA, I was getting the below error.

"NetBackup could not initiate a trust relationship with host hostname. Ensure that your Domain Name Service (DNS) resolves the hostname to the correct IP address and check network connectivity".

 

Found this article which stated it is due to nbatd service not running.

https://www.veritas.com/support/en_US/article.100014059

 

On checking the running services using ./bpps -x, I found that nbatd service is not found to be running. (Attachment- bpps output_1)

Tried restarting the services as mentioned in the article, 'If nbatd is not mentioned as running on the system, restarting NetBackup would solve the problem', it was not of any help as nbatd did not come up.

Tried a /usr/openv/netbackup/bin/nbatd as well, but no use.

Also, in the attachment (bpps output_1), we can see the nbwmc service running. However, on running bpps within the next 5 seconds, that service seems to be missing.

Then, tried the steps as mentioned in the marked solution in the discussion

https://vox.veritas.com/t5/NetBackup/Netbackup-could-not-initiate-a-trust-relationship-with-host/td-...

Deleted the folder with the master server name, restarted services, and tried /usr/openv/netbackup/bin/jnbSA.

New error pops up

"The host does not have Netbackup Hostname based security certificate installed. The certificate is mandatory to establish a secure connection"

Reverted the changes from the backup copy of the files.

 

Tried the steps as mentioned in the marked solution of the discussion,

https://vox.veritas.com/t5/NetBackup/nbatd-is-failing-to-start/m-p/689744

 

Error:

[root@prdspnbapp01 bin]# /usr/openv/netbackup/bin/admincmd/bpnbaz -ConfigureAuth

You will have to restart NetBackup services on this machine after the command completes successfully.

Do you want to continue(y/n)y

Gathering configuration information.

ERROR: Failed to start security services - /usr/openv/netbackup/bin/nbatd -f

Unable to fork child process.

On searching some more, I stumbled upon this. (Attachment- nbcertcmd)

I found out that a certificate has expired on March 27, 2019. I tried steps to renew it, but to no avail.

I changed the system date to February, and the console started up, however with an error message stating that the Netbackup Web Management service is not running.

 

Could I have some help with getting nbatd and nbwmc up and running, and the admin console running without having to change the system date to some old date?

Regards,

Ajo

7 Replies

Re: Cannot login to admin console using jnbSA

bpps output_1:

[root@prdspnbapp01 bin]# ./bpps -a
NB Processes
------------
root 15585 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root 15590 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root 15607 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbdisco
root 15725 1 42 11:50 ? 00:02:21 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 5
root 15796 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbevtmgr
root 15803 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbaudit
root 15927 1 0 11:50 ? 00:00:01 /usr/openv/netbackup/bin/nbemm
root 15933 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbrb
root 15964 1 0 11:50 pts/3 00:00:00 /usr/openv/netbackup/bin/bprd
root 15983 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/bpcompatd
root 15988 1 0 11:50 pts/3 00:00:00 /usr/openv/netbackup/bin/bpdbm
root 15996 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbjm
root 16001 15988 0 11:50 pts/3 00:00:00 /usr/openv/netbackup/bin/bpjobd
root 16013 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbpem
root 16034 16013 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 16044 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbstserv
root 16083 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbrmms
root 16168 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbsl
root 16218 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbim
root 16248 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbars
root 16449 15988 0 11:50 pts/3 00:00:00 /usr/openv/netbackup/bin/bpdbm
root 16450 15996 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbjm
root 16526 1 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/nbsvcmon
root 16797 16083 0 11:50 ? 00:00:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -DPSPROXY
root 17138 16013 0 11:51 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem_cleanup
nbwebsvc 18138 1 99 11:55 ? 00:02:40 /usr/openv/java/jre/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dnbwmc -Xrs -d64 -Xms512m -Xmx1024m -Djava.library.path=/usr/openv/lib -DNB_HOSTNAME=prdspnbapp01 -DNB_INSTALL_DIR=/usr/openv -DVAR_GLOBAL=/usr/openv/var/global -Djava.util.logging.config.file=/usr/openv/wmc/config/tomcat.logging.properties -Dvrts.NBJAVA_CONF=/usr/openv/wmc/config/nbj.conf -Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/wmc/config/Debug.properties -Dlog.conf.path=/usr/openv/netbackup/nblog.conf -Dvrtsat.home=/usr/openv/netbackup/sec/at -Dvrtsat.datadir=/usr/openv/var/vxss -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -XX:ErrorFile=/usr/openv/wmc/webserver/logs/hs_err_pid%p.log -Djava.endorsed.dirs=/usr/openv/wmc/webserver/endorsed -classpath /usr/openv/wmc/webserver/bin/bootstrap.jar:/usr/openv/wmc/webserver/bin/tomcat-juli.jar -Dcatalina.base=/usr/openv/wmc/webserver -Dcatalina.home=/usr/openv/wmc/webserver -Djava.io.tmpdir=/usr/openv/wmc/webserver/temp org.apache.catalina.startup.Bootstrap start


MM Processes
------------
root 15944 1 0 11:50 pts/3 00:00:00 /usr/openv/volmgr/bin/ltid
root 15951 1 0 11:50 pts/3 00:00:00 vmd -v
root 16454 15944 0 11:50 pts/3 00:00:00 tldd -v
root 16455 15951 0 11:50 pts/3 00:00:00 rdevmi -sockfd 14 -p 50 -r
root 16471 15944 0 11:50 pts/3 00:00:00 avrd -v
root 16481 1 0 11:50 pts/3 00:00:00 tldcd -v

 

 

nbcertcmd:

[root@prdspnbapp01 bin]# ./nbcertcmd -getCAcertificate
nbcertcmd: The -getCACertificate operation failed for server prdspnbapp01.
EXIT STATUS 8506: The certificate has expired.
[root@prdspnbapp01 bin]# ./nbcertcmd -getCAcertificate
nbcertcmd: The -getCACertificate operation failed for server prdspnbapp01.
EXIT STATUS 8506: The certificate has expired.
[root@prdspnbapp01 bin]# ./nbcertcmd -getCAcertificate
nbcertcmd: The -getCACertificate operation failed for server prdspnbapp01.
EXIT STATUS 26: client/server handshaking failed
[root@prdspnbapp01 bin]# ./nbcertcmd -getcertificate
EXIT STATUS 5950: Certificate already exists.
[root@prdspnbapp01 bin]# ./nbcertcmd -listcacertdetails
Subject Name : /CN=nbatd/OU=root@prdspnbapp01/O=vx
Start Date : Jan 28 09:29:48 2014 GMT
Expiry Date : Jan 23 10:44:48 2034 GMT
SHA1 Fingerprint : AB:E5:C9:48:78:0C:2A:CF:69:2A:C7:9B:CC:93:A4:8A:39:06:4A:31

Operation completed successfully.
[root@prdspnbapp01 bin]# ./nbcertcmd -listcertdetails
Master Server : prdspnbapp01
Host ID : be819324-0697-43a6-9f92-957b74409288
Issued By : /CN=broker/OU=root@prdspnbapp01/O=vx
Serial Number : 0x6abadd960000000a
Expiry Date : Mar 27 09:53:05 2019 GMT
SHA1 Fingerprint : 58:AB:F0:07:F5:C4:59:9A:6C:F0Smiley Very Happy0:24:5E:93:44:F5:58:F7:78:2F

Operation completed successfully.

 

Tags (2)

Re: Cannot login to admin console using jnbSA

My knowledge about NBU 8.x security is unfortunately non-existent...
I was hoping that @Amol_Nair could assist here.

Please read through this discussion that seems to be more relevant than the NBU 7.6 discussion:
https://vox.veritas.com/t5/NetBackup/How-to-renew-NBU-812-Master-Server-security-certificate/td-p/85...

Re: Cannot login to admin console using jnbSA

Try to start nbatd manually:

# export LD_LIBRARY_PATH=/usr/openv/netbackup/sec/at/lib/
# /usr/openv/netbackup/bin/private/nbatd -c /usr/openv/var/global/vxss/eab/data

and after that try to start WMC:

# /usr/openv/netbackup/bin/nbwmc

 

Re: Cannot login to admin console using jnbSA

Tried both:

# export LD_LIBRARY_PATH=/usr/openv/netbackup/sec/at/lib/
# /usr/openv/netbackup/bin/private/nbatd -c /usr/openv/var/global/vxss/eab/data

 Output after that:

[root@prdspnbapp01 ~]# ps -ef|grep nbatd
root 6254 14830 0 18:06 pts/1 00:00:00 grep nbatd
[root@prdspnbapp01 ~]#
[root@prdspnbapp01 ~]#
[root@prdspnbapp01 ~]# /usr/openv/netbackup/bin/bpps -a
NB Processes
------------
root 1147 16013 0 Apr08 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem_email
root 4613 16083 0 18:00 ? 00:00:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -DPSPROXY
nbwebsvc 6347 1 99 18:06 ? 00:00:04 /usr/openv/java/jre/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dnbwmc -Xrs -d64 -Xms512m -Xmx1024m -Djava.library.path=/usr/openv/lib -DNB_HOSTNAME=prdspnbapp01 -DNB_INSTALL_DIR=/usr/openv -DVAR_GLOBAL=/usr/openv/var/global -Djava.util.logging.config.file=/usr/openv/wmc/config/tomcat.logging.properties -Dvrts.NBJAVA_CONF=/usr/openv/wmc/config/nbj.conf -Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/wmc/config/Debug.properties -Dlog.conf.path=/usr/openv/netbackup/nblog.conf -Dvrtsat.home=/usr/openv/netbackup/sec/at -Dvrtsat.datadir=/usr/openv/var/vxss -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -XX:ErrorFile=/usr/openv/wmc/webserver/logs/hs_err_pid%p.log -Djava.endorsed.dirs=/usr/openv/wmc/webserver/endorsed -classpath /usr/openv/wmc/webserver/bin/bootstrap.jar:/usr/openv/wmc/webserver/bin/tomcat-juli.jar -Dcatalina.base=/usr/openv/wmc/webserver -Dcatalina.home=/usr/openv/wmc/webserver -Djava.io.tmpdir=/usr/openv/wmc/webserver/temp org.apache.catalina.startup.Bootstrap start
root 15585 1 0 Apr08 ? 00:00:07 /usr/openv/netbackup/bin/vnetd -standalone
root 15590 1 0 Apr08 ? 00:00:02 /usr/openv/netbackup/bin/bpcd -standalone
root 15607 1 0 Apr08 ? 00:01:36 /usr/openv/netbackup/bin/nbdisco
root 15725 1 9 Apr08 ? 09:49:04 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 5
root 15796 1 0 Apr08 ? 00:00:38 /usr/openv/netbackup/bin/nbevtmgr
root 15803 1 0 Apr08 ? 00:01:13 /usr/openv/netbackup/bin/nbaudit
root 15927 1 0 Apr08 ? 00:10:41 /usr/openv/netbackup/bin/nbemm
root 15933 1 0 Apr08 ? 00:01:00 /usr/openv/netbackup/bin/nbrb
root 15964 1 0 Apr08 pts/3 00:00:18 /usr/openv/netbackup/bin/bprd
root 15983 1 0 Apr08 ? 00:00:03 /usr/openv/netbackup/bin/bpcompatd
root 15988 1 0 Apr08 pts/3 00:00:03 /usr/openv/netbackup/bin/bpdbm
root 15996 1 0 Apr08 ? 00:00:43 /usr/openv/netbackup/bin/nbjm
root 16001 15988 0 Apr08 pts/3 00:00:31 /usr/openv/netbackup/bin/bpjobd
root 16013 1 0 Apr08 ? 00:00:27 /usr/openv/netbackup/bin/nbpem
root 16034 16013 0 Apr08 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 16044 1 0 Apr08 ? 00:01:04 /usr/openv/netbackup/bin/nbstserv
root 16083 1 0 Apr08 ? 00:00:39 /usr/openv/netbackup/bin/nbrmms
root 16168 1 0 Apr08 ? 00:01:01 /usr/openv/netbackup/bin/nbsl
root 16218 1 0 Apr08 ? 00:00:26 /usr/openv/netbackup/bin/nbim
root 16248 1 0 Apr08 ? 00:03:30 /usr/openv/netbackup/bin/nbars
root 16449 15988 0 Apr08 pts/3 00:00:01 /usr/openv/netbackup/bin/bpdbm
root 16450 15996 0 Apr08 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbjm
root 16526 1 0 Apr08 ? 00:03:49 /usr/openv/netbackup/bin/nbsvcmon
root 17138 16013 0 Apr08 ? 00:00:01 /usr/openv/netbackup/bin/nbproxy dblib nbpem_cleanup


MM Processes
------------
root 15944 1 0 Apr08 pts/3 00:00:12 /usr/openv/volmgr/bin/ltid
root 15951 1 0 Apr08 pts/3 00:00:15 vmd -v
root 16454 15944 0 Apr08 pts/3 00:00:00 tldd -v
root 16471 15944 0 Apr08 pts/3 00:00:11 avrd -v
root 16481 1 0 Apr08 pts/3 00:00:00 tldcd -v

 

Nbatd is still not up.

Nbwmc dies out after a while automatically.

[root@prdspnbapp01 ~]# /usr/openv/netbackup/bin/bpps -a
NB Processes
------------
root 1147 16013 0 Apr08 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem_email
root 4613 16083 0 18:00 ? 00:00:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -DPSPROXY
root 15585 1 0 Apr08 ? 00:00:07 /usr/openv/netbackup/bin/vnetd -standalone
root 15590 1 0 Apr08 ? 00:00:02 /usr/openv/netbackup/bin/bpcd -standalone
root 15607 1 0 Apr08 ? 00:01:36 /usr/openv/netbackup/bin/nbdisco
root 15725 1 9 Apr08 ? 09:49:05 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 5
root 15796 1 0 Apr08 ? 00:00:38 /usr/openv/netbackup/bin/nbevtmgr
root 15803 1 0 Apr08 ? 00:01:13 /usr/openv/netbackup/bin/nbaudit
root 15927 1 0 Apr08 ? 00:10:41 /usr/openv/netbackup/bin/nbemm
root 15933 1 0 Apr08 ? 00:01:00 /usr/openv/netbackup/bin/nbrb
root 15964 1 0 Apr08 pts/3 00:00:18 /usr/openv/netbackup/bin/bprd
root 15983 1 0 Apr08 ? 00:00:03 /usr/openv/netbackup/bin/bpcompatd
root 15988 1 0 Apr08 pts/3 00:00:03 /usr/openv/netbackup/bin/bpdbm
root 15996 1 0 Apr08 ? 00:00:43 /usr/openv/netbackup/bin/nbjm
root 16001 15988 0 Apr08 pts/3 00:00:31 /usr/openv/netbackup/bin/bpjobd
root 16013 1 0 Apr08 ? 00:00:27 /usr/openv/netbackup/bin/nbpem
root 16034 16013 0 Apr08 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 16044 1 0 Apr08 ? 00:01:04 /usr/openv/netbackup/bin/nbstserv
root 16083 1 0 Apr08 ? 00:00:39 /usr/openv/netbackup/bin/nbrmms
root 16168 1 0 Apr08 ? 00:01:01 /usr/openv/netbackup/bin/nbsl
root 16218 1 0 Apr08 ? 00:00:26 /usr/openv/netbackup/bin/nbim
root 16248 1 0 Apr08 ? 00:03:30 /usr/openv/netbackup/bin/nbars
root 16449 15988 0 Apr08 pts/3 00:00:01 /usr/openv/netbackup/bin/bpdbm
root 16450 15996 0 Apr08 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbjm
root 16526 1 0 Apr08 ? 00:03:49 /usr/openv/netbackup/bin/nbsvcmon
root 17138 16013 0 Apr08 ? 00:00:01 /usr/openv/netbackup/bin/nbproxy dblib nbpem_cleanup


MM Processes
------------
root 15944 1 0 Apr08 pts/3 00:00:12 /usr/openv/volmgr/bin/ltid
root 15951 1 0 Apr08 pts/3 00:00:15 vmd -v
root 16454 15944 0 Apr08 pts/3 00:00:00 tldd -v
root 16471 15944 0 Apr08 pts/3 00:00:11 avrd -v
root 16481 1 0 Apr08 pts/3 00:00:00 tldcd -v

 

Tags (2)

Re: Cannot login to admin console using jnbSA

You really shouldn't be running bpnbaz configureAuth and other commands without really understanding what things it affects. With the new certificates in place, just running any such random command would eventually end up breaking everything.

In your case the 1st thing that we need to concentrate on is getting nbatd running on the master server, without nbatd running the bpnbaz command failing is pretty understandable.

Could you check if there are any files with .0 in the below path. You could directly run the below command to validate this information.

ls -la /usr/openv/var/global/vxss/eab/data/root/.VRTSat/profile/certstore/*.0

If you see a couple of such files kindly move them to say /tmp. *** Only the .0 files. Please do not move or touch anything else.

Once you move it try to restart NetBackup services including pbx and check whether nbatd starts. If it still fails to start then I would say reviewing nbatd logs would give us the correct path to proceed ahead. 

Highlighted

Re: Cannot login to admin console using jnbSA

Hi

Already got the same issue. cA certificate is expired and cannot be renew... you need to raise a support case. TSE will provide you the correct procedure to reinstall nbwebsvc service.

Otherwise if you send me a Personnal message I can provide you my own. I strongly recommend to be assisted by TSE for such operation.

Re: Cannot login to admin console using jnbSA

CA certificate would not really expire.. Its issued until 2038. In your case I believe it would either have been the tomcat certificate or the host id certficate that may have expired.. And I guess when you raised a support case the complete reconfiguration of nbwmc was done..

Out here since nbatd itself is not running trying to reconfigure nbwmc would not really help. But yes logging a support call is definitely recommended in such scenarios rather than running random command which may eventually complicate things further and we may need to go with a complete catalog recovery with fresh installation to get things back in place.