Highlighted

Netbackup - Unable to login (CRL)

Hi all,

So I used netbackup approx a week ago, logged in without issue (it's been in the same environment for years). I've tried to login to the console this morning but it errors with...

Unable to login - status 7656 (the revocation status of the host certificate cannot be verified using the certificate revocation list CRL) because the CRL is not updated. It is older than seven days.

 

I've tried various solutions suggested from other posts, tried an install repair but nothing has worked so far. 

Can anyone help?

14 Replies
Highlighted

Re: Netbackup - Unable to login (CRL)

Can you give some more details about your environment? How is access controlled? Are you using RBAC?

 

Highlighted

Re: Netbackup - Unable to login (CRL)

Okay - so can we assume you have tried this command:

nbcertcmd -getCrl -server <nbumaster>

Can then you provide the output of "nbcertcmd -listallcertificates" - feel free to obsufcate hostnames and fingerprints if required - I'm more interested in seeing the various expiry dates.

Highlighted

Re: Netbackup - Unable to login (CRL)

Hi David,

Yes i did try that command and got the following ....


C:\Program Files\Veritas\NetBackup\bin>nbcertcmd.exe -getCRL -server (hostname removed)
Failed to fetch certificate revocation list for (hostname removed).net. 5
982: The certificate revocation list is unavailable.
Successfully refreshed security level for (hostname removed).
EXIT STATUS 9305: CRL retrieval operation was partially successful.

 

C:\Program Files\Veritas\NetBackup\bin>nbcertcmd.exe -listallcertificates
[
{
"Subject Name": "/CN=nbatd/OU=root@(hostname removed)/O=vx",
"Start Date": "Sep 11 10:27:10 2014 GMT",
"Expiry Date": "Sep 06 11:42:10 2034 GMT",
"SHA1 Fingerprint": "removed",
"Certificate Path": "C:\\Program Files\\Veritas\\NetBackup\\var\\webtrusts
tore\\cacert.pem"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=e746c11c-6180-4263-9220-fbf590cb3987/OU=NBU_HOSTS/O=v
x",
"Expiry Date": "Dec 2 03:45:59 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x4a62513300000027",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\vxss\\at"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=(hostname removed)/OU=NBU_Machines@(hostname removed)O=vx",
"Expiry Date": "Feb 3 10:19:03 2021 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x6fc9036200000002",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\vxss\\at"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=(hostname removed)/OU=TOMCAT@(hostname removed)/O=vx",
"Expiry Date": "Dec 2 03:50:54 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x7666559f0000002a",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\global\\vx
ss\\tomcatcreds\\nbwebsvc"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=(hostname removed)/OU=NBU_Machines@(hostname removed)/O=vx",
"Expiry Date": "Dec 2 03:50:51 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x4b9a9f9d00000029",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\global\\vx
ss\\websvccreds\\at\\nbwebsvc"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=nbwebsvc/OU=NBU_HOSTS@(hostname removed)/O
=vx",
"Expiry Date": "Dec 2 03:50:50 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x744b868800000028",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\global\\vx
ss\\nbcertservice\\nbwebsvc"
}
]

Highlighted

Re: Netbackup - Unable to login (CRL)

Nothing appears to be a problem with your certificates.

Are you attempting to run the console from the master server itself? Or from another host?

That said, this may be an issue with the tomcat certificate (even though it appears valid). Increase logging for this (create a reg key HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ CurrentVersion \ Config, called: ENABLE_NBCURL_VERBOSE (as a DWORD with a vlue of 1). Then attempt the command "nbcertcmd -ping". In the nbcert log file (you may need to create the directory prior) if you see something like this, the the tomcat issue is the problem:
 * Server certificate:
 *        subject: CN=nbumaster; OU=TOMCAT@nbumaster; O=vx
 *        start date: 2017-01-31 21:59:12 GMT
 *        expire date: 2018-01-31 23:14:12 GMT
 *        issuer: CN=broker; OU=root@nbumaster; O=vx
 *        SSL certificate verify result: certificate has expired (10), continuing anyway.

If this is the case, you will need to renew the tomcat certificate - but I would strongly suggest you get help from Veritas support to perform this operation (the scope for really stuffing things up is there).

Highlighted

Re: Netbackup - Unable to login (CRL)

Hi David,

Ran the command and got the following back...

C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -ping
Fetched data = 1581071641294.

And yes i'm trying to open the console from the master server - I've been accessing it in the same way for years without issue.

Thanks for all your help so far!

Highlighted

Re: Netbackup - Unable to login (CRL)

What were the contents of the nblog file just after the ping?

<INSTALL_PATH>\Veritas\NetBackup\logs\nbcert\APP_ADMINS.<date>_00001.log

 

Highlighted

Re: Netbackup - Unable to login (CRL)

attached as the log file was too large to paste directly in the message. Hostname has been removed and replaced with '(hostname)' 

Highlighted

Re: Netbackup - Unable to login (CRL)

...NetBackup is trying every hour to auto update CRL:

$ grep "<16>" output.txt 
00:38:46.906 [2536.8704] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
00:38:46.906 [2536.8704] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
00:38:46.922 [2536.6636] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
01:38:48.193 [8768.8360] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
01:38:48.193 [8768.8360] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
01:38:48.193 [8768.8736] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
02:38:49.184 [1208.2692] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
02:38:49.184 [1208.2692] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
02:38:49.199 [1208.1676] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
03:38:49.581 [9124.8388] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
03:38:49.581 [9124.8388] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
03:38:49.597 [9124.3064] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
04:38:51.132 [6552.8332] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
04:38:51.132 [6552.8332] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
04:38:51.132 [6552.1604] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
05:38:51.872 [8392.2580] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
05:38:51.872 [8392.2580] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
05:38:51.888 [8392.2604] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
06:38:52.488 [8388.7724] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
06:38:52.488 [8388.7724] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
06:38:52.503 [8388.108] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
07:38:53.883 [8744.8568] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
07:38:53.883 [8744.8568] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
07:38:53.899 [8744.160] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
08:38:55.100 [8252.2060] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982
08:38:55.100 [8252.2060] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982
08:38:55.116 [8252.8784] <16> nbcertcmd: Attempt to refresh CRLs was partially successful

.

Perhaps you might be able to try googling for "NetBackup 5982" and review the tech notes / posts and consider if any of the solutions might be appropriate for you.

Highlighted

Re: Netbackup - Unable to login (CRL)

Thanks for the reply - where is it trying to fetch the revocation list from?

Highlighted

Re: Netbackup - Unable to login (CRL)

AFAIK, NetBackup Servers and NetBackup Clients would attempt to fetch the CRL list(s) from the NetBackup Master Server - but I could well be wrong.

I'm wondering if your NetBackup Server has lost its own name for itself, or has a bad/missing/double/looping DNS entry, or a bad/missing/double hosts file entry, or maybe a corrupt NetBackup name cache.

You could try:

bpclntcmd -self
bpclntcmd -pn
bpclntcmd -ip x.x.x.x       #using IP of master
bpclntcmd -hn mastername
bpclntcmd -clear_host_cache
(then retry the first four commands above)
(then retry the usual nbcertcmd commands)
Highlighted

Re: Netbackup - Unable to login (CRL)

I went through those steps - all the details were correct, hostname, IP etc - i still followed the steps anyway but still get the same error message when trying to launch the console.

I've also checked the local host file and DNS - all look fine.

Highlighted

Re: Netbackup - Unable to login (CRL)

maybe it's time to open a support ticket?

Re: Netbackup - Unable to login (CRL)

Hi Jamie

I can offer three suggestions to check.

1. Review this articale and see if is relevant https://www.veritas.com/support/en_US/article.100044143 follw the suggestions in there.
2. Could the web services account used by NetBackup be locked or disabled? If so unlock and trry again.
3. Is it possible that the file permissions for the web services folders have been altered? If this is the case then the fix really needs assistance from support.

As @sdo suggested, if none of the above helps, log a support call and have Veritas look at the problem properly

Cheers

Highlighted

Re: Netbackup - Unable to login (CRL)

Thanks for the suggestions - i've checked the link, the service mentioned was not disabled and running. I also checked the local nbwebsvc account had not been disabled or pw had been reset etc - no joy here either.

Unfortunately after approx 15+ years of being with Veritas/Symantec we (reasonbly large org) have gone with another backup product and as such we no longer have support with Veritas hence why I'm on the forums instead of logging a case - i guess i will have to speed up the migration process!