02-04-2020 04:33 AM
Hi all,
So I used netbackup approx a week ago, logged in without issue (it's been in the same environment for years). I've tried to login to the console this morning but it errors with...
Unable to login - status 7656 (the revocation status of the host certificate cannot be verified using the certificate revocation list CRL) because the CRL is not updated. It is older than seven days.
I've tried various solutions suggested from other posts, tried an install repair but nothing has worked so far.
Can anyone help?
02-04-2020 12:19 PM
Can you give some more details about your environment? How is access controlled? Are you using RBAC?
02-04-2020 02:59 PM
Okay - so can we assume you have tried this command:
nbcertcmd -getCrl -server <nbumaster>
Can then you provide the output of "nbcertcmd -listallcertificates" - feel free to obsufcate hostnames and fingerprints if required - I'm more interested in seeing the various expiry dates.
02-05-2020 01:31 AM - edited 02-05-2020 01:34 AM
Hi David,
Yes i did try that command and got the following ....
C:\Program Files\Veritas\NetBackup\bin>nbcertcmd.exe -getCRL -server (hostname removed)
Failed to fetch certificate revocation list for (hostname removed).net. 5
982: The certificate revocation list is unavailable.
Successfully refreshed security level for (hostname removed).
EXIT STATUS 9305: CRL retrieval operation was partially successful.
C:\Program Files\Veritas\NetBackup\bin>nbcertcmd.exe -listallcertificates
[
{
"Subject Name": "/CN=nbatd/OU=root@(hostname removed)/O=vx",
"Start Date": "Sep 11 10:27:10 2014 GMT",
"Expiry Date": "Sep 06 11:42:10 2034 GMT",
"SHA1 Fingerprint": "removed",
"Certificate Path": "C:\\Program Files\\Veritas\\NetBackup\\var\\webtrusts
tore\\cacert.pem"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=e746c11c-6180-4263-9220-fbf590cb3987/OU=NBU_HOSTS/O=v
x",
"Expiry Date": "Dec 2 03:45:59 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x4a62513300000027",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\vxss\\at"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=(hostname removed)/OU=NBU_Machines@(hostname removed)O=vx",
"Expiry Date": "Feb 3 10:19:03 2021 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x6fc9036200000002",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\vxss\\at"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=(hostname removed)/OU=TOMCAT@(hostname removed)/O=vx",
"Expiry Date": "Dec 2 03:50:54 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x7666559f0000002a",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\global\\vx
ss\\tomcatcreds\\nbwebsvc"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=(hostname removed)/OU=NBU_Machines@(hostname removed)/O=vx",
"Expiry Date": "Dec 2 03:50:51 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x4b9a9f9d00000029",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\global\\vx
ss\\websvccreds\\at\\nbwebsvc"
},
{
"Issued By": "/CN=broker/OU=root@(hostname removed)/O=vx",
"Subject Name": "/CN=nbwebsvc/OU=NBU_HOSTS@(hostname removed)/O
=vx",
"Expiry Date": "Dec 2 03:50:50 2020 GMT",
"SHA1 Fingerprint": "removed",
"Serial Number": "0x744b868800000028",
"Certificate Dir": "C:\\Program Files\\Veritas\\NetBackup\\var\\global\\vx
ss\\nbcertservice\\nbwebsvc"
}
]
02-06-2020 03:27 PM
Nothing appears to be a problem with your certificates.
Are you attempting to run the console from the master server itself? Or from another host?
That said, this may be an issue with the tomcat certificate (even though it appears valid). Increase logging for this (create a reg key HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ CurrentVersion \ Config, called: ENABLE_NBCURL_VERBOSE (as a DWORD with a vlue of 1). Then attempt the command "nbcertcmd -ping". In the nbcert log file (you may need to create the directory prior) if you see something like this, the the tomcat issue is the problem:
* Server certificate:
* subject: CN=nbumaster; OU=TOMCAT@nbumaster; O=vx
* start date: 2017-01-31 21:59:12 GMT
* expire date: 2018-01-31 23:14:12 GMT
* issuer: CN=broker; OU=root@nbumaster; O=vx
* SSL certificate verify result: certificate has expired (10), continuing anyway.
If this is the case, you will need to renew the tomcat certificate - but I would strongly suggest you get help from Veritas support to perform this operation (the scope for really stuffing things up is there).
02-07-2020 02:37 AM
Hi David,
Ran the command and got the following back...
C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -ping
Fetched data = 1581071641294.
And yes i'm trying to open the console from the master server - I've been accessing it in the same way for years without issue.
Thanks for all your help so far!
02-07-2020 06:06 PM
What were the contents of the nblog file just after the ping?
<INSTALL_PATH>\Veritas\NetBackup\logs\nbcert\APP_ADMINS.<date>_00001.log
02-11-2020 01:19 AM
attached as the log file was too large to paste directly in the message. Hostname has been removed and replaced with '(hostname)'
02-11-2020 01:37 AM
...NetBackup is trying every hour to auto update CRL:
$ grep "<16>" output.txt 00:38:46.906 [2536.8704] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 00:38:46.906 [2536.8704] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 00:38:46.922 [2536.6636] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 01:38:48.193 [8768.8360] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 01:38:48.193 [8768.8360] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 01:38:48.193 [8768.8736] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 02:38:49.184 [1208.2692] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 02:38:49.184 [1208.2692] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 02:38:49.199 [1208.1676] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 03:38:49.581 [9124.8388] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 03:38:49.581 [9124.8388] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 03:38:49.597 [9124.3064] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 04:38:51.132 [6552.8332] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 04:38:51.132 [6552.8332] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 04:38:51.132 [6552.1604] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 05:38:51.872 [8392.2580] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 05:38:51.872 [8392.2580] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 05:38:51.888 [8392.2604] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 06:38:52.488 [8388.7724] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 06:38:52.488 [8388.7724] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 06:38:52.503 [8388.108] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 07:38:53.883 [8744.8568] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 07:38:53.883 [8744.8568] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 07:38:53.899 [8744.160] <16> nbcertcmd: Attempt to refresh CRLs was partially successful 08:38:55.100 [8252.2060] <16> CurlCrlFetcher::fetchCrl: Failed to fetch CRL. CURL request failed. error = 5982 08:38:55.100 [8252.2060] <16> CrlRefreshTask::doTask: Failed to fetch certificate revocation list for (hostname). error = 5982 08:38:55.116 [8252.8784] <16> nbcertcmd: Attempt to refresh CRLs was partially successful
.
Perhaps you might be able to try googling for "NetBackup 5982" and review the tech notes / posts and consider if any of the solutions might be appropriate for you.
02-11-2020 01:51 AM
Thanks for the reply - where is it trying to fetch the revocation list from?
02-11-2020 02:48 AM
AFAIK, NetBackup Servers and NetBackup Clients would attempt to fetch the CRL list(s) from the NetBackup Master Server - but I could well be wrong.
I'm wondering if your NetBackup Server has lost its own name for itself, or has a bad/missing/double/looping DNS entry, or a bad/missing/double hosts file entry, or maybe a corrupt NetBackup name cache.
You could try:
bpclntcmd -self bpclntcmd -pn bpclntcmd -ip x.x.x.x #using IP of master bpclntcmd -hn mastername bpclntcmd -clear_host_cache (then retry the first four commands above) (then retry the usual nbcertcmd commands)
02-11-2020 03:17 AM
I went through those steps - all the details were correct, hostname, IP etc - i still followed the steps anyway but still get the same error message when trying to launch the console.
I've also checked the local host file and DNS - all look fine.
02-11-2020 06:34 AM
maybe it's time to open a support ticket?
02-11-2020 05:33 PM
Hi Jamie
I can offer three suggestions to check.
1. Review this articale and see if is relevant https://www.veritas.com/support/en_US/article.100044143 follw the suggestions in there.
2. Could the web services account used by NetBackup be locked or disabled? If so unlock and trry again.
3. Is it possible that the file permissions for the web services folders have been altered? If this is the case then the fix really needs assistance from support.
As @sdo suggested, if none of the above helps, log a support call and have Veritas look at the problem properly
Cheers
02-12-2020 02:43 AM
Thanks for the suggestions - i've checked the link, the service mentioned was not disabled and running. I also checked the local nbwebsvc account had not been disabled or pw had been reset etc - no joy here either.
Unfortunately after approx 15+ years of being with Veritas/Symantec we (reasonbly large org) have gone with another backup product and as such we no longer have support with Veritas hence why I'm on the forums instead of logging a case - i guess i will have to speed up the migration process!