06-19-2020 07:08 AM
Hello Community,
When Configuring Cloud Catalyst on Azure , We are getting below Issue.
Login credentials or certificate verification failed for server cloudcatalyst.server . . Please see tpconfig logs for further details.
Can anyone point to a resolution . Below is tpcommand logs
18:14:22.124 [89091] <2> nbclnt_curl_prefnet::helper_check_connect_status: nbcurl CONNECT FROM xxx.xxx.xxx.xxx.port TO xxx.xxx.xxx.xxx.port fd = 11 18:14:22.155 [89091] <4> NBClientCURL::performCurlOperation: HTTP response header : HTTP/1.1 200 Cache-Control: private Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Request-ID: 594E0A50B12633E6 Content-Type: text/vnd.netbackup+plain; version=3.0;charset=UTF-8 Content-Length: 4553 Date: Fri, 19 Jun 2020 12:44:21 GMT 18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: provider Name = CloudProvider 18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Read user preferences and initialize server handle 18:14:22.157 [89091] <2> catalyst.server.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting chunksize value to 67108864 for Dedupe storage 18:14:22.157 [89091] <8> catalyst.server.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Write Buffer size value to 67108864 for Dedupe storage 18:14:22.157 [89091] <8> catalyst.server.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Read Buffer size value to 67108864 for Dedupe storage 18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: CA_PATH=[/usr/openv/var/global/wmc/cloud/cacert.pem] 18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Building server URL 18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Blob Service Endpoint = [https://mycloudentrypoint] 18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Blob Service Endpoint For Data = [https://mycloudentrypoint] 18:14:22.158 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Initialize server handle 18:14:22.158 [89091] <2> catalyst.server.com: Azure: Check for CRL list 18:14:22.158 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler:Performing Certificate data collection for URL: https://mycloudentrypoint 18:14:22.158 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::checkCRLDirectory: Crl directory exists : /usr/openv/var/global/wmc/cloud/CrlStore 18:14:22.523 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 18:14:22.523 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 18:14:24.554 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 18:14:24.554 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 18:14:26.586 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 18:14:26.587 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 18:14:28.622 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 18:14:28.622 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 18:14:30.658 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 18:14:30.658 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: CURLcode value is: [77] 18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Resolved local IP xxx.xxx.xxx.xxx 18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Resolved remote IP xxx.xxx.xxx.xxx.port 18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Port used to connect: 443 18:14:32.658 [89091] <16> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Failed to download, curl error 77 18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspiazure.so err 2060203 18:14:32.659 [89091] <16> catalyst.server.com: metering: Failed to open a new session, return: 2060203 18:14:32.659 [89091] <2> catalyst.server.com: metering: leaving stspi_open_server 18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimetering.so err 2060203 18:14:32.659 [89091] <16> catalyst.server.com: [throttling_open_server_v7]fail to open server of the next plugin in stack, return code: 2060203 18:14:32.659 [89091] <2> catalyst.server.com: [throttling_open_server_v7]leave. 18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspithrottling.so err 2060203 18:14:32.659 [89091] <16> catalyst.server.com: MSDPCLD:Failed to open a cloud server. Error code: 2060203 18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimsdpcld.so err 2060203 18:14:32.659 [89091] <16> catalyst.server.com: gateway: Failed to open a new session, return: 2060203 18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspigateway.so err 2060203 18:14:32.659 [89091] <16> Valid_STS_Server: Failed to open server connection to type PureDisk_azure_rawd server cloudserver: Error = 2060203 A connection to a cloud storage provider could not be established
Solved! Go to Solution.
06-26-2020 11:57 AM
Since we were time bound on this. This is how I did it .
1. Ran the Process of Decommissioning using nbdecomission command
2. Ran the install again on the Catalyst Server.
3 Tried the Configure the Storage Server again. We got the same error as show in screenshot, but this time tpcommand showed different error
pd_mount:exit: (2060029:authorization faliure) open_server :pd_mount() failed (2060029:authorization faliure) MSDPCLD : Authentication to spad failed, synchronising the credentials now
This looked similar to the error which we get while configrung MSDP Pool
Ran below command
/usr/openv/pdde/pdconfigure/scripts/installers/PDDE_deleteConfig.sh
After this The Reconfiguration attempt to configure cloud went fine.
Very much wanted to know what the root cause was , but now its resolved.
Thank you every one for your inputs and suggestions.
06-19-2020 07:46 AM - edited 06-19-2020 07:51 AM
The curl error 77 should point you in the right direction.
That is an SSL cert issue which would mean that there is a problem reading the the SSL CA cert. That could be a bad path, but that doesn't look like it since it notes that the CRL path exists.
It could also be access/permission issues, that's what I would look at first.
Came across this as well, maybe take a look.
https://www.veritas.com/content/support/en_US/article.100032774
06-19-2020 08:34 AM
I would follow the steps in this article:
https://www.veritas.com/support/en_US/article.100042691
Unfortunately it requires a factory reset, but it's going to be the most effective method of reconfiguring the CloudCatalyst to get it up and running again.
06-19-2020 11:06 AM
Thanks for the advice, I tried the technote and only thing I could find was the ndmp entry which I removed.
Is there any additional logs I can check to further investigate this.
When you say its Access/permissions , any logs which might reflect this to start with ?
06-19-2020 12:44 PM
What version of netbackup are running in your environment?
06-20-2020 06:15 AM
8.2
06-21-2020 03:08 PM
Is this a BYO CloudCatalyst, or a CloudCatalyst appliance?
06-22-2020 12:42 AM
06-22-2020 10:46 AM
What level of logging are the tpcommand logs you've shown so far?
If you haven't increased them, do so and then attach the new logs as a .txt or paste them here. Here is how you can increase the logging if you haven't done so yet.
The vm.conf
file contains configuration entries for media and device management. NetBackup can create this file, but if it does not exist, you must create it.
06-23-2020 04:50 AM
Below is the Verbose TPCommand output ( Trimmed ) I have attached full logs as it was long
CS_throttle:azure_rawd:CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure. 16:26:17.168 [173598] <2> catalystserver.com: [throttling_open_server_v7]the server name to the next plugin in stack is CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure. 16:26:17.168 [173598] <4> catalystserver.com: libsts setupp() 20/06/23 16:26:17: CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure claimed by /usr/openv/lib/ost-plugins/libstspimetering.so 16:26:17.168 [173598] <2> catalystserver.com: metering: entry stspi_open_server 16:26:17.168 [173598] <2> catalystserver.com: metering: Server Name = CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure 16:26:17.168 [173598] <2> catalystserver.com: metering: using default metering directory /usr/openv/var/global/wmc/cloud/meter 16:26:17.168 [173598] <2> catalystserver.com: metering: Reading config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf) 16:26:17.168 [173598] <2> catalystserver.com: metering: Closing config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf) 16:26:17.168 [173598] <2> catalystserver.com: metering: Reading config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf) 16:26:17.168 [173598] <2> catalystserver.com: metering: Closing config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf) 16:26:17.168 [173598] <4> catalystserver.com: libsts setupp() 20/06/23 16:26:17: CS_Azure:azure_rawd:my-azure claimed by /usr/openv/lib/ost-plugins/libstspiazure.so 16:26:17.168 [173598] <2> catalystserver.com: Azure: Entering [stspi_open_server] 16:26:17.168 [173598] <2> catalystserver.com: Azure: stspi_open_server: Azure server name: CS_Azure:azure_rawd:my-azure 16:26:17.168 [173598] <2> catalystserver.com: Azure: stspi_open_server: Azure stype: azure_rawd, server name: my-azure, sts server name: azure_rawd:my-azure 16:26:17.169 [173598] <2> getCrlCheckLevelFromConfig: crlCheckFlag value read from bpconf : 1 16:26:17.170 [173598] <2> nbclnt_curl_prefnet::helper_check_connect_status: nbcurl CONNECT FROM ip.xxx.xxx.xxx.55560 TO ip.xxx.xxx.xxx.1556 fd = 11 16:26:17.195 [173598] <4> NBClientCURL::performCurlOperation: HTTP response header : HTTP/1.1 200 Cache-Control: private Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Request-ID: C88A13FB79DCF073 Content-Type: text/vnd.netbackup+plain; version=3.0;charset=UTF-8 Content-Length: 4553 Date: Tue, 23 Jun 2020 10:56:16 GMT 16:26:17.197 [173598] <2> catalystserver.com: Azure: stspi_open_server: provider Name = azure 16:26:17.197 [173598] <2> catalystserver.com: Azure: stspi_open_server: Read user preferences and initialize server handle 16:26:17.197 [173598] <2> catalystserver.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting chunksize value to 67108864 for Dedupe storage 16:26:17.197 [173598] <8> catalystserver.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Write Buffer size value to 67108864 for Dedupe storage 16:26:17.197 [173598] <8> catalystserver.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Read Buffer size value to 67108864 for Dedupe storage 16:26:17.197 [173598] <2> catalystserver.com: Azure: stspi_open_server: CA_PATH=[/usr/openv/var/global/wmc/cloud/cacert.pem] 16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Building server URL 16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Blob Service Endpoint = [https://emmhostnameloc.blob.core.windows.net] 16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Blob Service Endpoint For Data = [https://emmhostnameloc.blob.core.windows.net] 16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Initialize server handle 16:26:17.198 [173598] <2> catalystserver.com: Azure: Check for CRL list 16:26:17.198 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler:Performing Certificate data collection for URL: https://emmhostnameloc.blob.core.windows.net 16:26:17.198 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::checkCRLDirectory: Crl directory exists : /usr/openv/var/global/wmc/cloud/CrlStore 16:26:17.888 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 16:26:17.888 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 16:26:19.932 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 16:26:19.932 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 16:26:21.976 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 16:26:21.976 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 16:26:24.021 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 16:26:24.021 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 16:26:26.069 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77 16:26:26.069 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download 16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: CURLcode value is: [77] 16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Resolved local IP ip.xxx.xxx.xxx 16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Resolved remote IP ip.xxx.xxx.xxx 16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Port used to connect: 443 16:26:28.069 [173598] <16> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Failed to download, curl error 77 16:26:28.069 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspiazure.so err 2060203 16:26:28.070 [173598] <16> catalystserver.com: metering: Failed to open a new session, return: 2060203 16:26:28.070 [173598] <2> catalystserver.com: metering: leaving stspi_open_server 16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimetering.so err 2060203 16:26:28.070 [173598] <16> catalystserver.com: [throttling_open_server_v7]fail to open server of the next plugin in stack, return code: 2060203 16:26:28.070 [173598] <2> catalystserver.com: [throttling_open_server_v7]leave. 16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspithrottling.so err 2060203 16:26:28.070 [173598] <16> catalystserver.com: MSDPCLD:Failed to open a cloud server. Error code: 2060203 16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimsdpcld.so err 2060203 16:26:28.070 [173598] <16> catalystserver.com: gateway: Failed to open a new session, return: 2060203 16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspigateway.so err 2060203 16:26:28.070 [173598] <16> Valid_STS_Server: Failed to open server connection to type PureDisk_azure_rawd server my-azure: Error = 2060203 A connection to a cloud storage provider could not be established
06-25-2020 10:16 AM
This TechNote seems to be very closely related to the issue you're seeing. Are you using self-signed certs from NBU or external CA? Either way, I think this might resolve the issue you're seeing (even though the exact SSL error code is different, the problem seems to be the same).
06-26-2020 11:57 AM
Since we were time bound on this. This is how I did it .
1. Ran the Process of Decommissioning using nbdecomission command
2. Ran the install again on the Catalyst Server.
3 Tried the Configure the Storage Server again. We got the same error as show in screenshot, but this time tpcommand showed different error
pd_mount:exit: (2060029:authorization faliure) open_server :pd_mount() failed (2060029:authorization faliure) MSDPCLD : Authentication to spad failed, synchronising the credentials now
This looked similar to the error which we get while configrung MSDP Pool
Ran below command
/usr/openv/pdde/pdconfigure/scripts/installers/PDDE_deleteConfig.sh
After this The Reconfiguration attempt to configure cloud went fine.
Very much wanted to know what the root cause was , but now its resolved.
Thank you every one for your inputs and suggestions.
06-26-2020 12:00 PM
Thanks for letting us know how you resolved it!