cancel
Showing results for 
Search instead for 
Did you mean: 

Login credentials or certificate verification failed for server cloudcatayst

iVaibhav
Level 4

Hello Community, 

When Configuring Cloud Catalyst on Azure , We are getting below Issue. 

Login credentials or certificate verification failed for server cloudcatalyst.server . . Please see tpconfig logs for further details.

image.pngCan anyone point to a resolution . Below is tpcommand logs

 

18:14:22.124 [89091] <2> nbclnt_curl_prefnet::helper_check_connect_status: nbcurl CONNECT FROM xxx.xxx.xxx.xxx.port TO xxx.xxx.xxx.xxx.port fd = 11
18:14:22.155 [89091] <4> NBClientCURL::performCurlOperation: HTTP response header : HTTP/1.1 200  Cache-Control: private Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Request-ID: 594E0A50B12633E6 Content-Type: text/vnd.netbackup+plain; version=3.0;charset=UTF-8 Content-Length: 4553 Date: Fri, 19 Jun 2020 12:44:21 GMT
18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: provider Name =  CloudProvider
18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Read user preferences and initialize server handle
18:14:22.157 [89091] <2> catalyst.server.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting chunksize value to 67108864 for Dedupe storage
18:14:22.157 [89091] <8> catalyst.server.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Write Buffer size value to 67108864 for Dedupe storage
18:14:22.157 [89091] <8> catalyst.server.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Read Buffer size value to 67108864 for Dedupe storage
18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: CA_PATH=[/usr/openv/var/global/wmc/cloud/cacert.pem]
18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Building server URL
18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Blob Service Endpoint = [https://mycloudentrypoint]
18:14:22.157 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Blob Service Endpoint For Data = [https://mycloudentrypoint]
18:14:22.158 [89091] <2> catalyst.server.com: Azure: stspi_open_server: Initialize server handle
18:14:22.158 [89091] <2> catalyst.server.com: Azure: Check for CRL list
18:14:22.158 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler:Performing Certificate data collection for URL: https://mycloudentrypoint
18:14:22.158 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::checkCRLDirectory: Crl directory exists : /usr/openv/var/global/wmc/cloud/CrlStore
18:14:22.523 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
18:14:22.523 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
18:14:24.554 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
18:14:24.554 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
18:14:26.586 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
18:14:26.587 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
18:14:28.622 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
18:14:28.622 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
18:14:30.658 [89091] <16> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
18:14:30.658 [89091] <2> catalyst.server.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: CURLcode value is: [77]
18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Resolved local IP xxx.xxx.xxx.xxx
18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Resolved remote IP xxx.xxx.xxx.xxx.port
18:14:32.658 [89091] <2> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Port used to connect: 443
18:14:32.658 [89091] <16> catalyst.server.com: Azure: CloudRevocationCheck::crlHandler: Failed to download, curl error 77
18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspiazure.so err 2060203
18:14:32.659 [89091] <16> catalyst.server.com: metering: Failed to open a new session, return: 2060203
18:14:32.659 [89091] <2> catalyst.server.com: metering: leaving stspi_open_server
18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimetering.so err 2060203
18:14:32.659 [89091] <16> catalyst.server.com: [throttling_open_server_v7]fail to open server of the next plugin in stack, return code: 2060203
18:14:32.659 [89091] <2> catalyst.server.com: [throttling_open_server_v7]leave.
18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspithrottling.so err 2060203
18:14:32.659 [89091] <16> catalyst.server.com: MSDPCLD:Failed to open a cloud server. Error code: 2060203
18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimsdpcld.so err 2060203
18:14:32.659 [89091] <16> catalyst.server.com: gateway: Failed to open a new session, return: 2060203
18:14:32.659 [89091] <16> catalyst.server.com: libsts opensvh() 20/06/19 18:14:32: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspigateway.so err 2060203
18:14:32.659 [89091] <16> Valid_STS_Server: Failed to open server connection to type PureDisk_azure_rawd server cloudserver: Error = 2060203 A connection to a cloud storage provider could not be established

 

1 ACCEPTED SOLUTION

Accepted Solutions

@EthanH.

Since we were time bound on this. This is how I did it .

1. Ran the Process of Decommissioning using nbdecomission command

2. Ran the install again on the Catalyst Server.

3 Tried the Configure the Storage Server again. We got the same error as show in screenshot, but this time tpcommand showed different error

pd_mount:exit: (2060029:authorization faliure)

open_server :pd_mount() failed (2060029:authorization faliure)

MSDPCLD : Authentication to spad failed, synchronising the credentials now

This looked similar to the error which we get while configrung MSDP Pool

Ran below command

/usr/openv/pdde/pdconfigure/scripts/installers/PDDE_deleteConfig.sh

After this The Reconfiguration attempt to configure cloud went fine.

Very much wanted to know what the root cause was , but now its resolved.

Thank you every one for your inputs and suggestions.

 

View solution in original post

12 REPLIES 12

Krutons
Moderator
Moderator
   VIP   

The curl error 77 should point you in the right direction.

That is an SSL cert issue which would mean that there is a problem reading the the SSL CA cert. That could be a bad path, but that doesn't look like it since it notes that the CRL path exists.
It could also be access/permission issues, that's what I would look at first.

Came across this as well, maybe take a look.
https://www.veritas.com/content/support/en_US/article.100032774

EthanH
Level 4

I would follow the steps in this article: 

https://www.veritas.com/support/en_US/article.100042691

Unfortunately it requires a factory reset, but it's going to be the most effective method of reconfiguring the CloudCatalyst to get it up and running again.

@Krutons.

Thanks for the advice, I tried the technote and only thing I could find was the ndmp entry which I removed.

Is there any additional logs I can check to further investigate this.

When you say its Access/permissions , any logs which might reflect this to start with ?

Krutons
Moderator
Moderator
   VIP   

What version of netbackup are running in your environment?

8.2

Is this a BYO CloudCatalyst, or a CloudCatalyst appliance?

@EthanH 

Its BYOD Cloud Catalyst

Krutons
Moderator
Moderator
   VIP   

What level of logging are the tpcommand logs you've shown so far?

If you haven't increased them, do so and then attach the new logs as a .txt or paste them here. Here is how you can increase the logging if you haven't done so yet.

Verbose settings for logs in /usr/openv/volmgr/debug:

For these logs, the only option is to add VERBOSE into /usr/openv/volmgr/vm.conf.   No numerical value is accepted.
 
About the vm.conf configuration file

The vm.conf file contains configuration entries for media and device management. NetBackup can create this file, but if it does not exist, you must create it.

@Krutons 

Below is the Verbose TPCommand output ( Trimmed ) I have attached full logs as it was long

CS_throttle:azure_rawd:CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure.
16:26:17.168 [173598] <2> catalystserver.com: [throttling_open_server_v7]the server name to the next plugin in stack is CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure.
16:26:17.168 [173598] <4> catalystserver.com: libsts setupp() 20/06/23 16:26:17: CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure claimed by /usr/openv/lib/ost-plugins/libstspimetering.so
16:26:17.168 [173598] <2> catalystserver.com: metering: entry stspi_open_server
16:26:17.168 [173598] <2> catalystserver.com: metering: Server Name = CS_meter:azure_rawd:CS_Azure:azure_rawd:my-azure
16:26:17.168 [173598] <2> catalystserver.com: metering: using default metering directory /usr/openv/var/global/wmc/cloud/meter
16:26:17.168 [173598] <2> catalystserver.com: metering: Reading config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf)
16:26:17.168 [173598] <2> catalystserver.com: metering: Closing config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf)
16:26:17.168 [173598] <2> catalystserver.com: metering: Reading config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf)
16:26:17.168 [173598] <2> catalystserver.com: metering: Closing config file (/usr/openv/var/global/wmc/cloud/cloudstore.conf)
16:26:17.168 [173598] <4> catalystserver.com: libsts setupp() 20/06/23 16:26:17: CS_Azure:azure_rawd:my-azure claimed by /usr/openv/lib/ost-plugins/libstspiazure.so
16:26:17.168 [173598] <2> catalystserver.com: Azure: Entering [stspi_open_server]
16:26:17.168 [173598] <2> catalystserver.com: Azure: stspi_open_server: Azure server name: CS_Azure:azure_rawd:my-azure
16:26:17.168 [173598] <2> catalystserver.com: Azure: stspi_open_server: Azure stype: azure_rawd, server name: my-azure, sts server name: azure_rawd:my-azure
16:26:17.169 [173598] <2> getCrlCheckLevelFromConfig: crlCheckFlag value read from bpconf : 1
16:26:17.170 [173598] <2> nbclnt_curl_prefnet::helper_check_connect_status: nbcurl CONNECT FROM ip.xxx.xxx.xxx.55560 TO ip.xxx.xxx.xxx.1556 fd = 11
16:26:17.195 [173598] <4> NBClientCURL::performCurlOperation: HTTP response header : HTTP/1.1 200  Cache-Control: private Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Request-ID: C88A13FB79DCF073 Content-Type: text/vnd.netbackup+plain; version=3.0;charset=UTF-8 Content-Length: 4553 Date: Tue, 23 Jun 2020 10:56:16 GMT
16:26:17.197 [173598] <2> catalystserver.com: Azure: stspi_open_server: provider Name =  azure
16:26:17.197 [173598] <2> catalystserver.com: Azure: stspi_open_server: Read user preferences and initialize server handle
16:26:17.197 [173598] <2> catalystserver.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting chunksize value to 67108864 for Dedupe storage
16:26:17.197 [173598] <8> catalystserver.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Write Buffer size value to 67108864 for Dedupe storage
16:26:17.197 [173598] <8> catalystserver.com: Azure: CAzureConfigFileHandler::CAzureConfigFileHandler:Overwriting Read Buffer size value to 67108864 for Dedupe storage
16:26:17.197 [173598] <2> catalystserver.com: Azure: stspi_open_server: CA_PATH=[/usr/openv/var/global/wmc/cloud/cacert.pem]
16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Building server URL
16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Blob Service Endpoint = [https://emmhostnameloc.blob.core.windows.net]
16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Blob Service Endpoint For Data = [https://emmhostnameloc.blob.core.windows.net]
16:26:17.198 [173598] <2> catalystserver.com: Azure: stspi_open_server: Initialize server handle
16:26:17.198 [173598] <2> catalystserver.com: Azure: Check for CRL list
16:26:17.198 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler:Performing Certificate data collection for URL: https://emmhostnameloc.blob.core.windows.net
16:26:17.198 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::checkCRLDirectory: Crl directory exists : /usr/openv/var/global/wmc/cloud/CrlStore
16:26:17.888 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
16:26:17.888 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
16:26:19.932 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
16:26:19.932 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
16:26:21.976 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
16:26:21.976 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
16:26:24.021 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
16:26:24.021 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
16:26:26.069 [173598] <16> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: ERROR: Failed to download, curl error 77
16:26:26.069 [173598] <2> catalystserver.com: Azure: Crl_Options::addCurlEasyPerform: Sleeping for 2 seconds before retrying download
16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: CURLcode value is: [77]
16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Resolved local IP ip.xxx.xxx.xxx
16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Resolved remote IP ip.xxx.xxx.xxx
16:26:28.069 [173598] <2> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Port used to connect: 443
16:26:28.069 [173598] <16> catalystserver.com: Azure: CloudRevocationCheck::crlHandler: Failed to download, curl error 77
16:26:28.069 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspiazure.so err 2060203
16:26:28.070 [173598] <16> catalystserver.com: metering: Failed to open a new session, return: 2060203
16:26:28.070 [173598] <2> catalystserver.com: metering: leaving stspi_open_server
16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimetering.so err 2060203
16:26:28.070 [173598] <16> catalystserver.com: [throttling_open_server_v7]fail to open server of the next plugin in stack, return code: 2060203
16:26:28.070 [173598] <2> catalystserver.com: [throttling_open_server_v7]leave.
16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspithrottling.so err 2060203
16:26:28.070 [173598] <16> catalystserver.com: MSDPCLD:Failed to open a cloud server. Error code: 2060203
16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspimsdpcld.so err 2060203
16:26:28.070 [173598] <16> catalystserver.com: gateway: Failed to open a new session, return: 2060203
16:26:28.070 [173598] <16> catalystserver.com: libsts opensvh() 20/06/23 16:26:28: v11_open_server failed in plugin /usr/openv/lib/ost-plugins/libstspigateway.so err 2060203
16:26:28.070 [173598] <16> Valid_STS_Server: Failed to open server connection to type PureDisk_azure_rawd server my-azure: Error = 2060203 A connection to a cloud storage provider could not be established

@iVaibhav 

This TechNote seems to be very closely related to the issue you're seeing. Are you using self-signed certs from NBU or external CA? Either way, I think this might resolve the issue you're seeing (even though the exact SSL error code is different, the problem seems to be the same).

@EthanH.

Since we were time bound on this. This is how I did it .

1. Ran the Process of Decommissioning using nbdecomission command

2. Ran the install again on the Catalyst Server.

3 Tried the Configure the Storage Server again. We got the same error as show in screenshot, but this time tpcommand showed different error

pd_mount:exit: (2060029:authorization faliure)

open_server :pd_mount() failed (2060029:authorization faliure)

MSDPCLD : Authentication to spad failed, synchronising the credentials now

This looked similar to the error which we get while configrung MSDP Pool

Ran below command

/usr/openv/pdde/pdconfigure/scripts/installers/PDDE_deleteConfig.sh

After this The Reconfiguration attempt to configure cloud went fine.

Very much wanted to know what the root cause was , but now its resolved.

Thank you every one for your inputs and suggestions.

 

Thanks for letting us know how you resolved it!