Netbackup Console slow after deleting MSDP

[Adnan_M_F]

Hello,

We had a to delete and reconfigure a MSDP pool as there was a storage failure. The master/media for this pool are the same server. 

We followed the below article to delete the pool:

https://www.veritas.com/support/en_US/article.000011549

Everything went well, except after running step 13 i.e. PDDE_deleteConfig, the netbackup console/application (on server which is both master/media) has become rather slow and takes time to load anything.

After running the PDDE command and restarting the server, the Netbackup Database Manager and Netback Request Daemon services were failing to start with windows error that it took long time to start. We changed the registry Control key and added a "ServicesPipeTimeout" key to increase the timeout. This worked and both services automatically started after restart.

We managed to create a new MSDP pool on the same server, but the netbackup admin console is just not responsive. Backup jobs on other media servers do seem to be running. All critical netbackup services are up as well but the netbackup just does not run well and sometimes goes into "Not responding".

Any help would be much appreciated. 

Regards,

Adnan

[PatS729]

hi,

the problem you have mentioned is absolutely irrelevant to NBU console performance. I would suggest try using Java Console or NetBackup Administration Console (you need to install it seperately from NetBackup Installation) to see if makes any difference.

[Adnan_M_F]

Thanks Pat, we have the admin console installed on another media server as well and its the same issue. No matter what you click, the policies, STU, Devices, anything on the console it goes on to loading and half of the time "Not Responding". 

[PatS729 · Accepted solution]

Hi,

You will need to look into <install_path>\VERITAS\NetBackup\logs\nbconsole and admin logs to investigate it further. Upload both logs here else, create support case with Veritas.

[Adnan_M_F]

Hello Pat,

Thank you.

I've attached both the nbconsole and admin logs, taken while starting the console which took a while and then clicking on few items in the menu which also took time to making the console go blank.

Regards,

Adnan

[PatS729]

Hi,

Not sure why this is appearing in nbconsole log

23:02:51.477 [11088.12612] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS
 status: FAILED, (24) BAD_VERSION; system: (10036) A blocking operation is currently executing. ; FROM 0.0.0.0 TO nbserver.com 172.0.10.23 bprd VIA pbx
 status: FAILED, (44) CONNECT_TIMEOUT; system: (10060) Connection timed out.; FROM 0.0.0.0 TO nbserver.com 172.0.10.23 bprd VIA vnetd
 status: SUCCESS; FROM 0.0.0.0 TO nbserver.com 172.0.10.23 bprd
23:02:51.477 [11088.12612] <2> logconnections: BPRD CONNECT FROM 172.0.10.23.63556 TO 172.0.10.23.13720 fd = 1520
23:02:51.477 [11088.12612] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:1918] found via getnameinfo OUR_HOST=nbserver.com IPSTR=172.0.10.23

How are we resolving hostname ? using DNS or host entries ?

[Adnan_M_F]

We're using host entries & we do update our DNS too. Would a newer console maintenance update help? 

[Adnan_M_F]

Pat, thank you so much for your help. You guided us to the right logs "nbconsole" & "admin" to investigate. We found the below recurring each time we clicked anywhere on the console in the "nbconsole":

07:54:19.372 [9272.8528] <2> parse_packet: INTTYPE Index 43 Field m_nKbPerSec Value <53067>
07:54:20.152 [11088.12612] <2> bprd_connect: Ignoring reserved port request.
07:54:20.152 [11088.12612] <2> bprd_connect: Ignoring local_name nbserver.com
07:54:20.152 [11088.12612] <2> bprd_connect: Ignoring owner_name admin.
07:54:20.168 [11088.12612] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:54:20.168 [11088.12612] <2> logconnections: BPRD CONNECT FROM 172.0.10.23.52877 TO 172.0.10.23.1556 fd = 2480
07:54:20.168 [11088.12612] <2> Unknown@VssInit: ++++ ENTERING ++++
07:54:20.168 [11088.12612] <2> Unknown@VssInit: (../../libVnbat/vss_auth.cpp,1397): ARGS: ReqVersion="60", BrokerName="", BrokerPort="0"
07:54:20.168 [11088.12612] <2> Unknown@VssInitWithAtHandle: ++++ ENTERING ++++
07:54:20.168 [11088.12612] <2> Unknown@VssGetFQDNHostName: ++++ ENTERING ++++
07:54:20.168 [11088.12612] <2> Unknown@VssGetFQDNHostName: (../../libVnbat/vss_fqdn.cpp,82): ARGS: InputName="", FullNameSize="1024"
07:54:20.168 [11088.12612] <2> Unknown@VssGetFQDNHostName: (../../libVnbat/vss_fqdn.cpp,90): Input name has ZERO length, returning SUCCESS
07:54:20.168 [11088.12612] <2> Unknown@VssGetFQDNHostName: ---- EXITING ----
07:54:20.168 [11088.12612] <2> Unknown@VssInitWithAtHandle: ---- EXITING ----
07:54:20.168 [11088.12612] <2> Unknown@VssInit: ---- EXITING ----
07:54:20.168 [11088.12612] <2> Unknown@VssImportCredential: ++++ ENTERING ++++
07:54:20.168 [11088.12612] <2> Unknown@VssImportCredential: (../../libVnbat/vss_auth.cpp,3005): ARGS: FileName="C:\Users\admin\AppData\Local\Symantec\NetBackup\36342991.conf"
07:54:20.168 [11088.12612] <2> Unknown@VssCredentialCache::findEntry: ++++ ENTERING ++++
07:54:20.168 [11088.12612] <2> Unknown@VssCredentialCache::findEntry: (../../libVnbat/vss_credentialcache.cpp,199): ARGS: FileName=C:\Users\admin\AppData\Local\Symantec\NetBackup\36342991.conf
07:54:20.168 [11088.12612] <2> Unknown@VssCredentialCache::findEntry: (../../libVnbat/vss_credentialcache.cpp,222): Found cached entry for credential:C:\Users\admin\AppData\Local\Symantec\NetBackup\36342991.conf
07:54:20.168 [11088.12612] <2> Unknown@VssCopyIdentity: ++++ ENTERING ++++
07:54:20.168 [11088.12612] <2> Unknown@VssCopyIdentity: ---- EXITING ----
07:54:20.168 [11088.12612] <2> Unknown@VssCredentialCache::findEntry: (../../libVnbat/vss_credentialcache.cpp,273): Broker:"nbserver.com!1556!nbatd" BrokerPort:"1556" DomainType:"nt" Domain:"CORP" Name:"admin"
07:54:20.168 [11088.12612] <2> Unknown@VssCredentialCache::findEntry: ---- EXITING ----
07:54:20.168 [11088.12612] <2> Unknown@VssImportCredential: ---- EXITING ----

It just went on showing some issue with credentials and we realised an "Admin Error" done during the re-configuration of the MSDP pool which was initially giving us a credential failed error. So we had changed the host property attribute "access control" from "prohibited" to "automatic". It had nothing to do with the MSDP re-configuration, which now we realize :)

But thanks a lot for your help. Much appreciated. 

[Adnan_M_F]

Just wanted to reiterate here for anyone reading this thread that the issue had NOTHING TO DO WITH MSDP. It was the master server host property where we did an administrative mistake and chose the "Access Control" to be automatic, which by default is supposed to be "Prohibited". After putting it back to prohibited, it worked for us.