nbsl.exe spawning hundreds of cmd.exe and nbproxy.exe Netbackup 8.1.2

Hi Forum Guru's

I have noticed recently that our master server over time ends up with hundreds/thousands of cmd and nbproxy that are not terminating gracefully. The only way to get them all cleared is either a server reboot or issuing bpdown then bpup.

We have OpsAnaytics 2k8R2 8.1.2

Netbackup Master 2k8R2 8.1.2

Media Servers 2k12R2 8.1.2

A bug report for knowledge base article 100007812 BUG REPORT: nbproxy policy manager processes become orpaned and are not getting cleaned up for version 7.1 decribes the issue exactly.

I have followed this article Article: 100038207 to up the logging on nbsl and server logging on the OpsCenter server but the logs are not revealing much. There is however a reocurring error on the master nbsl log

snip<>

0,51216,132,132,507364,1551791637794,83592,56404,0:,94:NB error code: 25 NB localized Error text: cannot connect on socket(NBUErrorTranslator.cpp:47),37:NBUErrorTranslator::getErrorDetails(),1
0,51216,132,132,507365,1551791637794,83592,56404,0:,170:Error:ClientQueryFacetImpl:Smiley TongueopulateOutput():#229:../ClientQueryFacetImpl.cpp(482): errNo:25 localizedStr:cannot connect on socket additionalStrSmiley SadExceptionManager.cpp:97),39:ExceptionManager::fillExceptionObject(),1
0,51216,132,132,507366,1551791637794,83592,56404,0:,63:Exception name:NBSLOpException(CORBAExceptionTranslator.cpp:47),43:CORBAExceptionTranslator::getErrorDetails(),1
0,51216,132,132,507367,1551791637794,83592,56404,0:,69:Rethrowing NBSLOpException exception(CORBAExceptionTranslator.cpp:85),43:CORBAExceptionTranslator::getErrorDetails(),1

snip<>

On the Opscenter log all looks pretty good. I hope someone can throw a different light on this as I cannot see the tree for the woods at the moment. Thanks in advance.

5 Replies

Re: nbsl.exe spawning hundreds of cmd.exe and nbproxy.exe Netbackup 8.1.2

I have logged a case with Veritas support. Let's see what they make of it.

Re: nbsl.exe spawning hundreds of cmd.exe and nbproxy.exe Netbackup 8.1.2

Good choice!
The snippet of the 'raw' log does not show source or destination IPs that's involved in the socket errors.

Re: nbsl.exe spawning hundreds of cmd.exe and nbproxy.exe Netbackup 8.1.2

Any updates from support on the issue ? I am facing similar issue.

Re: nbsl.exe spawning hundreds of cmd.exe and nbproxy.exe Netbackup 8.1.2

Wild guess... what is your IPv4 TCP dymanic ephemeral port range on master ?

netsh int ipv4 show dynamicport tcp

(FYI - NetBackup does not use UDP, and does not generally use IPv6 - so don't worry about the other available ranges - we are only intested in the IPv4 TCP range)

Re: nbsl.exe spawning hundreds of cmd.exe and nbproxy.exe Netbackup 8.1.2

Have seen lot of nbproxy.exe if the connection to opscenter has been unstable,

in that case we have found that killing the nbsl tree helps.

Usually there starts a new nbsl after a short while.

Not ideal, but a workaround when you can't stop netbackup

The standard questions: Have you checked: 1) What has changed. 2) The manual 3) If there are any tech notes or VOX posts regarding the issue