nbsl.exe spawning hundreds of cmd.exe and nbproxy.exe Netbackup 8.1.2

[Slartybardfast]

Hi Forum Guru's

I have noticed recently that our master server over time ends up with hundreds/thousands of cmd and nbproxy that are not terminating gracefully. The only way to get them all cleared is either a server reboot or issuing bpdown then bpup.

We have OpsAnaytics 2k8R2 8.1.2

Netbackup Master 2k8R2 8.1.2

Media Servers 2k12R2 8.1.2

A bug report for knowledge base article 100007812 BUG REPORT: nbproxy policy manager processes become orpaned and are not getting cleaned up for version 7.1 decribes the issue exactly.

I have followed this article Article: 100038207 to up the logging on nbsl and server logging on the OpsCenter server but the logs are not revealing much. There is however a reocurring error on the master nbsl log

snip<>

0,51216,132,132,507364,1551791637794,83592,56404,0:,94:NB error code: 25 NB localized Error text: cannot connect on socket(NBUErrorTranslator.cpp:47),37:NBUErrorTranslator::getErrorDetails(),1
0,51216,132,132,507365,1551791637794,83592,56404,0:,170:Error:ClientQueryFacetImpl::populateOutput():#229:../ClientQueryFacetImpl.cpp(482): errNo:25 localizedStr:cannot connect on socket additionalStr:(ExceptionManager.cpp:97),39:ExceptionManager::fillExceptionObject(),1
0,51216,132,132,507366,1551791637794,83592,56404,0:,63:Exception name:NBSLOpException(CORBAExceptionTranslator.cpp:47),43:CORBAExceptionTranslator::getErrorDetails(),1
0,51216,132,132,507367,1551791637794,83592,56404,0:,69:Rethrowing NBSLOpException exception(CORBAExceptionTranslator.cpp:85),43:CORBAExceptionTranslator::getErrorDetails(),1

snip<>

On the Opscenter log all looks pretty good. I hope someone can throw a different light on this as I cannot see the tree for the woods at the moment. Thanks in advance.

[Slartybardfast]

I have logged a case with Veritas support. Let's see what they make of it.

[Marianne]

Good choice!
The snippet of the 'raw' log does not show source or destination IPs that's involved in the socket errors.

[dipendra]

Any updates from support on the issue ? I am facing similar issue.

[sdo]

Wild guess... what is your IPv4 TCP dymanic ephemeral port range on master ?

netsh int ipv4 show dynamicport tcp

(FYI - NetBackup does not use UDP, and does not generally use IPv6 - so don't worry about the other available ranges - we are only intested in the IPv4 TCP range)

[Michael_G_Ander]

Have seen lot of nbproxy.exe if the connection to opscenter has been unstable,

in that case we have found that killing the nbsl tree helps.

Usually there starts a new nbsl after a short while.

Not ideal, but a workaround when you can't stop netbackup

[Slartybardfast · Accepted solution]

All I think I found a solution/work around for my issue. nbsl process was spawning literally muliple hundreds of nbproxy.exe so many that I actually ran out of ephemeral ports. I did extend the ports (not my first choice but needs must). But I found when I cleared all the media servers from the "VMware Access Hosts" in the Master server Host properties the 7000+ nbproxy processes droppped back to 550 connected processes. I could be completely in the wrong park here and I am sure someone will be along to correct me. But this has worked in this environment.