03-05-2019 05:20 AM
Hi Forum Guru's
I have noticed recently that our master server over time ends up with hundreds/thousands of cmd and nbproxy that are not terminating gracefully. The only way to get them all cleared is either a server reboot or issuing bpdown then bpup.
We have OpsAnaytics 2k8R2 8.1.2
Netbackup Master 2k8R2 8.1.2
Media Servers 2k12R2 8.1.2
A bug report for knowledge base article 100007812 BUG REPORT: nbproxy policy manager processes become orpaned and are not getting cleaned up for version 7.1 decribes the issue exactly.
I have followed this article Article: 100038207 to up the logging on nbsl and server logging on the OpsCenter server but the logs are not revealing much. There is however a reocurring error on the master nbsl log
snip<>
0,51216,132,132,507364,1551791637794,83592,56404,0:,94:NB error code: 25 NB localized Error text: cannot connect on socket(NBUErrorTranslator.cpp:47),37:NBUErrorTranslator::getErrorDetails(),1
0,51216,132,132,507365,1551791637794,83592,56404,0:,170:Error:ClientQueryFacetImpl::populateOutput():#229:../ClientQueryFacetImpl.cpp(482): errNo:25 localizedStr:cannot connect on socket additionalStr:(ExceptionManager.cpp:97),39:ExceptionManager::fillExceptionObject(),1
0,51216,132,132,507366,1551791637794,83592,56404,0:,63:Exception name:NBSLOpException(CORBAExceptionTranslator.cpp:47),43:CORBAExceptionTranslator::getErrorDetails(),1
0,51216,132,132,507367,1551791637794,83592,56404,0:,69:Rethrowing NBSLOpException exception(CORBAExceptionTranslator.cpp:85),43:CORBAExceptionTranslator::getErrorDetails(),1
snip<>
On the Opscenter log all looks pretty good. I hope someone can throw a different light on this as I cannot see the tree for the woods at the moment. Thanks in advance.
Solved! Go to Solution.
02-18-2020 05:25 AM
All I think I found a solution/work around for my issue. nbsl process was spawning literally muliple hundreds of nbproxy.exe so many that I actually ran out of ephemeral ports. I did extend the ports (not my first choice but needs must). But I found when I cleared all the media servers from the "VMware Access Hosts" in the Master server Host properties the 7000+ nbproxy processes droppped back to 550 connected processes. I could be completely in the wrong park here and I am sure someone will be along to correct me. But this has worked in this environment.
03-06-2019 01:22 AM
I have logged a case with Veritas support. Let's see what they make of it.
03-06-2019 01:52 AM
Good choice!
The snippet of the 'raw' log does not show source or destination IPs that's involved in the socket errors.
10-08-2019 02:12 AM
Any updates from support on the issue ? I am facing similar issue.
10-08-2019 02:19 AM
Wild guess... what is your IPv4 TCP dymanic ephemeral port range on master ?
netsh int ipv4 show dynamicport tcp
(FYI - NetBackup does not use UDP, and does not generally use IPv6 - so don't worry about the other available ranges - we are only intested in the IPv4 TCP range)
10-14-2019 06:43 AM
Have seen lot of nbproxy.exe if the connection to opscenter has been unstable,
in that case we have found that killing the nbsl tree helps.
Usually there starts a new nbsl after a short while.
Not ideal, but a workaround when you can't stop netbackup
02-18-2020 05:25 AM
All I think I found a solution/work around for my issue. nbsl process was spawning literally muliple hundreds of nbproxy.exe so many that I actually ran out of ephemeral ports. I did extend the ports (not my first choice but needs must). But I found when I cleared all the media servers from the "VMware Access Hosts" in the Master server Host properties the 7000+ nbproxy processes droppped back to 550 connected processes. I could be completely in the wrong park here and I am sure someone will be along to correct me. But this has worked in this environment.