ERROR: Connection rejected by PBX exchange

[____111]

Hi 

I got a problem,when I start the java console(jnbSA) from the master server,then I got the error (see below)

Then i view the log as below

/usr/openv/netbackup/logs/user_ops/nbjlogs/jbp.root.jnbSA.19328.log

ICACHE.PBXIOPCredsHandler-> ServiceContext registered.

[21-11-2013 21:26:12:533] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:18243

[21-11-2013 21:26:12:534] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

[21-11-2013 21:26:12:535] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl

[21-11-2013 21:26:12:540] pbxiop:ERROR: Connection rejected by PBX exchange

[21-11-2013 21:26:12:540] pbxiop:INFO: Retrying to connect to null

[21-11-2013 21:26:13:543] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:32614

[21-11-2013 21:26:13:543] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

[21-11-2013 21:26:13:544] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl

[21-11-2013 21:26:13:546] pbxiop:ERROR: Connection rejected by PBX exchange

[21-11-2013 21:26:13:547] pbxiop:INFO: Retrying to connect to null

[21-11-2013 21:26:14:548] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:21025

[21-11-2013 21:26:14:548] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

[21-11-2013 21:26:14:549] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl

[21-11-2013 21:26:14:549] pbxiop:ERROR: Connection rejected by PBX exchange

[21-11-2013 21:26:14:550] pbxiop:INFO: Retrying to connect to null

[21-11-2013 21:26:15:552] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:37459

[21-11-2013 21:26:15:552] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

[21-11-2013 21:26:15:553] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl

[21-11-2013 21:26:15:578] pbxiop:ERROR: Connection rejected by PBX exchange

[21-11-2013 21:26:15:579] pbxiop:INFO: Retrying to connect to null

[21-11-2013 21:26:16:581] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:57349

[21-11-2013 21:26:16:581] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

[21-11-2013 21:26:16:582] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl

[21-11-2013 21:26:16:585] pbxiop:ERROR: Connection rejected by PBX exchange

[21-11-2013 21:26:16:586] pbxiop:INFO: Retrying to connect to null

[21-11-2013 21:26:17:588] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:36825

[21-11-2013 21:26:17:588] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

[21-11-2013 21:26:17:589] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl

[21-11-2013 21:26:17:590] pbxiop:ERROR: Connection rejected by PBX exchange

[21-11-2013 21:26:17:590] pbxiop:INFO: Retrying to connect to null

(ObjSvc)org.jacorb.orb.Reference._non_existent

    Request ID: 104

    Server: nbumaster

    Time Taken: 6073ms

    Successful: 

    Reply Status: Unknown

    SystemException: org.omg.CORBA.TRANSIENT

vxlogview -o 103 -t 00:10:00

11/21/2013 22:47:00.324 [Info] hand_off looking for proxy for = nbsl

11/21/2013 22:47:00.324 [Warning] No proxy found.

11/21/2013 22:47:00.324 [Info] PBX_Client_Proxy::handle_close

11/21/2013 22:47:01.327 [Info] PBX_Manager:: handle_input with fd = 7

11/21/2013 22:47:01.327 [Info] PBX_Client_Proxy::parse_line, line = ack=1  From 172.17.122.41

11/21/2013 22:47:01.328 [Info] PBX_Client_Proxy::handle_input read returned: EWOULDBLOCK

11/21/2013 22:47:01.328 [Info] PBX_Client_Proxy::parse_line, line = extension=nbsl  From 172.17.122.41

11/21/2013 22:47:01.328 [Info] hand_off looking for proxy for = nbsl

11/21/2013 22:47:01.328 [Warning] No proxy found.

11/21/2013 22:47:01.328 [Info] PBX_Client_Proxy::handle_close

/opt/VRTSpbx/bin/pbxcfg -p

Auth User:0 : root

Secure Mode: false

Debug Level: 10

Port Number: 1556

PBX service is not cluster configured

ps -ef | grep pbx

root      5069     1  0 Nov18 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange

root     19623 19112  0 23:05 pts/6    00:00:00 grep pbx

So i need help to troubleshot the problem about why it can't connect to pbx.

Thank U

[Swati_Joshi]

Hi,

Have you followed the steps below?

http://www.symantec.com/docs/TECH191634

Thanks,

Swati

[rizwan84tx]

Looks lik NetBackup Service layer (NBSL) process has problem; hope restarted nbsl should help.

21-11-2013 21:26:12:534] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

[21-11-2013 21:26:12:535] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl

[21-11-2013 21:26:12:540] pbxiop:ERROR: Connection rejected by PBX exchange

[Marianne]

Please show us output of bpps -x.

Have you checked that the OS firewall (iptables) is disabled on the master?

Does the master have /etc/hosts entry for itself?

e.g:
127.0.0.1       localhost
<IP-address>  nbumaster

[____111]

bpps -x

NB Processes

------------

root      5091     1  0 Nov18 ?        00:00:03 /usr/openv/netbackup/bin/vnetd -standalone

root      5094     1  0 Nov18 ?        00:00:03 /usr/openv/netbackup/bin/bpcd -standalone

root      5164     1  0 Nov18 ?        00:15:15 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      5357     1  0 Nov18 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd

root      5422     1  0 Nov18 ?        00:00:01 /usr/openv/netbackup/bin/nbcssc -a NetBackup

MM Processes

------------

Shared Symantec Processes

-------------------------

root      5069     1  0 Nov18 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange

cat /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1                localhost.localdomain localhost

::1             localhost6.localdomain6 localhost6

172.17.122.41 nbumaster

172.17.122.42 nbumaster2

172.17.122.43 nbu01med

172.17.122.45 db2client

172.17.122.46 oraclient

172.17.122.25 winclient

172.17.122.26 winclient2

 service iptables status

Firewall is stopped.

thank U

[____111]

Hi

I had restarted the netbackup service then the output below（bpps -X）

NB Processes

------------

root      5091     1  0 Nov18 ?        00:00:03 /usr/openv/netbackup/bin/vnetd -standalone

root      5094     1  0 Nov18 ?        00:00:04 /usr/openv/netbackup/bin/bpcd -standalone

root      5164     1  0 Nov18 ?        00:15:20 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      5357     1  0 Nov18 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd

root      5422     1  0 Nov18 ?        00:00:01 /usr/openv/netbackup/bin/nbcssc -a NetBackup

MM Processes

------------

Shared Symantec Processes

-------------------------

root      5069     1  0 Nov18 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange

But the problem is still exist.

[____111]

I do but nothing change.the problem is still exist.

[Marianne]

Most of NBU master server processes are not running.

What is output of 'hostname' on this server? It should be 'nbumaster'

Please create the following log folders, then restart NBU:

admin bpdbm bprd

Do: 
netbackup stop
/opt/VRTSpbx/bin/vxpbx_exchanged stop

(check with bpps that all processes have stopped)

/opt/VRTSpbx/bin/vxpbx_exchanged start
netbackup

check for running processes with bpps -x.

If processes such as nbemm, bprd, bpdbm, nbpem, nbrb, etc are not running, copy log files in above folders to text files that correspond with process name (e.g. admin.txt) and post these logs as File attachment.

Also post output of this command after restart attempt:

vxlogview -o 111 -t 00:10:00

[____111]

[root@nbumaster logs]# ls -l

total 84

drwxr-xr-x 2 root root 4096 Nov 22 00:27 admin

drwxr-xr-x 2 root root 4096 Nov 18 00:56 bmrd

drwxr-xr-x 2 root root 4096 Aug 14 00:00 bmrsetup

drwxr-xr-x 2 root root 4096 Nov 22 00:27 bpdbm

drwxr-xr-x 2 root root 4096 Nov 22 00:27 bprd

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbars

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbaudit

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbemm

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbevtmgr

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbftclnt

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbim

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbjm

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbkms

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbpem

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbrb

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbrmms

drwxr-xr-x 2 root root 4096 Nov 22 00:00 nbsl

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbstserv

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbsvcmon

drwxr-xr-x 2 root root 4096 Nov 21 20:45 nbutils

drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbvault

[root@nbumaster logs]# pwd

/usr/openv/logs

[root@nbumaster logs]# /opt/VRTSpbx/bin/vxpbx_exchanged stop

Stopped Symantec Private Branch Exchange

[root@nbumaster logs]# /usr/openv/netbackup/bin/goodies/netbackup stop

stopping the NetBackup CloudStore Service Container

stopping the NetBackup compatibility daemon

stopping the NetBackup Database Server

stopping the NetBackup client daemon

stopping the NetBackup network daemon

[root@nbumaster logs]# bpps -x

NB Processes

------------

MM Processes

------------

Shared Symantec Processes

-------------------------

[root@nbumaster logs]# /opt/VRTSpbx/bin/vxpbx_exchanged start

Started Symantec Private Branch Exchange

[root@nbumaster logs]# /usr/openv/netbackup/bin/goodies/netbackup start

NetBackup Authentication daemon started.

NetBackup network daemon started.

NetBackup client daemon started.

NetBackup SAN Client Fibre Transport daemon started.

NetBackup Database Server started.

NetBackup Authorization daemon started.

NetBackup Event Manager started.

NetBackup Audit Manager started.

NetBackup Deduplication Manager started.

NetBackup Deduplication Engine started.

NetBackup Enterprise Media Manager started.

NetBackup Resource Broker started.

Rebuilding device nodes.

Media Manager daemons started.

NetBackup request daemon started.

NetBackup compatibility daemon started.

NetBackup Job Manager started.

NetBackup Policy Execution Manager started.

NetBackup Storage Lifecycle Manager started.

NetBackup Indexing Manager started.

NetBackup Remote Monitoring Management System started.

NetBackup Key Management daemon started.

NetBackup Service Layer started.

NetBackup Agent Request Server started.

NetBackup Bare Metal Restore daemon not started.

NetBackup Vault daemon started.

NetBackup CloudStore Service Container started.

NetBackup Service Monitor started.

NetBackup Bare Metal Restore Boot Server daemon started.

[root@nbumaster logs]# bpps -x

NB Processes

------------

root     20558     1  0 00:34 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone

root     20561     1  0 00:34 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone

root     20631     1  2 00:34 ?        00:00:01 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root     20765     1  0 00:35 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd

root     20787     1  2 00:35 ?        00:00:00 /usr/openv/netbackup/bin/nbsl

root     20831     1  0 00:35 pts/1    00:00:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup

root     20849     1  3 00:35 ?        00:00:00 /usr/openv/netbackup/bin/bmrbd

MM Processes

------------

Shared Symantec Processes

-------------------------

root     20535     1  0 00:33 ?        00:00:00 /opt/VRTSpbx/bin/pbx_exchange

[root@nbumaster logs]# vxlogview -o 111 -t 00:10:00 >/tmp/1122.log

[root@nbumaster logs]# cat /tmp/1122.log

11/22/2013 00:34:53.358 [EMMServer::EMMServer]  

11/22/2013 00:34:53.365 [DSMFSMORBConfig::DSMFSMORBConfig] Setting DSM ORB thread count to <10>

11/22/2013 00:34:53.375 [REMORBConfig::REMORBConfig] Setting REM ORB thread count to <10>

11/22/2013 00:34:53.375 [FATClientORBConfig::FATClientORBConfig] Setting FTClient ORB thread count to <10>

11/22/2013 00:34:53.550 [EMMMain] <ENTER>

11/22/2013 00:34:53.552 [Error] V-111-1049 EMMServer generic error = controlledRun() failed -> doShouldStart(), doInitialize() or doRun() failed

11/22/2013 00:34:53.584 [FATClientORBConfig::shutServant] Calling fini() on FAT Client

11/22/2013 00:34:53.584 [FATClientORBConfig::shutServant] Done fini() on FAT Client

11/22/2013 00:34:53.587 [REMORBConfig::shutServant] Calling fini() on REM

11/22/2013 00:34:53.587 [REMORBConfig::shutServant] Done fini() on REM

11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Calling fini() on FSM

11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Done fini() on FSM

11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Calling fini() on DSM

11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Done fini() on DSM

11/22/2013 00:34:53.587 [EMMServer::fini] EMM Server shutting down

11/22/2013 00:34:53.587 [Info] V-111-1003 [EMMServer::fini] EMMServer shutting down

11/22/2013 00:34:53.587 [EMMServer::fini] EMM Server shut down complete

11/22/2013 00:34:53.593 [EMMMain] <EXIT>

11/22/2013 00:34:53.593 [Log::deinit]  

11/22/2013 00:34:53.597 [EMMServer::~EMMServer]  

The admin&bprd&bpdbm log is not generating,is it anything option steps missing？

Thank U for help.

[Marianne]

We see that there is an attempt to start bpdbm and bprd:

NetBackup Database Server started.
....
NetBackup request daemon started.

If nothing is logged in these folders, there is something very wrong with your installion.

Is this a new installation that has never worked?
If so - have you checked the installation log to see if all was successful?
(Look for install_trace.####  in /usr/openv/tmp)
 

What is output of 'hostname'?
Show us contents of bp.conf file. 
nbumaster should be first 'SERVER'.

Try to manually start bprd and see if anything is logged in bprd log:

/usr/openv/netbackup/bin/initbprd
 

[____111]

1.This is not a new installation.It's working normally before

2.[root@nbumaster bpdbm]# hostname

nbumaster

3.[root@nbumaster bpdbm]# cat /usr/openv/netbackup/bp.conf 

SERVER = nbumaster

CLIENT_NAME = nbumaster

SERVER = nbu01med 

CONNECT_OPTIONS = localhost 1 0 2

USE_VXSS = PROHIBITED

VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY

EMMSERVER = nbumaster

HOST_CACHE_TTL = 3600

VXDBMS_NB_DATA = /usr/openv/db/data

LIST_FS_IMAGE_HEADERS = NO

TELEMETRY_UPLOAD = NO

REQUIRED_INTERFACE = nbumaster

4.[root@nbumaster logs]# /usr/openv/netbackup/bin/initbprd

[root@nbumaster logs]# pwd

/usr/openv/logs

[root@nbumaster logs]# cd bprd

[root@nbumaster bprd]# ls

[root@nbumaster bprd]# ls -l

total 0

[root@nbumaster bprd]# 

It's not generate log as well.Is it the path of logfile is correct？(/usr/openv/logs/bprd)

Thank U

[Marianne]

Oops - seems your log folders are in the wrong place!

Logs for legacy processes must be in /usr/openv/netbackup/logs.

So, please create admin bpdbm and bprd under ..netbackup/logs, then try to stop and start NBU again.

There should be something in the logs this time.

[Marianne · Accepted solution]

bprd log was sent via PM:

bprd: NetBackup evaluation expired: expiration = Thu Nov 21 02:05:10 2013, it is now Fri Nov 22 02:05:10 2013.....

So - this is why NBU processes have stopped and cannot be started again.