Forum Discussion

____111's avatar
____111
Level 3
12 years ago

ERROR: Connection rejected by PBX exchange

Hi 

I got a problem,when I start the java console(jnbSA) from the master server,then I got the error (see below)

CN0EIRHSNVR85KZ_$YAFWY3.jpg

 

Then i view the log as below

/usr/openv/netbackup/logs/user_ops/nbjlogs/jbp.root.jnbSA.19328.log

 

ICACHE.PBXIOPCredsHandler-> ServiceContext registered.
[21-11-2013 21:26:12:533] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:18243
[21-11-2013 21:26:12:534] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl
[21-11-2013 21:26:12:535] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl
[21-11-2013 21:26:12:540] pbxiop:ERROR: Connection rejected by PBX exchange
[21-11-2013 21:26:12:540] pbxiop:INFO: Retrying to connect to null
[21-11-2013 21:26:13:543] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:32614
[21-11-2013 21:26:13:543] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl
[21-11-2013 21:26:13:544] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl
[21-11-2013 21:26:13:546] pbxiop:ERROR: Connection rejected by PBX exchange
[21-11-2013 21:26:13:547] pbxiop:INFO: Retrying to connect to null
[21-11-2013 21:26:14:548] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:21025
[21-11-2013 21:26:14:548] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl
[21-11-2013 21:26:14:549] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl
[21-11-2013 21:26:14:549] pbxiop:ERROR: Connection rejected by PBX exchange
[21-11-2013 21:26:14:550] pbxiop:INFO: Retrying to connect to null
[21-11-2013 21:26:15:552] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:37459
[21-11-2013 21:26:15:552] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl
[21-11-2013 21:26:15:553] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl
[21-11-2013 21:26:15:578] pbxiop:ERROR: Connection rejected by PBX exchange
[21-11-2013 21:26:15:579] pbxiop:INFO: Retrying to connect to null
[21-11-2013 21:26:16:581] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:57349
[21-11-2013 21:26:16:581] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl
[21-11-2013 21:26:16:582] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl
[21-11-2013 21:26:16:585] pbxiop:ERROR: Connection rejected by PBX exchange
[21-11-2013 21:26:16:586] pbxiop:INFO: Retrying to connect to null
[21-11-2013 21:26:17:588] pbxiop:INFO: Created bound socket 0.0.0.0/0.0.0.0:36825
[21-11-2013 21:26:17:588] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl
[21-11-2013 21:26:17:589] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl
[21-11-2013 21:26:17:590] pbxiop:ERROR: Connection rejected by PBX exchange
[21-11-2013 21:26:17:590] pbxiop:INFO: Retrying to connect to null
(ObjSvc)org.jacorb.orb.Reference._non_existent
    Request ID: 104
    Server: nbumaster
    Time Taken: 6073ms
    Successful: 
    Reply Status: Unknown
    SystemException: org.omg.CORBA.TRANSIENT
 
vxlogview -o 103 -t 00:10:00
 
11/21/2013 22:47:00.324 [Info] hand_off looking for proxy for = nbsl
11/21/2013 22:47:00.324 [Warning] No proxy found.
11/21/2013 22:47:00.324 [Info] PBX_Client_Proxy::handle_close
11/21/2013 22:47:01.327 [Info] PBX_Manager:: handle_input with fd = 7
11/21/2013 22:47:01.327 [Info] PBX_Client_Proxy::parse_line, line = ack=1  From 172.17.122.41
11/21/2013 22:47:01.328 [Info] PBX_Client_Proxy::handle_input read returned: EWOULDBLOCK
11/21/2013 22:47:01.328 [Info] PBX_Client_Proxy::parse_line, line = extension=nbsl  From 172.17.122.41
11/21/2013 22:47:01.328 [Info] hand_off looking for proxy for = nbsl
11/21/2013 22:47:01.328 [Warning] No proxy found.
11/21/2013 22:47:01.328 [Info] PBX_Client_Proxy::handle_close
 
/opt/VRTSpbx/bin/pbxcfg -p
Auth User:0 : root
Secure Mode: false
Debug Level: 10
Port Number: 1556
PBX service is not cluster configured
 
ps -ef | grep pbx
root      5069     1  0 Nov18 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange
root     19623 19112  0 23:05 pts/6    00:00:00 grep pbx
 
So i need help to troubleshot the problem about why it can't connect to pbx.
 
Thank U
 
 
 
7.5
Backup and Recovery
Error messages
Linux
NetBackup
  • Marianne's avatar
    Marianne
    12 years ago

    bprd log was sent via PM:

    bprd: NetBackup evaluation expired: expiration = Thu Nov 21 02:05:10 2013, it is now Fri Nov 22 02:05:10 2013.....

    So - this is why NBU processes have stopped and cannot be started again.

  • rizwan84tx's avatar
    rizwan84tx
    Level 6
    12 years ago

    Looks lik NetBackup Service layer (NBSL) process has problem; hope restarted nbsl should help.

     

    21-11-2013 21:26:12:534] pbxiop:INFO: createSocket - Trying to connect to nbumaster:1556:nbsl

    [21-11-2013 21:26:12:535] pbxiop:INFO: Connected to exchange at:nbumaster:1556:nbsl
    [21-11-2013 21:26:12:540] pbxiop:ERROR: Connection rejected by PBX exchange
  • Marianne's avatar
    Marianne
    Level 6
    12 years ago

    Please show us output of bpps -x.

    Have you checked that the OS firewall (iptables) is disabled on the master?

    Does the master have /etc/hosts entry for itself?

    e.g:
    127.0.0.1       localhost
    <IP-address>  nbumaster

  • ____111's avatar
    ____111
    Level 3
    12 years ago

     

    bpps -x
    NB Processes
    ------------
    root      5091     1  0 Nov18 ?        00:00:03 /usr/openv/netbackup/bin/vnetd -standalone
    root      5094     1  0 Nov18 ?        00:00:03 /usr/openv/netbackup/bin/bpcd -standalone
    root      5164     1  0 Nov18 ?        00:15:15 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
    root      5357     1  0 Nov18 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
    root      5422     1  0 Nov18 ?        00:00:01 /usr/openv/netbackup/bin/nbcssc -a NetBackup
     
     
    MM Processes
    ------------
     
     
    Shared Symantec Processes
    -------------------------
    root      5069     1  0 Nov18 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange
     
    cat /etc/hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    127.0.0.1                localhost.localdomain localhost
    ::1             localhost6.localdomain6 localhost6
    172.17.122.41 nbumaster
    172.17.122.42 nbumaster2
    172.17.122.43 nbu01med
    172.17.122.45 db2client
    172.17.122.46 oraclient
    172.17.122.25 winclient
    172.17.122.26 winclient2
     

     

     service iptables status
    Firewall is stopped.
     
     
    thank U
  • ____111's avatar
    ____111
    Level 3
    12 years ago

    Hi

    I had restarted the netbackup service then the output below（bpps -X）

     

    NB Processes
    ------------
    root      5091     1  0 Nov18 ?        00:00:03 /usr/openv/netbackup/bin/vnetd -standalone
    root      5094     1  0 Nov18 ?        00:00:04 /usr/openv/netbackup/bin/bpcd -standalone
    root      5164     1  0 Nov18 ?        00:15:20 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
    root      5357     1  0 Nov18 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
    root      5422     1  0 Nov18 ?        00:00:01 /usr/openv/netbackup/bin/nbcssc -a NetBackup
     
     
    MM Processes
    ------------
     
     
    Shared Symantec Processes
    -------------------------
    root      5069     1  0 Nov18 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange
     
    But the problem is still exist.

     

  • Marianne's avatar
    Marianne
    Level 6
    12 years ago

    Most of NBU master server processes are not running.

    What is output of 'hostname' on this server? It should be 'nbumaster'

    Please create the following log folders, then restart NBU:

    admin bpdbm bprd

    Do: 
    netbackup stop
    /opt/VRTSpbx/bin/vxpbx_exchanged stop

    (check with bpps that all processes have stopped)

    /opt/VRTSpbx/bin/vxpbx_exchanged start
    netbackup

    check for running processes with bpps -x.

    If processes such as nbemm, bprd, bpdbm, nbpem, nbrb, etc are not running, copy log files in above folders to text files that correspond with process name (e.g. admin.txt) and post these logs as File attachment.

    Also post output of this command after restart attempt:

    vxlogview -o 111 -t 00:10:00

  • ____111's avatar
    ____111
    Level 3
    12 years ago

     

    [root@nbumaster logs]# ls -l
    total 84
    drwxr-xr-x 2 root root 4096 Nov 22 00:27 admin
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 bmrd
    drwxr-xr-x 2 root root 4096 Aug 14 00:00 bmrsetup
    drwxr-xr-x 2 root root 4096 Nov 22 00:27 bpdbm
    drwxr-xr-x 2 root root 4096 Nov 22 00:27 bprd
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbars
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbaudit
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbemm
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbevtmgr
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbftclnt
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbim
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbjm
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbkms
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbpem
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbrb
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbrmms
    drwxr-xr-x 2 root root 4096 Nov 22 00:00 nbsl
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbstserv
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbsvcmon
    drwxr-xr-x 2 root root 4096 Nov 21 20:45 nbutils
    drwxr-xr-x 2 root root 4096 Nov 18 00:56 nbvault
     
     
    [root@nbumaster logs]# pwd
    /usr/openv/logs
     
     
    [root@nbumaster logs]# /opt/VRTSpbx/bin/vxpbx_exchanged stop
    Stopped Symantec Private Branch Exchange

     

     

    [root@nbumaster logs]# /usr/openv/netbackup/bin/goodies/netbackup stop
    stopping the NetBackup CloudStore Service Container
    stopping the NetBackup compatibility daemon
    stopping the NetBackup Database Server
    stopping the NetBackup client daemon
    stopping the NetBackup network daemon
    [root@nbumaster logs]# bpps -x
    NB Processes
    ------------
     
     
    MM Processes
    ------------
     
     
    Shared Symantec Processes
    -------------------------
     
     
     
     
     
    [root@nbumaster logs]# /opt/VRTSpbx/bin/vxpbx_exchanged start
    Started Symantec Private Branch Exchange
    [root@nbumaster logs]# /usr/openv/netbackup/bin/goodies/netbackup start
    NetBackup Authentication daemon started.
    NetBackup network daemon started.
    NetBackup client daemon started.
    NetBackup SAN Client Fibre Transport daemon started.
    NetBackup Database Server started.
    NetBackup Authorization daemon started.
    NetBackup Event Manager started.
    NetBackup Audit Manager started.
    NetBackup Deduplication Manager started.
    NetBackup Deduplication Engine started.
    NetBackup Enterprise Media Manager started.
    NetBackup Resource Broker started.
    Rebuilding device nodes.
    Media Manager daemons started.
    NetBackup request daemon started.
    NetBackup compatibility daemon started.
    NetBackup Job Manager started.
    NetBackup Policy Execution Manager started.
    NetBackup Storage Lifecycle Manager started.
    NetBackup Indexing Manager started.
    NetBackup Remote Monitoring Management System started.
    NetBackup Key Management daemon started.
    NetBackup Service Layer started.
    NetBackup Agent Request Server started.
    NetBackup Bare Metal Restore daemon not started.
    NetBackup Vault daemon started.
    NetBackup CloudStore Service Container started.
    NetBackup Service Monitor started.
    NetBackup Bare Metal Restore Boot Server daemon started.
    [root@nbumaster logs]# bpps -x
    NB Processes
    ------------
    root     20558     1  0 00:34 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
    root     20561     1  0 00:34 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
    root     20631     1  2 00:34 ?        00:00:01 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
    root     20765     1  0 00:35 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
    root     20787     1  2 00:35 ?        00:00:00 /usr/openv/netbackup/bin/nbsl
    root     20831     1  0 00:35 pts/1    00:00:00 /usr/openv/netbackup/bin/nbcssc -a NetBackup
    root     20849     1  3 00:35 ?        00:00:00 /usr/openv/netbackup/bin/bmrbd
     
     
    MM Processes
    ------------
     
     
    Shared Symantec Processes
    -------------------------
    root     20535     1  0 00:33 ?        00:00:00 /opt/VRTSpbx/bin/pbx_exchange

     

     

     

    [root@nbumaster logs]# vxlogview -o 111 -t 00:10:00 >/tmp/1122.log
    [root@nbumaster logs]# cat /tmp/1122.log
    11/22/2013 00:34:53.358 [EMMServer::EMMServer]  
    11/22/2013 00:34:53.365 [DSMFSMORBConfig::DSMFSMORBConfig] Setting DSM ORB thread count to <10>
    11/22/2013 00:34:53.375 [REMORBConfig::REMORBConfig] Setting REM ORB thread count to <10>
    11/22/2013 00:34:53.375 [FATClientORBConfig::FATClientORBConfig] Setting FTClient ORB thread count to <10>
    11/22/2013 00:34:53.550 [EMMMain] <ENTER>
    11/22/2013 00:34:53.552 [Error] V-111-1049 EMMServer generic error = controlledRun() failed -> doShouldStart(), doInitialize() or doRun() failed
    11/22/2013 00:34:53.584 [FATClientORBConfig::shutServant] Calling fini() on FAT Client
    11/22/2013 00:34:53.584 [FATClientORBConfig::shutServant] Done fini() on FAT Client
    11/22/2013 00:34:53.587 [REMORBConfig::shutServant] Calling fini() on REM
    11/22/2013 00:34:53.587 [REMORBConfig::shutServant] Done fini() on REM
    11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Calling fini() on FSM
    11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Done fini() on FSM
    11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Calling fini() on DSM
    11/22/2013 00:34:53.587 [DSMFSMORBConfig::shutServants] Done fini() on DSM
    11/22/2013 00:34:53.587 [EMMServer::fini] EMM Server shutting down
    11/22/2013 00:34:53.587 [Info] V-111-1003 [EMMServer::fini] EMMServer shutting down
    11/22/2013 00:34:53.587 [EMMServer::fini] EMM Server shut down complete
    11/22/2013 00:34:53.593 [EMMMain] <EXIT>
    11/22/2013 00:34:53.593 [Log::deinit]  
    11/22/2013 00:34:53.597 [EMMServer::~EMMServer]  

     

    The admin&bprd&bpdbm log is not generating,is it anything option steps missing？

     

    Thank U for help.smiley

  • Marianne's avatar
    Marianne
    Level 6
    12 years ago

    We see that there is an attempt to start bpdbm and bprd:

    NetBackup Database Server started.
    ....
    NetBackup request daemon started.

    If nothing is logged in these folders, there is something very wrong with your installion.

    Is this a new installation that has never worked?
    If so - have you checked the installation log to see if all was successful?
    (    Look for install_trace.####  in /usr/openv/tmp)
     

    What is output of 'hostname'?
    Show us contents of bp.conf file. 
    nbumaster should be first 'SERVER'.

    Try to manually start bprd and see if anything is logged in bprd log:

    /usr/openv/netbackup/bin/initbprd
     

  • ____111's avatar
    ____111
    Level 3
    12 years ago

    1.This is not a new installation.It's working normally before

    2.[root@nbumaster bpdbm]# hostname

    nbumaster
     
    3.[root@nbumaster bpdbm]# cat /usr/openv/netbackup/bp.conf 
    SERVER = nbumaster
    CLIENT_NAME = nbumaster
     
    SERVER = nbu01med 
    CONNECT_OPTIONS = localhost 1 0 2
    USE_VXSS = PROHIBITED
    VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY
    EMMSERVER = nbumaster
    HOST_CACHE_TTL = 3600
    VXDBMS_NB_DATA = /usr/openv/db/data
    LIST_FS_IMAGE_HEADERS = NO
    TELEMETRY_UPLOAD = NO
    REQUIRED_INTERFACE = nbumaster
     
    4.[root@nbumaster logs]# /usr/openv/netbackup/bin/initbprd
    [root@nbumaster logs]# pwd
    /usr/openv/logs
    [root@nbumaster logs]# cd bprd
    [root@nbumaster bprd]# ls
    [root@nbumaster bprd]# ls -l
    total 0
    [root@nbumaster bprd]# 
     
    It's not generate log as well.Is it the path of logfile is correct？(/usr/openv/logs/bprd)
     
    Thank U