Netbackup Error

[girish_nair1984]

Dear Friends

I am using Netbackup 7.0.

While doing Robot inventory, I am getting an error : EMM Database Error (Status 196) and The Enterprise Media Manager Server is not responding (Status 25).

The following Daemons has stopped running.

bmrd,nbsvcmon,nbars, spad,nbrb,spoold,nbemm,nbaudit,nbkms.

Kindly advice, how to rectify these error.

Thank You

[girish_nair1984]

While running this command tpautoconf –report_disc

I am getting an error : EMM interface initialization failed, status 77

[mph999]

Stop NBU (what's left of it)

Reboot Server

What NBU processes are running ?

Martin

[girish_nair1984]

I created a log folder (/usr/openv/netbackup/logs/admin) the following log was generated

15:24:02.823 [26398] <4> db_getSTUNITlist: emmserver_name = his1

15:24:02.823 [26398] <4> db_getSTUNITlist: emmserver_port = 1556

15:24:12.266 [26410] <2> bprdreq: INITIATING: version NetBackup 7.0 created: 2010070800

15:24:12.266 [26410] <2> logparams: /usr/openv/netbackup/bin/admincmd/bprdreq -M localhost -terminate

15:24:12.267 [26410] <2> bprdreq: VERBOSE = 0

15:24:12.269 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002

15:24:12.269 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+veritas_pbx,1,20,2,1,0+localhost.txt

15:24:12.270 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002

15:24:12.270 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+vnetd,1,20,2,1,0+localhost.txt

15:24:12.271 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002

15:24:12.271 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+bprd,1,20,2,1,0+localhost.txt

15:24:12.272 [26410] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:24:12.272 [26410] <2> logconnections: BPRD CONNECT FROM 127.0.0.1.19974 TO 127.0.0.1.1556 fd = 4

15:24:12.377 [26414] <2> nbpemreq: INITIATING: version NetBackup 7.0 created: 2010070800

15:24:12.381 [26414] <2> logparams: /usr/openv/netbackup/bin/admincmd/nbpemreq -suspend_scheduling

15:24:12.390 [26414] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:759)

15:24:13.306 [26410] <2> doreq: ...ok, received 0 from bprd

15:24:16.307 [26410] <2> bprdreq: EXIT status = 0

15:25:01.896 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:25:01.897 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:26:05.953 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:26:05.953 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:27:10.007 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:27:10.007 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:28:19.172 [27916] <2> nbpemreq: INITIATING: version NetBackup 7.0 created: 2010070800

15:28:19.172 [27916] <2> logparams: /usr/openv/netbackup/bin/admincmd/nbpemreq -resume_scheduling

15:28:19.180 [27916] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:759)

15:28:31.001 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::COMM_FAILURE

15:28:31.001 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/COMM_FAILURE:1.0'

TAO exception, minor code = 13 (failed to recv request response; ECONNRESET), completed = MAYBE

15:28:35.183 [28198] <4> db_getSTUNITlist: emmserver_name = his1

15:28:35.183 [28198] <4> db_getSTUNITlist: emmserver_port = 1556

15:29:12.327 [28263] <2> logparams: /usr/openv/netbackup/bin/admincmd/bpversion

15:29:12.331 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.331 [28263] <2> logconnections: BPCD CONNECT FROM 192.168.1.13.26281 TO 192.168.1.13.1556 fd = 3

15:29:12.332 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.333 [28263] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1716: 0: via PBX: VNETD CONNECT FROM 192.168.1.13.50004 TO 192.168.1.13.1556 fd = 4

15:29:12.333 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.540: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

15:29:12.373 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.557: 0: ipc_string: /tmp/vnet-28265346930952373257000000000-x3xOeR

15:29:12.413 [28263] <2> local_bpcr_connect: expected reserved port, got 1556

15:29:12.523 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.523 [28263] <2> logconnections: BPCD CONNECT FROM 192.168.1.13.43878 TO 192.168.1.13.1556 fd = 3

15:29:12.523 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.524 [28263] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1716: 0: via PBX: VNETD CONNECT FROM 192.168.1.13.22652 TO 192.168.1.13.1556 fd = 4

15:29:12.524 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.540: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

15:29:12.564 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.557: 0: ipc_string: /tmp/vnet-28269346930952564247000000000-pYOPo5

15:29:12.604 [28263] <2> local_bpcr_connect: expected reserved port, got 1556

15:29:34.259 [28198] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:29:34.259 [28198] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:29:35.064 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:29:35.064 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:29:40.064 [26398] <16> db_getSTUNITlist: (-) Translating EMM_ERROR_CorbaTransient(3000001) to 25 in the NetBackup context

15:29:40.064 [26398] <16> db_getSTUNITlist: error (25) initializing EMM interface.

15:29:40.064 [26398] <16> bpstsinfo/update ERROR: unable to retrieve stu list from server (25)

15:30:38.315 [28198] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:30:38.315 [28198] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

[girish_nair1984]

The following daemons are running

ltid

nbsl

bprd

vmd

bocompatd

nbstserv

nbsvsmon

NB_dbsrv

nbrmms

bpdbm

nbjm

nbpem

nbevtmgr

[Mark_Solutions]

Corba Transient Error .... Sounds like you may be out of memory ... reboot at least!

Are you really on 7.0? I would advise you patch at least or preferably upgrade

Hope this helps

[mph999]

I meant reboot, then tell us what processes are runing ...

[girish_nair1984]

Yes I am using Netbackup 7.0.

But we have 2 Netback Servers with same partition size. 1 is working perfectly fine while another one is showing the above mentioned error.s

[girish_nair1984]

After using this command /usr/openv/netbackup/bin/goodies/ netbackup stop

 and then bpps -a to verify the processes/daemons are running

NB Processes

------------

root      1066     1  0 12:13 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      1085  1066  0 12:13 ?        00:00:00 [NB_dbsrv] <defunct>

root      1217     1  0 12:13 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      1223  1217  0 12:13 ?        00:00:00 [NB_dbsrv] <defunct>

root      4714     1  0 12:28 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      4719  4714  0 12:28 ?        00:00:00 [NB_dbsrv] <defunct>

root      4966     1  0 12:28 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      4968  4966  0 12:28 ?        00:00:00 [NB_dbsrv] <defunct>

root     12441     1  0 Sep05 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root     12443 12441  0 Sep05 ?        00:00:00 [NB_dbsrv] <defunct>

root     13484     1  0 13:15 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem

root     28843 28802  0 15:34 pts/5    00:00:09 /usr/openv/java/jre/bin/java -Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -showversion -Dvrts.common.server.LCCTYPE="en_US" -Dvrts.common.server.LOG_CMDLINES=false -Djava.library.path=/usr/openv/lib -Dvrtsat.home=/usr/openv/netbackup/sec/at -Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/java/Debug.properties -Dvrts.nbe.HOSTNAME= -Dvrts.nbe.INST_REL=700000 -Dvrts.nbe.BUILDNO=20100707 -Djava.util.prefs.PreferencesFactory=vrts.common.launch.PropertiesPreferencesFactory -Xms36M -Xmx256M vrts.nbe.AdminConsole

root     28885     1  0 15:34 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     28887 28885  0 15:34 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     28893 28885  0 15:34 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     28967 28885  0 15:35 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     29471     1  0 15:40 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem

[Mark_Solutions]

Have you rebooted yet as we advised? Will clear those orphaned processes that may be causing the memory / Corba errors - if they persist after the reboot you may also need to delete the orphaned corba file - but let us know first before we cover that

[girish_nair1984]

@ Mark_Solutions.

Yes I have rebooted as per your advice, but still that error is there.

[Marianne]

Have you checked disk space on the master server?  

EMM will shut down when filesystem runs low on disk space to prevent corruption of database.

Is PBX running?
Use 'bpps -x' to list all processes.

After stopping and starting NBU, check EMM logs for the last 10 minutes (or enough minutes to capture stop and start):

vxlogview -o 111 -t 00:10:00

[badri]

by any chance was the server renamed or added to a different domain?

[girish_nair1984]

@ Marianne van den Berg

Please find the logs after running bpps -x command

---

NB Processes
------------
root      1066     1  0 Sep06 ?    00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf 
@/usr/openv/var/global/databases.conf -hn 7
root      1085  1066  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root      1217     1  0 Sep06 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv /usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      1223  1217  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root      4714     1  0 Sep06 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      4719  4714  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root     4966     1  0 Sep06 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv 
@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      4968  4966  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root     12441     1  0 Sep05 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv 
@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root     12443 12441  0 Sep05 ?        00:00:00 [NB_dbsrv] <defunct>
root     13484     1  0 Sep06 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root     15203     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root     15206     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root     15269     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbevtmgr
root     15535     1  0 10:22 pts/3    00:00:00 /usr/openv/netbackup/bin/bprd
root     15540     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root     15551     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbjm
root     15572     1  0 10:22 pts/3    00:00:00 /usr/openv/netbackup/bin/bpdbm
root     15579 15572  0 10:22 pts/3    00:00:00 /usr/openv/netbackup/bin/bpjobd
root     15589     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbpem
root     15642     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbstserv
root     15690     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbrmms
root     15765     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbsl
root     16035     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbsvcmon
root     16263 16222  1 10:23 pts/3    00:00:05 /usr/openv/java/jre/bin/java - 
Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -showversion - 
Dvrts.common.server.LCCTYPE="en_US" - 
Dvrts.common.server.LOG_CMDLINES=false - 
Djava.library.path=/usr/openv/lib - 
Dvrtsat.home=/usr/openv/netbackup/sec/at - 
Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/java/Debug.properties -Dvrts.nbe.HOSTNAME= - Dvrts.nbe.INST_REL=700000 -Dvrts.nbe.BUILDNO=20100707 -Djava.util.prefs.PreferencesFactory=vrts.common.launch.PropertiesPreferencesFactory -Xms36M -Xmx256M vrts.nbe.AdminConsole
root     16314     1  0 10:23 ?  00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 
en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16316 16314  0 10:23 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 
en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16321 16314  0 10:23 ?    00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16374 16314  0 10:23 ?     00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16878     1  0 10:27 pts/3    00:00:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
root     17300 15589  0 10:30 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root     29471     1  0 Sep06 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem


MM Processes
------------
root     15530     1  0 10:22 pts/3    00:00:00 /usr/openv/volmgr/bin/ltid
root     15694     1  0 10:22 pts/3    00:00:00 vmd


Shared Symantec Processes
-------------------------
root      3972     1  0 Mar20 ?         00:01:37 /opt/VRTSpbx/bin/pbx_exchange
 

---

Output of vxlogview -o 111 -t 00:10:00

---

[root@his1 ~]# /usr/openv/netbackup/bin/vxlogview -o 111 -t 00:10:00
09/11/2012 10:22:30.691 [EMMServer::EMMServer]  
09/11/2012 10:22:30.691 [DSMFSMORBConfig::DSMFSMORBConfig] Setting DSM ORB thread count to <10>
09/11/2012 10:22:30.691 [REMORBConfig::REMORBConfig] Setting REM ORB thread count to <10>
09/11/2012 10:22:30.691 [FATClientORBConfig::FATClientORBConfig] Setting FTClient ORB thread count to <10>
09/11/2012 10:22:30.774 [EMMMain] <ENTER>
09/11/2012 10:22:30.776 [EMMServer::isLocalHostEMMServer]  
09/11/2012 10:22:30.778 [EMMServer::isLocalHostEMMServer] LocalHost = <his1>, EmmServer = <his1>
09/11/2012 10:22:30.778 [NBEmmSvc::doShouldStart] Should start
09/11/2012 10:22:30.924 [NBEmmSvc::doInit] <ENTER>
09/11/2012 10:22:30.926 [NBEmmSvc::doInit] NumberOrbThreads = <10>, NumberDbConnections = <4>, NumberDbBrowseConnections = <3>
09/11/2012 10:22:30.927 [EMMServer::SetOrb] Orb is set
09/11/2012 10:22:30.927 [NBEmmSvc::doInit] Extra orb options:  -ORBCollocation no
09/11/2012 10:22:30.927 [EMMServer::init] Server initializing VxEMM
09/11/2012 10:22:30.927 [Info] V-111-1000 [EMMServer::init] EMMServer initializing = VxEMM
09/11/2012 10:22:30.927 [EMMServer::init] Option: -DbServer *** Unrecognized option -DbServer ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbServer *** Unrecognized option -DbServer ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: NB_his1 *** Unrecognized option NB_his1 ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: NB_his1 *** Unrecognized option NB_his1 ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: -DbPort *** Unrecognized option -DbPort ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbPort *** Unrecognized option -DbPort ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: 13785 *** Unrecognized option 13785 ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: 13785 *** Unrecognized option 13785 ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: -DbName *** Unrecognized option -DbName ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbName *** Unrecognized option -DbName ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: NBDB *** Unrecognized option NBDB ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: NBDB *** Unrecognized option NBDB ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: -ConnectString *** Unrecognized option -ConnectString ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -ConnectString *** Unrecognized option -ConnectString ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=******;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB *** Unrecognized option PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=nbusql;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=******;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB *** Unrecognized option PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=nbusql;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB ***
09/11/2012 10:22:30.928 [EMMServer::run] Server running
09/11/2012 10:22:30.928 [Info] V-111-1002 [EMMServer::run] EMMServer running
09/11/2012 10:22:49.923 V-111-1051 [EMMServer::run] Can not establish initial DB connection
09/11/2012 10:22:49.923 [Error] V-111-1049 EMMServer generic error = Can not establish initial DB connection
09/11/2012 10:22:49.923 [Error] V-111-1049 EMMServer generic error = controlledRun() failed -> doShouldStart(), doInitialize() or doRun() failed
09/11/2012 10:22:49.923 [FATClientORBConfig::shutServant] Calling fini() on FAT Client
09/11/2012 10:22:49.923 [FATClientORBConfig::shutServant] Done fini() on FAT Client
09/11/2012 10:22:49.923 [REMORBConfig::shutServant] Calling fini() on REM
09/11/2012 10:22:49.923 [REMORBConfig::shutServant] Done fini() on REM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Calling fini() on FSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Done fini() on FSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Calling fini() on DSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Done fini() on DSM
09/11/2012 10:22:49.923 [EMMServer::fini] EMM Server shutting down
09/11/2012 10:22:49.923 [Info] V-111-1003 [EMMServer::fini] EMMServer shutting down
09/11/2012 10:22:49.995 [Error] V-111-1049 EMMServer generic error = Exception caught for IORTable clean up <NotFound>
09/11/2012 10:22:49.995 [EMMServer::fini] EMM Server shut down complete
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.008 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.009 [EMMMain] <EXIT>
09/11/2012 10:22:50.009 [Log::deinit]  
09/11/2012 10:22:50.009 [EMMServer::~EMMServer]  
 

---

[Mark_Solutions]

It sounds like there is something wrong with SQL Anywhere daemon or your server.conf file - see this tech note which describes it :

http://www.symantec.com/docs/HOWTO33625

As the EMM database starts to come up it doesn't seem to understand anything in that file so it is either that or it may be SQL Anywhere itself.

It is the same sirt of error you can get after an upgrade - take a look through this thread:

https://www-secure.symantec.com/connect/forums/nbu-update-7101-failed-initialize-emm-connection-195

To summarise ... the above was a schema mismatch after a patch .. i dont see that in your EMM log but maybe it doesn't go back far enough?

So first run:

./nbdbms_start_server located in /usr/openv/db/bin.

You may be able to see in clear text why EMM doesn't start as intended.

Have you patched or anything prior to this error? Has anything changed? Have you plenty of disk space?

[Marianne]

Seems you did NOT reboot as requested...
pbx_exchange has been up and running since 20 March. Other processes since 5 and 6 Sept....

You have not told us if you have checked/verified disk space on the master?

The errors in emm log looks like possible corruption of one of more config files. Please check size and timestamp of all of the following files - none of them should be 0 bytes:

/usr/openv/var/global/server.conf
/usr/openv/var/global/databases.conf
/usr/openv/db/data/vxdbms.conf

[mph999 · Accepted solution]

OK, this should not be taking this long to resolve.

Following on from Mariannes post, if the disk did fill up it is very likely that the transaction log has been toasted ...  There is no way to tell, so we just make a basic check, and if this fails, we rebuild it.

Do exactly these steps.

Stop NBU

Reboot server

Is NBU now working ... if yes, then the issue is fixed.

If No, do this :

stop NBU and then run

/usr/openv/db/bin/nbdb_ping

You should see a message saying :

Database [NBDB] is alive and well on server [NB_womble].   (Your server name will be different)

If you see a message saying the database is unavailable then you will need to recreate the transaction log to see if that resolves the issue.  To rebuild the transaction log, follow these steps exactly.

/usr/openv/netbackup/bin/goodies/netbackup stop

/usr/openv/db/bin/nbdb_admin -auto_start NONE

. /usr/openv/db/vxdbms_env.sh

cd /usr/openv/db/data

mv NBDB.log NBDB.log.bad

/usr/openv/db/bin/dbeng11 -f NBDB

/usr/openv/db/bin/nbdb_admin -auto_start NBDB

( /usr/openv/db/bin/nbdb_admin -auto_start BMRDB  -  If you use BMR)

/usr/openv/db/bin/nbdbms_start_server

/usr/openv/db/bin/nbdb_ping

If it now shows the DB as alive and well, hopefully issue is fixed and you can restart NBU

Martin

[girish_nair1984]

Dear friends

I have stop NBU and restarted the server but.

At this point during startup it is stuck showing message

Netbackup SAN Client fibre transport daemon started

It's been 45 mins till now, I am getting this message.

Please Help

[girish_nair1984]

I am not able to log into my Master Server now. I am using Red Hat Linux

[Marianne]

We have asked this about 3 or 4 times in the last couple of days:

HAVE YOU CHECKED DISK SPACE?

If you cannot login to the system, speak to your Linux support team. NBU nit coming up will not cause this. Sounds like there is something else wron on this server?

Have you tried remote login (using something like putty) ?
Can you ping the server from your desktop or other machines?