09-06-2012 03:50 AM
Dear Friends
I am using Netbackup 7.0.
While doing Robot inventory, I am getting an error : EMM Database Error (Status 196) and The Enterprise Media Manager Server is not responding (Status 25).
The following Daemons has stopped running.
bmrd,nbsvcmon,nbars, spad,nbrb,spoold,nbemm,nbaudit,nbkms.
Kindly advice, how to rectify these error.
Thank You
Solved! Go to Solution.
09-12-2012 01:29 PM
OK, this should not be taking this long to resolve.
Following on from Mariannes post, if the disk did fill up it is very likely that the transaction log has been toasted ... There is no way to tell, so we just make a basic check, and if this fails, we rebuild it.
Do exactly these steps.
Stop NBU
Reboot server
Is NBU now working ... if yes, then the issue is fixed.
If No, do this :
stop NBU and then run
/usr/openv/db/bin/nbdb_ping
You should see a message saying :
Database [NBDB] is alive and well on server [NB_womble]. (Your server name will be different)
If you see a message saying the database is unavailable then you will need to recreate the transaction log to see if that resolves the issue. To rebuild the transaction log, follow these steps exactly.
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/db/bin/nbdb_admin -auto_start NONE
. /usr/openv/db/vxdbms_env.sh
cd /usr/openv/db/data
mv NBDB.log NBDB.log.bad
/usr/openv/db/bin/dbeng11 -f NBDB
/usr/openv/db/bin/nbdb_admin -auto_start NBDB
( /usr/openv/db/bin/nbdb_admin -auto_start BMRDB - If you use BMR)
/usr/openv/db/bin/nbdbms_start_server
/usr/openv/db/bin/nbdb_ping
If it now shows the DB as alive and well, hopefully issue is fixed and you can restart NBU
Martin
09-06-2012 04:11 AM
While running this command tpautoconf –report_disc
I am getting an error : EMM interface initialization failed, status 77
09-06-2012 04:13 AM
Stop NBU (what's left of it)
Reboot Server
What NBU processes are running ?
Martin
09-06-2012 04:37 AM
15:24:02.823 [26398] <4> db_getSTUNITlist: emmserver_name = his1
15:24:02.823 [26398] <4> db_getSTUNITlist: emmserver_port = 1556
15:24:12.266 [26410] <2> bprdreq: INITIATING: version NetBackup 7.0 created: 2010070800
15:24:12.266 [26410] <2> logparams: /usr/openv/netbackup/bin/admincmd/bprdreq -M localhost -terminate
15:24:12.267 [26410] <2> bprdreq: VERBOSE = 0
15:24:12.269 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002
15:24:12.269 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+veritas_pbx,1,20,2,1,0+localhost.txt
15:24:12.270 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002
15:24:12.270 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+vnetd,1,20,2,1,0+localhost.txt
15:24:12.271 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002
15:24:12.271 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+bprd,1,20,2,1,0+localhost.txt
15:24:12.272 [26410] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:24:12.272 [26410] <2> logconnections: BPRD CONNECT FROM 127.0.0.1.19974 TO 127.0.0.1.1556 fd = 4
15:24:12.377 [26414] <2> nbpemreq: INITIATING: version NetBackup 7.0 created: 2010070800
15:24:12.381 [26414] <2> logparams: /usr/openv/netbackup/bin/admincmd/nbpemreq -suspend_scheduling
15:24:12.390 [26414] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:759)
15:24:13.306 [26410] <2> doreq: ...ok, received 0 from bprd
15:24:16.307 [26410] <2> bprdreq: EXIT status = 0
15:25:01.896 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT
15:25:01.897 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO
15:26:05.953 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT
15:26:05.953 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO
15:27:10.007 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT
15:27:10.007 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO
15:28:19.172 [27916] <2> nbpemreq: INITIATING: version NetBackup 7.0 created: 2010070800
15:28:19.172 [27916] <2> logparams: /usr/openv/netbackup/bin/admincmd/nbpemreq -resume_scheduling
15:28:19.180 [27916] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:759)
15:28:31.001 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::COMM_FAILURE
15:28:31.001 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/COMM_FAILURE:1.0'
TAO exception, minor code = 13 (failed to recv request response; ECONNRESET), completed = MAYBE
15:28:35.183 [28198] <4> db_getSTUNITlist: emmserver_name = his1
15:28:35.183 [28198] <4> db_getSTUNITlist: emmserver_port = 1556
15:29:12.327 [28263] <2> logparams: /usr/openv/netbackup/bin/admincmd/bpversion
15:29:12.331 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:29:12.331 [28263] <2> logconnections: BPCD CONNECT FROM 192.168.1.13.26281 TO 192.168.1.13.1556 fd = 3
15:29:12.332 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:29:12.333 [28263] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1716: 0: via PBX: VNETD CONNECT FROM 192.168.1.13.50004 TO 192.168.1.13.1556 fd = 4
15:29:12.333 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.540: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
15:29:12.373 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.557: 0: ipc_string: /tmp/vnet-28265346930952373257000000000-x3xOeR
15:29:12.413 [28263] <2> local_bpcr_connect: expected reserved port, got 1556
15:29:12.523 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:29:12.523 [28263] <2> logconnections: BPCD CONNECT FROM 192.168.1.13.43878 TO 192.168.1.13.1556 fd = 3
15:29:12.523 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:29:12.524 [28263] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1716: 0: via PBX: VNETD CONNECT FROM 192.168.1.13.22652 TO 192.168.1.13.1556 fd = 4
15:29:12.524 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.540: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
15:29:12.564 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.557: 0: ipc_string: /tmp/vnet-28269346930952564247000000000-pYOPo5
15:29:12.604 [28263] <2> local_bpcr_connect: expected reserved port, got 1556
15:29:34.259 [28198] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT
15:29:34.259 [28198] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO
15:29:35.064 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT
15:29:35.064 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO
15:29:40.064 [26398] <16> db_getSTUNITlist: (-) Translating EMM_ERROR_CorbaTransient(3000001) to 25 in the NetBackup context
15:29:40.064 [26398] <16> db_getSTUNITlist: error (25) initializing EMM interface.
15:29:40.064 [26398] <16> bpstsinfo/update ERROR: unable to retrieve stu list from server (25)
15:30:38.315 [28198] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT
15:30:38.315 [28198] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO
09-06-2012 04:41 AM
The following daemons are running
ltid
nbsl
bprd
vmd
bocompatd
nbstserv
nbsvsmon
NB_dbsrv
nbrmms
bpdbm
nbjm
nbpem
nbevtmgr
09-06-2012 04:45 AM
Corba Transient Error .... Sounds like you may be out of memory ... reboot at least!
Are you really on 7.0? I would advise you patch at least or preferably upgrade
Hope this helps
09-06-2012 04:56 AM
I meant reboot, then tell us what processes are runing ...
09-06-2012 04:57 AM
Yes I am using Netbackup 7.0.
But we have 2 Netback Servers with same partition size. 1 is working perfectly fine while another one is showing the above mentioned error.s
09-06-2012 05:12 AM
After using this command /usr/openv/netbackup/bin/goodies/ netbackup stop
and then bpps -a to verify the processes/daemons are running
NB Processes
------------
root 1066 1 0 12:13 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 1085 1066 0 12:13 ? 00:00:00 [NB_dbsrv] <defunct>
root 1217 1 0 12:13 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 1223 1217 0 12:13 ? 00:00:00 [NB_dbsrv] <defunct>
root 4714 1 0 12:28 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 4719 4714 0 12:28 ? 00:00:00 [NB_dbsrv] <defunct>
root 4966 1 0 12:28 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 4968 4966 0 12:28 ? 00:00:00 [NB_dbsrv] <defunct>
root 12441 1 0 Sep05 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 12443 12441 0 Sep05 ? 00:00:00 [NB_dbsrv] <defunct>
root 13484 1 0 13:15 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 28843 28802 0 15:34 pts/5 00:00:09 /usr/openv/java/jre/bin/java -Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -showversion -Dvrts.common.server.LCCTYPE="en_US" -Dvrts.common.server.LOG_CMDLINES=false -Djava.library.path=/usr/openv/lib -Dvrtsat.home=/usr/openv/netbackup/sec/at -Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/java/Debug.properties -Dvrts.nbe.HOSTNAME= -Dvrts.nbe.INST_REL=700000 -Dvrts.nbe.BUILDNO=20100707 -Djava.util.prefs.PreferencesFactory=vrts.common.launch.PropertiesPreferencesFactory -Xms36M -Xmx256M vrts.nbe.AdminConsole
root 28885 1 0 15:34 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile
root 28887 28885 0 15:34 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile
root 28893 28885 0 15:34 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile
root 28967 28885 0 15:35 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile
root 29471 1 0 15:40 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
09-06-2012 06:34 AM
Have you rebooted yet as we advised? Will clear those orphaned processes that may be causing the memory / Corba errors - if they persist after the reboot you may also need to delete the orphaned corba file - but let us know first before we cover that
09-08-2012 10:44 PM
@ Mark_Solutions.
Yes I have rebooted as per your advice, but still that error is there.
09-08-2012 11:53 PM
Have you checked disk space on the master server?
EMM will shut down when filesystem runs low on disk space to prevent corruption of database.
Is PBX running?
Use 'bpps -x' to list all processes.
After stopping and starting NBU, check EMM logs for the last 10 minutes (or enough minutes to capture stop and start):
vxlogview -o 111 -t 00:10:00
09-09-2012 03:42 AM
by any chance was the server renamed or added to a different domain?
09-10-2012 11:44 PM
Please find the logs after running bpps -x command
NB Processes
------------
root 1066 1 0 Sep06 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf
@/usr/openv/var/global/databases.conf -hn 7
root 1085 1066 0 Sep06 ? 00:00:00 [NB_dbsrv] <defunct>
root 1217 1 0 Sep06 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv /usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 1223 1217 0 Sep06 ? 00:00:00 [NB_dbsrv] <defunct>
root 4714 1 0 Sep06 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 4719 4714 0 Sep06 ? 00:00:00 [NB_dbsrv] <defunct>
root 4966 1 0 Sep06 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv
@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 4968 4966 0 Sep06 ? 00:00:00 [NB_dbsrv] <defunct>
root 12441 1 0 Sep05 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv
@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root 12443 12441 0 Sep05 ? 00:00:00 [NB_dbsrv] <defunct>
root 13484 1 0 Sep06 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 15203 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root 15206 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root 15269 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/nbevtmgr
root 15535 1 0 10:22 pts/3 00:00:00 /usr/openv/netbackup/bin/bprd
root 15540 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/bpcompatd
root 15551 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/nbjm
root 15572 1 0 10:22 pts/3 00:00:00 /usr/openv/netbackup/bin/bpdbm
root 15579 15572 0 10:22 pts/3 00:00:00 /usr/openv/netbackup/bin/bpjobd
root 15589 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/nbpem
root 15642 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/nbstserv
root 15690 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/nbrmms
root 15765 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/nbsl
root 16035 1 0 10:22 ? 00:00:00 /usr/openv/netbackup/bin/nbsvcmon
root 16263 16222 1 10:23 pts/3 00:00:05 /usr/openv/java/jre/bin/java -
Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -showversion -
Dvrts.common.server.LCCTYPE="en_US" -
Dvrts.common.server.LOG_CMDLINES=false -
Djava.library.path=/usr/openv/lib -
Dvrtsat.home=/usr/openv/netbackup/sec/at -
Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/java/Debug.properties -Dvrts.nbe.HOSTNAME= - Dvrts.nbe.INST_REL=700000 -Dvrts.nbe.BUILDNO=20100707 -Djava.util.prefs.PreferencesFactory=vrts.common.launch.PropertiesPreferencesFactory -Xms36M -Xmx256M vrts.nbe.AdminConsole
root 16314 1 0 10:23 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1
en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root 16316 16314 0 10:23 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1
en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root 16321 16314 0 10:23 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root 16374 16314 0 10:23 ? 00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root 16878 1 0 10:27 pts/3 00:00:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
root 17300 15589 0 10:30 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root 29471 1 0 Sep06 ? 00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
MM Processes
------------
root 15530 1 0 10:22 pts/3 00:00:00 /usr/openv/volmgr/bin/ltid
root 15694 1 0 10:22 pts/3 00:00:00 vmd
Shared Symantec Processes
-------------------------
root 3972 1 0 Mar20 ? 00:01:37 /opt/VRTSpbx/bin/pbx_exchange
Output of vxlogview -o 111 -t 00:10:00
[root@his1 ~]# /usr/openv/netbackup/bin/vxlogview -o 111 -t 00:10:00
09/11/2012 10:22:30.691 [EMMServer::EMMServer]
09/11/2012 10:22:30.691 [DSMFSMORBConfig::DSMFSMORBConfig] Setting DSM ORB thread count to <10>
09/11/2012 10:22:30.691 [REMORBConfig::REMORBConfig] Setting REM ORB thread count to <10>
09/11/2012 10:22:30.691 [FATClientORBConfig::FATClientORBConfig] Setting FTClient ORB thread count to <10>
09/11/2012 10:22:30.774 [EMMMain] <ENTER>
09/11/2012 10:22:30.776 [EMMServer::isLocalHostEMMServer]
09/11/2012 10:22:30.778 [EMMServer::isLocalHostEMMServer] LocalHost = <his1>, EmmServer = <his1>
09/11/2012 10:22:30.778 [NBEmmSvc::doShouldStart] Should start
09/11/2012 10:22:30.924 [NBEmmSvc::doInit] <ENTER>
09/11/2012 10:22:30.926 [NBEmmSvc::doInit] NumberOrbThreads = <10>, NumberDbConnections = <4>, NumberDbBrowseConnections = <3>
09/11/2012 10:22:30.927 [EMMServer::SetOrb] Orb is set
09/11/2012 10:22:30.927 [NBEmmSvc::doInit] Extra orb options: -ORBCollocation no
09/11/2012 10:22:30.927 [EMMServer::init] Server initializing VxEMM
09/11/2012 10:22:30.927 [Info] V-111-1000 [EMMServer::init] EMMServer initializing = VxEMM
09/11/2012 10:22:30.927 [EMMServer::init] Option: -DbServer *** Unrecognized option -DbServer ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbServer *** Unrecognized option -DbServer ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: NB_his1 *** Unrecognized option NB_his1 ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: NB_his1 *** Unrecognized option NB_his1 ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: -DbPort *** Unrecognized option -DbPort ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbPort *** Unrecognized option -DbPort ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: 13785 *** Unrecognized option 13785 ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: 13785 *** Unrecognized option 13785 ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: -DbName *** Unrecognized option -DbName ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbName *** Unrecognized option -DbName ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: NBDB *** Unrecognized option NBDB ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: NBDB *** Unrecognized option NBDB ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: -ConnectString *** Unrecognized option -ConnectString ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -ConnectString *** Unrecognized option -ConnectString ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=******;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB *** Unrecognized option PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=nbusql;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=******;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB *** Unrecognized option PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=nbusql;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB ***
09/11/2012 10:22:30.928 [EMMServer::run] Server running
09/11/2012 10:22:30.928 [Info] V-111-1002 [EMMServer::run] EMMServer running
09/11/2012 10:22:49.923 V-111-1051 [EMMServer::run] Can not establish initial DB connection
09/11/2012 10:22:49.923 [Error] V-111-1049 EMMServer generic error = Can not establish initial DB connection
09/11/2012 10:22:49.923 [Error] V-111-1049 EMMServer generic error = controlledRun() failed -> doShouldStart(), doInitialize() or doRun() failed
09/11/2012 10:22:49.923 [FATClientORBConfig::shutServant] Calling fini() on FAT Client
09/11/2012 10:22:49.923 [FATClientORBConfig::shutServant] Done fini() on FAT Client
09/11/2012 10:22:49.923 [REMORBConfig::shutServant] Calling fini() on REM
09/11/2012 10:22:49.923 [REMORBConfig::shutServant] Done fini() on REM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Calling fini() on FSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Done fini() on FSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Calling fini() on DSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Done fini() on DSM
09/11/2012 10:22:49.923 [EMMServer::fini] EMM Server shutting down
09/11/2012 10:22:49.923 [Info] V-111-1003 [EMMServer::fini] EMMServer shutting down
09/11/2012 10:22:49.995 [Error] V-111-1049 EMMServer generic error = Exception caught for IORTable clean up <NotFound>
09/11/2012 10:22:49.995 [EMMServer::fini] EMM Server shut down complete
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.008 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.009 [EMMMain] <EXIT>
09/11/2012 10:22:50.009 [Log::deinit]
09/11/2012 10:22:50.009 [EMMServer::~EMMServer]
09-11-2012 04:56 AM
It sounds like there is something wrong with SQL Anywhere daemon or your server.conf file - see this tech note which describes it :
http://www.symantec.com/docs/HOWTO33625
As the EMM database starts to come up it doesn't seem to understand anything in that file so it is either that or it may be SQL Anywhere itself.
It is the same sirt of error you can get after an upgrade - take a look through this thread:
https://www-secure.symantec.com/connect/forums/nbu-update-7101-failed-initialize-emm-connection-195
To summarise ... the above was a schema mismatch after a patch .. i dont see that in your EMM log but maybe it doesn't go back far enough?
So first run:
./nbdbms_start_server located in /usr/openv/db/bin.
You may be able to see in clear text why EMM doesn't start as intended.
Have you patched or anything prior to this error? Has anything changed? Have you plenty of disk space?
09-12-2012 12:45 PM
Seems you did NOT reboot as requested...
pbx_exchange has been up and running since 20 March. Other processes since 5 and 6 Sept....
You have not told us if you have checked/verified disk space on the master?
The errors in emm log looks like possible corruption of one of more config files. Please check size and timestamp of all of the following files - none of them should be 0 bytes:
/usr/openv/var/global/server.conf
/usr/openv/var/global/databases.conf
/usr/openv/db/data/vxdbms.conf
09-12-2012 01:29 PM
OK, this should not be taking this long to resolve.
Following on from Mariannes post, if the disk did fill up it is very likely that the transaction log has been toasted ... There is no way to tell, so we just make a basic check, and if this fails, we rebuild it.
Do exactly these steps.
Stop NBU
Reboot server
Is NBU now working ... if yes, then the issue is fixed.
If No, do this :
stop NBU and then run
/usr/openv/db/bin/nbdb_ping
You should see a message saying :
Database [NBDB] is alive and well on server [NB_womble]. (Your server name will be different)
If you see a message saying the database is unavailable then you will need to recreate the transaction log to see if that resolves the issue. To rebuild the transaction log, follow these steps exactly.
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/db/bin/nbdb_admin -auto_start NONE
. /usr/openv/db/vxdbms_env.sh
cd /usr/openv/db/data
mv NBDB.log NBDB.log.bad
/usr/openv/db/bin/dbeng11 -f NBDB
/usr/openv/db/bin/nbdb_admin -auto_start NBDB
( /usr/openv/db/bin/nbdb_admin -auto_start BMRDB - If you use BMR)
/usr/openv/db/bin/nbdbms_start_server
/usr/openv/db/bin/nbdb_ping
If it now shows the DB as alive and well, hopefully issue is fixed and you can restart NBU
Martin
09-13-2012 04:05 AM
Dear friends
I have stop NBU and restarted the server but.
At this point during startup it is stuck showing message
Netbackup SAN Client fibre transport daemon started
It's been 45 mins till now, I am getting this message.
Please Help
09-13-2012 04:13 AM
I am not able to log into my Master Server now. I am using Red Hat Linux
09-13-2012 04:24 AM
We have asked this about 3 or 4 times in the last couple of days:
HAVE YOU CHECKED DISK SPACE?
If you cannot login to the system, speak to your Linux support team. NBU nit coming up will not cause this. Sounds like there is something else wron on this server?
Have you tried remote login (using something like putty) ?
Can you ping the server from your desktop or other machines?