EMM interface initialization failed, status = 92

[NBU35]

Hi

we have master server

cat /usr/openv/netbackup/bin/version
NetBackup-Solaris9 6.5.6
 

uname -a
SunOS asprod11 5.9 Generic_122300-30 sun4u sparc SUNW,Sun-Fire-V890

and media server

 /usr/openv/netbackup/bin/version
NetBackup-Solaris10 6.5.6
 

uname -a
SunOS astst09 5.10 Generic_144488-09 sun4v sparc SUNW,SPARC-Enterprise-T5220

backups are failing and connectivity is fine .

when we run vmoprcmd on master server we get following errors.

 vmoprcmd -d -h server.nbu.com
EMM database error (196)

on media server

EMM interface initialization failed, status = 92
 

and tldd deamon is not running on media server now after bouncing the netbackup and PBX service , earlier it was running

please suggest

[NBU35]

also getiing following now on media server

EMM interface initialization failed, status = 333
 on running vmoprcmd , tpautoconf -t , tpconfig -d

[Mark_Solutions]

is this a new install or has this suddenly happened?

If suddenly then try rebooting the Master for a clean startup.

It tends to indicate a loop back error so also check the hosts file, it should have its own loop back addres:

127.0.0.1 localhost

[NBU35]

It has come suddenly and

127.0.0.1 localhost

entry is already there , is there any other solution to it except reboot ?

[NBU35]

11:39:12.186 [529] <16> ConnectToServerInterface: CORBA::EXCEPTION: Cannot get object reference ptr JOBINST for interface.exception = NO_PERMISSION
11:39:12.186 [529] <16> initializeJmComm: RequestInitialResources : ConnectToServerInterface on server [asprod11.aldc.att.com] fails [25]
11:39:12.186 [529] <2> RequestInitialResources: returning
11:39:12.186 [529] <4> nbjm_media_request: Error from RequestMultipleResources, Master asprod11.aldc.att.com, error 25, resourceAllocated 0
11:39:12.186 [529] <2> main: Sending [EXIT STATUS 25] to NBJM
11:39:12.186 [529] <2> bptm: EXITING with status 25 <----------
11:41:14.416 [1220] <2> bptm: INITIATING (VERBOSE = 1): -w -c astst09.aldc.att.com -den 6 -rt 8 -rn 0 -stunit astst09_lto -cl astst09 -bt 1318606871 -b astst09.aldc.att.com_1318606871 -st 0 -cj 3 -p TestDev -reqid -1317919740 -jm -brm -hostname astst09.aldc.att.com -ru root -rclnt astst09.aldc.att.com -rclnthostname astst09.aldc.att.com -rl 4 -rp 5356800 -sl Full -ct 0 -maxfrag 2048 -v -mediasvr astst09.aldc.att.com -no_callback -connect_options 0x01010100 -jobid 15682 -jobgrpid 15682 -masterversion 650000 -bpbrm_shm_id 184549668 -blks_per_buffer 512 -shm
11:41:14.420 [1220] <4> bptm: emmserver_name = asprod11.aldc.att.com
11:41:14.420 [1220] <4> bptm: emmserver_port = 1556
11:41:14.464 [1220] <2> Orb::init: initializing ORB EMMlib_Orb with: dbstunitq -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb EMMlib_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:739)
11:42:13.760 [1220] <16> emmlib_initializeEx: (-) Exception! CORBA::NO_PERMISSION
11:42:13.760 [1220] <16> bptm: (-) Translating EMM_ERROR_Authorization_Denied(3000007) to 118 in the NetBackup context
11:42:13.761 [1220] <16> bptm: error (118) initializing EMM interface.
11:42:13.761 [1220] <16> main: (-) Translating ERROR UNDEFINED(118) to 196 in the Media context
11:42:13.761 [1220] <2> bptm: error(196) initializing EMM interface.

11:42:13.761 [1220] <2> bptm: EXITING with status 47 <----------
11:42:15.225 [1287] <2> bptm: INITIATING (VERBOSE = 1): -unload -dn A000_F16_S7_D187 -dp /dev/rmt/0cbn -dk 2000539 -m GC2028 -mk 4002111 -mds 8 -alocid 7504221 -jobid -1317919741 -jm
 

also

nbemmcmd -getemmserver
NBEMMCMD, Version:6.5.6
Failed to initialize EMM connection.  Verify that network access to the EMM server is available and that the services nbemm and pbx_exchange are running on the EMM server. (195)
Command did not complete successfully.

 bpclntcmd -pn
expecting response from server master.nbu.com
astst09-bkup *NULL* 130.8.70.38 52686
 

please suggest

[Mark_Solutions]

Either reboot or work through this - it is a useful thread:

https://www-secure.symantec.com/connect/articles/troubleshooting-emm-error-196

[Marianne]

If master server hostname is "asprod11" (output from uname -a), and media server hostname is "astst09" , then where does master.nbu.com and server.nbu.com fit into the picture?

Please post bp.conf entries on master and media servers as well as output of 'nbemmcmd -listhosts -verbose' on the master.

You seem to have multiple NICs in the servers? bpclntcmd -pn shows 'astst09-bkup'.

Please double-check /etc/hosts entries as well as /etc/hostname.* entries plus 'ifconfig -a' output on master and media server to see if anything has changed.

[Karthikeyan_Sun]

Are the entries on EMM.Conf is correct ?

Correct EMM.conf entries

NUM_DB_CONNECTIONS=<value>

NUM_DB_BROWSE_CONNECTIONS=<value>      --->correct

NUM_ORB_THREADS=<value>

[Mark_Solutions]

This all looks a little confusing - it may be a FQDN issue or, reading betweem the lines, it may just be that the Media Server is not listed as a Server and Media Server in the Master Servers host properties - or correctly registered with EMM.

Check your hosts files and try and get everything neat and tidy so that both Master and Media are know to themselves and each other by their FDQN and short name, that the Media Server is properly registered with EMM (probably wont be able to do this until the Master is sorted out) and then restart the daemons on both.

This may have been caused by something as simple as a DNS change?

Hope this helps

[NBU35]

sorry for the mess up made by me above.. required details are below

we found the folloing in the BPbkar logs . Please  mention what is location of emm.conf file.

18:20:58.996 [202] <16> bpbkar setup_shm: ERR - Could not open file /usr/openv/netbackup/db/config/shm/astst09.aldc.att.com_1318976158 to get shared memory information. Errno = 2: No such file or directory
18:20:58.996 [202] <16> bpbkar sighandler: ERR - bpbkar killed by SIGPIPE
18:20:58.996 [202] <2> bpbkar sighandler: INF - ignoring additional SIGPIPE signals
18:20:58.996 [202] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 40: network connection broken
18:20:58.997 [202] <4> bpbkar Exit: INF - EXIT STATUS 40: network connection broken
18:20:58.997 [202] <4> bpbkar Exit: INF - setenv FINISHED=0
 

[NBU35]

backup is some time failing with EC 47 after writing some data . some times fails with EC 37 with out writing any thing.

[Mark_Solutions]

Hi

Still a little confused but that error may relate to one that was fixed in 6.5.6:

http://www.symantec.com/docs/TECH66236

Looking at your outputs most of your listed servers are at 6.5.6 but these are below that level:

gaalpa1msgusr7a.ugd.att.com
gaalpa1msgusr7c.ugd.att.com
gaalpa1msgusr7e.ugd.att.com
gaalpa1egisxaa1.itservices.sbc.com
gaalpa1egifil01.itservices.sbc.com
asp9004a.aldc.att.com
asprd172.aldc.att.com
aspc0017.aldc.att.com
aspc0018.aldc.att.com
aspc0019.aldc.att.com
asprd469-ebr.aldc.att.com
asprd470-ebr.aldc.att.com
p1xan1d1-ebr.vci.att.com

Your original Corba Error may relate to this one:

http://www.symantec.com/docs/TECH150396

You can increase the Corba Logging using this tech note:

http://www.symantec.com/docs/TECH54432

Lots to go at here but it does look like you may have more than just one issue

Hope this helps

[Yasuhisa_Ishika]

You said "backup is some time failing with EC 47 after writing some data". This sounds like EMM issue have gone, and EMM can respond to daemons like nbpem, nbrb and so on. Is this right? Have you already checked whether disk full condition was occured? When disk usage reaches 98% or so, EMM become stopped.

[NBU35]

found following in BPTM logs

07:36:30.972 [10642] <16> emmlib_initializeEx: (-) Exception! CORBA::NO_PERMISSION
07:36:30.972 [10642] <16> bptm: (-) Translating EMM_ERROR_Authorization_Denied(3000007) to 118 in the NetBackup context
07:36:30.973 [10642] <16> bptm: error (118) initializing EMM interface.
07:36:30.973 [10642] <16> main: (-) Translating ERROR UNDEFINED(118) to 196 in the Media context
07:36:30.973 [10642] <2> bptm: error(196) initializing EMM interface.
 

[Mark_Solutions]

Again you have Corba errors in this - although 196 just means it has missed its backup window, presumably as it had failed previosuly and this was a re-try?

Check out the links on my earlier post