Highlighted

Unable to start vmd on master server

Hello, I have recently upgrade NBU to 8.1.1 in VCS environment. After a couple of days, we realized that vmd was not running on master server (all devices are configured in media servers). When we try to start vmd starting ltid we receive the following error: "Failed to validate this host with the EMM server". Everything is working fine, except that we are not able to run inverntory robots or work with storage devices from the console (for inventory we just run vmupdate from the media server).

When vmd is configured in debug mode, we see the following in the log:

10:47:43.550 [25100532] <4> vmd: INITIATING
10:47:43.550 [25100532] <2> mm_getnodename: cached_hostname shortnamehostname.com.uy, cached_method 3
10:47:43.565 [25100532] <2> mm_getnodename: (6) hostname shortnamehostname.com.uy (from cached_hostname)
10:47:43.565 [25100532] <4> vmd: Host name is shortnamehostname.com.uy
10:47:45.590 [25100532] <4> vmd: emmlib_initialize bkserverag, 1556, <status 0>
10:47:45.590 [25100532] <2> mm_getnodename: (0) hostname shortnamehostname.com.uy (from cached_hostname)
10:47:45.618 [25100532] <16> emmlib_UpdateHostEx: (0) UpdateMachine failed, emmError = 2001071, nbError = 0
10:47:45.618 [25100532] <16> AddAndVerifyHost: Failed to update this host. EMM error code = 2001071
10:47:45.618 [25100532] <16> AddAndVerifyHost: (-) Translating EMM_ERROR_MachineAlreadyExistAsMismatchedType(2001071) to 136 in the Media context
10:47:45.618 [25100532] <16> vmd: Failed updating host information in EMM
10:47:45.619 [25100532] <16> vmd: terminating - the operation requested has failed (163)
10:47:45.619 [25100532] <16> uninitialize: (-) Invalid Connection ID.
10:47:45.619 [25100532] <16> emmlib_uninitialize: (0) Failed to release EMM session and database objects
10:47:45.619 [25100532] <16> vmd: terminating - daemon terminated (7)

 

The weird thing is that in vmd log the hostname is shown as fqn name however the hostname is listed using the shortname (bpclntcm -hn hostname)

 

Any help or comment will be very welcome

 

Thanks and best regards

Tags (2)
3 Replies

Re: Unable to start vmd on master server

Please show us output of these commands on the active master server node:

bpclntcmd -self

nbemmcmd -listhosts -verbose

Contents of bp.conf

Extremely important that we distinguish between physical and virtual hostnames in a clustered environment, so, if you need to replace actual hostnames, please use names where we can clearly see difference between nodename and virtual hostname. 
e.g:
node1  
node1.fqdn
virtmast
virtmast.fqdn
etc.

PS:
Are you aware that NetBackup 8.1.2 and later does not support Master or Media Server on AIX?

Re: Unable to start vmd on master server

Marianne,

 

Thanks for your reply, here the outputs:

 

root@bkmasterag) /> bpclntcmd -self
yp_get_default_domain failed: (6) internal yp server or client error
NIS does not seem to be running: (1) args to yp function are bad
gethostname() returned: bkserverag
host bkserverag: bkserverag at 172.26.151.125
aliases:     bkserverag     172.26.151.125
getfqdn(bkserverag) returned: bkserverag.corp.acme.com.uy
you have mail in /usr/spool/mail/root

 

(root@bkmasterag) /> nbemmcmd -listhosts -verbose
NBEMMCMD, Version: 8.1.1
The following hosts were found:
bkserverag
        MachineName = "bkserverag"
        FQName = "bkserverag.corp.acme.com.uy"
        MachineDescription = ""
        MachineNbuType = server (6)
bkserverag
        ClusterName = ""
        MachineName = "bkserverag"
        FQName = "bkserverag.corp.acme.com.uy"
        GlobalDriveSeed = "VEND:#.Smiley TongueROD:#.:IDX"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = master (3)
        MachineState = active for disk jobs (12)
        NetBackupVersion = 8.1.1.0 (811000)
        OperatingSystem = rs6000 (5)
        ScanAbility = 5
bkserverpar
        ClusterName = ""
        MachineName = "bkserverpar"
        FQName = "bkserverpar.corp.acme.com.uy"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x15
        MachineNbuType = media (1)
        MachineState = active for tape and disk jobs (14)
        MasterServerName = "bkserverag"
        NetBackupVersion = 8.1.1.0 (811000)
        OperatingSystem = rs6000 (5)
        ScanAbility = 5
smediaag
        ClusterName = ""
        MachineName = "smediaag"
        FQName = "smediaag.corp.acme.com.uy"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = media (1)
        MachineState = active for disk jobs (12)
        MasterServerName = "bkserverag"
        NetBackupVersion = 8.1.1.0 (811000)
        OperatingSystem = windows (11)
        ScanAbility = 5
bkserverpar
        MachineName = "bkserverpar"
        FQName = "bkserverpar.corp.acme.com.uy"
        MachineDescription = "PureDisk"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
svnxparndmp
        MachineName = "svnxparndmp"
        FQName = "svnxparndmp.corp.acme.com.uy"
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = ndmp (2)
bkmediaag.corp.acme.com.uy
        ClusterName = ""
        MachineName = "bkmediaag.corp.acme.com.uy"
        FQName = "bkmediaag.corp.acme.com.uy"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = media (1)
        MachineState = active for tape and disk jobs (14)
        MasterServerName = "bkserverag"
        NetBackupVersion = 8.1.1.0 (811000)
        OperatingSystem = rs6000 (5)
        ScanAbility = 5
smediapar
        ClusterName = ""
        MachineName = "smediapar"
        FQName = "smediapar.corp.acme.com.uy"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = media (1)
        MachineState = active for disk jobs (12)
        MasterServerName = "bkserverag"
        NetBackupVersion = 8.1.1.0 (811000)
        OperatingSystem = windows (11)
        ScanAbility = 5
COFC-CZJ6340BBN01
        MachineName = "COFC-CZJ6340BBN01"
        FQName = "COFC-CZJ6340BBN01"
        MachineDescription = "hp-StoreOnceCatalyst"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
COFC-CZJ64301XG01
        MachineName = "COFC-CZJ64301XG01"
        FQName = "COFC-CZJ64301XG01"
        MachineDescription = "hp-StoreOnceCatalyst"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
COFC-hpAgraciada
        MachineName = "COFC-hpAgraciada"
        FQName = "COFC-hpAgraciada"
        MachineDescription = "hp-StoreOnceCatalyst"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
COFC-hpParaguay
        MachineName = "COFC-hpParaguay"
        FQName = "COFC-hpParaguay"
        MachineDescription = "hp-StoreOnceCatalyst"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
svwvcenter5.corp.acme.com.uy
        MachineName = "svwvcenter5.corp.acme.com.uy"
        FQName = "svwvcenter5.corp.acme.com.uy"
        MachineDescription = ""
        MachineNbuType = virtual_machine (10)
bkmediaag.corp.ue.com.uy
        MachineName = "bkmediaag.corp.ue.com.uy"
        FQName = "bkmediaag.corp.ue.com.uy"
        MachineDescription = ""
        MachineNbuType = foreign_media (13)
svwveritasmsdp
        ClusterName = ""
        MachineName = "svwveritasmsdp"
        FQName = "svwveritasmsdp.corp.acme.com.uy"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = media (1)
        MachineState = active for disk jobs (12)
        MasterServerName = "bkserverag"
        NetBackupVersion = 8.1.1.0 (811000)
        OperatingSystem = windows (11)
        ScanAbility = 5
lvwnetintapp
        MachineName = "lvwnetintapp"
        FQName = "lvwnetintapp.corp.acme.com.uy"
        MachineDescription = ""
        MachineNbuType = remote_master (11)
svwveritasevault
        MachineName = "svwveritasevault"
        FQName = "svwveritasevault.corp.acme.com.uy"
        MachineDescription = "PureDisk"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
svwveritasmsdp
        MachineName = "svwveritasmsdp"
        FQName = "svwveritasmsdp.corp.acme.com.uy"
        MachineDescription = "PureDisk"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
COFC-hpAgraciada2
        MachineName = "COFC-hpAgraciada2"
        FQName = "COFC-hpAgraciada2"
        MachineDescription = "hp-StoreOnceCatalyst"
        MachineFlags = 0x2
        MachineNbuType = ndmp (2) (storage_server)
smediaag2.corp.acme.com.uy
        ClusterName = ""
        MachineName = "smediaag2.corp.acme.com.uy"
        FQName = "smediaag2.corp.acme.com.uy"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = media (1)
        MachineState = active for disk jobs (12)
        MasterServerName = "bkserverag"
        NetBackupVersion = 8.1.1.0 (811000)
        OperatingSystem = windows (11)
        ScanAbility = 5
Command completed successfully.

----------------------------------------------------------------------------------------------------

 

bp.conf:

SERVER = bkserverag
SERVER = bkmasterag
SERVER = bkmasterag.corp.acme.com.uy
SERVER = bkmasterpar
SERVER = bkmasterpar.corp.acme.com.uy
SERVER = bkserverpar
SERVER = bkserverpar.corp.acme.com.uy
SERVER = bkmediaag.corp.acme.com.uy
SERVER = pc149806
SERVER = pc149801
SERVER = pc154046
SERVER = pc154036
SERVER = pc149819
SERVER = pvwsopdis
SERVER = svwveritasops
SERVER = pc148636
SERVER = pc147171
SERVER = pc147076
SERVER = svnxparndmp
SERVER = svwtareasprog
SERVER = smediaag
SERVER = smediapar
SERVER = svwdevapp
SERVER = svwveritasmsdp
SERVER = svwnetctrlmprd
SERVER = pvwsyasanadm
SERVER = smediaag2.corp.acme.com.uy
SERVER = smediaag2
CLIENT_NAME = bkmasterag
CLUSTER_NAME = bkserverag
ALLOW_MEDIA_OVERWRITE = DBR
ALLOW_MEDIA_OVERWRITE = TAR
ALLOW_MEDIA_OVERWRITE = CPIO
ALLOW_MEDIA_OVERWRITE = ANSI
ALLOW_MEDIA_OVERWRITE = AOS/VS
ALLOW_MEDIA_OVERWRITE = MTF1
ALLOW_MEDIA_OVERWRITE = RS-MTF1
ALLOW_MEDIA_OVERWRITE = BE-MTF1
#VERBOSE = 5
SERVER_SENDS_MAIL = YES
CLIENT_CONNECT_TIMEOUT = 1800
CLIENT_READ_TIMEOUT = 10000
BPSTART_TIMEOUT = 9000
SLAVE_CONNECT_TIMEOUT = 3600
ALLOW_NON_RESERVED_PORTS = YES
RANDOM_PORTS = NO
CLIENT_RESERVED_PORT_WINDOW = 900 999
SERVER_RESERVED_PORT_WINDOW = 800 899
BPEND_TIMEOUT = 3600
KEEP_JOBS_HOURS = 720
KEEP_JOBS_SUCCESSFUL_HOURS = 720
# WAIT_IN_QUEUE = YES
# QUEUE_ON_ERROR = YES
INCOMPLETE_JOB_CLEAN_INTERVAL = 1
# BPTM_QUERY_TIMEOUT = 960
EMMSERVER = bkserverag
VXDBMS_NB_DATA = /usr/openv/db/data
MEDIA_UNMOUNT_DELAY = 180
INCOMPLETE_BKUP_JOB_CLEAN_INTERVAL = 1
BPDM_VERBOSE = 0
MEDIA_SERVER = bkserverpar
MEDIA_SERVER = bkmediaag.corp.acme.com.uy
MEDIA_SERVER = smediaag
MEDIA_SERVER = smediapar
MEDIA_SERVER = svwveritasmsdp
BPRD_VERBOSE = 0
BPSCHED_VERBOSE = 5
# NBPEM_VERBOSE = 5
# NBJM_VERBOSE = 5
# NBRB_VERBOSE = 5
BPTM_VERBOSE = 0
BPBRM_VERBOSE = 0
CLIENT_PORT_WINDOW = 0 0
USE_VXSS = PROHIBITED
SPS_REDIRECT_ALLOWED = dag2 smailmrpar2
SPS_REDIRECT_ALLOWED = dag2 smailmrag2
SPS_REDIRECT_ALLOWED = dag1 smailmrag1
SPS_REDIRECT_ALLOWED = dag1 smailmrpar1
SPS_REDIRECT_ALLOWED = dag2 SMAILMRPAR1
SPS_REDIRECT_ALLOWED = DAG svwmail1
SPS_REDIRECT_ALLOWED = DAG svwmail2
SPS_REDIRECT_ALLOWED = DAG svwmail3
SPS_REDIRECT_ALLOWED = DAG svwmail4
SPS_REDIRECT_ALLOWED = DAG svwmail5
SPS_REDIRECT_ALLOWED = DAG svwmail6
JOB_PRIORITY = 0 0 90000 90000 90000 90000 85000 85000 80000 80000 80000 80000 75000 75000 70000 70000 50000 50000 45000 0 0 0 0 0
AUTHENTICATION_DOMAIN = bkserverag "ADDED AUTOMATICALLY" PASSWD bkserverag 0
AUTHORIZATION_SERVICE = bkserverag 0
HOST_CACHE_TTL = 3600
FORCE_RESTORE_MEDIA_SERVER = bkserverag bkmediaag.corp.acme.com.uy
TRUSTED_MASTER = lvwnetintapp.corp.acme.com.uy
TRUSTED_MASTER = lvwnetintapp
BPCD_WHITELIST_PATH = /openv/scripts/scripts_ctrlm/LOG/bpdup_ag.ls
BPCD_WHITELIST_PATH = /openv/scripts/scripts_ctrlm/LOG/bpdup_par.ls
VM_PROXY_SERVER = smediaag
VM_PROXY_SERVER = smediapar
#ENABLE_NBCURL_VERBOSE = 1
VERBOSE = 0
TELEMETRY_UPLOAD = YES
WEBSVC_GROUP = nbwebgrp
WEBSVC_USER = nbwebsvc
VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY

-----------------------------------------------------------------------------------

 

Yes, we are aware of the AIX supported version, we will migrate to other platfom the master server

Thanks and best regards

 

Osvaldo

 

Re: Unable to start vmd on master server

your emm server is showing as EMMSERVER = bkserverag

Could you please check your /etc/host file and also do nslookup for the same host name.