Dead media servers coming back like Freddie Krueger

[SteveLaw]

Hi,

We had an environment in pre-production where the servers had a -w2k8 prefix added to their names and were tested in Netbackup before the old environment was turned off, the prefix removed and the environment going live. But I can't get rid of these servers - all the stuff I normally do won't get rid of them, they disappear but then pop back up again. 
I normally do this: 
Delete all policies referencing the servers
Delete SLPs & Storage Units
Ensure all disk/tape images are either migrated to another media server or have expired
Run nbemmcmd -deletehost -machinename mediaserver -machinetype 

Usually works but not for these ones. One of them is dwb004-w2k8, and I just ran nbdecommission -list_ref -oldserver dwb004-w2k8 to identify anything left in the Netbackup config, but got nothing useful back:

E:\Program Files\Veritas\NetBackup\bin>nbdecommission -list_ref -oldserver dwb004-w2k8

Server dwb004-w2k8 is a media server

Server dwb004-w2k8 is a member of the following media sharing groups:
UNRESTRICTED_SHARING_GROUP

Media referencing media server dwb004-w2k8

Media generation rules on host dwb004-w2k8:
cannot connect to vmd (70)

Drive paths found on media server dwb004-w2k8

Robots attached to dwb004-w2k8

Processing tape storage units

Processing basic disk storage units

Processing SnapVault/NearStore disk storage units

Processing advanced disks
dwb004-w2k8 is not a AdvancedDisk storage server

Processing shared disks
dwb004-w2k8 is not a SharedDisk storage server

Processing OST devices
dwb004-w2k8 does not reference any Open Storage server

Processing storage life cycles

Process backup policies

Processing Fibre Transports
dwb004-w2k8 is not a Fibre Transport server

Processing bp.conf file/Windows registry on the NetBackup media servers
Processing config file on bc20n0v0
Processing config file on swnb02.corpadds.net
Processing config file on swnb01.corpadds.net
Processing config file on dwb004.dcsit.net
Processing config file on deploy02.corpadds.net
Processing config file on deploy01.corpadds.net
Processing config file on tw05vm02
Processing config file on uwdb01.corpadds.net
Processing config file on swdb01.corpadds.net
Processing config file on tw03n3v0.dcsit.net
Processing config file on tw03vm01.dcsit.net
Processing config file on tw03n0v0
Processing config file on tw03vm02.dcsit.net
Processing config file on bc23n0v0.dcsit.net
Processing config file on bc13n0v0.dcsit.net
Processing config file on bc22n0v0.dcsit.net
Processing config file on tw03n2v1.dcsit.net
Processing config file on tw05n0m0.dcsit.net
Processing config file on tw03n1v0
Processing config file on bc11n2v1
Processing config file on tw03n1v1
Processing config file on bc11n1v1.dcsit.net
Processing config file on bc11n1v0.dcsit.net
Processing config file on bc11n2v0
Processing config file on bc11n0v1
Processing config file on bc11n0v0
Processing config file on bc02n0v0.dcsit.net
Processing config file on tw03n2v0.dcsit.net
Processing config file on tw03n0v1.dcsit.net
Processing config file on swa028
Processing config file on uwb008
Processing config file on uwb002
Processing config file on tw06n0v0.dcsit.net
Processing config file on swa018
Processing config file on ukdcsas01
Processing config file on swc004
Processing config file on swc003
Processing config file on swa025
Processing config file on swa029
Processing config file on swa027
Processing config file on uwa006
Processing config file on dlb001.dcsit.net
Processing config file on dlb030.dcsit.net
Processing config file on ilb030.dcsit.net
Processing config file on ulb025.dcsit.net
Processing config file on slb025.dcsit.net
Processing config file on swnb03
Processing config file on ulb030.dcsit.net
Processing config file on slb030.dcsit.net
Processing config file on ula030.dcsit.net
Processing config file on sla030.dcsit.net
Processing config file on slm040.dcsit.net
Processing config file on dlb040.dcsit.net
Processing config file on dlb041.dcsit.net
Processing config file on ulb040.dcsit.net
Processing config file on ulb041.dcsit.net
Processing config file on ula040.dcsit.net
Processing config file on dla040.dcsit.net
Processing config file on ulb051.dcsit.net
Processing config file on ula050.dcsit.net
Processing config file on ulb050.dcsit.net
Processing config file on s869bvdms001.emea.corpds.net
Processing config file on u869bvdms001.emea.corpds.net
Processing config file on ilb040.dcsit.net
Processing config file on ilb041.dcsit.net
Processing config file on ila040.dcsit.net
Processing config file on ula031.dcsit.net
Processing config file on ulb031.dcsit.net
Processing config file on slb031.dcsit.net
Processing config file on sla031.dcsit.net
Processing config file on sla040.dcsit.net
Processing config file on slb040.dcsit.net
Processing config file on slb041.dcsit.net
Processing config file on slb042.dcsit.net
Processing config file on D869DVABS001
Processing config file on U869DVABS001
Processing config file on slb050.dcsit.net
Processing config file on slb051.dcsit.net
Processing config file on sla050.dcsit.net
Processing config file on s869dvabs001.emea.corpds.net
Processing config file on dlb002.dcsit.net
Processing config file on uwb002-w2k8
unable to read host config for server uwb002-w2k8
Processing config file on swc004-w2k8
unable to read host config for server swc004-w2k8

Processing bp.conf file/Windows registry on the NetBackup master server
Processing config file on swm009.corpadds.net

E:\Program Files\Veritas\NetBackup\bin>

What have I got to do to slay these monsters?

Thanks :)  

[SteveLaw]

There are still references to these prefixed servers within the registry of the servers that were renamed:

[HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\ServiceManager\ServiceDef\MDMServer]
"InstallPath"="'C:\\Program Files\\Veritas\\NetBackup\\bin\\MDMServer'"
"EntryPoint"="MDMServer"
"Options"="-DbType ASA -DbServer VERITAS_NB_SWC004-W2K8 -DbName NBDB -DbHost SWC004-W2K8 -DbPort 13785"
"noOptions"="-DbType ASA -DbServer VERITAS_NB_SWC004-W2K8 -DbInst VERITAS -DbName NBDB"

[HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\VxDBMS\NB]
"SERVER_NAME"="NB_SWC004-W2K8"
"SERVICE_NAME"="SQLANYs_VERITAS_NB"
"INSTALL_PATH"="C:\\Program Files\\Veritas\\NetBackupDB"
"SERVER_PORT"="13785"
"VERSION"="2.10.004"

Could these be why these 'ghost' media servers keep reappearing? Can I just delete these keys?

[Mike_Gavrilov]

It looks like not all processes are stopped on dwb004-w2k8 and it adds himself to the NetBackup automatically. I've seen such situation when vmd wasn't stopped and was adding the media-server again and again. 

[Marianne]

Wait... Those registry entries are only for a Master server.
Was SWC004-W2K8 installed as a Master server?

And the machine renamed without completely uninstalling NBU?

[Marianne]

@SteveLaw

Have you seen attempts to help you?

Please help us to help you...

[Amol_Nair]

As you used nbemmcmd deletehost command I am assuming that this was added as a media server in the environment..

Kindly log on the machine "dwb004-w2k8" and check if this name is added in the vm.conf file as MM_SERVER_NAME

The location of the file is <install_path>\volmgr\vm.conf

If the name is present it would popup even if manually deleted..