Netbackup Master server still attempting to contact decommissioned clients

[GDYR_MPD]

I am currently running Netbackup Server 7.7.2 on RedHat Linux and a mix of media servers (HP-UX, AIX, RedHat). 

Several years ago many clients were taken off the list of servers to backup.  They were/are running Out-dated operating systems; hence had Netbackup client 4, 5, or 6 installed.  I am monitoring output from the Master server and noticed these client servers are still attempted to be contacted over a range of ports.  I added the clients to a test policy and deleted them.  No longer is the range of ports being contacted, only a single port 13782.

I have verified the clients are not in a policy.  I am finding some of the clients in /db/client/<client> and in /db/images/<client>.  I understand images are past backups and do not wish to touch that directory.

These clients may no longer exist and I would like my master server to stop attempting to contact them on port 13782.  My thought is that during upgrade processes through the years did not update the database tables properly. 

Has anyone else run into this?  Any idea how to fix.

[revarooo]

Shouldn't need the clients that are deleted in netbackup/db/clients directory. You can delete those or move them. Obviously do NOT delete anything from db/images

[areznik]

If you go to the Clients screen in the GUI, are they still there? 

Run this command for a few of the clients and see if you get anything back: 

nbemmcmd -listhosts -display_server -machinetype client -machinename <clientname> 

[sdo]

Or perhaps keep the Client Attributes, i.e. keep the contents of /db/client, and "offline" the no longer existing client name, and maybe this will cease the reach out?

How often was the reach out to old client names?  Hourly?  Every 12 hours?  Every 24 hours?  Maybe image cleanup is causing something deep inside NetBackup to attempt to resolve the name.

In fact, this rings a bell...

...I think you may need to create a load of dummy hosts file name entries like:

192.168.200.200  oldclientname

...and point the many old client names at an unused IP address... and I think this will stop the errors when internal NetBackup tries to resolve a name to an address when trundelling over the /db/images set.

[GDYR_MPD]

Result from command posted

NBEMMCMD, Version: 7.7.2
The function returned the following failure status:
invalid host name (136)
Command did not complete successfully.
 

[GDYR_MPD]

I used the Java GUI to remove the items from the clients directory, just incase additional items are being performed in the background.

Netbackup Management -> Host Properties -> Master Servers -> (double click) <server>

Select the Client Properties and removed the old visible hosts from the list.

Will see in the next 24 hours if an attempt is made to contact the client and post the result.

[GDYR_MPD]

It varies. 

Since the clients were deleted years ago I would guess it is attempting to connect at the same rate as the backups.  I monitored for 5 nights and some hosts showed 5 attempts.  Others showed up to 115 attempts.

[Mark_Solutions]

Normally the db\client (Client Attributes) and servers / additional servers list should do it ... but there is another place it can happen and that is with VMware query policies.

If the query says displayname=server1 and server1 doesnt exist anymore then it doesnt get backed up or throw an error  .. but it does check for it everytime the policy runs.

If you pipe out your polcies (bppllist -allpolicies -U>c>\policies.txt) and search for any of the clients being checked for that may also help.

Worth also making sure that none of them exist for any of your media servers too as many reports include what the media servers do as well as the Master.

Obviously if they were ever media servers or similar then storage units, storage unit groups, server groups, credentials etc can all come into play

Hope this helps

[sdo]

You are seeing the connect attempts where?   In the bpdbm log?   Can you show us a snippet/example?

I remember something similar where bpdbm was spending lots of time trying to resolve names.  In the end the workaround was to create dummy hosts file entries for the old client names that no longer existed.  I think I was on a Solaris 10 master with NetBackup v7.1.x.x or maybe one of the early 7.5.x.x releases at the time.

[GDYR_MPD]

Only appearing in the log directory nbproxy.

04:13:45.230 [12597] <2> dblibFacadeImpl::getHostInfo: Input:HostName: server006(dblibFacadeImpl.cpp:10724)
04:14:00.281 [12597] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED status: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO server006 10.10.123.229 bpcd VIA pbx status: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO server006 10.10.123.229 bpcd VIA vnetd status: FAILED, (42) CONNECT_REFUSED; system: (111) Connection refused; FROM 0.0.0.0 TO server006 10.10.123.229 bpcd
04:14:00.281 [12597] <2> local_bpcr_connect: Can't connect to client server006
04:14:00.281 [12597] <2> ConnectToBPCD: bpcd_connect_and_verify(server006, server006) failed: 25
04:14:00.281 [12597] <2> local_getHostInfo: Can't connect to server006. Errno = 115: Operation now in progress
04:14:00.281 [12597] <16> local_getAllBEorNBHostInfo: local_getHostInfo(nb_master, 67, server006, ...) failed: 25
 

[revarooo]

dblibFacadeImpl - that's definitely a policy manager operation, so a VM/HyperV operation may cause this, so as Mark recommended, check if you have a VM policy querying for clients

[nbutech · Accepted solution]

Run the command to identify the clients

bpplclients -allunique -l > output.txt

If they are not listed anywhere and you see it in logs you might then gather logs and send to Veritas support for the sql script to remove those client

[GDYR_MPD]

It looks like contact with support will be needed.