Restores fail with "Requested machine is not configured in NetBackup"

[m_3sth3r]

I'm running Netbackup 7.6.0.1 on oracle enterprise linux 5.8.

there is 1x Master/Media server, and 2 Media servers with multiple nics.  I gave each interface it's own name. (according to this technote http://www.symantec.com/business/support/index?page=content&id=TECH54733).  The three servers can see each other on all the interfaces.

master and media servers bp.conf

SERVER=master

SERVER=media1

SERVER=media2

SERVER=master-backup

SERVER=media1-backup

SERVER=media2-backup

SERVER=master-backup2

SERVER=media1-backup2

SERVER=media2-backup2

I have a client that is on the non-backup network, it's bp.conf looks as follow (it cannot access the other networks)

SERVER=master

SERVER=media1

SERVER=media2

CLIENT=clientx

The restores fail , every time.

I checked the logs and it's looking for media2-backup2  instead of media2 (vxlogview -o 143 -X "jobid=1234")  (it cannot connect to this server on that network interface)

 .407 [allocateMedia] INITIATING:
 .407 [MdsGlobalCache::update_stale] clearing stale_host_info_key_map
 .407 [MdsGlobalCache::update_stale] clearing stale_host_config_key_map
 .407 [MdsGlobalCache::update_stale] clearing stale density_type_map
 .407 [MdsGlobalCache::update_stale] clearing stale dg_capacity_stat_map
 .407 [MdsGlobalCache::update_stale] clearing stale dg_dvlist_map
 .407 [allocateMedia] masterServer = master, client = clientx, jobType = 2, capabilityFlags = 0, fatPipePreference = 0, statusOnly = 0
 .407 [allocateMedia] MediaRequest_Record: MediaServer = media2-backup2, PreferredMediaServer = , MediaId = P123L6, MediaKey = 0, AssignedTime = 1396159208, DriveName = , DrivePath = , PoolName = , Density = -1, RequiredNdmpTapeServer = , PreferredNdmpTapeServer = , MediaType = 2, MediaSubType = 0, StuNameForDupWrite =
 .407 [allocateMedia] skip_stu_list is empty
 .407 [allocateMedia] skip_med_list is empty
 .407 [allocateMedia] skip_drv_list is empty
 .407 [allocateMedia] skip_disk_volume_list is empty
 .407 [allocateMedia] alloc_med_list is empty
 .407 [allocateMedia] alloc_drv_list is empty
 .407 [allocateMedia] reserved_med_list is empty
 .407 [allocateMedia] umt_med_list is empty
 .407 [allocateMedia] reserved_dv_list is empty
 .407 [allocateMedia] disk_alloc_key_list is empty
 .407 [allocateMedia] unload_requested_drive_cache is empty
 .407 [BEGIN_MDS_DB_TRANSACTION] attempting to get database connection
 .407 [BEGIN_MDS_DB_TRANSACTION] got database connection
 .408 [BEGIN_MDS_DB_TRANSACTION] attempting to start database transaction
 .408 [BEGIN_MDS_DB_TRANSACTION] started database transaction
 .408 [allocate_by_tape_media] job_type = 2, capability_flags = 0, master_server_name = master, client_name = clientx, fat_required = 0
 .408 [sql_get_host_info_by_name] host_name = master, hc = 0
 .418 [sql_get_host_info_by_name] raw_host_info - raw_host_info_t: key = 1000002, parent_key = 1000002, fqname = master, state = 14, nbversion = 760100, nbtype = 3, cluster_key = 0, active_node_key = 0, flags = 119, rank = 3
 .418 [sql_get_host_info_by_name] out - host_info_t: key = 1000002, parent_key = 1000002, fqname = master, state = 14, nbversion = 760100, nbtype = 3, cluster_key = 0, cluster_fqname = , active_node_key = 0, flags = 119, raw_host_key = 1000002, raw_host_name = master, rank = 3
 .418 [sql_get_host_info_by_name] host_name = media2-backup2, hc = 0
 .428 [allocate_by_tape_media] could not get key for media server, MediaServer = media2-backup2
 .428 [END_MDS_DB_TRANSACTION] cur_err = 2005110, m_dbconn_stat = 0, m_dberr_stat = 0, m_closed_db_trx = 0
 .428 [END_MDS_DB_TRANSACTION] rolled back database transaction
 .428 [allocateMedia] EXIT INFO:
 .428 [allocateMedia] skip_stu_list is empty
 .428 [allocateMedia] skip_med_list is empty
 .428 [allocateMedia] skip_drv_list is empty
 .428 [allocateMedia] skip_disk_volume_list is empty
 .428 [allocateMedia] alloc_med_list is empty
 .429 [allocateMedia] alloc_drv_list is empty
 .429 [allocateMedia] reserved_med_list is empty
 .429 [allocateMedia] umt_med_list is empty
 .429 [allocateMedia] reserved_dv_list is empty
 .429 [allocateMedia] disk_alloc_key_list is empty
 .429 [allocateMedia] unload_requested_drive_cache is empty
 .429 [allocateMedia] EXIT STATUS = 2005110 (EMM_ERROR_MDS_NoMachine, Requested machine is not configured in NetBackup)

I updated the bp.conf on the master and media servers with:

FORCE_RESTORE_MEDIA_SERVER = media2-backup2 master
FORCE_RESTORE_MEDIA_SERVER = media2-backup1 master

and reread the config with :

/usr/openv/netbackup/bin/admincmd/bprdreq -rereadconfig

but the backups still fail with the same error.  I stopped netbackup on the media2-backup2 server, and then it tries to use media1-backup2. It should be using either master / media1 / media2.

When I run nbemmcmd listshosts I get this output

nbemmcmd -listhosts
NBEMMCMD, Version: 7.6.0.1
The following hosts were found:
server           master
master           master
media            media1
media            media2
Command completed successfully.

Should I add the other interface names as aliases?

what else am I missing?

[Marianne]

I would like to see the following logs:

On the master server: bprd
(If the log folder does not exist under /usr/openv/netbackup/logs, create it and restart NBU.)

On the client, dbclient and bpcd.
(If the log folders do not exist /usr/openv/netbackup/logs, create them and change permission on dbclient folder to 777 to allow oracle user to write to the folder.)

After next failure, copy logs to .txt files (e.g. bprd.txt) and upload as File attachments.
PS: Level 0 logs will be fine.

bprd log will show us restore request, search criteria, image ownership details, media server selection for restore, etc.. etc...

[m_3sth3r]

Marianne

While I'm collecting logs, we tried something else.  We explicitly changed the media owner on the tape media to the media server that should be doing the restores and the restore is running.

I have a storage unit group configured, that contains the 3 servers (master, media1, media2).  I also have a server group configured that contains the 3 servers (master, media1, media2).

On host properties - media the setting is enabled for "enable unrestricted media sharing for all media servers"

On the policy that performs the backup, , media owner is set to "any".

I'll still get the logs and post.

[Marianne]

Good choice to change media ownership to a specific media server.

bprd log would show us which media server was seen as image owner and also if and how FORCE_RESTORE... entry was interpreted.

[richierichess]

Interested in seeing what the log contains

[callmebruce]

Was the bprd demon restarted after adding the FORCE_RESTORE_MEDIA_SERVER entries were added?

I am guessing the backup was taken by the servers on the same network so would need to be restored by the same server as backed it up?

To force the restores to use the same servers as the backups:

Create file /usr/openv/netbackup/db/config/USE_BACKUP_MEDIA_SERVER_FOR_RESTORE 

and restart bprd.

Then the server that restores will have to be on the same network as the server that was backed up.

If you need to restore from the backup network images to the non-backup network you will need to override this setting with the -disk_media_server option for bprestore.

[m_3sth3r · Accepted solution]

So, the restore is now running

What I did see in the bprd log file from yesterday with the failures were the following:

09:44:24.271 [32463] <2> ParseConfigExA: Badly formed configuration entry on line 24: FORCE_RESTORE_MEDIA_SERVER = media2-backup

which then later became

13:56:24.675 [19084] <2> restorefiles: Because of FORCE_RESTORE_MEDIA_SERVER, changing server of bid clientx_1396388733 from media2 to master

I had configured the media host override in the GUI, clearly I eventually did it right. (the corresponding lines in the bp.conf)

FORCE_RESTORE_MEDIA_SERVER = media2 master
FORCE_RESTORE_MEDIA_SERVER = media2-backup master

[Marianne]

Having bprd log to troubleshoot restore issues is a must IMHO!