RMAN Duplicate hungs on controlfile restoring..

[Vitalii]

Hello,

oracle 12.1 + NetBackup 7.6 (8.1)

Restoring redirectione is allowed on Master

Permission 777 for /usr/openv/netbackup/logs/

$ rman

connect target sys/pwd@PROD
connect rcvcat rman/pwd@RCAT
connect auxiliary /

run {
SET UNTIL TIME = "TO_DATE('2019-06-03 08:00:00', 'YYYY-MM-DD HH24:MI:SS')";
allocate AUXILIARY channel ch01 type 'sbt_tape' PARMS 'ENV=(NB_ORA_SERV=esn-nbuma01,NB_ORA_CLIENT=prod)';
allocate channel ch0 device type disk;
duplicate target database to TEST
NOFILENAMECHECK;
}

rman log:

...

channel ch01: starting datafile backup set restore
channel ch01: restoring control file
channel ch01: reading from backup piece cntrl_2296_1_1009949090

..and hangs about 20 minutes after that error message:

Failed to process backup file <cntrl_2293_1_1009948044>

this is NetBackup Client log:

/usr/openv/netbackup/logs/user_ops/dbext/logs/30623.0.1559897226

......

Restore started Fri Jun 7 10:28:09 2019

10:28:13 (8562721.001) INF - Data socket = esn1-nbume02.IPC:/usr/openv/var/tmp/vnet-29506559896093223996000000000-RFhF2G;bbb9ccf92d611b121919c356ab837fb8;11;600
10:28:13 (8562721.001) INF - Name socket = esn1-nbume02.IPC:/usr/openv/var/tmp/vnet-29507559896093307990000000000-McYhoP;0835cc1dc8b478bb15cee5cb4087f3f6;11;600
10:28:13 (8562721.001) INF - Job id = 8562721
10:28:13 (8562721.001) INF - Backup id = prod_1559532203
10:28:13 (8562721.001) INF - Backup time = 1559532203
10:28:13 (8562721.001) INF - Policy name = SB_P0_ORACLE_ARCH_PPROD
10:28:13 (8562721.001) INF - Snapshot = 0
10:28:13 (8562721.001) INF - Frozen image = 0
10:28:13 (8562721.001) INF - Backup copy = 0
10:28:13 (8562721.001) INF - Master server = esn-nbuma01.local
10:28:13 (8562721.001) INF - Media server = esn1-nbume02.local
10:28:13 (8562721.001) INF - Encrypt = 0
10:28:13 (8562721.001) INF - Use shared memory = 0
10:28:13 (8562721.001) INF - Restore id = 8562721.001
10:28:13 (8562721.001) INF - Encrypt = 0
10:28:13 (8562721.001) INF - Client read timeout = 3600
10:28:13 (8562721.001) INF - Media mount timeout = 0
10:28:13 (8562721.001) INF - client = test
10:28:13 (8562721.001) INF - requesting_client = prod
10:28:13 (8562721.001) INF - browse_client = prod
10:28:23 /cntrl_2296_1_1009949090
10:28:23 /cntrl_2296_1_1009949090

Well, after reading many notes I'm worried about Master & Media Server name in log:

Master server = esn-nbuma01.local
Media server = esn1-nbume02.local

there is already no this names in network, but these ones exists enstead:

Master server = esn-nbuma01
Media server = esn1-nbume02

I've tried change order of rows in /usr/openv/netbackup/bp.conf

SERVER = esn-nbuma01
SERVER = esn1-nbume01

SERVER = esn1-nbume02
SERVER = esn2-nbume01
SERVER = esn2-nbume02
CLIENT_NAME = prod
CONNECT_OPTIONS = localhost 0 0 2

no lucky.

[Vitalii]

this one also configured:

oracle:TEST> env|grep NB_ORA
NB_ORA_CLIENT=prod

[Michal_Mikulik1]

Hello,

to simplify troubleshooting, perform these tests with a simple file first. (so work with Standard policy backups, not Oracle policy backups).

regards

Michal

[Vitalii]

Hi and thanks for your answer!

restoring of simple file works.

[Vitalii]

Could the issue be related to different Additional (or Media) servers that connected to Master server? It seems PROD and TEST DB servers have the same Master, but different Additional (or Media) servers. 

How can I chech Oracle backup policy, if file restoration (simple restore) works as expected? Thanks!

[Marianne]

Could you please share the following logs (from the same date and time) that will contain restore info?

On master: bprd (if the folder does not exist, yoou need to create it and restart NBU, followed by another restore attempt)

On destination client: dbclient.

Please copy these logs to bprd.txt and dbclient.txt and upload the files.

About SERVER entries - best to have the same entries in the source and destination client.
The restore instruction will in most cases be sent to same media server that performed the backup (we will see evidence in bprd log once all the restore parameters have been verified by the master). 

We may require more logs, but these 2 will give us a good indication of how hostnames are used and resolved in NBU. 

[Vitalii]

Hello, thanks for your answer!

Unfortunatelly, I have no possibility to get any Server side logs. No server logs no Server console. I only had access to client side and logs from /usr/openv/netbackup/logs/user_ops/dbext/logs and I've posted it in the begining of topic. Probably it's really poor information and it's hard to fix the the issue. I think, the topic could be hung on this step for some time. It was urgent task and I couldn't wait. That's why I have duplicated database from active oracle standby database thru the network (Net8). Hopefully I can return to the issue later and can understand the reason why NetBackup redirection worked before for 3 similar configurations (I've already restored some databases such way), but not with new one. At least I know for sure, that I need server logs and I'll try to get it for new test restore attempt. I wonder if you could provide absolute paths for bprd.txt and dbclient.txt.  Thanks one more time!

[Marianne]

Logging is explained in the Troubleshooting chapter of NetBackup for Oracle Administrator's Guide 
. 
Look for:  
Enabling the debug logs manually (UNIX)
To create the NetBackup for Oracle for UNIX database agent logs manually
1 Create the following directories on the client:
...
■ dbclient
For any backup or restore using RMAN.
/usr/openv/netbackup/logs/dbclient

The path on the master for bprd is incorrect in the manual -  correct path below:

On the NetBackup server or servers, create the debug log directories for the legacy processes that interact with the Oracle agent.
On the master server:
/usr/openv/netbackup/logs/bprd
If you need to create the bprd log folder, NBU must be restarted. 

There will be date-stamped log files inside these folders that can be opened with a text editor. (They are not called bprd.txt or dbclient.txt. )
I need you to locate the log files that contain restore attempt info and rename/copy them to .txt files for easy identification.