rman restore issue

[Kai2209]

Hi,
we are having a problem to restore data using rman. First we tried to restore the backup from one server to another but it failed. Then we tried to restore on the same server, but failed also. I guess it is a problem with name resolution but DNS works and hosts files are fine to me.

Below is the rman log...

...
Starting restore at 13-APR-17

channel ch1: starting datafile backup set restore
channel ch1: restoring control file
channel ch1: reading from backup piece lev2_v3s04qlb_1_1
channel ch1: ORA-19870: error while restoring backup piece lev2_v3s04qlb_1_1
ORA-19507: failed to retrieve sequential file, handle="lev2_v3s04qlb_1_1", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Failed to open backup file for restore.

failover to previous backup

channel ch1: starting datafile backup set restore
channel ch1: restoring control file
channel ch1: reading from backup piece lev1_uvs026lh_1_1
channel ch1: ORA-19870: error while restoring backup piece lev1_uvs026lh_1_1
ORA-19507: failed to retrieve sequential file, handle="lev1_uvs026lh_1_1", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Failed to open backup file for restore.

failover to previous backup
...

Can someone please help in this, thanks!

[Michael_G_Ander]

There seems to be some name confusion between fraamdblite1p and fraamdblite1d

Make sure that the name (NB_ORA_CLIENT) used by the restore script is the same as the name used for storing the backup images (bplist is a good test for this)

Do remember to send the NB_ORA_CLIENT on all the channels on all the pieces in the restore script.

There is a rather good tech note about alernate Oracle client restore, which seems is what you trying to do.

[Michal_Mikulik1]

Hello,

all attempts ended after 900 sec timeout with error:

20:44:35.108 [24698] <16> xbsa_GetObject: ERR - VxBSAGetObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
20:44:35.108 [24698] <16> int_StartJob: ERR - Failed to open backup file for restore.

I recommend to test these:

- mere filesystem restore initiated from the client

- telnet to port 1556 from client to Master/Media Servers and back

- reverse DNS lookup for the client on the server(s) and for the server(s) on the client

Regards

Michal

[Marianne]

Is this a node in a cluster? Or a multi-homed server (with 2 hostnames)?

We see this in dbclient log:

Node name: fraamdblite1d
Client Host: fraamdblite1p

13:15:14.363 [22299] <8> dbc_GetServerClientConfig: WARNING - NBU's client name= <fraamdblite1p> differs from gethostname()= <fraamdblite1d>

We see IP address 10.64.2.22 is used to connect to the master server:

13:15:24.038 [22354] <2> logconnections: BPRD CONNECT FROM 10.64.2.22.37474 TO 10.64.2.40.1556 fd = 9

Which hostname is associated with this IP address?

We can see from the image ID that the backup was done for client name fraamdblite1p.

No response from the master server in 15 minutes.

You need to check bprd log on the master server to see how incoming IP address from client was resolved to hostname, how the catalog was searched for image, how permissions were interpreted, etc.

If bprd log folder does not exist on the master, create the folder and restart NBU.
Retry the restore and check bprd for errors.

About restore to different client: http://www.veritas.com/docs/000041053

PS:

Please try to post logs with .txt extentions.

[Kai2209]

first of all thanks for the replies.

The client fraamdblite1p (10.64.2.52) is productive and the client fraamdblite1d is development (10.64.2.52). The master server is 10.64.2.40. We want to restore a backup of 1p to 1d. But even on the same server we get this error.

I have attached the bprd of the master server. Hope this helps.

I already created a No.Restrictions file in altnames directory. Ii can do a bplist on both clients and master server. Nslookup shows everything as expected...

thanks again...

[Marianne]

See these entries in bprd log:

13:23:37.866 [14874] <2> get_type_of_client_list_restore: list and restore not specified for fraamdblite1p
13:23:37.866 [14874] <2> process_request: client restore requests disallowed

5:28:31.152 [20485] <2> get_type_of_client_list_restore: list and restore not specified for fraamdblite1d
15:28:31.152 [20485] <2> process_request: client restore requests disallowed

It seems client browse is allowed in your environment, not restores.

Check Master Server Host properties -> Master -> Client Attributes.

Check global client restore permissions (Allow client restore) and individual client restore permissions. 

[Kai2209]

thanks Marianne, i do not know why this has been unchecked. It was helpful and we are able to resore files partially but not complete :(

...

Oracle instance shut down
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 04/19/2017 10:29:44
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-19870: error while restoring backup piece lev1_uirvq8qt_1_1
ORA-19507: failed to retrieve sequential file, handle="lev1_uirvq8qt_1_1", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Failed to open backup file for restore.
                                                                              
RMAN> **end-of-file**

[Marianne]

I will have a look at the logs when time permits.

Did the restore attempt show up in Activity Monitor?
If so, please post all text in Details tab. (This will make it easier to locate PIDs in the logs).

[Marianne]

There seems to be a problem with resource allocation and release.

I see this about 6 times in bprd log:

09:55:02.662 [2965] <2> jmcomm_ReleaseAllResourcesForJob: releaseAllResources returns [805]
09:55:02.662 [2965] <2> jmcomm_ReleaseAllResourcesForJob: returning
09:55:02.662 [2965] <16> free_allocated_resources: Free resource allocations failed.

What else is running on the master/media server?

Are other backups and restores (e.g. file level restore) working fine?

[Kai2209]

attached you will find  a snippet of the activity log

the details of the failed jobs are...

19.04.2017 09:55:14 - begin Restore
19.04.2017 09:55:15 - restoring from image fraamdblite1p_1490046928
19.04.2017 09:55:15 - requesting resource @aaaah
19.04.2017 09:55:15 - granted resource  MediaID=@aaaah;DiskVolume=LSU-DE;DiskPool=dd-de-dp;Path=LSU-DE;StorageServer=dd-de.xyz.com;MediaServer=fraadmops1b
19.04.2017 09:55:17 - Error bpbrm (pid=9204) bpcd on fraamdblite1d exited with status 59: access to the client was not allowed
19.04.2017 10:11:17 - restored from image fraamdblite1p_1490046928; restore time: 0:16:02
19.04.2017 10:11:18 - end Restore; elapsed time 0:16:04
Oracle policy restore error  (2801)

19.04.2017 10:14:31 - begin Restore
19.04.2017 10:14:32 - restoring from image fraamdblite1p_1490219787
19.04.2017 10:14:32 - requesting resource @aaaah
19.04.2017 10:14:32 - granted resource  MediaID=@aaaah;DiskVolume=LSU-DE;DiskPool=dd-de-dp;Path=LSU-DE;StorageServer=dd-de.xyz.com;MediaServer=fraadmops1b
19.04.2017 10:14:33 - Error bpbrm (pid=8052) bpcd on fraamdblite1d exited with status 59: access to the client was not allowed
19.04.2017 10:30:33 - restored from image fraamdblite1p_1490219787; restore time: 0:16:01
19.04.2017 10:30:34 - end Restore; elapsed time 0:16:03
Oracle policy restore error  (2801)

[Marianne]

Error bpbrm (pid=9204) bpcd on fraamdblite1d exited with status 59: access to the client was not allowed

It seems fraamdblite1d does not have SERVER entry for master/media server hostname in bp.conf, or else it does not resolve the master/media incoming IP address to hostname that matches SERVER entry in bp.conf exactly.

Create bpcd log folder on fraamdblite1d and test connection as follows on master/media server:

bptestbpcd -client fraamdblite1d -verbose -debug

Please post output of command as well as bpcd log.

[Kai2209]

i have restartet the NBU processes already yesterday, does it make sense to restart the master server?

The other backups and restores (file level) are running fine. Oracle rman backup is also fine...

[Marianne]

It is possible that the destination client may be the 'resource' issue.

Please resolve the status 59 issue as per my previous post.

[Kai2209]

i can see the failed restores are handled by the media server (fraadmops1) and the sucessful restores by the master (enigmab, franbp1) server

i did the command from master and from media server but no result in bpcd of the client fraamdlite1d

so the media server seems to be the problem...

[Marianne]

bptestbpcd: terminating - cannot open debug file: Permission denied (13)
file open failed

So, the media server is a Windows server... Have you opened cmd with 'Run as Administrator'?

Where have you created bpcd folder? Same /usr/openv/netbackup/logs folder where dbclient also exists? With root as owner?

I find it strange that none of the bptestbpcd tests write anything to bpcd on the client.

Try other methods to test forward and reverse name lookup. Especially on the client:

bpclntcmd -hn <media-hostname>

bpclntcmd -ip <media-IP>

Check that the hostname resolved by -ip option exist as SERVER or MEDIA_SERVER in client's bp.conf.

[Kai2209]

i am so sorry Marianne, pls. disregsard the first part of my message i hab a typo in the cleint name, The result is

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client fraamdblite1d -verbose -debug
15:08:40.576 [8752.3720] <2> bptestbpcd: VERBOSE = 0
15:08:40.577 [8752.3720] <2> ConnectionCache::connectAndCache: Acquiring new connection for host enigmab, quer
y type 223
15:08:40.596 [8752.3720] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:08:40.597 [8752.3720] <2> logconnections: BPDBM CONNECT FROM 10.64.8.84.64412 TO 10.64.8.35.1556 fd = 728
15:08:40.609 [8752.3720] <2> db_CLIENTsend: reset client protocol version from 0 to 7
15:08:40.677 [8752.3720] <2> db_end: Need to collect reply
15:08:40.678 [8752.3720] <2> db_freeEXDB_INFO: ?
15:08:40.697 [8752.3720] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:08:40.700 [8752.3720] <2> logconnections: BPCD CONNECT FROM 10.64.4.44.64413 TO 10.64.2.22.1556 fd = 728
15:08:40.714 [8752.3720] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:08:40.716 [8752.3720] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1776: 0: via PBX: VNETD CONNECT FRO
M 10.64.4.44.64414 TO 10.64.2.22.1556 fd = 744
15:08:40.717 [8752.3720] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.445: 0: VN_R
EQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
15:08:40.757 [8752.3720] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.462: 0: ipc_
string: /tmp/vnet-27485492607320864436000000450-BvT18J
15:08:40.962 [8752.3720] <2> ConnectToBPCD: bpcd_connect_and_verify(fraamdblite1d, fraamdblite1d) failed: 46
<16>bptestbpcd main: Function ConnectToBPCD(fraamdblite1d) failed: 46
15:08:40.965 [8752.3720] <16> bptestbpcd main: Function ConnectToBPCD(fraamdblite1d) failed: 46
<2>bptestbpcd: server not allowed access
15:08:40.966 [8752.3720] <2> bptestbpcd: server not allowed access
<2>bptestbpcd: EXIT status = 46
15:08:40.968 [8752.3720] <2> bptestbpcd: EXIT status = 46
server not allowed access

[Marianne]

server not allowed access

Same as status 59.

You need bpcd on the client (or manual reverse lookup of media server IP on client) to see how client resolves media server IP to hostname.

Ensure resolved hostname exists as SERVER (or MEDIA_SERVER) in client's bp.conf.

[Kai2209]

Hallo,

except our problem with the "global client restore permissions" the problem was a routing issue on our windows media server. Since we forced the restore to use the unix master/media server everything was working fine.

Thanks to everybody especially to Marianne for this graet support !!

[nmoralesEC]

Hello, I hope it can help something, I'm really new to NBU however the first experience I had to restore RMAN ORACLE happened something very similar very similar. The problem was, the target server had other backup software previously, the solution was to format the target environment, install only NBU, configure bp.conf, run oracle_link, and launch the RMAN function. It was the easiest solution, because the environment was testing, the complex was to work to make the two software work at the same time. Luck