NBU7 strange restore good before but fail now for 11g(10g still ok)

[symanesee]

hannel d1: restoring control file
released channel: d1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 12/10/2011 11:41:39
ORA-19870: error while restoring backup piece /c-1255836038-20111209-02
ORA-19507: failed to retrieve sequential file, handle="/c-1255836038-20111209-02", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Backup file </c-1255836038-20111209-02> not found in NetBackup catalog

compare dbclient log now and before

before when restored good :

....

02:30:32.764 [23084] <2> vnet_vnetd_daemon_socket: ../../libvlibs/vnet_vnetd.c.479: VN_REQUEST_DAEMON_SOCKET
: 2 0x00000002
02:30:32.764 [23084] <2> vnet_vnetd_daemon_socket: ../../libvlibs/vnet_vnetd.c.493: ipc_string: /tmp/vnet-23
104323109819401252000000002-KZgdy0
02:30:32.845 [23084] <2> int_ReadData: INF - reading buffer # 1 of size 262144
02:30:33.755 [23084] <2> int_ReadData: INF - Got end-of-file

02:30:33.755 [23084] <8> int_ReadData: WRN - Failed to set client read timeout.
02:30:34.271 [23084] <2> int_CloseImage: INF - Restore - closing </c-1255836038-20111205-02>
02:30:53.431 [23084] <4> sbtend: INF - --- END of SESSION ---

Now failed:

02:30:06.786 [29503] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
02:30:06.827 [29503] <2> logconnections: BPRD CONNECT FROM 127.0.0.1.10896 TO 127.0.0.1.13724
02:30:06.953 [29503] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOC
KET: 6 0x00000006
02:30:06.953 [29503] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
02:30:06.994 [29503] <2> logconnections: BPRD CONNECT FROM 127.0.0.1.63284 TO 127.0.0.1.13724
02:30:07.119 [29503] <4> VxBSAQueryObject: INF - No match was found for query
02:30:07.119 [29503] <2> int_FindBackupImage: INF - /c-1255836038-20111209-02 not found
02:30:07.119 [29503] <16> int_StartJob: ERR - Backup file </c-1255836038-20111209-02> not found in NetBackup
catalog

seems the key point :

vnet_vnetd_daemon_socket: ../../libvlibs/vnet_vnetd.c.493: ipc_string: /tmp/vnet-23104323109819401252000000002-KZgdy0

[Zahid_Haseeb]

1.) Share the Activity Monitor Details.

2.) Did you have both Oracle machines on the Network, means 10G and 11G ?

If yes then are you able to restore normal / Flat file backup from 11G machine ? if not then first check the Network connectivity, DNS resolution.

[Marianne]

Have you verified that the Rman catalog and the NetBackup images database are in sync?

This looks like the image might have expired in NBU:

'Backup file </c-1255836038-20111209-02> not found in NetBackup catalog'

Run the following on NBU master to list backups for this client for the period that you are looking for:

'bplist -C <client-name> -t 4 -s <start-date> -e <end-date> -R /

[symanesee]

Thanks for replies!

Activity Monitor reveals none abnormal.
 

10G and 11G are on the same nbu server, 10g restored OK

bplist told good result for two client macines (10g & 11g)

I noticed the failed dbclient.log for 11g

<4> VxBSAQueryObject: INF - No match was found for query

wonder any tools can diagnose VxBSAQueryObject & ipc_string: /tmp/vnet-...

[symanesee]

yeah i noticed the owner of bplist output is incorret

-rw-rw---- oracle    oinstall     131072 Dec 10 22:36 /c-1255836038-20111210-02

the 'oracle' should be ora11,the group is ok, is it the cause ?

[symanesee]

the above owner maybe not incorret or shoud be such, local is ora11, but before is good

[Marianne]

bplist is not incorrect - this particular backup was done by oracle user. NetBackup has no way of knowing this other than the user id that is sent at the time of backup. Image ownership cannot be changed.

The restore can only be done by the user who performed the backup - in your case, oracle.

If the backup was done with 'group' or 'any' permissions, the restore can be done as different user.

See http://www.symantec.com/docs/TECH141983

extract:

With NetBackup 6.5.4 you can set permissions on the backup image when the backup is written.  The permissions can be set to USER, GROUP or ANY.

SEND 'BKUP_IMAGE_PERM=USER|GROUP|ANY' when backup is written to set the permissions.
 

So, if the backup was not done this way, only the oracle user can restore this backup.

[symanesee]

from the output of bplist

-rw-rw---- oracle    oinstall     131072 Dec 10 22:36 /c-1255836038-20111210-02

I can restore by another user in group oinstall other than oracle before Dec 7

in fact I now even cannot restore backups that I ever can

[symanesee]

bprd log said on Dec 6

02:30:18.491 [23096] <2> fileslist: begin db communication
02:30:18.491 [23096] <2> fileslist: criteria sent to db mgr
02:30:18.565 [23096] <2> process_request: EXIT STATUS 0

now

02:30:19.494 [5960] <2> fileslist: begin db communication
02:30:19.494 [5960] <2> fileslist: criteria sent to db mgr
02:30:19.545 [5960] <2> fileslist: unexpected return value from db_FLISTreceive: no entity was found 227
02:30:19.545 [5960] <2> process_request: fileslist failed - EC status = no entity was found
02:30:19.545 [5960] <2> process_request: EXIT STATUS 227

[symanesee]

get_image_by_file all were OK

on Dec 6

02:30:18.328 [23088] <2> process_request: EXIT STATUS 0

now

02:30:19.344 [5952] <2> process_request: EXIT STATUS 0

yeah but gid in filelist is different

on Dec 6

02:30:18.447 [23096] <2> fileslist:    client_uid = 1002
02:30:18.447 [23096] <2> fileslist:    client_gid = 505

now

02:30:19.450 [5960] <2> fileslist:    client_uid = 1002
02:30:19.450 [5960] <2> fileslist:    client_gid = 1002

apparently this the cause , I'll confirm it soon

[symanesee]

but

# id oracle
uid=505(oracle) gid=505(oinstall) groups=505(oinstall),504(dba),1002(asmadmin),1004(asmdba)

no problem

why

 id ora11
uid=1002(ora11) gid=505(oinstall) groups=505(oinstall),504(dba),1002(asmadmin),1004(asmdba)

cannot

even # id ora11
uid=1002(ora11) gid=505(oinstall) groups=505(oinstall)

still

08:55:42.509 [27851] <2> fileslist:    client_gid = 1002

the same even if restart netbackup

[symanesee]

so how to change something somewhere that bprd can gives correct log ?

let

08:55:42.509 [27851] <2> fileslist:    owner = ora11
08:55:42.509 [27851] <2> fileslist:    group = asmadmin

back to be

02:30:18.447 [23096] <2> fileslist:    owner = ora11
02:30:18.447 [23096] <2> fileslist:    group = oinstall

[Marianne]

It seems your backups are performed by more than one user that belongs to more than one group?  Did you do a listing for more than one image? More info from bprd is needed to determine why there is a difference.
Permissions are written at the time of backup, it cannot be changed afterwards.

You have two choices:

1. Restore as same user that did the backup.
2. Set image ownership when doing backups (as per TN mentioned above). Image ownership is not the same as file-level permissions.

[symanesee]

thanks anyhow, but your choices too less, I dropped the user and then recreated it, the group's name became the hoped. Although I'm not willing it's a bug, sometime if it appears again after I adjust the policy ...

[Marianne]

If you are not happy with my response, please open a support call with Symantec.

I have been assisting customers for more than 12 years with Oracle restores - that is how NBU works.
For many years we only had option 1.
Symantec has now added option 2 for us.

[symanesee]

not unhappy,but only want to know why the group changed

for your choice 1 I already have the same user used so impossible

for choice 2 I hope to be still able to use previous backups, also impossible

thanks again for your kind support