RMAN Backup failing with error code 84

[aksriniv]

we are using netbackup 7.1 and aix oracle client and we are using rman to backup the oracle database

Master Server running windows 2008 x64 netbackup 7.1

Client running aix 5.3 oracle version 9.2 64 bit.

we have 5 sids in the same server and we have individual policies for each sid.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch00 channel at 08/30/2011 21:25:57
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: Error received from media manager layer, error text:
   VxBSAEndTxn: Failed with error:
   The transaction was aborted.
 

[Will_Restore]

though not shown in your log above usually means a bad tape. 

[aksriniv]

all our data are backed up to disk.

[Marianne]

Please share the following info:

Information in Details Tab of Activity Monitor.
bptm log on media server
Info on disk storage type - basic dsu, advanced dsu, msdp, pddo?

[aksriniv]

8/31/2011 12:27:30 AM - Info nbjm(pid=23288) starting backup job (jobid=9713) for client moh8014e, policy TOBIN_MART2_NONPROD, schedule Default-Application-Backup 
8/31/2011 12:27:30 AM - Info nbjm(pid=23288) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=9713, request id:{29B67F53-0997-4705-870B-A4742F35FA12}) 
8/31/2011 12:27:30 AM - requesting resource MPS2647_DD880_DISK
8/31/2011 12:27:30 AM - requesting resource mps2647.NBU_CLIENT.MAXJOBS.moh8014e
8/31/2011 12:27:30 AM - requesting resource mps2647.NBU_POLICY.MAXJOBS.TOBIN_MART2_NONPROD
8/31/2011 12:27:30 AM - granted resource mps2647.NBU_CLIENT.MAXJOBS.moh8014e
8/31/2011 12:27:30 AM - granted resource mps2647.NBU_POLICY.MAXJOBS.TOBIN_MART2_NONPROD
8/31/2011 12:27:30 AM - granted resource MediaID=@aaaah;DiskVolume=mps2647lsu;DiskPool=mps2647_lsu_dp;Path=mps2647lsu;StorageServer=cyrddr01u...
8/31/2011 12:27:30 AM - granted resource MPS2647_DD880_DISK
8/31/2011 12:27:30 AM - estimated 0 Kbytes needed
8/31/2011 12:27:30 AM - Info nbjm(pid=23288) started backup job for client moh8014e, policy TOBIN_MART2_NONPROD, schedule Default-Application-Backup on storage unit MPS2647_DD880_DISK
8/31/2011 12:27:30 AM - started process bpbrm (19192)
8/31/2011 12:27:31 AM - Info bpbrm(pid=19192) moh8014e is the host to backup data from    
8/31/2011 12:27:31 AM - Info bpbrm(pid=19192) reading file list from client       
8/31/2011 12:27:31 AM - connecting
8/31/2011 12:27:33 AM - Info bpbrm(pid=19192) listening for client connection        
8/31/2011 12:27:39 AM - Info bpbrm(pid=19192) INF - Client read timeout = 300     
8/31/2011 12:27:39 AM - Info bpbrm(pid=19192) accepted connection from client        
8/31/2011 12:27:39 AM - connected; connect time: 00:00:08
8/31/2011 12:27:40 AM - Info bphdb(pid=921778) Backup started          
8/31/2011 12:27:40 AM - Info bptm(pid=25476) start           
8/31/2011 12:27:40 AM - Info bptm(pid=25476) using 262144 data buffer size       
8/31/2011 12:27:40 AM - Info bptm(pid=25476) setting receive network buffer to 1049600 bytes     
8/31/2011 12:27:40 AM - Info bptm(pid=25476) using 30 data buffers        
8/31/2011 12:27:43 AM - Info bptm(pid=25476) start backup          
8/31/2011 12:27:43 AM - Info bptm(pid=25476) backup child process is pid 26044.25972      
8/31/2011 12:27:43 AM - Info bptm(pid=26044) start           
8/31/2011 12:27:43 AM - begin writing
8/31/2011 12:27:47 AM - Info bphdb(pid=921778) dbclient(pid=921778) wrote first buffer(size=262144)        
8/31/2011 12:32:16 AM - Critical bptm(pid=25476) image write failed: error 2060046: plugin error     
8/31/2011 12:32:16 AM - Critical bptm(pid=25476) sts_get_image_prop failed: error 2060046: plugin error      
8/31/2011 12:32:16 AM - Critical bptm(pid=25476) image delete failed: error 2060046: plugin error     
8/31/2011 12:32:20 AM - Error bptm(pid=25476) cannot write image to disk, Invalid argument     
8/31/2011 12:32:20 AM - Info bptm(pid=25476) EXITING with status 84 <----------       
8/31/2011 12:32:23 AM - end writing; write time: 00:04:40
media write error(84)
8/31/2011 12:32:28 AM - Info bphdb(pid=921778) done. status: 84: media write error 

[Will_Restore]

A similar error with v6.5 PureDisk Dedup option:

http://www.symantec.com/docs/TECH65921

Perhaps a Symantec support request is in order to resolve this issue.

[Marianne]

Seems this is a DataDomain STU?  "requesting resource MPS2647_DD880_DISK"

I see 'plugin error' : image write failed: error 2060046: plugin error   

Are ANY backups working to this STU?

Have you verified that the plugin is recognised by the media server? Run  'bpstsinfo -pi' from cmd.

[aksriniv]

we have created a SLP for RMAN backups and for cofigured that STU for backups and for duplication

we have stu of the remote site.

Plugin Info:
        Plugin Name: libstspibasicdisk.dll
        Prefix: STSBasicDisk:
        Label: Basic Disk Plugin
        Prefix: AdvancedDisk:
        Label: Advanced Disk Plugin
        Build Version: 11
        Build Version Minor: 1
        Operating Version: 11
        Vendor Version: 710000 NetBackup_7.1

        Plugin Name: libstspicp.dll
        Prefix: ClientDirect:
        Label: Copy STS-I Plugin
        Build Version: 11
        Build Version Minor: 1
        Operating Version: 11
        Vendor Version:

        Plugin Name: libstspinbostpxy.dll
        Prefix: NewProxy:
        Label: NewProxy STS-I Plugin
        Build Version: 11
        Build Version Minor: 1
        Operating Version: 11
        Vendor Version:

        Plugin Name: libstspinearstore.dll
        Prefix: ntap:
        Label: Nearstore Plugin
        Build Version: 9
        Build Version Minor: 1
        Operating Version: 9
        Vendor Version: Protocol Version = 2

        Plugin Name: libstspiproxy.dll
        Prefix: *
        Label: STS-I Proxy
        Build Version: 9
        Build Version Minor: 1
        Operating Version: 9
        Vendor Version:

        Plugin Name: libstspishmem.dll
        Prefix: ShMem:
        Label: Shared Memory STS-I Plugin
        Build Version: 0
        Build Version Minor: 0
        Operating Version: 0
        Vendor Version:

        Plugin Name: libstspiDataDomain.dll
        Prefix: DataDomain:
        Label: Data Domain Plugin
        Build Version: 11
        Build Version Minor: 1
        Operating Version: 11
        Vendor Version: 2.3.1.0-223945

        Plugin Name: libstspipd.dll
        Prefix: PureDisk:
        Label: OpenStorage PureDisk Plugin
        Build Version: 11
        Build Version Minor: 1
        Operating Version: 11
        Vendor Version: 6.5.0.0 00:13:42 Jan 18 2011

media Server and the Master server are the same. we use ost plugin to connect the data domain 890 and we have LSU created in data domain and configured disk pools in the netbackup through open storage.

[Marianne]

Are any normal OS backups working to the DD?

First confirm that Standard or MS-Windows test backups directly to DD stu is working fine.

[aksriniv]

Yes

[Marianne]

Please ask your DD guys to check the DD logs for errors. The plugin error points to some problem on the DD.

We have seen a similar error when the private network between the local and remote DD was configured incorrectly.

[Anton_Panyushki]

I'd redirect these backups to a good old tape and kick an Oracle policy. If this backup job fails, DD plugin is to blame. I presume There is NetBackup 7.1 client and agent for Oracle on the client.

Another good thing you might do is to check media manager layer with sbttest http://download.oracle.com/docs/cd/B10500_01/server.920/a96566/rcmtroub.htm

[aksriniv]

I Have logged a case with data domain since we donot have tape backup  iam waiting for the reponse i will post the solution once i have it from them.

[aksriniv · Accepted solution]

This is resolved after datadomain gave us OS upgrade patch.