Rman Backup Failures in 11g Oracle (SAP) with 7.20 (22) BrTools

[Jesse_Josserand]

We have recently implemented the "_rman_chan_failo = yes" fix for Rman Channel drops which previously caused BrTools to see failures, but now warnings. This is working in most places across my landscape; however, in a few systems, I'm still seeing failures as follows after a long backup run is nearly complete...

09:13:12.562 [44761456] <2> int_WriteData: INF - writing buffer # 1093000 of size 262144

09:13:13.245 [44761456] <16> writeToServer:

09:13:13.245 [44761456] <16> dbc_put: ERR - failed sending data to server

09:13:13.245 [44761456] <4> closeApi: INF - EXIT STATUS 6: the backup failed to back up the requested files

09:13:13.246 [44761456] <16> VxBSASendData: ERR - Could not do a bsa_put().

09:13:13.246 [44761456] <16> xbsa_SendData: ERR - VxBSASendData: Failed with error:

Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the server

09:13:13.249 [44761456] <16> dbc_put: ERR - invalid handle received from the application

09:13:13.250 [44761456] <4> closeApi: INF - EXIT STATUS 6: the backup failed to back up the requested files

09:13:13.250 [44761456] <16> VxBSASendData: ERR - Could not do a bsa_put().

09:13:13.250 [44761456] <16> xbsa_SendData: ERR - VxBSASendData: Failed with error:

Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the server

09:13:13.251 [44761456] <16> dbc_put: ERR - invalid handle received from the application

09:13:13.251 [44761456] <4> closeApi: INF - EXIT STATUS 6: the backup failed to back up the requested files

09:13:13.251 [44761456] <16> VxBSASendData: ERR - Could not do a bsa_put().

09:13:13.251 [44761456] <16> xbsa_SendData: ERR - VxBSASendData: Failed with error:

Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the server

...when I get these, my Netbackup team says "script problem" or "permission issues on files being backed up"; however, neither of those are the case here - scripts have not changed and permissions are wide open for backups. Further, non-Rman backups succeed fully with no problems! The database reflected in the errors above is a very large database (12TB of datafiles, 10.6TB of actual data).

Any help appreciated!!!

FYI, the oracle associated backup run log is pretty meaningless in troubleshooting this failure.

I've uploaded the oracle Rman backup run log that indicated the failure, plus the 2 dbclient netbackup logs that are related. I've also uploaded an example oracle run log of the previous non-Rman backup run log that succeeded... for comparison.

[Jesse_Josserand]

Just fyi... there are 8 databases on this host of the following sizes and most backup just fine with essentially the exact same configuration files (with diffs specific to the database only).

de08u3543... AXD......... 347.69 216.17

de08u3543... ACD......... 7.32 2.87

de08u3543... AAD......... 61.22 40.51

de08u3543... ATD......... 67.25 31.94

de08u3543... ARS......... 16,144.36 15,255.82

de08u3543... GD2......... 149.41 81.07

de08u3543... ARM......... 12,006.36 10,683.57

de08u3543... ASD......... 157.22 59.43

...sizes above are in gigabytes and the 1st number is datafiles' size, while the second is data size.

[AAlmroth]

What happened at 09:13:13.245 February 15th?

All streams were cut off at the same time, indicating perhaps a network issue? System resource issue?

Check in the syslog on the AIX server to check whether you can see anything happening external to Oracle and NBU.

I see that you use NBU 6.5, could you please confirm that NB_CLT, NB_JAV and NB_ORA are at least at 6.5.4 where support for AIX 6.1 and Oracle 11gR2 was introduced.

There is also an EEB available for 6.5.4 and 6.5.5, but I'm not sure that is the exact issue you are experiencing though, but it could be worthwhile to apply, or upgrade to 6.5.6 if the master and media servers are at 6.5.6 or higher (7.x).

"RMAN-10038 error when attempting to backup Oracle version 11.2 using NetBackup 6.x."

http://www.symantec.com/docs/TECH77071

"NB_6.5.4_ET1862252_1_347226.zip is a NetBackup 6.5.4 EEB (Emergency Engineering Binary) for Oracle Clients"

http://www.symantec.com/docs/TECH124572

/A