Database Backup failing with error

[Varma_Chiluvuri]

I am a netbackup admin, the RMAN database backup is failing with below error

Starting backup at 12/15/2011 12:31:18
channel ch1: starting full datafile backup set
channel ch1: specifying datafile(s) in backup set
input datafile file number=00003 name=/oracle/E6Q/sapdata7/undo_1/undo.data1
input datafile file number=00010 name=/oracle/E6Q/sapdata1/sr3_4/sr3.data4
input datafile file number=00014 name=/oracle/E6Q/sapdata2/sr3_8/sr3.data8
input datafile file number=00018 name=/oracle/E6Q/sapdata2/sr3_12/sr3.data12
input datafile file number=00022 name=/oracle/E6Q/sapdata3/sr3_16/sr3.data16
input datafile file number=00026 name=/oracle/E6Q/sapdata4/sr3_20/sr3.data20
input datafile file number=00030 name=/oracle/E6Q/sapdata4/sr3_24/sr3.data24
input datafile file number=00034 name=/oracle/E6Q/sapdata5/sr3701_2/sr3701.data2
input datafile file number=00038 name=/oracle/E6Q/sapdata5/sr3701_6/sr3701.data6
input datafile file number=00042 name=/oracle/E6Q/sapdata6/sr3701_10/sr3701.data10
input datafile file number=00046 name=/oracle/E6Q/sapdata7/undo_2/undo.data2
input datafile file number=00050 name=/oracle/E6Q/sapdata1/sr3702_1/sr3702.data1
input datafile file number=00054 name=/oracle/E6Q/sapdata3/sr3702_5/sr3702.data5
input datafile file number=00058 name=/oracle/E6Q/sapdata5/sr3702_9/sr3702.data9
channel ch1: starting piece 1 at 12/15/2011 12:31:23
channel ch2: starting full datafile backup set
channel ch2: specifying datafile(s) in backup set
input datafile file number=00007 name=/oracle/E6Q/sapdata1/sr3_1/sr3.data1
input datafile file number=00011 name=/oracle/E6Q/sapdata1/sr3_5/sr3.data5
input datafile file number=00015 name=/oracle/E6Q/sapdata2/sr3_9/sr3.data9
input datafile file number=00019 name=/oracle/E6Q/sapdata3/sr3_13/sr3.data13
input datafile file number=00023 name=/oracle/E6Q/sapdata3/sr3_17/sr3.data17
input datafile file number=00027 name=/oracle/E6Q/sapdata4/sr3_21/sr3.data21
input datafile file number=00031 name=/oracle/E6Q/sapdata4/sr3_25/sr3.data25
input datafile file number=00035 name=/oracle/E6Q/sapdata5/sr3701_3/sr3701.data3
input datafile file number=00039 name=/oracle/E6Q/sapdata5/sr3701_7/sr3701.data7
input datafile file number=00043 name=/oracle/E6Q/sapdata6/sr3701_11/sr3701.data11
input datafile file number=00047 name=/oracle/E6Q/sapdata7/undo_3/undo.data3
input datafile file number=00051 name=/oracle/E6Q/sapdata1/sr3702_2/sr3702.data2
input datafile file number=00055 name=/oracle/E6Q/sapdata3/sr3702_6/sr3702.data6
input datafile file number=00001 name=/oracle/E6Q/sapdata1/system_1/system.data1
input datafile file number=00005 name=/oracle/E6Q/sapdata1/patrold_1/patrold.data1
channel ch2: starting piece 1 at 12/15/2011 12:31:23
RMAN-03009: failure of backup command on ch1 channel at 12/15/2011 12:46:29
ORA-19506: failed to create sequential file, name="20111215_E6Q_Data_4amu95kq_1_1", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   VxBSAValidateFeatureId: Failed with error:
   Server Status:  cannot connect on socket
channel ch1 disabled, job failed on it will be run on another channel
released channel: ch1
released channel: ch2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch2 channel at 12/15/2011 12:46:29
ORA-19506: failed to create sequential file, name="20111215_E6Q_Data_4bmu95kr_1_1", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   VxBSAValidateFeatureId: Failed with error:
   Server Status:  cannot connect on socket

RMAN>

Recovery Manager complete.
Policy:Ora_RMAN_E6QCutover
Job completed: 111215124630

[Varma_Chiluvuri]

Backup is failing with above error, the channels are not opening. Please suggest the solution

[Marianne]

'Server Status:  cannot connect on socket'

Two things to check:

vnetd port (13724) open in both directions between master and client.
forward and reverse lookup in both directions between master and client (add hosts entries if not on DNS).

For further troubleshooting, please ensure that you have the following logs enabled:

on master: bprd (restart NBU after creating folder).
on client: dbclient (remember to 'chmod 777 dbclient' if Unix/Linux client).

Please also mention NBU version on master, media, client as well as OS on all of them.

[Varma_Chiluvuri]

The backup was completeing successfully but nothing was backing up.

The channels were not opening.

Dec 17, 2011 6:00:08 AM - requesting resource sun140-nb-hcart3-robot-tld-3
Dec 17, 2011 6:00:08 AM - requesting resource sun107-nb.NBU_CLIENT.MAXJOBS.sape6qv-n1-nb
Dec 17, 2011 6:00:08 AM - requesting resource sun107-nb.NBU_POLICY.MAXJOBS.Ora_RMAN_E6QCutover
Dec 17, 2011 6:00:09 AM - granted resource  sun107-nb.NBU_CLIENT.MAXJOBS.sape6qv-n1-nb
Dec 17, 2011 6:00:09 AM - granted resource  sun107-nb.NBU_POLICY.MAXJOBS.Ora_RMAN_E6QCutover
Dec 17, 2011 6:00:09 AM - granted resource  sun140-nb-hcart3-robot-tld-3
Dec 17, 2011 6:00:09 AM - estimated 0 kbytes needed
Dec 17, 2011 6:00:14 AM - started process bpbrm (pid=11970)
Dec 17, 2011 6:00:14 AM - connecting
Dec 17, 2011 6:00:16 AM - connected; connect time: 0:00:00
Dec 17, 2011 6:00:21 AM - end writing
the requested operation was successfully completed  (0)

[Eric_Zhang]

any abnormal messages in  the <oracle_RMAN_script.out > file ?

[Marianne]

Please check your Policy type. It must be Oracle.

If Policy type is Standard or MS-Windows, the script will be backed up instead of being executed on the client.

Please also supply more info about your setup:
OS and NBU version on master and media server.
OS and NBU version on Oracle client.

[AAlmroth]

IN order to better understand what is going wrong we would need to look in several log files, namely bpcd on client and media server, bpbrm on the media server. These together with the logs that have already been suggested. bpcd should give us a better idea why the socket connection is failing.

Is this a proxy backup (snapshot)? In that case, please verify that exclude_list doesn't exclude the oracle directories. Based on your statement that the jobs succeeds but no files are being backed up. If it is a snapshot backup, we should perhaps also have a look in bpbkar logs.

Is it an Oracle RAC configuration?

/A