RMAN Script Failure

[Vivi]

Hi,everybody!

My RMAN script get the following output:

# more mes_czmes02_ora.sh.out

Script /usr/openv/netbackup/dbext/mes_czmes02_ora.sh
==== started on Fri May 11 14:16:45 CST 2012 ====

RMAN: /u1/oracle/product/10gr2/db_1/bin/rman
ORACLE_SID: rac2
ORACLE_USER: oracle
ORACLE_HOME: /u1/oracle/product/10gr2/db_1

NB_ORA_FULL: 1
NB_ORA_INCR: 0
NB_ORA_CINC: 0
NB_ORA_SERV: tsczbak02.trinasolar.com
NB_ORA_POLICY: MES_CZMES02_ORA

Full backup requested
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
You have new mail.
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21> 22> 23>
 24> 25> 26> 27> 28> 29> 30> 31> 32> 33> 34> 35> 36> 37> 38> RMAN> rget database: RAC
 (DBID=2261274443)
using target database control file instead of recovery catalog

RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21> 22> 23>
 24> 25> 26> 27> 28> 29> 30> 31> 32> 33> 34> 35> 36> 37> 38>
allocated channel: ch00
channel ch00: sid=1804 instance=rac2 devtype=SBT_TAPE
channel ch00: Veritas NetBackup for Oracle - Release 7.1 (2011020313)

allocated channel: ch01
channel ch01: sid=1831 instance=rac2 devtype=SBT_TAPE
channel ch01: Veritas NetBackup for Oracle - Release 7.1 (2011020313)

Starting backup at 2012-05-11 14-16-49
channel ch00: starting incremental level 0 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00006 name=+DATA1/rac/datafile/mesproddb00.dbf
input datafile fno=00012 name=+DATA1/rac/datafile/mesproddb04.dbf
input datafile fno=00016 name=+DATA1/rac/datafile/mesproddb08.dbf
input datafile fno=00001 name=+DATA1/rac/datafile/system.dbf
input datafile fno=00002 name=+DATA1/rac/datafile/undotbs1.dbf
channel ch00: starting piece 1 at 2012-05-11 14-16-50
channel ch01: starting incremental level 0 datafile backupset
channel ch01: specifying datafile(s) in backupset
input datafile fno=00025 name=+DATA1/rac/datafile/mesproddb14.dbf
input datafile fno=00033 name=+DATA1/rac/datafile/mesproddb19.dbf
input datafile fno=00019 name=+DATA1/rac/datafile/mesreptdb02.dbf
input datafile fno=00039 name=+DATA1/rac/datafile/undotbs1_3.dbf
input datafile fno=00026 name=+DATA1/rac/datafile/streams01.dbf
channel ch01: starting piece 1 at 2012-05-11 14-16-50
RMAN-03009: failure of backup command on ch00 channel at 05/11/2012 14:17:45
ORA-19506: failed to create sequential file, name="bk_10829_1_783008210", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   VxBSAValidateFeatureId: Failed with error:
   Server Status:  cannot connect on socket
channel ch00 disabled, job failed on it will be run on another channel
released channel: ch00
released channel: ch01
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch01 channel at 05/11/2012 14:17:45
ORA-19506: failed to create sequential file, name="bk_10830_1_783008210", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   VxBSAValidateFeatureId: Failed with error:
   Server Status:  cannot connect on socket

RMAN>

Recovery Manager complete.

Script /usr/openv/netbackup/dbext/mes_czmes02_ora.sh
==== ended in error on Fri May 11 14:17:45 CST 2012 ====

# more /usr/openv/netbackup/bp.conf

SERVER = tsczbak02
SERVER = tsczbak03
CLIENT_NAME = czmes02
CONNECT_OPTIONS = localhost 1 0 2

# more /usr/openv/netbackup/logs/dbclient   ---------> see attached file

NBU 7.1

master server:windows2008R2

client:solaris10 SPARC

Thanks!!

[Marianne]

Server Status:  cannot connect on socket

Your client is unable to connect to bprd on master server.

3 things to check:

Proper forward and reverse hostname lookup on client for Master.
If Solaris client is not on DNS, add /etc/hosts entry for master on client.
Ensure hostname in /etc/hosts is spelled exactly the same (including case) as 1st SERVER entry in client's bp.conf (/usr/openv/netbackup/bp.conf).

Confirm that Master can resolve client's IP address to the hostname that appears in the Oracle policy.
When RMAN starts, client tries to connect to bprd on master (via PBX or vnetd). Master needs to resolve incoming IP address to hostname.

Open ports between client and master.
PBX (1556) will be tried first. Failure to connect on this port will try vnetd (13724).

[Oddy]

Hi,

According to the dbclient log does the client fail to connect to the bprd service on the master.
Make sure you have opened following ports bidirectionally in the firewall
13720 (bprd)
13724 (vnetd>
13782 (bpcd)

To make sure that it does connect, try to telnet to the ports.
I.e from the solaris client run
telnet tsczbak02 13720
I assume that tszbak02 is the master server.

If this fails verify that the bprd port is open in the firewall and try again.
If you can telnet to the bprd port on the master, try
su - <oracle user>
and test the telnet again. If that fails make sure that the /etc/hosts and /etc/nsswitch.conf is readable for everyone.

Create following log folders
on master
<install dir>\netbackup\logs\bprd
on media server
<install dir>\netbackup\logs\bpbrm
<install dir>\netbackup\logs\bptm
on client
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/bphdb
/usr/openv/netbackup/logs/dbclient

I hope this helps.

/Oddy

[Marianne]

Oddy, NO, No, NO, No!!!!!!!!

As from NBU 6.0 NBU comms between server and client is ONLY via vnetd.
As from 7.0.1, connection will first be attempted on PBX (1556), then fail back to vnetd (13724) if PBX fails.

See http://www.symantec.com/docs/TECH136090

[Vivi]

Thank you two response!

I add a send statement to my RMAN script.

SEND 'NB_ORA_POLICY=MES_CZMES02_ORA,NB_ORA_SERV=tsczbak02.trinasolar.com';

then my backup job running normally.

according to the dbclient log, ServerName=czmes03 ClientName=czmes02 policy=trinasolar-mes-oracle.

actually,my servername=tsczbak02.trinasolar and policy=MESZ_CZMES02_ORA

so,what cause this issue?

=================================part of dbclient log============================

10:50:55.905 [29563] <4> dbc_GetClassNameFromConfig: user's backup policy= <trinasolar-mes-oracle>
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - entering GetEnv - NBBSA_SCHEDULE
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - returning -
10:50:55.905 [29563] <4> dbc_GetScheduleNameFromConfig: user's schedule= <Default-Application-Backup>
10:50:55.905 [29563] <4> dbc_GetServerClientConfig: entering dbc_GetServerClientConfig.
10:50:55.905 [29563] <4> dbc_GetServerClientConfig: ServerName: <czmes03>, ClientName: <czmes02>
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - entering GetEnv - BSA_SERVICE_HOST
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - returning - czmes03
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - entering GetEnv - NBBSA_CLIENT_HOST
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - returning -
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - entering GetEnv - NBBSA_VERBOSE_LEVEL
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - returning - 5
10:50:55.905 [29563] <2> sbtinit: INF - entering
10:50:55.905 [29563] <2> xbsa_GetEnv: INF - entering
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - entering GetEnv - NBBSA_CLIENT_HOST
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - returning - czmes02
10:50:55.905 [29563] <2> xbsa_GetEnv: INF - leaving (0)

10:50:55.905 [29563] <2> xbsa_GetEnv: INF - entering
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - entering GetEnv - NBBSA_SCHEDULE
10:50:55.905 [29563] <4> VxBSAGetEnv: INF - returning - Default-Application-Backup
10:50:55.905 [29563] <2> xbsa_GetEnv: INF - leaving (0)

10:50:55.905 [29563] <2> int_GetOracleVersion: INF - entering
10:50:55.905 [29563] <2> int_GetOracleVersion: INF - ORACLE_HOME= </u1/oracle/product/10gr2/db_1>
10:50:55.905 [29563] <2> int_GetOracleVersion: INF - Found </u1/oracle/product/10gr2/db_1/bin/rman>
10:50:55.905 [29563] <2> int_GetOracleVersion: INF - leaving
10:50:55.905 [29563] <2> int_LogSystemInfo: INF - entering
10:50:55.905 [29563] <2> int_LogSystemInfo: INF -

=====================================end===============================

[Marianne · Accepted solution]

Double-check bp.conf on the client - these entries are read from bp.conf:

dbc_GetServerClientConfig: ServerName: <czmes03>, ClientName: <czmes02>

This says to me that czmes03  is 1st in the SERVER list.
Master server must ALWAYS be 1st in the list. This is how the Client knows where to send backup request.

Check 'normal' bp.conf in /usr/openv/netbackup as well as Oracle user's home directory.

The policy is probably hard-coded in (user's) bp.conf as well.

[Vivi]

I have delete the bp.conf in Oracle home directory.and comment the send statement in RMAN script.

and backup running normal.

Thanks a lot!!!!