Oracle RMAN Backup Issue

[zubair_hassan1]

I am facing a following error while executing oracle rman backup for HPUX client. NBU version is 7 and master and media servers are on windows 2008. I have entries in bp.conf for master server and also media server. I am using media server storage unit. Given below is the RMAN script output. I have tried almost everything. name resolution seems alright and i have verified it with bpclntcmd. Linking was also performed for oracle but nothing seems to be working. Any help would be appreciated.

RMAN: /ora1/oracle/product/10.2.0/db_1/bin/rman
ORACLE_SID: RMHDB
ORACLE_USER: oracle
ORACLE_HOME: /ora1/oracle/product/10.2.0/db_1

NB_ORA_FULL: 1
NB_ORA_INCR: 0
NB_ORA_CINC: 0
NB_ORA_SERV: netbackupmaster.rmh.med.local
NB_ORA_POLICY: ERP1_Oracle_Full_Backup

Full backup requested
(c)Copyright 1983-2000 Hewlett-Packard Co.,  All Rights Reserved.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-1992 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1986 Digital Equipment Corp.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2000 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.
(c)Copyright 1991-2000 Isogon Corporation, All Rights Reserved.

                           RESTRICTED RIGHTS LEGEND
Use, duplication, or disclosure by the U.S. Government is subject to
restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in
Technical Data and Computer Software clause in DFARS 252.227-7013.

                           Hewlett-Packard Company
                           3000 Hanover Street
                           Palo Alto, CA 94304 U.S.A.

Rights for non-DOD U.S. Government Departments and Agencies are as set
forth in FAR 52.227-19(c)(1,2).

Open Source Software has been installed on this system.  See
/opt/OpenSource/etc/copyright
^[[3g        ^[H        ^[H        ^[H        ^[H        ^[H        ^[H        ^[H        ^[H        ^[H
Recovery Manager: Release 10.2.0.1.0 - Production on Wed Sep 7 16:11:39 2011

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: RMHDB (DBID=1521051378)
using target database control file instead of recovery catalog

RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21> 22> 23> 24> 25> 26> 27> 28> 29> 30> 31> 32> 33> 34> RMAN> 2> 3> 4> 5> 6> 7> 8>
9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21> 22> 23> 24> 25> 26> 27> 28> 29> 30> 31> 32> 33> 34>
allocated channel: ch00
channel ch00: sid=959 devtype=SBT_TAPE
channel ch00: Veritas NetBackup for Oracle - Release 7.0 (2010010419)

Starting backup at 07-SEP-11
channel ch00: starting incremental level 0 datafile backupset
channel ch00: specifying datafile(s) in backupset
input datafile fno=00059 name=/ora1/oracle/product/10.2.0/db_1/dbs/D:oracleoradataORCLarsystem.dbf
input datafile fno=00070 name=/ora1/ejdata/dm_dbtree_index01.dbf
input datafile fno=00071 name=/ora1/oracle/product/10.2.0/db_1/dbs/patch_data01.dbf
input datafile fno=00072 name=/ora1/oracle/product/10.2.0/db_1/dbs/patch_index01.dbf
input datafile fno=00080 name=/ora1/oradata/RMHDB/tmp.dbf
channel ch00: starting piece 1 at 07-SEP-11
released channel: ch00
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch00 channel at 09/07/2011 16:12:39
ORA-19506: failed to create sequential file, name="bk_10612_1_761242303", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   VxBSAValidateFeatureId: Failed with error:
   Server Status:  cannot connect on socket

RMAN> RMAN>

Recovery Manager complete.
logout

Script /usr/openv/netbackup/bin/hot_database_backup.sh
==== ended in error on Wed Sep 7 16:12:40 WAT 2011 ====

[Will_Restore]

Did you add Netbackup for Oracle license to the Master server?

Also check the dbclient log on the database client for details.

[RiaanBadenhorst]

Check the bpcd logs on both master, media, and client (look for your client's name/ip and look for discrepancies)

Check the bprd log on the master (you might note here that the master isn't resolving the client correctly, might find a long name instead of a short name or vice versa)

Check the bpbrm log on the media server (if this log has entries about the client then comms with the master is probably fine. Make sure the media has no issues with resolving the client name>ip and vice versa)

[Michael_G_Ander]

For database backups you often have to increase CLIENT_CONNECT_TIMEOUT & CLIENT_READ_TIMEOUT

Another thing I have seen giving problems on unix is the permissions on ../netbackup/logs/user_ops/ and below

There might also be an indication of what problem is in the ../netabckup/logs/bphdb folder especially if you have verbose=5 in the bp.conf.

Hope this helps you

Regards

Michael

[jim_dalton]

It says you're not using rman ...is that intentional?

Did client install and linking succeed?

Set up logging on ora end as per manual , up verbosity as above and we'll see where we are.

Jim

[Nicolai]

Do the Oracle host have multiple NIC's installed ?. The error message say "cannot connect on socket" which indicate to me that the traffic may go in the wrong direction.

[zubair_hassan1]

wrobbins:

We have capacity based license installed and as per the NBU master licenses it does show oracle extension in the listed licenses.

Riaan Badenhorst 

Client resolution is fine for OS based backups but there seems to be some issue with rman based backups. In bpdbm logs "client_db: client client_name not available". This happens when i use media server storage unit. I tried to test it with master server storage unit, RMAN log had thrown the same error but bpdbm didnt show this db_client not available.

I will verify the other logs and let you know the output.

Michael Simcorp

I think you are reffering to the logs on the client side. I will verify the permissions. I remember i had already changed the default time out values to higher number.

jim dalton

Where it says "Not using rman"? I will definitely change the log level to 5.

I will let you all know the result after making the recommended changes and then we will move forward with the troubleshooting. Thanks to all for time and effort.

Regards,

-Zubair

[zubair_hassan1]

Dear Nicolai,

There used to be multiple NW interfaces on the srever but at the moment it has only one IP address assigned to one NIC. And as mentioned before, OS and file system backups are running fine. 

[Marianne · Accepted solution]

"Server Status:  cannot connect on socket "

Comms with database backups work different in comparison with OS backup.

When rman starts, a backup request is sent from the client directly to bprd on the master server (via vnetd).
The master then needs to resolve the client's IP address to a hostname that appears in Client list of policy.

Please ensure that you have bprd log folder on the master. If it does not exist, create the folder and restart NBU. After next failure, check bprd log on master for incoming request from client's IP. If master is unable to resolve client's IP, add to hosts file. Ensure that master can resolve client's IP address to the Client name contained in the policy.

Please also create the following log folders on the client: bpcd dbclient.
Remember to chmod 777 dbclient.

[jim_dalton]

It says it here Zubair:

"connected to target database: RMHDB (DBID=1521051378)
using target database control file instead of recovery catalog"

Recovery catalog - Im thinking is rman.

Im no rman expert but I think you also need to register the DB in rman so that rman will allow the DB to send it backup information.

Jim

[RiaanBadenhorst]

Hi Jim,

RMAN database isn't a requirement :)