cancel
Showing results for 
Search instead for 
Did you mean: 

Oracle Backups failing with status 6

Andre_Torres
Level 5
Partner Accredited

Hi everyone,

 

I've migrated from an old Red Hat Master Server to a new NetBackup Appliance 5230 Master Server. I've done the following procedure to do the backups on new server:

1)Stop NBU services on client
2)Change SERVER entry on bp.conf on client pointing to new Master Server
3)Change hosts file adding IP of new Master Server
4)Start NetBackup services on client
5)Run bpclntcmd -clear_host_cache
 
Tested doing a file system backup and it worked. But when I try to do an Oracle Backup it gives me error 6
 
error6.png
 
As if it is a Script error, I've contacted the DBA Oracle and he said it gives him this error:
 
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch00 channel at 01/07/2014 02:03:24
ORA-19506: failed to create sequential file, name="bkp_dbf_587_1_836185699", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
 
 
 
Just for you all to know it worked on my old server, the only changes were the ones done which I said before.
23 REPLIES 23

sri_vani
Level 6
Partner

Please verify this link: http://www.symantec.com/business/support/index?page=content&id=TECH147617

If nt resolved your issue please attach dbclient log in .txt format

 

Solution

This issue's cause was incorrect permissions on the bp.conf file.

Modification of the permission to 644 resolved the problem.

 

Will_Restore
Level 6

client RMAN script  NB_ORA_SERV=   entry needs to point to the new Master

sri_vani
Level 6
Partner

yes

Andre_Torres
Level 5
Partner Accredited

sri vani, it was already on permission 644.

 

 

dbclient log:

 

01:28:44.143 [9100] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
01:28:44.143 [9100] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
01:28:44.143 [9100] <2> int_GetMMInfo: INF - Initialized Signal
01:28:44.143 [9100] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
01:28:50.154 [9100] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
01:28:50.155 [9100] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
01:28:51.156 [9100] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
01:28:51.156 [9100] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
01:28:54.160 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
01:28:54.162 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
01:28:54.162 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
01:28:54.163 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
01:28:54.167 [9100] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:28:54.168 [9100] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.15929 TO 10.27.253.249.1556 fd = 17
01:28:54.193 [9100] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
01:28:54.197 [9100] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:28:54.198 [9100] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.62601 TO 10.27.253.249.1556 fd = 18
01:29:28.626 [9247] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
01:29:28.626 [9247] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
01:29:28.626 [9247] <2> int_GetMMInfo: INF - Initialized Signal
01:29:28.626 [9247] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
01:29:34.635 [9247] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_arc>
01:29:34.635 [9247] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_arc> will be used for this API session.
01:29:35.635 [9247] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-arc>
01:29:35.636 [9247] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-arc> will be used for non-proxy API session.
01:29:39.641 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
01:29:39.643 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
01:29:39.643 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
01:29:39.644 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
01:29:39.646 [9247] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:29:39.647 [9247] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.29615 TO 10.27.253.249.1556 fd = 17
01:29:39.672 [9247] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
01:29:39.677 [9247] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:29:39.678 [9247] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.33800 TO 10.27.253.249.1556 fd = 18
01:43:56.740 [9100] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/9100.0.1389065334
01:43:56.740 [9100] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/9100.0.1389065334>
01:43:56.740 [9100] <16> CreateNewImage: ERR - serverResponse() failed
01:43:56.740 [9100] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_585_1_836184533.
01:43:56.740 [9100] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
01:43:59.744 [9100] <4> sbtend: INF - --- END of SESSION ---
01:43:59.745 [9100] <8> close_image: Session being terminated abnormally, cleaning up
01:43:59.745 [9100] <4> close_image: INF - backup FAILED
01:43:59.745 [9100] <4> close_image: INF ---- end of Backup ---
 
01:43:59.745 [9100] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
01:43:59.745 [9100] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
01:44:41.901 [9247] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/9247.0.1389065379
01:44:41.901 [9247] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/9247.0.1389065379>
01:44:41.901 [9247] <16> CreateNewImage: ERR - serverResponse() failed
01:44:41.901 [9247] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_arch_586_1_836184579.
01:44:41.901 [9247] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
01:44:45.913 [9247] <4> sbtend: INF - --- END of SESSION ---
01:44:45.914 [9247] <8> close_image: Session being terminated abnormally, cleaning up
01:44:45.914 [9247] <4> close_image: INF - backup FAILED
01:44:45.914 [9247] <4> close_image: INF ---- end of Backup ---
 
01:44:45.914 [9247] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
01:44:45.914 [9247] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
01:48:09.890 [11482] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
01:48:09.891 [11482] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
01:48:09.891 [11482] <2> int_GetMMInfo: INF - Initialized Signal
01:48:09.891 [11482] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
01:48:15.901 [11482] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
01:48:15.901 [11482] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
01:48:16.903 [11482] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
01:48:16.903 [11482] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
01:48:19.908 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
01:48:19.911 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
01:48:19.911 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
01:48:19.912 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
01:48:19.922 [11482] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:48:19.923 [11482] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.39060 TO 10.27.253.249.1556 fd = 17
01:48:19.947 [11482] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
01:48:19.951 [11482] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:48:19.951 [11482] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.27983 TO 10.27.253.249.1556 fd = 18
02:03:22.281 [11482] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/11482.0.1389066499
02:03:22.281 [11482] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/11482.0.1389066499>
02:03:22.281 [11482] <16> CreateNewImage: ERR - serverResponse() failed
02:03:22.281 [11482] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_587_1_836185699.
02:03:22.282 [11482] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
02:03:25.287 [11482] <4> sbtend: INF - --- END of SESSION ---
02:03:25.287 [11482] <8> close_image: Session being terminated abnormally, cleaning up
02:03:25.287 [11482] <4> close_image: INF - backup FAILED
02:03:25.287 [11482] <4> close_image: INF ---- end of Backup ---
 
02:03:25.287 [11482] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
02:03:25.287 [11482] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
11:06:10.992 [31600] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
11:06:10.992 [31600] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
11:06:10.992 [31600] <2> int_GetMMInfo: INF - Initialized Signal
11:06:10.992 [31600] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
11:06:17.001 [31600] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
11:06:17.001 [31600] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
11:06:18.003 [31600] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
11:06:18.003 [31600] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
11:06:21.010 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
11:06:21.012 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
11:06:21.012 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
11:06:21.013 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
11:06:21.017 [31600] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:06:21.017 [31600] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.59670 TO 10.27.253.249.1556 fd = 17
11:06:21.042 [31600] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
11:06:21.046 [31600] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:06:21.047 [31600] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.18961 TO 10.27.253.249.1556 fd = 18
11:21:22.870 [31600] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/31600.0.1389099981
11:21:22.870 [31600] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/31600.0.1389099981>
11:21:22.870 [31600] <16> CreateNewImage: ERR - serverResponse() failed
11:21:22.870 [31600] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_588_1_836219180.
11:21:22.870 [31600] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
11:21:25.875 [31600] <4> sbtend: INF - --- END of SESSION ---
11:21:25.875 [31600] <8> close_image: Session being terminated abnormally, cleaning up
11:21:25.875 [31600] <4> close_image: INF - backup FAILED
11:21:25.875 [31600] <4> close_image: INF ---- end of Backup ---
 
11:21:25.875 [31600] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
11:21:25.875 [31600] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
11:43:07.678 [2429] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
11:43:07.679 [2429] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
11:43:07.679 [2429] <2> int_GetMMInfo: INF - Initialized Signal
11:43:07.679 [2429] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
11:43:13.690 [2429] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
11:43:13.690 [2429] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
11:43:14.692 [2429] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
11:43:14.692 [2429] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
11:43:17.699 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/06b/c413626b+0,1,402,0,1,0+localhost.txt
11:43:17.700 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
11:43:17.701 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
11:43:17.702 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
11:43:17.703 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
11:43:17.707 [2429] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:43:17.708 [2429] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.48403 TO 10.27.253.249.1556 fd = 17
11:43:17.728 [2429] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
11:43:17.732 [2429] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:43:17.733 [2429] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.33124 TO 10.27.253.249.1556 fd = 18
11:44:18.420 [2582] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
11:44:18.421 [2582] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
11:44:18.421 [2582] <2> int_GetMMInfo: INF - Initialized Signal
11:44:18.421 [2582] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
11:44:24.430 [2582] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
11:44:24.430 [2582] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
11:44:25.432 [2582] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
11:44:25.432 [2582] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
11:44:28.436 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/06b/c413626b+0,1,402,0,1,0+localhost.txt
11:44:28.437 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
11:44:28.439 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
11:44:28.439 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
11:44:28.440 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
11:44:28.442 [2582] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:44:28.443 [2582] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.57942 TO 10.27.253.249.1556 fd = 17
11:44:28.466 [2582] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
11:44:28.470 [2582] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:44:28.471 [2582] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.32691 TO 10.27.253.249.1556 fd = 18
11:58:19.922 [2429] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/2429.0.1389102197
11:58:19.922 [2429] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/2429.0.1389102197>
11:58:19.922 [2429] <16> CreateNewImage: ERR - serverResponse() failed
11:58:19.922 [2429] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_589_1_836221396.
11:58:19.922 [2429] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
11:58:22.929 [2429] <4> sbtend: INF - --- END of SESSION ---
11:58:22.929 [2429] <8> close_image: Session being terminated abnormally, cleaning up
11:58:22.929 [2429] <4> close_image: INF - backup FAILED
11:58:22.929 [2429] <4> close_image: INF ---- end of Backup ---
 
11:58:22.929 [2429] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
11:58:22.929 [2429] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
11:59:30.536 [2582] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/2582.0.1389102268
11:59:30.536 [2582] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/2582.0.1389102268>
11:59:30.536 [2582] <16> CreateNewImage: ERR - serverResponse() failed
11:59:30.536 [2582] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_590_1_836221467.
11:59:30.536 [2582] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
11:59:33.542 [2582] <4> sbtend: INF - --- END of SESSION ---
11:59:33.542 [2582] <8> close_image: Session being terminated abnormally, cleaning up
11:59:33.542 [2582] <4> close_image: INF - backup FAILED
11:59:33.542 [2582] <4> close_image: INF ---- end of Backup ---
 
11:59:33.542 [2582] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
11:59:33.543 [2582] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
15:21:05.361 [26318] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
15:21:05.361 [26318] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
15:21:05.361 [26318] <2> int_GetMMInfo: INF - Initialized Signal
15:21:05.361 [26318] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
15:21:11.372 [26318] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
15:21:11.373 [26318] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
15:21:12.373 [26318] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
15:21:12.373 [26318] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
15:21:15.378 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
15:21:15.380 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
15:21:15.380 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
15:21:15.381 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
15:21:15.385 [26318] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:21:15.387 [26318] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.44735 TO 10.27.253.249.1556 fd = 17
15:21:15.408 [26318] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
15:21:15.412 [26318] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:21:15.413 [26318] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.65387 TO 10.27.253.249.1556 fd = 18
 

Andre_Torres
Level 5
Partner Accredited

It points a variable filled by bphdb, looking for it, it is pointing to my new server.

Andre_Torres
Level 5
Partner Accredited

Add the appliance master server name as a first server to the bp.conf file on the client.

-Done

 

Check that /etc/hosts entry for the master is correct.

-Done

 

what is the output of bpcltncmd -pn

expecting response from server ptrbkpapl01
ptrbkpapl01 ptrbkpapl01 10.27.240.63 46976
 
 

bpcltncmd -self  command from client

yp_get_default_domain failed: (12) Local domain name not set
NIS does not seem to be running: (1) Request arguments bad
gethostname() returned: titus
host titus: titus.petrobras.com.br at 10.27.246.198
aliases:     titus.petrobras.com.br     titus     10.27.246.198
getfqdn(titus) returned: titus.petrobras.com.br
 

 

and also check the logs of dbclient(on client)

-See attachment

 

and bprd on master server

-See attachment

sri_vani
Level 6
Partner

I see the below error on dbclient log:

<16> readCommFile: ERR - timed out after 900 seconds while reading from

can you plz try Increase the CLIENT_READ_TIMEOUT to 5400 to give more time and try a backup

sri_vani
Level 6
Partner

1)Add the appliance master server name as a first server to the bp.conf file on the client.

2)Check that /etc/hosts entry for the master is correct.

To verify if there any name resolution issues:

what is the output of 

bpcltncmd -pn

bpcltncmd -self  command from client

and also check the logs of dbclient(on client) and bprd on master server

 

Andre_Torres
Level 5
Partner Accredited

same error. I don't think it's a timeout error, because I can do a file system backup.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

In addition to what everyone else has said up to now - we can see that there is a comms problem between client and master server:

Client is trying to connect to bprd on master server via pbx (port 1556):

 logconnections: BPRD CONNECT FROM 10.27.252.41.39060 TO 10.27.253.249.1556 fd = 17

It gets no response from the master server:

Failed with error:    Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the server
Two things to check:
1. Port connectivity on port 1556 in both directions. If there is a firewall, check that this port is open.
2. Forward and reverse lookup in both directions.
Have you added hosts entry on the appliance for the client?
Can the master server resolve client's IP address to hostname that appears in the Oracle policy?
 
Please post bprd log file on the master server after backup attempt has failed.
Copy the log to bprd.txt and upload as File Attachment.
 

Andre_Torres
Level 5
Partner Accredited

Have you added hosts entry on the appliance for the client?

-Yes


Can the master server resolve client's IP address to hostname that appears in the Oracle policy?

-Yes

 

Please post bprd log file on the master server after backup attempt has failed.

Copy the log to bprd.txt and upload as File Attachment.

-Posted before, can't post it as .txt because in txt mode it's 25MB.

sri_vani
Level 6
Partner

Can you please try this :

Change the file modes of the following directories to 777 - "chmod 777 <path>" where <path> is:
/usr/openv/netbackup/logs/user_ops
/usr/openv/netbackup/logs/user_ops/dbext
/usr/openv/netbackup/logs/user_ops/dbext/logs and try backup

sri_vani
Level 6
Partner

I suspect this wld be the reason:

http://www.symantec.com/business/support/index?page=content&id=TECH196439

Review the NB_ORA_CLIENT, NB_ORA_POLICY, and NB_ORA_SCHED variables configured for use by RMAN and NetBackup.  Ensure they do not contain any spaces which will result in an argument parsing problem on the master server.

Andre_Torres
Level 5
Partner Accredited

These already were in permission 777

Andre_Torres
Level 5
Partner Accredited

No spaces. As I said before, it worked before on my old server, and NB_ORA_CLIENT, NB_ORA_POLICY, and NB_ORA_SCHED were not changed.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Please help us to understand the different IP addresses that we see in the logs.

In dbclient log, we see that the client IP address is 10.27.252.41, trying to connect to bprd at IP address 10.27.253.249. This IP address should the master's IP.

01:28:54.168 [9100] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.15929 TO 10.27.253.249.1556 fd = 17

 

When I look at bprd log, the master IP seems to be 10.27.240.63.

01:28:01.068 [27604] <2> logconnections: BPRD ACCEPT FROM 10.27.243.247.60434 TO 10.27.240.63.1556 fd = 9

 

You can see that the master is logging connections to itself on IP address 10.27.240.63.

What is weird about the backup attempt from the client at 01:28, (requesting backup of policy name petrobras_titus_db_ora_jbossrep_hot_full_dia)  is that we see a backup request for the SAME policy name from HOST=PTRWSVGW01 IPSTR=10.27.243.247 at more or less the same time.

01:29:19.094 [27822] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:1904] found via getnameinfo OUR_HOST=PTRWSVGW01 IPSTR=10.27.243.247
 
01:29:19.095 [27822] <2> process_request: immediate backup request = 329199 55 root other NONE PTRWSVGW01 PTRWSVGW01 NONE petrobras_titus_db_ora_jbossrep_hot_full_dia Diario_Full 0 0 13 0 C C C C C 0 0 -1 0 0

 

Can you help us to understand this descrepancy in IP addresses and hostnames?

Will_Restore
Level 6

Yes this is really confusing as none of those match this response

 

bpcltncmd -self  command from client

host titus: titus.petrobras.com.br at 10.27.246.198

Andre_Torres
Level 5
Partner Accredited

Both Master and Client have multiple vLans.

 

10.27.252.41 is the IP of Backup vLan in the client

 

10.27.253.249 is the IP of Backup vLan in my old Master Server

 

10.27.240.63 is the IP of Management vLan of my new Master Server.

 

 

Client should use 10.27.252.41 and Master should use 10.27.252.163

Will_Restore
Level 6

>>Client should use 10.27.252.41 and Master should use 10.27.252.163

 

Well in that case you need to set up name resolution so both Master and Client see themselves and each other that way as the current output shows otherwise.