06-13-2017 02:21 AM - edited 06-13-2017 03:09 AM
The oracle database backups using the Rman script are failing with status 29 for the parent stream and status 21 for the child streams.
It was working fine and all of a sudden since the last weekend, it has started failing.
I have attached the detailed status of the failed job along with the bphdb and oracle logs from the client server as well as the bptestbpcd, bpclntcmd command output from the master, media and client server.
Master Server OS: Linux NBU Version: 8.0
Media Server OS: Netbackup Appliance 2.7.3 NBU Version: 7.7.3
client Server OS: Win Server 2008 R2 NBU Version: 7.6.1
BPHDB job log
ERR - failed executing command <"D:\oracle\local\bin\cscorafullbackup.bat">
21:16:43.658 [13788.16732] <16> bphdb: ERR - exit status: <1>
21:16:43.658 [13788.16732] <16> bphdb: ERR - Error code: 29
21:16:43.658 [13788.16732] <16> bphdb: ERR - bphdb exit status = 29: failed trying to exec a command
RMAN-03009: failure of backup command on ch00 channel at 06/12/2017 21:22:06
ORA-19506: failed to create sequential file, name="HOTDB_BSLDB_urqs6kurf_s27514_p1_t946502511", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: socket open failed
Recovery Manager complete.
Return code: 1
Can anybody please help me with the issue.
06-13-2017 03:18 AM
Was something changed in the weekend either on the Oracle database or the Windows machine ? Talk with your DBA and OS administrators, if you havn't already.
Seen Oracle upgrade/patchning cause issues and the same with Windows patches, especially security ones.
Does the stderr file in the bphdb folder contain anything ?
I would try to run D:\oracle\local\bin\cscorafullbackup.bat by hand to see if there was some output, that indicated the problem with the script.
06-13-2017 08:29 PM
After some investigation it has been found that the netbackup process
"C:\Program Files\Veritas\pdde\mtstrmd.exe" --config "C:\Program Files\Veritas\NetBackup\bin\ost-plugins\mtstrm.conf" was holding one port open.
And after having restarted the service associated with this process: “NetBackup Deduplication Multi-Threaded Agent” and launching the backups, the database backups seems to be running fine now and we aren't getting the "Socket Open Failed" error message any more.
I believe restarting the service freed the port and the backups are running fine now.
Many thanks for your help.