03-12-2019 07:27 PM
After netbackup upgrade from 7.7.3 to 8.1.1 RMAN backup logs are not getting deleted after completion of backup.
We have netbackup master win 2012 R2 with NBU version 8.1.1
and 2 Linux ( RHEL ) media servers with NBU version 8.1.1
and RMAN clients are solaris containers with OS version 5.10 and few are 5.11
We are running backup with separate backup nic.
Below is the RMAN log message which was provided by oracle team. But from our Netbackup console all parent and child shows as success.
RMAN-03009: failure of backup command on ORA_SBT_TAPE_7 channel at 03/10/2019 20:36:06
ORA-19509: failed to delete sequential file, handle="ARCH_Backup_IOP_set247628_piece1_copy1_20190310_qcts43g4_1_1", parms="ENV=(NB_ORA_POLICY=DB_RMAN_QA1_ORAARCH,NB_ORA_SERV=backupmaster01.domain.com)"
ORA-27027: sbtremove2 returned error
ORA-19511: Error received from media manager layer, error text:
Failed to remove, ARCH_Backup_IOP_set247628_piece1_copy1_20190310_qcts43g4_1_1, from image catalog.
ORA-27191: sbtinfo2 returned error
03-12-2019 08:16 PM
I am wondering which part of RMAN script is deleting the log, is it DELETE OBSOLETE?
Can you check if you did not have NBU Catalog running at the same time, I seen this issue when NBU Catalog is happening and RMAN cleans out some old backups through expiry - this is not allowed during Catalog backups
03-12-2019 08:17 PM
Oh, there is also a relevant technote https://www.veritas.com/support/en_US/article.100015746
03-12-2019 09:07 PM
It is happening all the time when no catalog backup running also same error. We have enabled bprd logs and reviewd and we don't see any clues. This is happening after the upgrade of master and media servers from 7.7.3 to 8.1.1.
03-12-2019 09:09 PM
Netbackup client versions are at 7.6 and 7.7 version
03-12-2019 09:36 PM
Ok, but you still did not say which RMAN command gives you grief. It is important to establish which command gives the error.
03-12-2019 10:25 PM - edited 03-12-2019 10:26 PM
Checked with oracle team they are not using command DELETE OBSOLETE in the script. Attached the script log.
We are facing same issue in older oracle databases 9i version backup is configured using crontab. Once log backup is completed there is no message appearing backup completed successfully to delete the backed up logs due to which backgound cron jobs not getting terminated for every hour.
ps -ef | grep script
oracle 19999 19978 0 11:20:01 ? 0:09 /bin/ksh /u01/home/oracle/scripts/arch_to_disk/arch_netback_1.ksh tidq D
oracle 16797 16777 0 15:20:01 ? 0:06 /bin/ksh /u01/home/oraclel/scripts/arch_to_disk/arch_netback_1.ksh kidq
oracle 24893 24852 0 07:20:01 ? 0:12 /bin/ksh /u01/home/oracle/scripts/arch_to_disk/arch_netback_1.ksh kiqr D
03-12-2019 10:38 PM
Sorry but troubleshooting does not work this way - somebody need to do the ground work and analyse the bphdb log and check which part of the script produces the error. The script itself is useful but does not point to the piece which breaks.
03-12-2019 11:27 PM
Could you please share the script, the full RMAN log as well as dbclient log on the Oracle client?
03-13-2019 12:14 AM
Wait!
I think you have realized where the problem is:
@bkrishna11 wrote:
Checked with oracle team they are not using command DELETE OBSOLETE in the script. Attached the script log.
We are facing same issue in older oracle databases 9i version backup is configured using crontab. Once log backup is completed there is no message appearing backup completed successfully to delete the backed up logs due to which backgound cron jobs not getting terminated for every hour.
ps -ef | grep script
oracle 19999 19978 0 11:20:01 ? 0:09 /bin/ksh /u01/home/oracle/scripts/arch_to_disk/arch_netback_1.ksh tidq D
oracle 16797 16777 0 15:20:01 ? 0:06 /bin/ksh /u01/home/oraclel/scripts/arch_to_disk/arch_netback_1.ksh kidq
oracle 24893 24852 0 07:20:01 ? 0:12 /bin/ksh /u01/home/oracle/scripts/arch_to_disk/arch_netback_1.ksh kiqr D
You need to trace the script output and NBU logs to see where it is getting stuck.
NBU logs:
dbclient on Oracle client
bprd and bpdbm on the master server (NBU needs to be restarted after these logs are created).
Have a look at this TN : https://www.veritas.com/content/support/en_US/article.100000534.html
This TN explains why image cleanup takes a long time and is not directly related to 8.x version:
_%t'
as the end of the format statement?03-13-2019 05:15 AM
03-13-2019 09:13 AM
Hi Mikail,
thanks for your suggestion, i would be generating required logs and will share.
03-13-2019 09:26 AM - edited 03-13-2019 09:27 AM
.Hi Marianne,
I have gone through the provided link for catalaog and image expiry. But this is not new configuration and same RMAN script we are using for years and all of sudden after upgrading to master and media servers from 7.7.3 to 8.1.1 immedeatly issue started.
Client contains multiple nics but we are using backup nic for data transfer.
I will generate dbclient log and provide output.
03-14-2019 12:03 AM
Hi Amol
( Inorder to isolate where the problem
lies I would suggest start by commenting out the last line i.e.” delete expired” from your script. Since crosscheck archivelogs would still be included after the backup runs you can connect to rman and do a list backup and determine how many pieces were marked as expired and trigger a manual delete expired archivelog command and monitor the time taken.)
I will check DBA team on this.
I have attached dbclient log error messages in attached notepad i could see error messages like not deleting images from catalog.
We have configured client with backup interface name in policy so what would be order in bp.conf file
it should be
backupmaster01
backupmaster01-backup
or vice versa I mean first entry should be master server name in bp.conf in this case it should be production interface or backup interface.
03-14-2019 12:41 AM
Hello,
this type of problem is better to solve with support - especially when the only change which led to the error was the upgrade.
I have quickly browsed the discussion and have several notes:
- use backup NIC names in REQUIRED_INTERFACE parameter only. Dont use them as policy client names, NB_ORA_CLIENT values etc.
- AFAIK backup NIC is used for data flow only, not for metadata flow. So it makes no sense to use it also for Master Server
Regards
Michal