Forum Discussion

Arshad_Khateeb's avatar
Arshad_Khateeb
Level 5
10 years ago

Oracle Database backup hungs!

Configuration Setup

Master Server - SunOS 5.9     UNIX    Master Server    6.5.6    Connected

Media Server - SunOS 5.10     UNIX    Media Server    6.5.6    Connected

Client - SunOS 5.9     UNIX    Client    6.5.6    Connected

 

Since last month we are experiencing this oracle db backup issue. All the child job used to complete except parent. It just gets hung.

dbclient log

> tail log.051915 
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - Media Information for Backup File : <iwhprod_20150518214547_db_a8q7ao2l_1_1>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - Media Sharing Mode : <Multiple Concurrent Users>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - File Ordering Mode : <Sequential file access>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - Media ID : <AL4417>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - File Creation Date and Time : <1432045250>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - File Expiration Date and Time : <1440080450>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - Comment : <Backup ID : rome_pres-bu_1432045250>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - File Creation Method : <Stream>
10:52:20.027 [10505] <2> int_DumpSbtInfo: INF - leaving
10:52:20.027 [10505] <2> sbtinfo2: INF - leaving

 

rman log

> tail iwhprod_20150518214547_db_bkup_inc0.log
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00
RMAN-06731: command backup:100.0% complete, time left 00:00:00

 

 

 

7.1.x and Earlier
Agent for Oracle on Windows or Linux Servers
Backup and Recovery
NetBackup
Solaris
  • Nicolai's avatar
    Nicolai
    Moderator
    10 years ago

    Nothing from the netbackup side you can do if RMAN hangs. Tell the DBA to look into it.

    And by the way: Netbackup 6.5 is out of service for a long time, time for upgrade.

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago
    With OS and NBU versions this old and no longer supported, my guess is that the same may be true for the Oracle version.... Probably good idea to check physical and logical resources on the client. With the environment this old, chances are that the aging Oracle server is battling with data growth.
  • Arshad_Khateeb's avatar
    Arshad_Khateeb
    Level 5
    10 years ago

    Thanks Nicolai and Marianne for your inputs!!!

    I do agree about the NBU environment being old. Our proposal is on table to upgarde NBU to 7.6.x. We'll be doing this upgrade in next quarter.

    Regarding DBA need to check from their end.......they are checking and let me know what they should check or what we need here to know.

    Yesterday, we upgraded the memory on client bcoz it has lots of jobs running on daily basis. We thought that it might be intruppting the backups to complete.

    As stated above that, all child streams complete and its just parent job which hung everytime.

  • Arshad_Khateeb's avatar
    Arshad_Khateeb
    Level 5
    10 years ago

    They did their checks and nothing appeared to be holding the server except backup. 

    connected; connect time: 0:00:00   -    This is where the parent backup job hung in GUI

    DBA too have a case open with Oracle. Looks like we can't do much from NBU :(

     

  • Nicolai's avatar
    Nicolai
    Moderator
    10 years ago

    They will need to do a trace on the RMAN process to see what is going.  Google "RMAN trace"

  • Arshad_Khateeb's avatar
    Arshad_Khateeb
    Level 5
    10 years ago

    DBA did RMAN trace while testing the Database backups. Here is where it started showing errors.....

    ----------------------------------------------------------------------------------------------------------------------------------------

    RMAN-06731: command backup:4.1% complete, time left 07:53:47
    RMAN-03009: failure of backup command on CH01 channel at 05/19/2015 22:31:16

    ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
    ORA-19511: Error received from media manager layer, error text:
       Failed to process backup file <iwhprod_20150519221015_db_a9q7c1nb_1_1>
    ORA-19502: write error on file "iwhprod_20150519221015_db_a9q7c1nb_1_1", blockno 138650625 (blocksize=512)
    ORA-27030: skgfwrt: sbtwrite2 returned error
    ORA-19511: Error received from media manager layer, error text:
       VxBSASendData: Failed with error:
       Server
    RMAN-12018: channel CH01 disabled, job failed on it will be run on another channel
    RMAN-03009: failure of backup command on CH02 channel at 05/19/2015 22:31:16

    ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
    ORA-19511: Error received from media manager layer, error text:
       Failed to process backup file <iwhprod_20150519221015_db_aaq7c1nb_1_1>
    ORA-19502: write error on file "iwhprod_20150519221015_db_aaq7c1nb_1_1", blockno 169535489 (blocksize=512)
    ORA-27030: skgfwrt: sbtwrite2 returned error
    ORA-19511: Error received from media manager layer, error text:
       VxBSASendData: Failed with error:
       Server
    RMAN-12018: channel CH02 disabled, job failed on it will be run on another channel
    RMAN-03009: failure of backup command on CH03 channel at 05/19/2015 22:31:16

    ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
    ORA-19511: Error received from media manager layer, error text:
       Failed to process backup file <iwhprod_20150519221015_db_abq7c1nb_1_1>
    ORA-19502: write error on file "iwhprod_20150519221015_db_abq7c1nb_1_1", blockno 143012865 (blocksize=512)
    ORA-27030: skgfwrt: sbtwrite2 returned error
    ORA-19511: Error received from media manager layer, error text:
       VxBSASendData: Failed with error:
       Server
    RMAN-12018: channel CH03 disabled, job failed on it will be run on another channel
    RMAN-08031: released channel: CH01
    RMAN-08031: released channel: CH02
    RMAN-08031: released channel: CH03
    RMAN-08031: released channel: CH04
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03009: failure of backup command on CH04 channel at 05/19/2015 22:31:17

    ORA-27192: skgfcls: sbtclose2 returned error - failed to close file

    ORA-19511: Error received from media manager layer, error text:
       Failed to process backup file <iwhprod_20150519221015_db_acq7c1nb_1_1>
    ORA-19502: write error on file "iwhprod_20150519221015_db_acq7c1nb_1_1", blockno 129326081 (blocksize=512)
    ORA-27030: skgfwrt: sbtwrite2 returned error
    ORA-19511: Error received from media manager layer, error text:
       VxBSASendData: Failed with error:
       Server

    RMAN> 

    Recovery Manager complete.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Database INC0 Backup of database iwhprod on server rome_pres
    The RMAN script completed with a RETURN CODE of 1
    End Time is 05/19/2015 22:31:19

  • Will_Restore's avatar
    Will_Restore
    Level 6
    10 years ago

    Similar problem in this old thread

    https://www-secure.symantec.com/connect/forums/oracle-fullbackups-status-6

    A few suggestions were offered but the originator never came back to mark a solution.