Backup job failed with EC 130(system error occurred)

[dixit47]

Oracle Policy failed with EC 130(system error occurred)

Detailed status:

05/11/2015 05:02:14 - Error bprd (pid=19352) Unable to write progress log </usr/openv/netbackup/logs/user_ops/dbext/logs/3051.0.1431338533> on client xxx.com. Policy=XXX_ORA_ARCH Sched=NONE
05/11/2015 05:02:14 - Error bprd (pid=19352) CLIENT XXX.com  POLICY XXX_ORA_ARCH  SCHED NONE  EXIT STATUS 130 (system error occurred)

OS /Version :

Master server :NetBackup-Solaris10 7.6.0.4
Media server : NetBackup-RedHat2.6.18 7.6.0.1
Client server : NetBackup-RedHat2.6.18 7.6.0.1

Experts : Could you please advice !!!

[dixit47]

Getting below RMAN logs :

RMAN-03009: failure of backup command on ORA_SBT_TAPE_1 channel at 05/11/2015 10:08:43
ORA-19506: failed to create sequential file, name="df_s22029_p1_t879382947", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
channel ORA_SBT_TAPE_1 disabled, job failed on it will be run on another channel
RMAN-03009: failure of backup command on ORA_SBT_TAPE_2 channel at 05/11/2015 10:08:43
ORA-19506: failed to create sequential file, name="df_s22030_p1_t879382947", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
channel ORA_SBT_TAPE_2 disabled, job failed on it will be run on another channel
RMAN-03009: failure of backup command on ORA_SBT_TAPE_3 channel at 05/11/2015 10:08:43
ORA-19506: failed to create sequential file, name="df_s22031_p1_t879382947", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
channel ORA_SBT_TAPE_3 disabled, job failed on it will be run on another channel
RMAN-03009: failure of backup command on ORA_SBT_TAPE_4 channel at 05/11/2015 10:08:43
ORA-19506: failed to create sequential file, name="df_s22032_p1_t879382947", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
channel ORA_SBT_TAPE_4 disabled, job failed on it will be run on another channel
RMAN-03009: failure of backup command on ORA_SBT_TAPE_5 channel at 05/11/2015 10:08:43
ORA-19506: failed to create sequential file, name="df_s22033_p1_t879382947", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
channel ORA_SBT_TAPE_5 disabled, job failed on it will be run on another channel
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_SBT_TAPE_6 channel at 05/11/2015 10:08:43
ORA-19506: failed to create sequential file, name="df_s22034_p1_t879382947", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve

Experts : can you help to fix this EC 130 issue

[dixit47]

Experts : can anyone help me to fix this EC 130 issue ?

[CRZ]

Previous cases reporting this issue discovered they were running up on limits set in the /etc/security/limits.conf file.  Could you check this file on the client and see if there are any settings for the Oracle user in there (like nproc)?

[Marianne]

Check comms between client and master : forward and reverse name lookup as well as port connectivity on PBX (port 1556). Check permissions on dbext log folder - it needs 777 permission. If dbclient log folder was created, ensure 777 permissions on this folder as well.

[dixit47]

@Criz :

o/p of file :

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file
@oinstall             soft    nofile          65536
@oinstall             hard    nofile          65536
@oinstall             soft    nproc           16384
@oinstall             hard    nproc           16384
@oinstall             soft    stack           32768
@oinstall             hard    stack           unlimited

Please suggest

[dixit47]

@Marianne : i saw your updates on another forum for same issue, i have checked comms between client and master : forward and reverse name lookup as well as port connectivity on PBX (port 1556).

dbext log folder : drwxrwxrwt : 777 Permission

yeah i have created the dbclient log folder with full permission

kindly suggest next course of action to fix this issue

[dixit47]

@Criz : Checked /etc/security/limits.conf file

o/p for ur ref. :

 /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file
@oinstall             soft    nofile          65536
@oinstall             hard    nofile          65536
@oinstall             soft    nproc           16384
@oinstall             hard    nproc           16384
@oinstall             soft    stack           32768
@oinstall             hard    stack           unlimited

kindly suggest !!

[Marianne]

So... then you know what I'm going to ask for next... bprd log on the master dbclient log on the client. Copy logs to .txt files and upload as file attachment. Curious to know if this has ever worked?

[dixit47]

Please find dbclient logs

[dixit47]

and bprd logs and let me know what need to be done to fix this issue

[Pangal]

Hi,

        I can see the following warning in db client log "NBBSA_CLIENT_HOST name= <Instance> differs from NBU config name"

     Make sure client name are exactly same in bp.conf file and one used by rman .
 

[dixit47]

@pangal : Both names are same(RMAN & bp.conf entries), Already checked !!

[Pangal]

Hi,

 1) I guess i have seen this error in my environment , it is communication issue ,Execute bpclntcmd commands on client to check connectivity

   Client name should be exactly same everwhere (With same case  ) bpconf file, rman and netbackup policy

   if shortname( everwhere shortname should be used), if full name( everywhere fullname should be used)

 2)  Also check the bpps output ,there might be stuck process on the client ,

[Marianne]

Was backup still in progress when you collected dbclient log? I don't see any failures. Only successful archive log and control file backups. I do not see any attempt to backup db's as per your opening post : "df_s22034_p1_t879382947" at 10:08. First backup attempt in dbclient log was shortly after 11:32 for archive logs. You need to show us dbclient log for failed backup attempt.