Client Oracle error 245

[issmag]

hello,

i have a probleme on one Oracle Client. at 19h30 the backup return the error 245.

the backup is launched by a RMAN Script. the schedule is an application Schedule (Tivoli 8.3).

the server client is AIX 6.1 Netbackup client version is 7.5

the Master server is Windows 2008 R2. Netbackup server version is 7.5.0.6

in this policy we have 47 clients (all of them pass without any problem, and with the same script

the same client is backing up successfully with the same policy at different time. when we change the backup time for the same instance at 20h30 the backup pass successfully !!!

there is no logs in bpdhp folder. it's empty.

please find in the attachement the dbclient log

) .

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

the script is :

$ORACLE_HOME/bin/rman "TARGET=/" "CATALOG=rman/rman@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vob.app)(PORT=1522))(CONNECT_DATA=(SI

D=rmax)))" CMDFILE='/production/Exploit/vob/log/RMAN_BCKP_DB_INCR/20150225193007042610/000001.log' LOG='/production/Exploit/vob/log

/RMAN_BCKP_DB_INCR/20150225193007042610/000002.log' APPEND >>/production/Exploit/vob/log/RMAN_BCKP_DB_INCR/20150225193007042610/000

003.log 2>&1

RMAN> 2> 3> 4> 5> 6>

contenu du fichier /production/Exploit/vob/log/RMAN_BCKP_DB_INCR/20150225193007042610/000001.log

run {

  allocate channel chnl1 device type="SBT_TAPE";

  backup incremental level=1 cumulative as backupset format="%d_DB_INCR_20150225193007_%U_%t" tag "DB_INCR_DB" database plus archiv

elog tag "DB_INCR_ARCH" delete all input;

  backup current controlfile format="%d_DB_INCR_20150225193007_%U_%t" tag "DB_INCR_CONTROL";

}

thanks for your help

[Michael_G_Ander]

What policy type does the job details show in the 1930 job ?

bphdb is empty as you start the backup outside netbackup control, normally bphdb starts the scrip calling rman

A little confused as to where Tivoli fit in this

[issmag]

hello,

the policy type is Oracle

[Marianne]

We need to see the cmd file to see if policy and/or client name is specified. The cmd file will hopefully explain why we are seeing this error: serverResponse: ERR - server exited with status 245: the specified policy is not of the correct client type One more thing : we need bprd log on the master server. This will show us client connection attempt and master's attempt to find requested policy for resolved client name. We need the section of bprd log around the time that the failure is seen.

[issmag]

hello,

here is the policy propreties :

C:\>bppllist 4_Oracle -U
------------------------------------------------------------

Policy Name:       4_Oracle

  Policy Type:         Oracle
  Active:              yes
  Effective date:      06/03/2013 11:00:37
  Block Incremental:   no
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           4_BRT_1An
  Volume Pool:         NetBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    yes
  Application Discovery:      no
  Discovery Lifetime:      28800 seconds
ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  Linux         SuSE2.6.16    s2-in0pj-a006
                 RS6000        AIX61         UXEAUFR157B-v1907
                 RS6000        AIX61         UXEAUFR159-v1907
                 RS6000        AIX61         UXEAUFR646-v1907
                 Linux         RedHat2.6.18  UXEAUFR681A-v1901
                 Linux         RedHat2.6.18  UXEAUFR682A-v1901
                 RS6000        AIX61         UXVESYN641-v1907
                 RS6000        AIX53         uxeaufr153-v1907
                 Linux         SuSE2.6.16    uxeaufr180-v1901
                 Linux         RedHat2.6.18  uxeaufr188a-v1901
                 Linux         RedHat2.6.18  uxeaufr183-v1901
                 Solaris       Solaris9      UXVESYN601-v1901
                 Solaris       Solaris9      UXEAUFR700-v1901
                 RS6000        AIX61         uxeaufr165-v1907
                 HP9000-700    HP-UX11.23    VNSDB1-v1908
                 Linux         SuSE2.6.16    s2-in0ej-a007-v1901
                 Linux         SuSE2.6.16    s2-in0pj-a006-v1901
                 RS6000        AIX53         S3-IN0DJ-D000-v1907
                 RS6000        AIX53         S3-IN0DJ-D001-v1907
                 RS6000        AIX53         S3-IN0IJ-D000-v1907
                 Linux         SuSE2.6.16    UVBACKO191-v1901
                 Linux-IA64    SuSE2.6       uvbacko581-v1901
                 Linux         RedHat2.6.18  uveaufr286-v1901
                 Linux         RedHat2.6.18  uveaufr847-v1901
                 RS6000        AIX61         UXBACKO140-v1907
                 RS6000        AIX61         UXBACKO142-v1907
                 RS6000        AIX53         UXBACKO143-v1907
                 RS6000        AIX61         UXBACKO144-v1907
                 RS6000        AIX61         UXBACKO145-v1907
                 RS6000        AIX53         UXBACKO146-v1907
                 RS6000        AIX53         UXBACKO147-v1907
                 RS6000        AIX53         UXBACKO148-v1907
                 Linux         SuSE2.6.16    uxbacko387-v1901
                 RS6000        AIX53         UXBACKO650-v1907
                 RS6000        AIX61         UXBACKO651-v1907
                 Solaris       Solaris9      UXDEVRE2-v1901
                 RS6000        AIX61         S3-IN0EJ-EA04-v1907
                 RS6000        AIX61         S3-IN4DJ-GT02-v1907
                 RS6000        AIX53         psiuxeaufr143-v1907
                 RS6000        AIX53         psiuxeaufr147-v1907
                 RS6000        AIX53         psiuxeaufr148-v1907
                 RS6000        AIX53         psiuxeaufr149-v1907
                 RS6000        AIX53         psiuxeaufr155-v1907
                 RS6000        AIX53         psiuxeaufr160-v1907
                 RS6000        AIX61         S3-IN0EJ-MM01-v1907
                 RS6000        AIX61         S3-IN0EJ-B000-v1907
                 RS6000        AIX61         uxeaufr158a-v1907

  Include:      (none defined)

  Schedule:              ORACLEBACKUP
    Type:                Application Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     0 (1 week)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Sunday     00:00:00  -->  Sunday     24:00:00
          Monday     00:00:00  -->  Monday     24:00:00
          Tuesday    00:00:00  -->  Tuesday    24:00:00
          Wednesday  00:00:00  -->  Wednesday  24:00:00
          Thursday   00:00:00  -->  Thursday   24:00:00
          Friday     00:00:00  -->  Friday     24:00:00
          Saturday   00:00:00  -->  Saturday   24:00:00

C:\>

the server on error is uxeaufr158a-v1907

please find in the attachement the bprd logs.

[Amol_Nair]

is there any other policy created for this same client i.e. "uxeaufr158a-v1907" which is not an Oracle policy. I mean is there a Standard policy type as well for this client to take filesystem backups of this client. if yes then add SEND parameters in the Rman script to ensure that the child jobs in backup uses the correct policy name. ensure that you specify the following 3 parameters in the Send parameters in the rman script. NB_ORA_POLICY, NB_ORA_CLIENT and NB_ORA_SERV

[Marianne]

dbclient log posted above is for 022315 (23 Feb).

bprd logs posted in zip-file are for 021915 (19 Feb), 022415 (24 Feb) and 022515 (25 Feb).

We need dbclient log on the client and bprd log on the master for the same date in order to cross-check on master when we see status 245 in dbclient log.

[issmag]

hello,

thanks for your return,

marianne, the bprd log 022415 has been writen at the same date as the dbclient log...

[Marianne]

There is no record of status 245 in  bprd log 022415. 

Is there a big time difference between the master and this client?

Is this the only client that fails intermittently?

The reason for my questions is because there are LOTS of hostname-lookup-related errors and network errors in bprd logs.

Some examples:

<16> connected_peer: vnet_cached_getnameinfo() for 10.161.71.108 failed, Unknown error (11001) 

<16> connected_peer: vnet_cached_getnameinfo() for 10.160.0.78 failed, Unknown error (11001)

<16> connected_peer: vnet_cached_getnameinfo() for 10.10.135.78 failed, Unknown error (11001)

<16> connected_peer: vnet_cached_getnameinfo() for 10.161.9.192 failed, Unknown error (11001)

<16> connected_peer: vnet_cached_getnameinfo() for 10.161.9.193 failed, Unknown error (11001) 

00:00:51.565 [21224.22820] <32> process_request: get_string() failed - network read error (10053)
00:00:51.565 [13420.17300] <32> process_request: get_string() failed - network read error (10053)
00:00:51.565 [16828.20272] <32> process_request: get_string() failed - network read error (10053)
00:00:51.565 [14540.15564] <32> process_request: get_string() failed - network read error (10053)

[issmag]

sorry marianne, the log are no longer avalaible :(

the customer changes the job time to 20:30, so the backup works... but, he want to know and to correct the issue..

could you please, check the log 021915, there is many entries for the client uxeaufr158a-v107 at  20:29

thanks a lot.

[Marianne]

I have checked all of the bprd logs - no sign of status 245 in any of them.

All I can suggest is to collect matching logs if/when the problem is seen again.
We need the matching bprd entries when error is seen in dbclient log.

[issmag]

hello,

here is in  the attachement, a newer logs (client : dbclient) and from  master bprd  log.

thanks a  lot for your help.

[Marianne]

What time was the failure seen?  And on which day?
It takes up way too much time when we don't know where to look...

These files are big and we try to help inbetween our normal day jobs.

We just need one dbclient and corresponding bprd log that will contain evidence of failure.

The logs you have posted are not for the same days:
dbclient : 9 and 10 March.
bprd: 8 March.

[issmag]

hello,

the error occured on 09 and 10 march.

the bprd content the 09 and 10 March. (please see the modification propreties).

[Marianne]

How does bprd log for 8 March contain client connections on 9 and 10 March? You are welcome to check the logs and post here when you find something.

[issmag]

hello,

appologies marianne,

so please find in the attachement, the 09 march bprd's log.

[issmag]

hello,

any update please ?

[Marianne]

bprd log file is massive. 
Just trying to unzip it makes my little 4GB ram laptop hang....

All of us trying to help here have busy day time jobs.

As per my previous post - best if you find the relevant sections in dbclient and bprd logs and then copy & paste it in a separate file.
Or log a support call with Symantec.
They have staff and tools to look at massive logs...