Forum Discussion

Carawain's avatar
Carawain
Level 3
8 years ago

client backup daily error (status code41)

Hello,

I need help because one client got daily backup error (status 41).

Impacted policies are standard policy which backup oracle database folder and system policy with full schedule so when the amount of data to backup is high.

the bpbkar process hung when it arrive to "begin writing state".

sometimes there a trace of backup flow like 32 Kilobytes so less performance but the backup is so long and also canceled. 

i tried to redirect the backup policies on several other media server available but the issue is the same so i know that the issue is on client side.

i tried many policy/client configuration found on the foum topic:

-increse client timeout

-activate the multiple datastream

-as the database policies have to be encrypted i tried to disable the encryption option and check the server processor load but nothing monopolize cpu ressource.

-i tried to disable client/side deduplication too but no more emprouvement.

so bpbkar stay indefinitely active on the server until a timeout occurs with status code 41.

Master and media server are in 7.6.1.2 version

client is an AIX 5 3 in 7.5.0.7 version

 If anyone have an idea to reveal the issue?

Thanks.

 

7.5
AIX
  • Marianne's avatar
    Marianne
    8 years ago

    Snippets of logs do not help when full data transfer path needs to be followed.

    If you cannot upload the full logs here (as .txt files), then best to log a Support call with Veritas where you can submit/upload complete log files. 

     

10 Replies

Sort By
  • Michael_G_Ander's avatar
    Michael_G_Ander
    Level 6
    8 years ago

    First of, a file backup of a running Oracle database is not a good idea as the database will be inconsistent in a restore situation.

    My suggestion is exclude the database files (usually called .dbf) and use either agent based or rman dump to backup the oracle database. Confer with your DBA about which solution is best.

     

    • Genericus's avatar
      Genericus
      Moderator
      8 years ago

      Mark that as a solution! Michael is absolutely correct. In fact, I would suggest that for simplicity, you seperate the database and log files into their own directories for simplicity in excluding them.

      excluding /oracle/db and /oracle/logs will be easier than exluding *.dbf, since it will have to scan for them...

       

      • Carawain's avatar
        Carawain
        Level 3
        8 years ago

        Hello

        thank for all reply.

        I'll check with dba administrator if we can improve this backup but we have a lot of database backup policies implemented as this one with status code 0.

        I've found this related technote: https://www.veritas.com/support/en_US/article.000009698

        I've launch the below custom bpbkar command to simulate the backup :

        ./bpbkar -dt 0 -r 888 -nocont /u07/flash_recovery_area/DBUVC170_ORAVAL15/

        Here the result of bpcd and bpbkar output with verbosity = 5:

        bpcd

        18:00:48.932 [1290352] <2> setup_debug_log: switched debug log file for bpcd
18:00:48.939 [1290352] <2> bpcd main: VERBOSE = 5
18:00:48.939 [1290352] <2> logparams: /usr/openv/netbackup/bin/bpcd -terminate 
18:00:48.939 [823324] <16> daemon_select_and_accept: got terminate request: 0
18:00:48.939 [823324] <2> bpcd main: got terminate request
18:00:48.973 [1290352] <2> init_resilient_cache: [vnet_nbrntd.c:869] Initialize resilient cache. 0 0x0
18:00:48.981 [1290352] <4> bpcd main: bpcd terminated
18:00:49.021 [823324] <4> bpcd main: EXIT status = 0
18:00:59.512 [1290374] <2> setup_debug_log: switched debug log file for bpcd
18:00:59.512 [1290374] <2> bpcd main: VERBOSE = 5
18:00:59.512 [1290374] <2> logparams: /usr/openv/netbackup/bin/bpcd -standalone 
18:00:59.512 [1290374] <2> SetMaxDataLimit: maximum data size: current=9223372036854775807 max=9223372036854775807
18:00:59.522 [823326] <2> setup_debug_log: switched debug log file for bpcd
18:00:59.573 [823326] <2> vnet_pcache_init_table: [vnet_private.c:235] starting cache size 200 0xc8
18:00:59.582 [823326] <8> file_to_cache_item: [vnet_addrinfo.c:6577] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/1ff/ffffffff+bpcd,1,2,0,1,0+.txt
18:00:59.601 [823326] <2> retry_getaddrinfo: [vnet_addrinfo.c:908] using SVC=bpcd PORT=13782
18:00:59.601 [823326] <2> retry_getaddrinfo: [vnet_addrinfo.c:912] adjusted service name 13782
18:00:59.601 [823326] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1632] found via getaddrinfo NAME=NULL SVC=bpcd
18:00:59.604 [823326] <2> daemon_startup_listeners: 4 listening for legacy service bpcd
18:00:59.615 [823326] <2> daemon_startup_listeners: listening for vnetd service bpcd
18:00:59.822 [823326] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.132: pbxRegisterEx successful at XXX.X.112.141:1556/bpcd, returns with 0 alt_addrs
18:00:59.822 [823326] <2> daemon_resource_checker: last_brk = 0, new_brk = 110073440
18:00:59.822 [823326] <2> daemon_resource_checker: last max_fd = 0, new_fd = 10
18:00:59.823 [823326] <16> resync_host_cache: bpclntcmd fork pid = 1822740
18:00:59.842 [823326] <2> daemon_check_for_zombies: waited for 1 child processes including: 1822740

         

        bpbkar

        18:01:39.303 [1822748] <2> logparams: ./bpbkar -dt 0 -r 888 -nocont /u07/flash_recovery_area/DBUVC170_ORAVAL15/ 
18:01:39.303 [1822748] <2> bpbkar main: hole_granularity = 256.
...
18:01:39.352 [1822748] <2> bpbkar SelectFile: INF - cwd = NULL
18:01:39.352 [1822748] <2> bpbkar SelectFile: INF - path = /u07/flash_recovery_area/DBUVC170_ORAVAL15
18:01:39.352 [1822748] <2> bpbkar resolve_path: INF - Actual mount point of /u07/flash_recovery_area is /u07/flash_recovery_area
18:01:39.352 [1822748] <2> bpbkar SelectFile: INF - Resolved_path = /u07/flash_recovery_area/DBUVC170_ORAVAL15
18:01:39.353 [1822748] <2> bpbkar resolve_path: INF - Actual mount point of /u07/flash_recovery_area/DBUVC170_ORAVAL15 is /u07/flash_recovery_area/DBUVC170_ORAVAL15
18:01:39.361 [1822748] <2> bpbkar SelectFile: INF - cwd = /u07/flash_recovery_area/DBUVC170_ORAVAL15
18:01:39.361 [1822748] <2> bpbkar SelectFile: INF - path = autobackup.tar
18:01:39.362 [1822748] <4> check_file_sparseness: Device changing from 0 to 5
...
19:25:32.086 [1822748] <2> bpbkar resolve_path: INF - Actual mount point of / is /
19:25:32.086 [1822748] <4> bpbkar expand_wildcards: end backup for filelist /u07/flash_recovery_area/DBUVC170_ORAVAL15/
19:25:32.087 [1822748] <4> bpbkar main: INF - Client completed sending data for backup

19:25:32.087 [1822748] <2> bpbkar main: INF - Total Size:710314321 + 333GB
19:25:32.087 [1822748] <2> bpbkar delete_old_files_recur: INF - checking files in directory /usr/openv/netbackup/hardlink_info for prefix = hardlinks_ and older than 30 days
19:25:32.087 [1822748] <2> bpbkar delete_old_files_recur: INF - checking files in directory /usr/openv/netbackup/hardlink_info/root for prefix = hardlinks_ and older than 30 days
19:25:32.096 [1822748] <2> bpbkar delete_old_files_recur: INF - checking files in directory /usr/openv/netbackup/logs/user_ops for prefix = jbp and older than 3 days
19:25:32.096 [1822748] <2> bpbkar delete_old_files_recur: INF - checking files in directory /usr/openv/netbackup/logs/user_ops/nbjlogs for prefix = jbp and older than 3 days
19:25:32.096 [1822748] <4> bpbkar Exit: INF - bpbkar exit normal
19:25:32.097 [1822748] <4> bpbkar Exit: INF - EXIT STATUS 0: the requested operation was successfully completed

        So when we simulate bpbkar on local client no issue the backup get successfully in 1,5 hours but when we launch the backup over the network something is stuck on bpbkar side.

         

        I need to understand if there's network error or system/client issue.