cancel
Showing results for 
Search instead for 
Did you mean: 

Backup failure

We have tried excluding the files and changing various parameters and services restart but neither it was not helpful,

 


Mar 11, 2021 3:00:00 AM - Info nbjm (pid=7759) starting backup job (jobid=1622615) for client usphi327-ebr, policy NIX_Daily, schedule Full
Mar 11, 2021 3:00:00 AM - Info nbjm (pid=7759) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1622615, request id:{28ED0A14-8248-11EB-A2A5-520EA6222B03})
Mar 11, 2021 3:00:00 AM - requesting resource usphi730_usphidd002_stu
Mar 11, 2021 3:00:00 AM - requesting resource usphi727-ebr.NBU_CLIENT.MAXJOBS.usphi327-ebr
Mar 11, 2021 3:00:00 AM - requesting resource usphi727-ebr.NBU_POLICY.MAXJOBS.NIX_Daily
Mar 11, 2021 3:00:00 AM - granted resource usphi727-ebr.NBU_CLIENT.MAXJOBS.usphi327-ebr
Mar 11, 2021 3:00:00 AM - granted resource usphi727-ebr.NBU_POLICY.MAXJOBS.NIX_Daily
Mar 11, 2021 3:00:00 AM - granted resource MediaID=@aaaad;DiskVolume=usphidd002_stu;DiskPool=usphidd002_stu_dp;Path=usphidd002_stu;StorageServer=usphidd002-ebr2;MediaServer=usphi730-ebr
Mar 11, 2021 3:00:00 AM - granted resource usphi730_usphidd002_stu
Mar 11, 2021 3:00:01 AM - Info bpbrm (pid=29832) usphi327-ebr is the host to backup data from
Mar 11, 2021 3:00:01 AM - Info bpbrm (pid=29832) reading file list for client
Mar 11, 2021 3:00:01 AM - estimated 0 kbytes needed
Mar 11, 2021 3:00:01 AM - Info nbjm (pid=7759) started backup (backupid=usphi327-ebr_1615453201) job for client usphi327-ebr, policy NIX_Daily, schedule Full on storage unit usphi730_usphidd002_stu
Mar 11, 2021 3:00:01 AM - started process bpbrm (pid=29832)
Mar 11, 2021 3:00:01 AM - connecting
Mar 11, 2021 3:00:02 AM - Info bpbrm (pid=29832) starting bpbkar on client
Mar 11, 2021 3:00:02 AM - Info bpbkar (pid=12214) Backup started
Mar 11, 2021 3:00:02 AM - Info bpbrm (pid=29832) bptm pid: 29840
Mar 11, 2021 3:00:02 AM - Info bptm (pid=29840) start
Mar 11, 2021 3:00:02 AM - Info bptm (pid=29840) using 262144 data buffer size
Mar 11, 2021 3:00:02 AM - Info bptm (pid=29840) using 30 data buffers
Mar 11, 2021 3:00:02 AM - connected; connect time: 0:00:00
Mar 11, 2021 3:00:03 AM - Info bptm (pid=29840) start backup
Mar 11, 2021 3:00:03 AM - Info bptm (pid=29840) backup child process is pid 29862
Mar 11, 2021 3:00:03 AM - begin writing
Mar 11, 2021 3:58:56 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/lib/nfs/rpc_pipefs] is in a different file system from [/var]. Skipping
Mar 11, 2021 3:58:56 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/lib/fireeye/xagt/__7dea7cf6_@.sock] is a socket special file. Skipping
Mar 11, 2021 3:58:56 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/lib/amazon/ssm/ipc/health] is a socket special file. Skipping
Mar 11, 2021 3:58:56 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/lib/amazon/ssm/ipc/termination] is a socket special file. Skipping
Mar 11, 2021 3:59:24 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/run/rpcbind.sock] is a socket special file. Skipping
Mar 11, 2021 3:59:24 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/run/acpid.socket] is a socket special file. Skipping
Mar 11, 2021 3:59:24 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/run/vmware/guestServicePipe] is a socket special file. Skipping
Mar 11, 2021 3:59:24 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/run/abrt/abrt.socket] is a socket special file. Skipping
Mar 11, 2021 3:59:24 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/run/dbus/system_bus_socket] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/public/cleanup] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/public/showq] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/public/flush] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/proxymap] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/proxywrite] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/local] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/virtual] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/discard] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/rewrite] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/retry] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/bounce] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/defer] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/smtp] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/trace] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/error] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/anvil] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/relay] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/scache] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/verify] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/tlsmgr] is a socket special file. Skipping
Mar 11, 2021 4:00:11 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/var/spool/postfix/private/lmtp] is a socket special file. Skipping
Mar 11, 2021 4:01:47 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/usr/openv/var/vnetd/terminate_bpcd.uds] is a socket special file. Skipping
Mar 11, 2021 4:01:47 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/usr/openv/var/vnetd/bpcd.uds] is a socket special file. Skipping
Mar 11, 2021 4:01:47 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/usr/openv/var/vnetd/terminate_vnetd.uds] is a socket special file. Skipping
Mar 11, 2021 4:02:41 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/home] is in a different file system from [/]. Skipping
Mar 11, 2021 4:02:41 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/sasconfig] is in a different file system from [/]. Skipping
Mar 11, 2021 4:02:41 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/int/sap/bw/bi/b3p/o] is on file system type NFS. Skipping
Mar 11, 2021 4:02:41 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/int/sap/bw/bi/b1p/o] is on file system type NFS. Skipping
Mar 11, 2021 4:02:41 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/int/terra] is on file system type NFS. Skipping
Mar 11, 2021 4:02:42 AM - Info bpbrm (pid=29832) from client usphi327-ebr: TRV - [/install_files] is on file system type NFS. Skipping
Mar 11, 2021 4:02:42 AM - Error bpbrm (pid=29832) db_FLISTsend failed: file read failed (13)
Mar 11, 2021 4:02:44 AM - Error bptm (pid=29840) media manager terminated by parent process
Mar 11, 2021 4:02:50 AM - Info bpbkar (pid=12214) done. status: 13: file read failed
Mar 11, 2021 4:02:50 AM - end writing; write time: 1:02:47
file read failed (13)

Tags (2)
6 Replies

Re: Backup failure

hi  @Prabhu05 

Create directory /usr/openv/netbackup/logs/bpbkar

Add VERBOSE = 5 on the client bp.conf 

Re-run the backup, after failed backup inspect bpbkar log files in /usr/openv/netbackup/logs/bpbkar

Also take a look at this tech note - it will help identifying if the backup gets stuck in a specific area of the file system:

https://www.veritas.com/support/en_US/article.100003560

Re: Backup failure

HI @Prabhu05 

Beyond the good suggestion by @Nicolai, the other question is what are you trying to achieve?

My reading based on the output you have provided is that your backup selection is just "/" on a *nix server. 

If you want to backup the entire server, try changing this to ALL_LOCAL_DRIVES, then create the exclude list after that to exclude the file systems you don't want. The system looks to be running SAP based on the file systems being skipped, so I'm guessing you may want to exclude the database files from a standard backup. 

David

Re: Backup failure

@Prabhu05 

You can ignore all of the 'Info' messages. This is normal, as the 'skipped' files and folders either do not need to be backed up (socket special file) or will be backed up as part of a different filesystem.
As per @davidmoline , best to have a look at how your Backup Selection is configured.

The real issue is seen just over an hour into the backup:

Mar 11, 2021 4:02:42 AM - Error bpbrm (pid=29832) db_FLISTsend failed: file read failed (13)
Mar 11, 2021 4:02:44 AM - Error bptm (pid=29840) media manager terminated by parent process
Mar 11, 2021 4:02:50 AM - Info bpbkar (pid=12214) done. status: 13: file read failed

bpbrm on the media server was the first process to report an error.
I would therefore like to see level 3 logs of bpbrm on the media server as well as level 3 bpbkar on the client.
Level 3 bptm log on the media server may also be helpful to determine up to when data from the client was received.

If you have 'ALL_LOCAL_DRIVES' in the Backup Selection with 'Allow Multiple Data Streams' in the Policy Attributes, NBU will start a separate job for each filesystem, which will make it easier to determine where the backup process is 'getting stuck'.

PS:
Only change policy before a FULL backup is due to avoid Incrementals running as Full.

Re: Backup failure

"My reading based on the output you have provided is that your backup selection is just "/" on a *nix server. 
If you want to backup the entire server, try changing this to ALL_LOCAL_DRIVES..."

Are you saying its better to use ALL_LOCAL_DRIVES on a unix system?

We use "ALL_LOCAL_DRIVES" on our windows backups and "/" on our Unix backups just cause that seems to logically make sense.  Do they work differently?

 

Re: Backup failure

Hi @sclind 

Yes a single "/" will (in general) behave differently to ALL_LOCAL_DRIVES on unix.

The backup process on Unix by default (there are exceptions) will look to back up the file on the file system(s) you specify. If you only specify "/" that is all that will be protected. You may miss other mounted file systems - in your case it is skipping "/home, &  /sasconfig" as shown in your original log extract.

The exception is if you also select the "Cross mount points" in the policy attributes. In which case selecting "/" will backup the entire system (local drives and not NFS). I can see you do not have this attribute selected based on the log extract.

The ALL_LOCAL_DRIVES will determine what local file systems are available and back up all of them. This is done using the bpmount command on the client - so you can see what the client will try to back up - and then what you may want to exclude.

For further information there is a reasonable explanation found by clicking the help button when a policy is open at the attributes page, then selecting "Cross Mount Point (policy attribute)" or refer to the same topic in the Administration Guide volume 1.

David

Re: Backup failure

@Prabhu05 

Have you seen attempts to help you?