Full Backup failing with error 71, but incremental...

anash7 · ‎02-05-2014

Hello All,

My master server is running at NBU 7.1.0.4 on a Solaris box and my client is a Windows 2000 server with NBU 5.1 MP4.

Backup selection is All Local Drives - with only C drive present on the server. So C and system state are getting backed up.

The full backups and incrementals were running fine till now but since yesterday the full backup for C drive has started failing with error 71 but the full backup for system state is completing successfully. If I run a incremental backup, both C and System State are completing successfully.

Please assist.

Policy Type is MS-Windows.

Job Details:

SET SNAP_ID=xyzsrvcph704_1391531402

BACKUP C:\ USING \\?\Volume{c99c4d98-f0d0-41e1-944d-5cb04a849b00}\ OPTIONS:ALT_PATH_PREFIX=C:\Program Files\VERITAS\NetBackup\temp\_vrts_frzn_img_2204,FITYPE=MIRROR,MNTPOINT=C:\,FSTYPE=NTFS

02/05/2014 11:37:16 - Info bpbrm (pid=21641) telling media manager to start backup on client

02/05/2014 11:37:16 - Info bptm (pid=22052) using 262144 data buffer size

02/05/2014 11:37:16 - Info bptm (pid=22052) using 64 data buffers

02/05/2014 11:37:16 - Info bptm (pid=22052) start backup

02/05/2014 11:37:16 - Info bpbrm (pid=21641) spawning a brm child process

02/05/2014 11:37:16 - Info bpbrm (pid=21641) child pid: 22478

02/05/2014 11:37:16 - Info bptm (pid=22052) Waiting for mount of media id BL4444 (copy 1) on server bckcph2.

02/05/2014 11:37:16 - mounting BL4444

02/05/2014 11:37:17 - Info bpbrm (pid=21641) sending bpsched msg: CONNECTING TO CLIENT FOR xyzsrvcph704_1391596630

02/05/2014 11:37:17 - Info bptm (pid=22462) setting receive network buffer to 65536 bytes

02/05/2014 11:37:17 - Info bpbrm (pid=21641) start bpbkar on client

02/05/2014 11:37:17 - Info bpbkar (pid=0) Backup started

02/05/2014 11:37:17 - Info bpbrm (pid=21641) Sending the file list to the client

02/05/2014 11:37:17 - connecting

02/05/2014 11:37:17 - connected; connect time: 0:00:00

02/05/2014 11:37:19 - Info bpbrm (pid=22478) from client xyzsrvcph704: TRV - object not found for file system backup: C:

02/05/2014 11:37:19 - Info bpbrm (pid=21641) sending message to media manager: STOP BACKUP xyzsrvcph704_1391596630

02/05/2014 11:37:20 - Info bpbrm (pid=21641) media manager for backup id xyzsrvcph704_1391596630 exited with status 150: termination requested by administrator

02/05/2014 11:37:20 - end writing

none of the files in the file list exist (71)

RamNagalla · ‎02-05-2014

I am not finding much clue on this , as you are in 5.X version

may be a clean reboot of the client would help ... please try client reboot if possible..

looks like your media server is also in 5.X

02/05/2014 11:37:17 - Info bpbrm (pid=21641) sending bpsched msg: CONNECTING TO CLIENT FOR xyzsrvcph704_1391596630

you are running all out of support products... win2000 Netbackup 5.X.. please try to upgrade them as soon as possible..

anash7 · ‎02-05-2014

There is no separate media server, master itself is media. Master is Solaris 5.10

Nicolai · ‎02-05-2014

Take a look at the bpbkar debug log file in [install_path]/Netbackup/logs/bpbkar

If the directory does not exist, create it and re-run the backup

anash7 · ‎02-05-2014

bpbkar log doesnt have anything specific. In the bpbrm I do find the below:

10:30:10.838 [13324] <2> MNG: backup_cmd = /usr/openv/netbackup/bin/bpbkar bpbkar -r 16070400 -ru root -dt 0 -to 0 -clnt xyzsrvcph704 -class file_backup_nt_uat -sched Full_monthly -st FULL -bpstart_to 300 -bpend_to 300 -read_to 9600 -ckpt_time 900 -blks_per_buffer 511 -stream_count 2 -stream_number 2 -jobgrpid 981255 -use_otm -use_ofb -b xyzsrvcph704_1391592603 -kl 7 -fso -WOFB_enabled -WOFB_fim 0 -WOFB_usage 0 -WOFB_error 0 -ct 13

10:30:10.838 [13324] <2> bpbrm send_bpsched_connected_msg: sending bpsched msg: CONNECTED TO CLIENT FOR xyzsrvcph704_1391592603

10:30:11.769 [13324] <2> bpbrm read_backup_start: from client xyzsrvcph704: read client start message

10:30:11.770 [13324] <2> write_file_names: buffering file name 'SET SNAP_ID=xyzsrvcph704_1391531402' for output

10:30:13.726 [13324] <2> bpbrm handle_backup: ESTIMATE -1 -1 xyzsrvcph704_1391592603

10:30:13.726 [13324] <4> bpbrm handle_backup: from client xyzsrvcph704: TRV - object not found for file system backup: C:

10:30:13.729 [13324] <4> db_error_add_to_file: from client xyzsrvcph704: TRV - object not found for file system backup: C:

10:30:13.738 [13324] <2> bpbrm handle_backup: client xyzsrvcph704 EXIT STATUS = 71: none of the files in the file list exist

10:30:13.802 [13094] <2> bpbrm tell_mm: sending media manager msg: STOP BACKUP xyzsrvcph704_1391592603

10:30:14.313 [13094] <2> bpbrm read_media_msg: read from media manager: EXIT xyzsrvcph704_1391592603 150

10:30:14.313 [13094] <2> bpbrm process_media_msg: media manager for backup id xyzsrvcph704_1391592603 exited with status 150: termination requested by administrator

10:30:14.314 [13094] <2> bpbrm send_status_to_parent: EXIT xyzsrvcph704_1391592603 71 sent to parent process.

10:30:14.314 [13094] <2> bpbrm mail_status: attempting to send mail to root on xyzsrvcph704

10:32:21.941 [13885] <2> logparams: -backup -backup_copy 0 -b xyzsrvcph704_1391592740 -bt 1391592740 -c xyzsrvcph704 -cl Test_xyzsrvcph704 -ct 13 -dt 0 -fso -jobid 982152 -jobgrpid 982152 -masterversion 710000 -mediasvr xyzbckcph2 -mpx 0 -nbu -p NONE -ru root -rg root -sched Full_monthly -st 0 -S xyzbckcph2 -stream_count 2 -stream_number 1 -snaponly -secure 1 -use_ofb -use_otm -WOFB_enabled 1 -nbu

10:32:21.945 [13885] <2> read_client: opendir() failed: xyzsrvcph704: No such file or directory (2)

10:32:21.945 [13885] <2> verify_client: bpbrm.c.34815: db_getCLIENT failed for CLIENT: xyzsrvcph704

10:32:21.947 [13885] <2> bpbrm start_bpcd_stat: bpbrm.c.20683: bpcd_client: xyzsrvcph704

10:32:21.947 [13885] <2> bpbrm start_bpcd_stat: bpbrm.c.20686: bpcd_client_hostname: xyzsrvcph704

10:32:22.516 [13885] <2> verify_client: bpbrm.c.35079: *client_hostname: xyzsrvcph704

10:32:22.517 [13885] <2> read_client: opendir() failed: xyzsrvcph704: No such file or directory (2)

10:32:23.537 [13885] <2> local_getHostInfo: platform of client xyzsrvcph704 is PC, WindowsNT

10:32:24.283 [13885] <2> bpfis_start: DBG: BPFIS Commands [255] - /usr/openv/netbackup/bin/bpfis bpfis create -nbu -owner NBU -id xyzsrvcph704_1391592740 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 13 -clnt xyzsrvcph704 -rg root -class Test_xyzsrvcph704 -sched Full_monthly -fso -S xyzbckcph2 -WOFB -fim VSP

10:32:24.283 [13885] <2> bpbrm start_bpcd_stat: bpbrm.c.20683: bpcd_client: xyzsrvcph704

10:32:24.283 [13885] <2> bpbrm start_bpcd_stat: bpbrm.c.20686: bpcd_client_hostname: xyzsrvcph704

10:32:24.803 [13885] <2> pfi_start_client: command = /usr/openv/netbackup/bin/bpfis bpfis create -nbu -owner NBU -id xyzsrvcph704_1391592740 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 13 -clnt xyzsrvcph704 -rg root -class Test_xyzsrvcph704 -sched Full_monthly -fso -S xyzbckcph2 -WOFB -fim VSP

10:32:24.876 [13885] <2> pfi_start_client: read start message from xyzsrvcph704, msg=<INF - BACKUP START 4136>

10:32:24.977 [13885] <2> get_bpfis_msg: bpfis buff: INF - FIS_ID=xyzsrvcph704_1391592740

10:34:28.470 [22850] <2> logparams: -backup -S xyzbckcph2 -c xyzsrvcph704 -ct 13 -ru root -cl Test_xyzsrvcph704 -sched Full_monthly -bt 1391592867 -dt 0 -st 0 -b xyzsrvcph704_1391592867 -mediasvr xyzbckcph2 -jobid 982161 -jobgrpid 982152 -masterversion 710000 -maxfrag 100000 -reqid -1390498138 -mt 2 -to 0 -stunit bp17_VTLLPA30_xyzbckcph2_LPA -rl 6 -rp 16070400 -eari 0 -cj 10 -D 14 -rt 8 -rn 0 -pool Standard -mpx 3 -use_ofb -use_otm -WOFB_enabled 1 -nbu -jm -secure 1

10:34:28.474 [22850] <2> read_client: opendir() failed: xyzsrvcph704: No such file or directory (2)

10:34:29.538 [22850] <2> local_getHostInfo: platform of client xyzsrvcph704 is PC, WindowsNT

10:34:31.613 [22850] <2> bpbrm read_parent_msg: read from parent START BACKUP -c xyzsrvcph704 -ct 13 -ru root -cl Test_xyzsrvcph704 -sched Full_monthly -bt 1391592867 -dt 0 -st 0 -b xyzsrvcph704_1391592867 -jobid 982161 -jobgrpid 982152 -rl 6 -rp 16070400 -use_ofb -use_otm -WOFB_enabled 1 -nbu -secure 1 -kl 7 -rg root -fso -stream_count 2 -stream_number 1 -keyword AlmÀ filÀ backup -ckpt_time 900 -connect_options 16974338

10:34:31.614 [22850] <2> read_client: opendir() failed: xyzsrvcph704: No such file or directory (2)

10:34:31.615 [22850] <2> read_client: opendir() failed: xyzsrvcph704: No such file or directory (2)

10:34:31.615 [22850] <2> verify_client: bpbrm.c.34815: db_getCLIENT failed for CLIENT: xyzsrvcph704

Marianne · ‎02-05-2014

Please try to ugrade your client to 6.5 at least with patch 6.5.6.

Please show us policy for this client? On master:

bppllist <policy-name> -U

Please create bpfis log folder as well. We need to see what happened with snapshot creation.

Handy NetBackup Links

anash7 · ‎02-05-2014

The backup completed successfully now, when I re-ran to create the bpfis logs.
I will share the logs back here if the issue re-occurs. Will try to upgrade the client also.

Policy details:

/usr/openv/netbackup/bin# bppllist file_backup_nt_uat -U

------------------------------------------------------------

Policy Name: file_backup_nt_uat

Policy Type: MS-Windows

Active: yes

Effective date: 01/01/1970 01:00:00

Backup network drvs: no

Collect TIR info: no

Mult. Data Streams: yes

Client Encrypt: no

Checkpoint: yes

Interval: 15

Policy Priority: 10

Max Jobs/Policy: Unlimited

Disaster Recovery: 0

Collect BMR info: no

Residence: bp17_VTLLPA30_xyzbckcph2_LPA

Volume Pool: Standard

Server Group: *ANY*

Keyword: Alm fil backup

Data Classification: -

Residence is Storage Lifecycle Policy: no

Application Discovery: no

Discovery Lifetime: 0 seconds

Granular Restore Info: no

Ignore Client Direct: no

HW/OS/Client: PC Windows2000 xyzsrvcph701

PC Windows2000 xyzsrvcph702

PC Windows2000 xyzsrvcph704

Windows-x86 WindowsXP xyzsrvcph31

Windows-x86 WindowsXP xyzsrvcph32

Windows-x86 WindowsXP xyzsrvcph34

Windows-x86 WindowsXP xyzsrvcph36

Include: ALL_LOCAL_DRIVES

Schedule: Full_Yearly

Type: Full Backup

Frequency: every 7 days

Maximum MPX: 2

Synthetic: 0

PFI Recovery: 0

Retention Level: 12 (infinity)

Number Copies: 1

Fail on Error: 0

Residence: (specific storage unit not required)

Volume Pool: (same as policy volume pool)

Server Group: (same as specified for policy)

Residence is Storage Lifecycle Policy: 0

Daily Windows:

Schedule: Full_monthly

Type: Full Backup

Maximum MPX: 2

Synthetic: 0

PFI Recovery: 0

Retention Level: 6 (6 months)

Number Copies: 1

Fail on Error: 0

Residence: (specific storage unit not required)

Volume Pool: (same as policy volume pool)

Server Group: (same as specified for policy)

Calendar sched: Enabled

Allowed to retry after run day

Tuesday, Week 1

Residence is Storage Lifecycle Policy: 0

Daily Windows:

Tuesday 17:30:00 --> Wednesday 00:30:00

Schedule: Full_weekly

Type: Full Backup

Maximum MPX: 2

Synthetic: 0

PFI Recovery: 0

Retention Level: 3 (1 month)

Number Copies: 1

Fail on Error: 0

Residence: (specific storage unit not required)

Volume Pool: (same as policy volume pool)

Server Group: (same as specified for policy)

Calendar sched: Enabled

Allowed to retry after run day

Tuesday, Week 2

Tuesday, Week 3

Tuesday, Week 4

Tuesday, Week 5

Residence is Storage Lifecycle Policy: 0

Daily Windows:

Tuesday 17:30:00 --> Wednesday 00:30:00

Schedule: Incr_daily

Type: Differential Incremental Backup

Frequency: every 1 day

Maximum MPX: 2

Synthetic: 0

PFI Recovery: 0

Retention Level: 1 (2 weeks)

Number Copies: 1

Fail on Error: 0

Residence: (specific storage unit not required)

Volume Pool: (same as policy volume pool)

Server Group: (same as specified for policy)

EXCLUDE DATE 0 - 03/10/2007

Residence is Storage Lifecycle Policy: 0

Daily Windows:

Sunday 00:00:00 --> Sunday 08:00:00

Monday 00:00:00 --> Monday 08:00:00

Tuesday 00:00:00 --> Tuesday 08:00:00

Wednesday 00:00:00 --> Wednesday 08:00:00

Thursday 00:00:00 --> Thursday 08:00:00

Friday 00:00:00 --> Friday 08:00:00

Saturday 00:00:00 --> Saturday 08:00:00

root:xyzbckcph2:/usr/openv/netbackup/bin#

Nicolai · ‎02-05-2014

Actual the first error message is not status code 71 but status code 150: Terminated by administrator. This message can occur erroneous when the system is missing resources like disk or memory.

02/05/2014 11:37:20 - Info bpbrm (pid=21641) media manager for backup id xyzsrvcph704_1391596630 exited with status 150: termination requested by administrator
02/05/2014 11:37:20 - end writing
none of the files in the file list exist  (71)

Try to reboot xyzsrvcph704

VOX

Full Backup failing with error 71, but incremental running fine