02-12-2015 09:11 AM
Hi All,
We have a Aix server whose backup is consistanly getting a 14, I setup a bpbkar log and viewed it. I have just a portion of it below where the backup is getting a exit status of 14
I believe the issue is a journal.v1.dat file is there anyway to verify this?
8:00:51.499 [1028250] <4> ct_logfiles_add: add log file:curr=/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home/track_journal.v1.dat.p7nvc.vore.xxxx.stat
e.nj.us_1423522807 prev=/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home/track_journal.v1.dat
18:00:51.500 [1028250] <4> ct_cat_open: successfully open cur /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home/track_journal.v1.dat.p7nvc.vore.xxxx.stat
e.nj.us_1423522807 and pre /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home/track_journal.v1.dat .
18:00:51.929 [385146] <16> bpbkar: ERR - read server exit status = 14: file write failed
18:00:51.934 [557204] <4> bpbkar main: real locales <en_US en_US C en_US en_US en_US>
18:00:51.934 [557204] <4> bpbkar main: standardized locales - mnt_lc_messages <en_US> mnt_lc_ctype <en_US> mnt_lc_time <en_US> mnt_lc_collate <en_US> mnt_lc_numeric <en_US>
18:00:51.935 [557204] <2> bpbkar main: create backup id list: p7nvc.msss.xxxx.xxx_1423004406,p7nvc.msss.xxxx.xxx_1423436405,p7nvc.msss.xxxx.xxx_1423350005,p7nvc.msss.xxxx.xxx_
1423263605,p7nvc.msss.xxxx.xxx_1423177205,p7nvc.msss.xxxx.xxx_1423112443,p7nvc.msss.xxxx.xxx_1423026136,p7nvc.msss.xxxx.xxx_1423004409,p7nvc.msss.xxxx.xxx_1423436408,p
8mvc.vore.xxxx.state.nj.us_1423350008,p7nvc.msss.xxxx.xxx_1423263607,p7nvc.msss.xxxx.xxx_1423177208,p7nvc.msss.xxxx.xxx_1423112789,p7nvc.msss.xxxx.xxx_1423026100,p7nvc.vore.tre
as.state.nj.us_1423004408,p7nvc.msss.xxxx.xxx_1423436406,p7nvc.msss.xxxx.xxx_1423350006,p7nvc.msss.xxxx.xxx_1423263606,p7nvc.msss.xxxx.xxx_1423177206,p7nvc.vore.xxxx.state.nj.
us_1423112787,p7nvc.msss.xxxx.xxx_1423026097,p7nvc.msss.xxxx.xxx_1423004410,p7nvc.msss.xxxx.xxx_1423436409,p7nvc.msss.xxxx.xxx_1423350009,p7nvc....
18:00:51.935 [557204] <4> bpbkar main: For accelerator, we will be using both the 'mtime' and 'ctime' for incrementals, the current configuration is : 0, will be set to 1.
18:00:51.935 [557204] <2> logparams: bpbkar -r 2678400 -ru root -dt 86701 -to 0 -bpstart_time 1423523110 -clnt p7nvc.msss.xxxx.xxx -class p7nvc.msss.xxxx.xxx -sched incremental -st INCR -bpst
art_to 2500 -bpend_to 2500 -read_to 6000 -stream_count 11 -stream_number 7 -jobgrpid 29891 -blks_per_buffer 512 -tir -tir_plus -use_otm -fso -b p7nvc.msss.xxxx.xxx_1423522809 -kl 28 -fscp -S nbv40w1
-storagesvr nbv40w1 -bidlist bid@p7nvc.msss.xxxx.xxx_p7nvc.msss.xxxx.xxx_1423522809 -use_ofb
18:00:51.979 [385146] <4> ct_logfiles_deleteall: unlink log files /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/proc/track_journal.v1.dat.p7nvc.vore.xxxx.
log.020915 (27%)
...skipping...
18:00:50.481 [868588] <4> bpbkar: INF - Estimate:-1 -1
18:00:50.497 [385146] <16> bpbkar: ERR - Cannot write to tir_info_file, fd is NULL.
18:00:50.498 [385146] <16> bpbkar: ERR - bpbkar FATAL exit status = 14: file write failed
18:00:50.498 [385146] <4> bpbkar: INF - EXIT STATUS 14: file write failed
18:00:50.481 [868588] <4> bpbkar: INF - Estimate:-1 -1
18:00:50.497 [385146] <16> bpbkar: ERR - Cannot write to tir_info_file, fd is NULL.
18:00:50.498 [385146] <16> bpbkar: ERR - bpbkar FATAL exit status = 14: file write failed
18:00:50.498 [385146] <4> bpbkar: INF - EXIT STATUS 14: file write failed
18:00:50.621 [868588] <2> bpbkar add_to_filelist: starting sizeof(filelistrec) <128>
18:00:50.621 [868588] <4> bpbkar: INF - Throttle duration required = 512 usec.
18:00:50.623 [868588] <4> bpbkar: start to backup filelist / ,nb_fscp_enabled is 1
18:00:50.623 [868588] <4> bpbkar: INF - Processing /
18:00:50.623 [868588] <4> bpbkar: filelist /,is folder traclog path is /_track_log_root_.
18:00:50.623 [868588] <2> ct_cat_close: close current track journal
18:00:50.623 [868588] <2> ct_cat_close: close previous track journal
18:00:50.623 [868588] <2> mkdir_p: make dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/_track_log_root_) result(0)
18:00:50.623 [868588] <2> ct_set_fs_ops: successfully call ct_set_fs_ops
18:00:50.623 [868588] <2> ct_tmpfile_clean: open dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/_track_log_root_), result(1)
18:00:50.623 [868588] <2> ct_tmpfile_clean: keep file(.)
18:00:50.623 [868588] <2> ct_tmpfile_clean: keep file(..)
18:00:50.623 [868588] <2> ct_tmpfile_clean: keep file(track_journal.v1.dat)
18:00:50.638 [868588] <4> ct_cat_open: waiting to lock current track log
18:00:50.638 [868588] <4> ct_cat_open: lock current track log: success
18:00:50.638 [868588] <4> ct_logfiles_add: add log file:curr=/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/_track_log_root_/track_journal.v1.dat.p7nvc.vore
.msss.xxxx.xxx_1423522805 prev=/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/_track_log_root_/track_journal.v1.dat
18:00:50.638 [868588] <4> ct_cat_open: successfully open cur /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/_track_log_root_/track_journal.v1.dat.p7nvc.vore
log.020915 (12%)
8:11:10.206 [1089596] <16> bpbkar: ERR - Cannot write to tir_info_file, fd is NULL.
18:11:10.206 [1089596] <16> bpbkar: ERR - bpbkar FATAL exit status = 14: file write failed
18:11:10.206 [1089596] <4> bpbkar: INF - EXIT STATUS 14: file write failed
18:11:11.334 [1089596] <16> bpbkar: ERR - read server exit status = 14: file write failed
18:11:11.348 [1089596] <2> ct_cat_close: close current track journal
18:11:11.348 [1089596] <2> ct_cat_close: close previous track journal
18:11:11.360 [1089596] <4> ct_logfiles_deleteall: unlink log files /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/proc/track_journal.v1.dat.p7nvc.vore.xxxx
.state.nj.us_1423523426
18:11:11.360 [1089596] <4> ct_logfiles_deleteall: finish logfiles delete/rename
18:11:11.360 [1089596] <4> ct_manage: state(1) != CT_SUCCESS, returning
18:11:11.360 [1089596] <2> ct_fini: fini is called
18:11:11.360 [1089596] <4> bpbkar: INF - setenv FINISHED=0
23:31:29.157 [1167384] <4> is_excluded: Excluded /p7nvc/AllPreMoveBackups/p7nvc/filesystems-on-p7nvc/usr/ora/1110/product/11.1.0/db_1/.patch_storage/12419384_Jul_11_2011_02_07_14/backup/sysman/admin/emdrep/s
ql/core by exclude_list entry core
23:31:35.486 [1167384] <4> is_excluded: Excluded /p7nvc/AllPreMoveBackups/p7nvc/filesystems-on-p7nvc/usr/ora/1110/product/11.1.0/db_1/.patch_storage/12419384_Jul_11_2011_02_07_14/files/sysman/admin/emdrep/sq
l/core by exclude_list entry core
23:31:39.799 [1167384] <4> is_excluded: Excluded /p7nvc/AllPreMoveBackups/p7nvc/filesystems-on-p7nvc/usr/ora
Aix 6.1 server
NBU client level 7.6.0.3
Thanks for any help/advice
Solved! Go to Solution.
03-02-2015 11:58 PM
This has been going on since 12 Feb - almost 3 weeks and 31 comments later.
We have not been able to help you to fix this.
If the new policy fails, please log a Support call with Symantec.
PS:
I was looking for more than just part of the schedule ....... was looking for entire policy config.
02-12-2015 11:24 AM
You don't mention if the backup has ever worked before?
It looks like you are using accelerator. Disable accelerator then try the backup.
Does the backup then work? If so, try deleting the track journal file
02-12-2015 11:42 AM
Hi revarooo,
Thanks...no the backup has never worked, yes we are using the accelerator. Are there instructions on how to disable the accelerator? if so will that affect all backups on the master backup server? or is there a way just to disable the accelerator on one particular client?
I do think I will end of either deleting the track file or excluding it. I would have to notify the AIX admin of the server if I did that.
Best Regards
02-12-2015 12:01 PM
Accelerator option is in the Policy Attribnutes, if it's selected, unselect it and re-run the backup
02-12-2015 12:14 PM
Revaroo, Thanks I will try that and re-run the backup
02-13-2015 12:23 AM
Usage specifics of the bpbkar_path_tr touch file to enable enhanced debug logging of the bpbkar process
http://www.symantec.com/docs/TECH31513
You can also run bpbkar by hand to verify if its really journal.v1.dat who hangs up bpbkar
# /usr/openv/netbackup/bin/bpbkar -dt 0 -r 888 -nocont /data > /dev/null
Replace /data with whatever directory journal.v1.dat is located in.
Update:
Upon reading the thread again, have you tried to exclude /usr/openv/netbackup/track from the backup ?
02-13-2015 01:05 AM
bpbkar_path_tr is not actually needed from 7.6 onwards.
02-13-2015 06:13 AM
Thanks for the info !
02-17-2015 08:05 AM
Hi Revarooo,
I tried and was unable to Disable the accelerator. When I did try I got the following
A box with the message "Use accelerator" cannot be selected if accelerator is enabled under schedules". Disable "Accelerator force rescan in the schedule or create another policy without use accelerator"
Hi Nicolai,
I tried excluding the >>usr/openv/netbackup/track from the backup still got the same error (code 14)
cur /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home/track_journal.v1.dat.p7nvc.vore.xxxx.stat
e.nj.us_1423522807 and pre /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home/track_journal.v1.dat .
18:00:51.929 [385146] <16> bpbkar: ERR - read server exit status = 14: file write failed
If I delete the track journal file will that hurt anything? Is that something I should let the Aix Admin of the server do?
Thanks for your help
02-17-2015 08:17 AM
if you delete the track log next job will be the Normal Full backup and do create the new track log and proced further.
if backup got successfull after disabling the accelator , you can try deleting the track log and run the Full backup with acclelator.
A box with the message "Use accelerator" cannot be selected if accelerator is enabled under schedules". Disable "Accelerator force rescan in the schedule or create another policy without use accelerator"
above error is becaue you have "accelator force rescan" option is selected one of the policy scheudles, first unselect the force rescan option from scheules and uncheck the accelartor option from policy attributes.
02-17-2015 10:36 AM
Hi Nagalla,
Thanks for the response, I unselected the force rescan option from the backup schedule and unchecked the accelerator option from the policy attributes.
I also deleted the track log, I will update after the backup runs.
Best Regards
02-20-2015 05:07 AM
Hi All,
Nagalla,
The full backup ran and received the same error and this was without Accelerator and I deleted the track file.
Acpre /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home/track
18:00:35.598 [913464] <4> bpbkar: INF - Processing /proc
18:00:35.612 [913464] <16> bpbkar: ERR - Cannot write to tir_info_file, fd is NULL.
18:00:35.612 [913464] <16> bpbkar: ERR - bpbkar FATAL exit status = 14: file write failed
18:00:35.612 [913464] <4> bpbkar: INF - EXIT STATUS 14: file write failed
Also
pre /usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home//track_journal.v1.dat
02-20-2015 05:20 AM
Try to move the track log somewhere else:
How to redirect the NetBackup Accelerator track log to a different location
http://www.symantec.com/docs/HOWTO77409
02-20-2015 05:24 AM
when Accelerator is disalbed and deleted track log .. how it is looking for
/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home//track
Do you find this file in client ?
can you check that ?
Delete anything under /usr/openv/netbackup/track/ and try backup agian... if it failed,
please make sure bpbkar log directory is present in /usr/openv/netbackup/logs/ client
put VERBOSE = 5 in bp.conf of client
try backup again and collect & attach the log to this post
02-20-2015 07:33 AM
Thanks Marianne,
I redirected the track using the command below
# ln –s /vol1/track /usr/openv/netbackup/track
when Accelerator is disalbed and deleted track log .. how it is looking for
/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/home//track
Do you find this file in client ? yes and it failed (full backup)
Nagalla,
I did this
put VERBOSE = 5 in bp.conf of client
I will try this if the redirection of the track doesn’t work
Delete anything under /usr/openv/netbackup/track/ and try backup agian.
Thanks for your assistance
02-23-2015 03:04 AM
You will have to delete /usr/openv/netbackup/track folder before you will be able to create the link.
02-23-2015 03:54 PM
Hi Marianne,
Thanks, I deleted the /usr/openv/netbackup/track folder then issue the command below
ln –s /vol1/track /usr/openv/netbackup/track
The backup ran and failed this time it received code 35 cannot make required directory. Here are snippets of the error messages from the bpbkar.log (which is too huge to post )
18:00:35.126 [1065146] <16> ct_init: g_fs_ops->mkdir() fail, error (35)
18:00:35.126 [1065146] <2> fscp_init: Perform block level backup: 1
18:00:35.126 [1065146] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 35: cannot make required directory
18:00:35.126 [1065146] <4> bpbkar Exit: INF - EXIT STATUS 35: cannot make required directory
18:00:35.127 [1065146] <2> bpbkar Exit: INF - Close of stdout complete
18:00:35.127 [1065146] <16> bpbkar Exit: ERR - read server exit status = 6: the backup failed to back up the requested files
18:00:35.126 [1065146] <16> ct_init: g_fs_ops->mkdir() fail, error (35)
18:00:35.126 [1065146] <2> fscp_init: Perform block level backup: 1
18:00:35.126 [1065146] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 35: cannot make required directory
18:00:35.126 [1065146] <4> bpbkar Exit: INF - EXIT STATUS 35: cannot make required directory
18:00:35.127 [1065146] <2> bpbkar Exit: INF - Close of stdout complete
18:00:35.127 [1065146] <16> bpbkar Exit: ERR - read server exit status = 6: the backup failed to back up the requested files
18:00:35.127 [1065146] <2> bpbkar manage_tir_info: Start
18:00:35.127 [1065146] <2> bpbkar manage_tir_info: End
18:00:35.127 [1065146] <2> bpbkar manage_fscp_info: Start
18:00:35.127 [1065146] <2> bpbkar manage_fscp_info: status=6, checkpoint_restart=0
18:00:35.127 [1065146] <2> ct_cat_close: close current track journal
18:00:35.127 [1065146] <2> ct_cat_close: close previous track journal
18:00:35.127 [1065146] <4> ct_logfiles_deleteall: no log files
18:00:35.127 [1065146] <16> fscp_fini: ct_fini() failed, error (15)
18:00:36.543 [626878] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1) error(2)
18:00:36.544 [626878] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbu50w0) error(2)
18:00:36.544 [626878] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx) error(2)
18:00:36.544 [626878] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx//p7nvc.msss.xxxx.xxx) error(2)
18:00:36.544 [626878] <16> ct_init: g_fs_ops->mkdir() fail, error (35)
18:00:36.544 [626878] <2> fscp_init: Perform block level backup: 1
18:00:36.544 [626878] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 35: cannot make required directory
18:00:36.544 [626878] <4> bpbkar Exit: INF - EXIT STATUS 35: cannot make required directory
18:00:36.544 [626878] <2> bpbkar Exit: INF - Close of stdout complete
18:00:36.544 [626878] <16> bpbkar Exit: ERR - read server exit status = 6: the backup failed to back up the requested files
18:00:37.582 [1200218] <2> mkdir_p: Warning: Failed to make dir() error(2)
18:00:37.582 [1200218] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1) error(2)
18:00:37.582 [1200218] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbu50w0) error(2)
18:00:37.582 [1200218] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx) error(2)
18:00:37.582 [1200218] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx error(2)
18:00:37.582 [1200218] <16> ct_init: g_fs_ops->mkdir() fail, error (35)
18:00:37.582 [1200218] <2> fscp_init: Perform block level backup: 1
18:00:37.582 [1200218] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 35: cannot make required directory
18:00:37.582 [1200218] <4> bpbkar Exit: INF - EXIT STATUS 35: cannot make required directory
18:00:37.582 [1200218] <2> bpbkar Exit: INF - Close of stdout complete
18:00:37.583 [1200218] <16> bpbkar Exit: ERR - read server exit status = 6: the backup failed to back up the requested files
18:00:37.583 [1200218] <2> bpbkar manage_tir_info: Start
18:00:37.583 [1200218] <2> bpbkar manage_tir_info: End
18:00:37.583 [1200218] <2> bpbkar manage_fscp_info: Start
18:00:37.583 [1200218] <2> bpbkar manage_fscp_info: status=6, checkpoint_restart=0
18:00:37.583 [1200218] <2> ct_cat_close: close current track journal
18:00:37.583 [1200218] <2> ct_cat_close: close previous track journal
18:00:37.583 [1200218] <4> ct_logfiles_deleteall: no log files
18:00:37.583 [1200218] <16> fscp_fini: ct_fini() failed, error (15)
18:00:37.583 [1200218] <2> bpbkar manage_fscp_info: End, and the return value is (15)
Best regards
18:00:39.826 [417828] <2> mkdir_p: Warning: Failed to make dir() error(2)
18:00:39.826 [417828] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1) error(2)
18:00:39.827 [417828] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbu50w0) error(2)
18:00:39.827 [417828] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx) error(2)
18:00:39.827 [417828] <2> mkdir_p: Warning: Failed to make dir(/usr/openv/netbackup/track/nbv40w1/nbv40w1/p7nvc.msss.xxxx.xxx/p7nvc.msss.xxxx.xxx//p7nvc.msss.xxxx.xxx) error(2)
18:00:39.827 [417828] <16> ct_init: g_fs_ops->mkdir() fail, error (35)
18:00:39.827 [417828] <2> fscp_init: Perform block level backup: 1
18:00:39.827 [417828] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 35: cannot make required directory
18:00:39.827 [417828] <4> bpbkar Exit: INF - EXIT STATUS 35: cannot make required directory
18:00:39.827 [417828] <2> bpbkar Exit: INF - Close of stdout complete
18:00:39.828 [417828] <16> bpbkar Exit: ERR - read server exit status = 6: the backup failed to back up the requested files
18:00:39.828 [417828] <2> bpbkar manage_tir_info: Start
18:00:39.828 [417828] <2> bpbkar manage_tir_info: End
18:00:39.828 [417828] <2> bpbkar manage_fscp_info: Start
18:00:39.828 [417828] <2> bpbkar manage_fscp_info: status=6, checkpoint_restart=0
18:00:39.828 [417828] <2> ct_cat_close: close current track journal
18:00:39.828 [417828] <2> ct_cat_close: close previous track journal
18:00:39.828 [417828] <4> ct_logfiles_deleteall: no log files
18:00:39.828 [417828] <16> fscp_fini: ct_fini() failed, error (15)
18:00:39.828 [417828] <2> bpbkar manage_fscp_info: End, and the return value is (15)
02-23-2015 07:24 PM
02-25-2015 05:54 AM
Marianne,
This is a scheduled backup, and yes I can create the folder.
I am root
P7nvc[root]:/ >
02-25-2015 06:00 AM
Repeat of yesterday's question:
Can you as root user create folder exactly as per log using 'mkdir -p'?