07-15-2020 03:07 AM
Hi Team,
VM backups in our environment are failing with Error code 11 after writing some data.
Master Server is 7.5.0.7. It acts as both Master and Media servers.
I've also checked the logs bpcd, bprd, vnetd but they don't show much information. Please guide me with this issue.
Hiere is the error from Netbackup Detailed Status.
14/07/2020 23:36:12 - positioned 2434L3; position time: 00:00:01
14/07/2020 23:36:12 - begin writing
15/07/2020 00:25:22 - Info bpbkar32(pid=15264) bpbkar waited 202 times for empty buffer, delayed 214 times.
15/07/2020 00:25:27 - Error bpbrm(pid=11572) could not send server status message
15/07/2020 00:25:27 - Critical bpbrm(pid=11572) unexpected termination of client KOSLOAPP001
15/07/2020 00:25:28 - Error bpbrm(pid=11572) cannot send mail to root on client ktslobak001.kt.group.local
15/07/2020 00:25:28 - Info bpbkar32(pid=0) done. status: 11: system call failed
15/07/2020 00:25:28 - end writing; write time: 00:49:16
system call failed(11)
07-15-2020 03:31 AM
Have a look at the processes that are involved in this backup:
bpbrm
bpbkar32
So, the logs that you need to check first are bpbrm and bpbkar.
07-15-2020 04:15 AM
Hi Marianne,
Thanks for the update. I'm unable to open the file as it is too big. Can you suggest some editors please.
07-15-2020 04:39 AM
Here are the logs(bpbrm and bpbkar) as an attachment. Below are the backups that are failing with this error.
KOSLOAPP001
KWSLOWBE005L
KWSLOWBE053
KWSLOWBE002
KWSLOWBE004L
Please help me with this as I'm unable to resolve issue with KOSLOAPP001 and couldn't secure a backup for this client since one month.
07-15-2020 04:39 AM
Try TextPad.
Hopefully your logging level is not set to 5?
I only enable this logging level if requested by Veritas Support.
Level 3 is normally fine for 99% of troubleshooting.
07-15-2020 06:51 AM
My 7.5.x way-back machine isn't working so well this morning. In current terminology, a VMware policy has a backup host or a restore host. I think it used to be called a proxy host. At any rate, that's where bpbkar32 executes.
Q1: In your case, does bpbkar32 execute on the same master/media server where bpbrm executes?
Q2: Do you see anything significant in the Windows application event log? Did bpbkar32 or another NetBackup process crash?
For Windows client processes such as bpfis, bpbkar32, and tar32, the important logging level is the one you configure under Windows client host properties. It should be set to 2, which is the max.
Q3: If bpbkar32 didn't crash, what errors do you find in its log? If bpbkar32 did crash, what are the last 20 log lines before the crash? Please post such a summary here.
Q4: Are your Windows and VMware versions (VCenter, Esx server, VDDK) compatible with NetBackup 7.5.0.7?
07-15-2020 06:58 AM
Upon reflection, I think I may have been incorrect in assuming you are doing VMware policy backups. Are you actually doing MS-Windows policy backups? Please share your policy information. Q2 and Q3 in my previous post still apply.
07-15-2020 07:29 AM
Thank you Marianne. I've installed Textpad.
Logging level is 0. I've checked both the files but don't see anything that I think is the cause of the issue. May be I can't find it. Will you be able to check the log(bpbrm) which I uploaded?
07-16-2020 03:41 AM
The log file is way too big to download on my (slow) home network.
You say that you have uploaded (bpbrm and bpbkar), but there is only one large file?
What you need to look for in the logs are the PID's seen in Activity Monitor.
e.g. bpbrm log file for 15 July, look for PID 11572 in square brackets [ ] .
bpbkar on 15 July, look for all PID entries [15264].
So, if you want to troubleshoot failures that happened on 15 July, there is no use looking at a log dated 13 July (071320.log).
07-16-2020 07:22 AM
You may want to check if your master (which is I presume it is your backup host) if it has enough space, also verify its resources after the execution of backups.
Please share bpbrm (level=3 of verbosity) and you may want to set vxms logs on the master server to get more information.
To configure VxMS logging on a Windows backup host
install_path\NetBackup\logs\vxms
Note:
For logging to occur, the VxMS folder must exist.
Note:
If you have run the NetBackup mklogdir.bat command, the VxMS log directory already exists.
VXMS_VERBOSE
in the following location:
HKEY_LOCAL_MACHINE > SOFTWARE > Veritas > NetBackup > CurrentVersion > Config
VXMS_VERBOSE
to 0 or greater. Larger numbers result in more verbose logs (set it to 5 it's enough)more information: https://www.veritas.com/support/en_US/doc/21902280-127283730-0/v23649764-127283730
good luck