05-31-2020 07:37 PM
Hi,
I have backup problem recently and shown (28) failed trying to fork a process.
The backup size is around 13TB, backup type NDMP, storage DD4200.
It's not failed every backup, can anyone advise what is the problem?
Many thanks.
Regards,
Ken
05-31-2020 11:11 PM - edited 05-31-2020 11:12 PM
I cannot find any 'known issues' with status 28 for NDMP backups.
This means that something in this environment is causing it.
Please give us all relevant details, such as NDMP filer and OS version, backup policy details,
all text in Job Details of failed job, ndmpagent log on the ndmpagent log on the ndmp media server.
These logs on the media server may also be helpful: bpbrm and bptm.
These log folders do not exist by default - please create them if they do not exist.
Please copy logs to process name, e.g. ndmpagent.txt before uploading here.
05-31-2020 11:34 PM
Hi Marianne,
Which logging level need to set for those logs ? (ndmpagent, BMTM and BPBRM)
06-01-2020 01:21 AM
ndmpagent is a unified log and exist by default.
This log should be the most useful one. I cannot remember what the default logging level is for unified logs, but for initial troubleshooting, the default should be fine. I believe to only ever increase unified logging level when requested by Veritas Support.
For bptm and bpbrm, level 3 usually gives sufficient info.
06-02-2020 12:33 AM
Error message and ndmpagent log attached
06/02/2020 14:34:29 - Error bpbrm (pid=27488) db_FLISTsend failed: failed trying to fork a process (28)
06/02/2020 14:34:30 - Error ndmpagent (pid=17172) terminated by parent process
06/02/2020 14:34:30 - Info ndmpagent (pid=0) done
06/02/2020 14:34:30 - Info ndmpagent (pid=17172) Received ABORT request from bptm
06/02/2020 14:34:30 - Error ndmpagent (pid=17172) NDMP backup failed, path = /root_vdm_1/P7FS0001_vol1
06/02/2020 14:35:00 - Error ndmpagent (pid=17172) VNX5600VDM1: Medium error
06/02/2020 14:35:00 - Warning ndmpagent (pid=17172) VNX5600VDM1: Write failed on archive volume 1
06/02/2020 14:35:00 - Info ndmpagent (pid=17172) VNX5600VDM1: server_archive: emctar vol 1, 3405047 files, 0 bytes read, 2013001940992 bytes written
06/02/2020 14:35:00 - Error ndmpagent (pid=17172) VNX5600VDM1: Backup is aborted.
06/02/2020 14:35:12 - Info ndmpagent (pid=0) done. status: 28: failed trying to fork a process
06/02/2020 14:35:12 - end writing; write time: 8:38:07
failed trying to fork a process (28)
06-02-2020 02:32 AM
It looks like the bpbrm process that is reporting a problem:
Error bpbrm (pid=27488) db_FLISTsend failed: failed trying to fork a process (28)
We will need to see bpbrm log.
Have you tried to monitor media server resources during backup?
06-02-2020 03:04 AM
The backup being retry, can't restart to activate the bpbrm log.
The master sever is also act as media server, and overall memory is around 30% and CPU under 10%
06-02-2020 04:37 AM
No restart needed to activate bpbrm and/or bptm log.
06-02-2020 12:19 PM
Looks like the media server is busy ? Have you verified CPU and Memory on this media server ?
06-02-2020 05:36 PM
I have set the logging = Level 3 for BPBRM & BPTM, but no logs folder created at \Program Files\Veritas\NetBackup\logs .
06-02-2020 10:58 PM
Please have another look at my post on Monday:
"These log folders do not exist by default - please create them if they do not exist. "