02-12-2017 10:06 PM
Good day
Please advise vmdk backups to data domain are failign with error code 10.
(10) allocation failed
02/10/2017 20:28:27 - Info nbjm (pid=3824) starting backup job (jobid=184223) for client NRBOWEB101, policy Bosasa_VMDK, schedule Weekly
02/10/2017 20:28:27 - Info nbjm (pid=3824) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=184223, request id:{8C21EE5A-0689-4642-94D1-F51207284609})
02/10/2017 20:28:27 - requesting resource dp-DDNDC1-FC-stu
02/10/2017 20:28:27 - requesting resource nrecnetbk111.NBU_CLIENT.MAXJOBS.NRBOWEB101
02/10/2017 20:28:28 - granted resource nrecnetbk111.NBU_CLIENT.MAXJOBS.NRBOWEB101
02/10/2017 20:28:28 - granted resource MediaID=@aaaaf;DiskVolume=NRECNETBK120_STU;DiskPool=dp-DDNDC1-FC;Path=NRECNETBK120_STU;StorageServer=DFC-bcx-ndc1-fc;MediaServer=nrecnetbk120
02/10/2017 20:28:28 - granted resource dp-DDNDC1-FC-stu
02/10/2017 20:28:33 - estimated 80196442 kbytes needed
02/10/2017 20:28:33 - Info nbjm (pid=3824) started backup (backupid=NRBOWEB101_1486751308) job for client NRBOWEB101, policy Bosasa_VMDK, schedule Weekly on storage unit dp-DDNDC1-FC-stu using backup host nrecnetbk120
02/10/2017 20:29:05 - started process bpbrm (pid=1204)
02/10/2017 20:31:18 - connecting
02/10/2017 20:31:23 - connected; connect time: 0:00:00
02/10/2017 20:33:02 - Info bptm (pid=6300) start
02/10/2017 20:37:06 - end writing
02/10/2017 20:38:30 - Error bpbrm (pid=1204) could not send server status message
02/10/2017 20:38:36 - Info bpbkar32 (pid=0) done
02/10/2017 20:38:41 - Info bpbkar32 (pid=0) done. status: 10: allocation failed
02/13/2017 08:02:43 - job 184223 was restarted as job 188135
allocation failed (10)
02-12-2017 10:38 PM
More info please...
Are all VM backups failing using this media server? Or just this one VM?
Or only when media server is heavily loaded?
Can you monitor resources on media server nrecnetbk120 during backup period?
Are backups going through for other VM hosts?
Or using another media server?
Please ensure that all of these log folders exists on backup hosts/ media server nrecnetbk120:
bpbrm bpbrm bpbkar
02-13-2017 12:00 AM
Hi Marianne
its only few VM's are failing using this media server, we do have only 1 media server for VMDK backups.
all log folders are exists.
do you we need to monitor resources on clinet or media server ?
02-13-2017 12:10 AM
The Media server.
Since the media server is the VMware backup host, it is performing both the function of media server (bptm and bpbrm) and the client (bpbkar).
This is the NBU explanation of status 10:
Explanation: The system memory allocation fails because of insufficient system memory available. A possible cause is that the system is overloaded with too many processes and not enough physical or virtual memory.
Recommended action: Free up memory by terminating any unneeded processes that consume memory. Add more swap space or physical memory.
Please also read the following chapter in NetBackup Backup Planning and Performance Tuning Guide : Chapter 11 OS-related tuning factors for Windows
02-13-2017 01:32 AM
It's a memory allocation issue :
Info bpbkar32 (pid=0) done. status: 10: allocation failed
http://www.veritas.com/docs/000090335
02-13-2017 01:51 AM
Hi Marriane
(bptm and bpbrm) and the client (bpbkar) logs already exists and server config is as below.
do you think we can schedule backups to different times to monitor or is there any settings that i need to change?
Windows 2012
128GB Ram
98GB swap
11 Core Cpu's
02-13-2017 02:22 AM
Please monitor resources during peak backup hours before making any changes.
And PLEASE take time and go through the Windows chapter of the Perf Tuning Guide.
In the meantime, you may want to try the NOSHM touch-file.
I personally experienced Status 10 on 1 media server a couple of years ago(2 identical media servers, error only on 1 server).
NOSHM fixed my problem. See http://www.veritas.com/docs/000026448
02-13-2017 02:28 AM
Will do.
Thank you so much
02-22-2017 12:55 AM
09-08-2020 09:33 AM
Ran into this issue with some of my backups. After troubleshooting and testing, I found my issue to be related to client side encryption. So another area to check if you experience this issue. Make sure you encryption key was successful placed on client side.