Highlighted

(10) allocation failed

Good day

Please advise vmdk backups to data domain are failign with error code 10.

(10) allocation failed

02/10/2017 20:28:27 - Info nbjm (pid=3824) starting backup job (jobid=184223) for client NRBOWEB101, policy Bosasa_VMDK, schedule Weekly
02/10/2017 20:28:27 - Info nbjm (pid=3824) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=184223, request id:{8C21EE5A-0689-4642-94D1-F51207284609})
02/10/2017 20:28:27 - requesting resource dp-DDNDC1-FC-stu
02/10/2017 20:28:27 - requesting resource nrecnetbk111.NBU_CLIENT.MAXJOBS.NRBOWEB101
02/10/2017 20:28:28 - granted resource  nrecnetbk111.NBU_CLIENT.MAXJOBS.NRBOWEB101
02/10/2017 20:28:28 - granted resource  MediaID=@aaaaf;DiskVolume=NRECNETBK120_STU;DiskPool=dp-DDNDC1-FC;Path=NRECNETBK120_STU;StorageServer=DFC-bcx-ndc1-fc;MediaServer=nrecnetbk120
02/10/2017 20:28:28 - granted resource  dp-DDNDC1-FC-stu
02/10/2017 20:28:33 - estimated 80196442 kbytes needed
02/10/2017 20:28:33 - Info nbjm (pid=3824) started backup (backupid=NRBOWEB101_1486751308) job for client NRBOWEB101, policy Bosasa_VMDK, schedule Weekly on storage unit dp-DDNDC1-FC-stu using backup host nrecnetbk120
02/10/2017 20:29:05 - started process bpbrm (pid=1204)
02/10/2017 20:31:18 - connecting
02/10/2017 20:31:23 - connected; connect time: 0:00:00
02/10/2017 20:33:02 - Info bptm (pid=6300) start
02/10/2017 20:37:06 - end writing
02/10/2017 20:38:30 - Error bpbrm (pid=1204) could not send server status message
02/10/2017 20:38:36 - Info bpbkar32 (pid=0) done
02/10/2017 20:38:41 - Info bpbkar32 (pid=0) done. status: 10: allocation failed
02/13/2017 08:02:43 - job 184223 was restarted as job 188135
allocation failed  (10)

 

8 Replies
Highlighted

Re: (10) allocation failed

More info please...

Are all VM backups failing using this media server? Or just this one VM?
Or only when media server is heavily loaded? 
Can you monitor resources on media server nrecnetbk120 during backup period?

Are backups going through for other VM hosts?
Or using another media server?

Please ensure that all of these log folders exists on backup hosts/ media server  nrecnetbk120:
bpbrm bpbrm bpbkar

 

 

Highlighted

Re: (10) allocation failed

Hi Marianne

its only few VM's are failing using this media server, we do have only 1 media server for VMDK backups.

all log folders are exists.

do you we need to monitor resources on clinet or media server ?

Highlighted

Re: (10) allocation failed

The Media server.

Since the media server is the VMware backup host, it is performing both the function of media server (bptm and bpbrm) and the client (bpbkar).

This is the NBU explanation of status 10:

Explanation: The system memory allocation fails because of insufficient system memory available. A possible cause is that the system is overloaded with too many processes and not enough physical or virtual memory.

Recommended action: Free up memory by terminating any unneeded processes that consume memory. Add more swap space or physical memory.

Please also read the following chapter in NetBackup Backup Planning and Performance Tuning Guide  :  Chapter 11 OS-related tuning factors for Windows

Highlighted

Re: (10) allocation failed

It's a memory allocation issue :

Info bpbkar32 (pid=0) done. status: 10: allocation failed

http://www.veritas.com/docs/000090335

 

Highlighted

Re: (10) allocation failed

Hi Marriane

(bptm and bpbrm) and the client (bpbkar) logs already exists and server config is as below.

do you think we can schedule backups to different times to monitor or is there any settings that i need to change?

Windows 2012

128GB Ram

98GB swap

11 Core Cpu's

 

 

Highlighted

Re: (10) allocation failed

Please monitor resources during peak backup hours before making any changes.

And PLEASE take time and go through the Windows chapter of the Perf Tuning Guide.

In the meantime, you may want to try the NOSHM touch-file.
I personally experienced Status 10 on 1 media server a couple of years ago(2 identical media servers, error only on 1 server). 
NOSHM fixed my problem. See http://www.veritas.com/docs/000026448

 

Highlighted

Re: (10) allocation failed

Will do.

Thank you so much

Highlighted

Re: (10) allocation failed

@RaghavTuppathi any update here?