allocation failed error 10;

[Spartacus81]

Friends, recently come across this issue and was wondering if you share your experiences and knowledge...

not sure if the memory allocation is on media server or on the client.. checked spoold.log file and couldnt find anything relevant there.. any suggestion/idea will be appreciated..

09/18/2012 09:43:15 - requesting resource gssuanbmas01.NBU_CLIENT.MAXJOBS.mspdurfp01
09/18/2012 09:43:15 - requesting resource gssuanbmas01.NBU_POLICY.MAXJOBS.gssuanbmed02_MS_Sites
09/18/2012 09:43:15 - granted resource  gssuanbmas01.NBU_CLIENT.MAXJOBS.mspdurfp01
09/18/2012 09:43:15 - granted resource  gssuanbmas01.NBU_POLICY.MAXJOBS.gssuanbmed02_MS_Sites
09/18/2012 09:43:15 - granted resource  MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=gssuanbmed02_Pool2;Path=PureDiskVolume;StorageServ...
09/18/2012 09:43:15 - granted resource  gssuanbmed02_Pool2
09/18/2012 09:43:16 - estimated 0 kbytes needed
09/18/2012 09:43:16 - Info nbjm (pid=24366) started backup (backupid=mspdurfp01_1347957796) job for client mspdurfp01, policy gssuanbmed02_MS_Sites, schedule Manual_Full on storage unit gssuanbmed02_Pool2
09/18/2012 09:43:17 - started process bpbrm (pid=21321)
09/18/2012 09:43:23 - connecting
09/18/2012 09:43:24 - connected; connect time: 0:00:00
09/18/2012 09:43:43 - begin writing
09/18/2012 09:43:55 - end writing; write time: 0:00:12
09/18/2012 09:45:39 - Info bpbrm (pid=21321) mspdurfp01 is the host to backup data from
09/18/2012 09:45:39 - Info bpbrm (pid=21321) reading file list from client
09/18/2012 09:45:40 - Info bpbrm (pid=21321) accelerator enabled
09/18/2012 09:45:41 - Info bpbrm (pid=21321) There is no complete backup image match with track journal, a regular full backup will be performed.
09/18/2012 09:45:42 - Info bpbrm (pid=21321) starting bpbkar on client
09/18/2012 09:45:42 - Info bpbkar (pid=1672) Backup started
09/18/2012 09:45:42 - Info bpbrm (pid=21321) bptm pid: 21324
09/18/2012 09:45:43 - Info bptm (pid=21324) start
09/18/2012 09:45:43 - Info bptm (pid=21324) using 262144 data buffer size
09/18/2012 09:45:43 - Info bptm (pid=21324) using 64 data buffers
09/18/2012 09:45:43 - Info gssuanbmed02 (pid=21324) Using OpenStorage client direct to backup from client mspdurfp01 to gssuanbmed02
09/18/2012 09:46:01 - Critical bptm (pid=21324) image open failed: error 2060013: no more entries
09/18/2012 09:46:10 - Error bptm (pid=21324) media manager terminated by parent process
09/18/2012 09:46:13 - Info bpbkar (pid=1672) done. status: 10: allocation failed
termination requested by administrator  (150)
 

[watsons]

We usually see this "allocation failed (10)" when accessing image/media database (process bpdbm), but yours look new to me.. bpbkar throwing that error out.

That technote about accessing image/media database is: http://www.symantec.com/docs/TECH19038, but I doubt it's going to help.

This type of error has to do with memory allocation, consider checking the memory usage in the client end to avoid high memory usage during backup.

[Spartacus81]

Thanks Watson..

I tried backing up the c:\temp\1.txt with both client side deduplication and media server deduplication to make sure it worked and that worked..

Basically these servers sitting at remote site and has only 2Meg link and over 1TB of data.. i have done the seeding and updated the pd.conf file on the client (fp_cahce_client_policy) to laod the fingerprints once the subsequent backup followed the seed run.. so that only data that has changed get transferred over the link..i had set the number of client streams to 2.. both servers are in pretty healthy states with 16GB (W2K8 64Bit) and 2GB (W2K3 32Bit) memory... reducing the number of streams on each client to 1 did the magic and backups are now running okay.. C: drive backup has completed took 1hr 30mins to backup 8GB over a 2meg link.. but looking at the stats: dedup was 95% which  means the data was there but the cache hits is 0.0% whereas it should be more than 0% because thats the fp_cache_client_policy attribute does loads the fingerprints in the memory.. any idea?

09/18/2012 12:22:19 - Info bpbkar (pid=1792) accelerator sent 8945674240 bytes out of 8945674240 bytes to server, optimization 0.0%
09/18/2012 12:22:19 - Info bpbkar (pid=1792) bpbkar waited 33542 times for empty buffer, delayed 165037 times.
09/18/2012 12:22:32 - Info gssuanbmed02 (pid=23207) StorageServer=PureDisk:gssuanbmed02; Report=PDDO Stats for (gssuanbmed02): scanned: 8736404 KB, CR sent: 402977 KB, CR sent over FC: 0 KB, dedup: 95.4%, cache hits: 0 (0.0%)
09/18/2012 12:22:33 - Info gssuanbmed02 (pid=23207) Using the media server to write NBU data for backup mspdurfp01_1347961826 to gssuanbmed02
09/18/2012 12:22:37 - Info bptm (pid=23207) EXITING with status 0 <----------
09/18/2012 12:22:37 - Info gssuanbmed02 (pid=23207) StorageServer=PureDisk:gssuanbmed02; Report=PDDO Stats for (gssuanbmed02): scanned: 2008 KB, CR sent: 681 KB, CR sent over FC: 0 KB, dedup: 66.1%
09/18/2012 12:22:37 - Info bpbrm (pid=23187) validating image for client mspdurfp01
09/18/2012 12:22:38 - Info bpbkar (pid=1792) done. status: 0: the requested operation was successfully completed
the requested operation was successfully completed  (0)

[watsons]

Are you saying the w2k8 has 16GB of memory while the w2k3 only has 2GB memory?? That is kind of a huge difference between them. Can you increase memory on w2k3, or make sure no other process are taking up memory when the backup is running?

I am not familiar with analyzing the dedup rate and cache hit, still learning about it  - sorry can't comment much on that. But for that job, it seems to complete successfully, so you didn't get the "allocation error" anymore?

[Spartacus81 · Accepted solution]

@ Watson: I know its a huge difference but i ran multiple tests and to be honest the memory on the W2K3 server was hardly even used... yes both jobs ran okay the only change i made was to reduce the number of streams from 2 to 1 and it worked.. though jobs are still running but they are nearly finished..