Highlighted

backup - allocation failed (10) - but enough memory

hi all,

we are experiencing backup failures with  error code 10,but we were monitoring memory usage and it was on average level. problem is only with FS /var which has around 2mil. files. Capture.PNG

UX team says maxdsiz_64bit is at max value. 
******** [root]:/var: >kmtune -q maxdsiz_64bit
Parameter Current Dyn Planned Module Version
==============================================================================
maxdsiz_64bit 4294967296 - 4294967296

Client HP-UX 11.11 64bit virtual has NB version 6.5. Two media servers  HP-UX 11.31 64bit physical. 

From my findings every related topic suggest to free up memory we don't know where to move with this. 

I would appreciate any suggestions. I attached log also,hopefully correctly. Thanks.

 

Tags (2)
19 Replies

Re: backup - allocation failed (10) - but enough memory

Hey

Please try to run ls -la /var |wc -l

does this command end successfully? If not this will be related to OS - revert back to admins of this HPUX machine.

Also please enable bpbkar logging on client end. BTW you are aware that this NBU version is end of life - I assume... time to move forward...

Re: backup - allocation failed (10) - but enough memory

Hi,

thanks for reply. I have already asked ux team to give us number of files under /var. [root]:/var: >find . -type f | wc -l
2293734

ls wont work as it doesnt search rekursively I believe. I will update with bpbkar. 

Thanks.

 

 

Re: backup - allocation failed (10) - but enough memory

You will see same error if a kernel setting is to low - e.g semaphore.

Please see this tech note - it mention AIX, but cause could be the same:

AIX Client filesystem backup fails with Error Status 10 "allocation failed"

http://www.veritas.com/docs/000023785

That said, 2 mill files in /var is ridiculous, I bet most of those 2 mill files are left over and can be deleted.

Re: backup - allocation failed (10) - but enough memory

Hi, 

we have tried contact hpux team with this, they replied with:

"there are many parameters in ulimit. , SO which parameter you want set for ulimit?
For which user id you want to made changes in ulimit?" also "Provided vendor recommedations are for AIX server & not for hpux"

this issue is ongoing for 2 months,we already checked many articles on this issue,thats why I have posted here.

Re: backup - allocation failed (10) - but enough memory

hi,

i am adding log file from bpbkar. Thanks.

Re: backup - allocation failed (10) - but enough memory

The bpbkar logs appears to run thru a list of socket special files.. have you tried excluding the dir

/var/opt/dce/rpc/local

from backup and see if that helps?

Another long shot: Check also if /swap or /tmp partition is nearly full. 

Re: backup - allocation failed (10) - but enough memory

You have to be a bit rude here and tell them its their job to figure that out. Their reply is completly helpless

Re: backup - allocation failed (10) - but enough memory

Hi,

so we added /var/opt/dce/rpc/local in exclude list. It didn't help. Attaching new log for anyone to look at. Thanks in advance for any ideas.

Re: backup - allocation failed (10) - but enough memory

Hey

I will repeat my self - have unix team to run this

time ls -la /var |wc -l
does this command end successfully? Do not run find only just above one... share outcome...
Also can you run ulimit -a on this box as the same user on which NBU is running ( I assume root it is) and provide outputs?

Re: backup - allocation failed (10) - but enough memory

has unix admin done this yet?

ulimit –d unlimited

Re: backup - allocation failed (10) - but enough memory

Hi,so they sent us outputs. For ulimit I'm not sure what those limits should be set to. Should I tell them to adjust some? Thank you.

<client>[root]:/rootSmiley Embarassed ls -la /var |wc -l

29

<client>[root]:/rootSmiley Embarassed ulimit -a

time(seconds) unlimited

file(blocks) unlimited

data(kbytes) 96732

stack(kbytes) 8192

memory(kbytes) unlimited

coredump(blocks) 4194303

nofiles(descriptors) 2048

 

Re: backup - allocation failed (10) - but enough memory

Hey

Sorry... there was a typo from my end...

instead of running this

time ls -la /var |wc -l

please run this

time ls -laR /var |wc -l

I forgot to add R which stands for recursive...

and about ulimit outputs on my HPUX in version HP-UX xxxx  B.11.31 U ia64 yyyyyy unlimited-user license

it looks like this

ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2015464
stack(kbytes) 131072
memory(kbytes) unlimited
coredump(blocks) 4194303

Re: backup - allocation failed (10) - but enough memory

Hi,no they havent done it. Should they do it for root user?also is it hp-ux comman?

Re: backup - allocation failed (10) - but enough memory

nofiles should be as a minimum, 8192 ....

Re: backup - allocation failed (10) - but enough memory

Hi,

so I have a new update. I suggested to hpux team to rise limits and provide ls -laR /var |wc -l.

They gave us this:

<client>[root]:/rootSmiley Embarassedls -laR /var |wc -l

ls: out of memory

83591

 

<client>[root]:/rootSmiley Embarassedulimit -d 2015464

sh: ulimit: The specified value exceeds the user's allowable limit.

 

<client>[root]:/rootSmiley Embarassedulimit -s 131072

sh: ulimit: The specified value exceeds the user's allowable limit.

Is there any help with this outside of getting more RAM?

 

Re: backup - allocation failed (10) - but enough memory

ls: out of memory

You can see that this is not a NetBackup issue.
NBU is suffering as a result of the same "out of memory " issue.

You really need to escalate the OS team for lack of response and commitment.

 

Re: backup - allocation failed (10) - but enough memory

They gave us this:

ls: out of memory

 

Either they are messing with you or don't understand their role. 

Present the technote to management and say you need some assistance.

https://www.veritas.com/support/en_US/article.000023785

Re: backup - allocation failed (10) - but enough memory

Hi,

hp-ux team is suggesting to reboot client server. Anyone thinks it will help? Thanks.

Re: backup - allocation failed (10) - but enough memory

Hey

Well they are (unix admins) SME on HPUX Smiley Wink so allow them to do the needful... again this is issue not related to NBU... if reboot will not help have them to purchase more memory to this server and clean up this folder from unnecessary files... Suggest them to open a case with HP as again NBU has nothing in common with this issue.