cancel
Showing results for 
Search instead for 
Did you mean: 

backup - allocation failed (10) - but enough memory

brad57
Level 3

hi all,

we are experiencing backup failures with  error code 10,but we were monitoring memory usage and it was on average level. problem is only with FS /var which has around 2mil. files. Capture.PNG

UX team says maxdsiz_64bit is at max value. 
******** [root]:/var: >kmtune -q maxdsiz_64bit
Parameter Current Dyn Planned Module Version
==============================================================================
maxdsiz_64bit 4294967296 - 4294967296

Client HP-UX 11.11 64bit virtual has NB version 6.5. Two media servers  HP-UX 11.31 64bit physical. 

From my findings every related topic suggest to free up memory we don't know where to move with this. 

I would appreciate any suggestions. I attached log also,hopefully correctly. Thanks.

 

19 REPLIES 19

quebek
Moderator
Moderator
   VIP    Certified

Hey

Please try to run ls -la /var |wc -l

does this command end successfully? If not this will be related to OS - revert back to admins of this HPUX machine.

Also please enable bpbkar logging on client end. BTW you are aware that this NBU version is end of life - I assume... time to move forward...

Hi,

thanks for reply. I have already asked ux team to give us number of files under /var. [root]:/var: >find . -type f | wc -l
2293734

ls wont work as it doesnt search rekursively I believe. I will update with bpbkar. 

Thanks.

 

 

Nicolai
Moderator
Moderator
Partner    VIP   

You will see same error if a kernel setting is to low - e.g semaphore.

Please see this tech note - it mention AIX, but cause could be the same:

AIX Client filesystem backup fails with Error Status 10 "allocation failed"

http://www.veritas.com/docs/000023785

That said, 2 mill files in /var is ridiculous, I bet most of those 2 mill files are left over and can be deleted.

Hi, 

we have tried contact hpux team with this, they replied with:

"there are many parameters in ulimit. , SO which parameter you want set for ulimit?
For which user id you want to made changes in ulimit?" also "Provided vendor recommedations are for AIX server & not for hpux"

this issue is ongoing for 2 months,we already checked many articles on this issue,thats why I have posted here.

hi,

i am adding log file from bpbkar. Thanks.

The bpbkar logs appears to run thru a list of socket special files.. have you tried excluding the dir

/var/opt/dce/rpc/local

from backup and see if that helps?

Another long shot: Check also if /swap or /tmp partition is nearly full. 

Nicolai
Moderator
Moderator
Partner    VIP   

You have to be a bit rude here and tell them its their job to figure that out. Their reply is completly helpless

Hi,

so we added /var/opt/dce/rpc/local in exclude list. It didn't help. Attaching new log for anyone to look at. Thanks in advance for any ideas.

quebek
Moderator
Moderator
   VIP    Certified

Hey

I will repeat my self - have unix team to run this

time ls -la /var |wc -l
does this command end successfully? Do not run find only just above one... share outcome...
Also can you run ulimit -a on this box as the same user on which NBU is running ( I assume root it is) and provide outputs?

has unix admin done this yet?

ulimit –d unlimited

Hi,so they sent us outputs. For ulimit I'm not sure what those limits should be set to. Should I tell them to adjust some? Thank you.

<client>[root]:/root:> ls -la /var |wc -l

29

<client>[root]:/root:> ulimit -a

time(seconds) unlimited

file(blocks) unlimited

data(kbytes) 96732

stack(kbytes) 8192

memory(kbytes) unlimited

coredump(blocks) 4194303

nofiles(descriptors) 2048

 

quebek
Moderator
Moderator
   VIP    Certified

Hey

Sorry... there was a typo from my end...

instead of running this

time ls -la /var |wc -l

please run this

time ls -laR /var |wc -l

I forgot to add R which stands for recursive...

and about ulimit outputs on my HPUX in version HP-UX xxxx  B.11.31 U ia64 yyyyyy unlimited-user license

it looks like this

ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2015464
stack(kbytes) 131072
memory(kbytes) unlimited
coredump(blocks) 4194303

Hi,no they havent done it. Should they do it for root user?also is it hp-ux comman?

mph999
Level 6
Employee Accredited

nofiles should be as a minimum, 8192 ....

Hi,

so I have a new update. I suggested to hpux team to rise limits and provide ls -laR /var |wc -l.

They gave us this:

<client>[root]:/root:>ls -laR /var |wc -l

ls: out of memory

83591

 

<client>[root]:/root:>ulimit -d 2015464

sh: ulimit: The specified value exceeds the user's allowable limit.

 

<client>[root]:/root:>ulimit -s 131072

sh: ulimit: The specified value exceeds the user's allowable limit.

Is there any help with this outside of getting more RAM?

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

ls: out of memory

You can see that this is not a NetBackup issue.
NBU is suffering as a result of the same "out of memory " issue.

You really need to escalate the OS team for lack of response and commitment.

 

They gave us this:

ls: out of memory

 

Either they are messing with you or don't understand their role. 

Present the technote to management and say you need some assistance.

https://www.veritas.com/support/en_US/article.000023785

Hi,

hp-ux team is suggesting to reboot client server. Anyone thinks it will help? Thanks.

quebek
Moderator
Moderator
   VIP    Certified

Hey

Well they are (unix admins) SME on HPUX ;) so allow them to do the needful... again this is issue not related to NBU... if reboot will not help have them to purchase more memory to this server and clean up this folder from unnecessary files... Suggest them to open a case with HP as again NBU has nothing in common with this issue.