03-29-2017 03:33 AM - edited 03-29-2017 03:38 AM
hi all,
we are experiencing backup failures with error code 10,but we were monitoring memory usage and it was on average level. problem is only with FS /var which has around 2mil. files.
UX team says maxdsiz_64bit is at max value.
******** [root]:/var: >kmtune -q maxdsiz_64bit
Parameter Current Dyn Planned Module Version
==============================================================================
maxdsiz_64bit 4294967296 - 4294967296
Client HP-UX 11.11 64bit virtual has NB version 6.5. Two media servers HP-UX 11.31 64bit physical.
From my findings every related topic suggest to free up memory we don't know where to move with this.
I would appreciate any suggestions. I attached log also,hopefully correctly. Thanks.
03-29-2017 03:56 AM
Hey
Please try to run ls -la /var |wc -l
does this command end successfully? If not this will be related to OS - revert back to admins of this HPUX machine.
Also please enable bpbkar logging on client end. BTW you are aware that this NBU version is end of life - I assume... time to move forward...
03-29-2017 06:33 AM - edited 03-29-2017 06:33 AM
Hi,
thanks for reply. I have already asked ux team to give us number of files under /var. [root]:/var: >find . -type f | wc -l
2293734
ls wont work as it doesnt search rekursively I believe. I will update with bpbkar.
Thanks.
03-29-2017 06:36 AM - edited 03-29-2017 06:39 AM
You will see same error if a kernel setting is to low - e.g semaphore.
Please see this tech note - it mention AIX, but cause could be the same:
AIX Client filesystem backup fails with Error Status 10 "allocation failed"
http://www.veritas.com/docs/000023785
That said, 2 mill files in /var is ridiculous, I bet most of those 2 mill files are left over and can be deleted.
03-29-2017 06:47 AM
Hi,
we have tried contact hpux team with this, they replied with:
"there are many parameters in ulimit. , SO which parameter you want set for ulimit?
For which user id you want to made changes in ulimit?" also "Provided vendor recommedations are for AIX server & not for hpux"
this issue is ongoing for 2 months,we already checked many articles on this issue,thats why I have posted here.
03-30-2017 03:53 AM
hi,
i am adding log file from bpbkar. Thanks.
03-30-2017 07:07 PM
The bpbkar logs appears to run thru a list of socket special files.. have you tried excluding the dir
/var/opt/dce/rpc/local
from backup and see if that helps?
Another long shot: Check also if /swap or /tmp partition is nearly full.
04-03-2017 03:31 AM
You have to be a bit rude here and tell them its their job to figure that out. Their reply is completly helpless
04-12-2017 02:47 AM
Hi,
so we added /var/opt/dce/rpc/local in exclude list. It didn't help. Attaching new log for anyone to look at. Thanks in advance for any ideas.
04-12-2017 03:40 AM
Hey
I will repeat my self - have unix team to run this
time ls -la /var |wc -l
does this command end successfully? Do not run find only just above one... share outcome...
Also can you run ulimit -a on this box as the same user on which NBU is running ( I assume root it is) and provide outputs?
04-17-2017 01:44 PM - edited 04-17-2017 01:47 PM
has unix admin done this yet?
ulimit –d unlimited
04-19-2017 01:42 AM
Hi,so they sent us outputs. For ulimit I'm not sure what those limits should be set to. Should I tell them to adjust some? Thank you.
<client>[root]:/root:> ls -la /var |wc -l
29
<client>[root]:/root:> ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 96732
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048
04-19-2017 01:55 AM - edited 04-19-2017 01:56 AM
Hey
Sorry... there was a typo from my end...
instead of running this
time ls -la /var |wc -l
please run this
time ls -laR /var |wc -l
I forgot to add R which stands for recursive...
and about ulimit outputs on my HPUX in version HP-UX xxxx B.11.31 U ia64 yyyyyy unlimited-user license
it looks like this
ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2015464
stack(kbytes) 131072
memory(kbytes) unlimited
coredump(blocks) 4194303
04-19-2017 02:13 AM
Hi,no they havent done it. Should they do it for root user?also is it hp-ux comman?
04-19-2017 02:29 AM
nofiles should be as a minimum, 8192 ....
04-20-2017 11:09 PM
Hi,
so I have a new update. I suggested to hpux team to rise limits and provide ls -laR /var |wc -l.
They gave us this:
<client>[root]:/root:>ls -laR /var |wc -l
ls: out of memory
83591
<client>[root]:/root:>ulimit -d 2015464
sh: ulimit: The specified value exceeds the user's allowable limit.
<client>[root]:/root:>ulimit -s 131072
sh: ulimit: The specified value exceeds the user's allowable limit.
Is there any help with this outside of getting more RAM?
04-20-2017 11:27 PM
ls: out of memory
You can see that this is not a NetBackup issue.
NBU is suffering as a result of the same "out of memory " issue.
You really need to escalate the OS team for lack of response and commitment.
04-21-2017 09:54 AM
They gave us this:
ls: out of memory
Either they are messing with you or don't understand their role.
Present the technote to management and say you need some assistance.
04-27-2017 06:49 AM - edited 04-27-2017 06:50 AM
Hi,
hp-ux team is suggesting to reboot client server. Anyone thinks it will help? Thanks.
04-27-2017 07:00 AM - edited 04-27-2017 07:00 AM
Hey
Well they are (unix admins) SME on HPUX ;) so allow them to do the needful... again this is issue not related to NBU... if reboot will not help have them to purchase more memory to this server and clean up this folder from unnecessary files... Suggest them to open a case with HP as again NBU has nothing in common with this issue.