04-13-2015 11:54 PM
Master solaris 10, cluster server.. NBU 7.6.0.3
Media server HP-UX B.11.31 U ia64 4235433303 unlimited-user license , NetBackup-HP-UX11.31 7.6.0.3
media server: cat /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS
262144
media server: cat /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS
40
earlier this media server was working perfectly fine, today I have recieved lots of error code 89 failures.
<16> create_shared_memory: could not allocate enough shared memory for backup buffers, No space left on device
14/04/2015 06:59:39 - Error bpbrm(pid=8318) Could not get shared memory for bpbrm child process communication, No space left on device (28)
14/04/2015 06:59:40 - Info bphdb(pid=0) done. status: 89: problems encountered during setup of shared memory
Please suggest
Solved! Go to Solution.
04-14-2015 12:11 AM
HP ...
Bit odd, this issue is more common on Solaris - has someone suddently turned on multiplexing ?
What is the sharded memory set to on the server, this is an OS kernel setting called ...
SHMMAX
It may need increasing, however, if this was working, something has changed - or the server is under more load / something welse using shared memory.
First idea is just to reboot, this should clear things out and if it was a one-off issue it may not reoccur.
It would be sensible to do this, as if it does n't retuurn it saves making changes that would therefore be unnecessary and may introduce other issues.
04-14-2015 12:11 AM
HP ...
Bit odd, this issue is more common on Solaris - has someone suddently turned on multiplexing ?
What is the sharded memory set to on the server, this is an OS kernel setting called ...
SHMMAX
It may need increasing, however, if this was working, something has changed - or the server is under more load / something welse using shared memory.
First idea is just to reboot, this should clear things out and if it was a one-off issue it may not reoccur.
It would be sensible to do this, as if it does n't retuurn it saves making changes that would therefore be unnecessary and may introduce other issues.
04-14-2015 12:21 AM
How to check setting SHMMAX
Multiplexing was already in place from long back. but issue has occured all of sudden today..
04-14-2015 12:22 AM
system state as of now.
Load averages: 0.12, 0.13, 0.13
327 processes: 260 sleeping, 66 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.10 0.8% 0.0% 3.2% 96.0% 0.0% 0.0% 0.0% 0.0%
1 0.14 2.2% 0.0% 4.8% 93.1% 0.0% 0.0% 0.0% 0.0%
2 0.11 1.0% 0.0% 4.2% 94.9% 0.0% 0.0% 0.0% 0.0%
3 0.11 1.2% 0.0% 1.8% 97.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.12 1.4% 0.0% 3.6% 95.1% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 2074632K (1658640K) real, 6475268K (5128448K) virtual, 1906204K free Page# 1/12
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
3 ? 660 root 168 20 350M 42496K sleep 70:19 9.90 9.88 bptm
3 ? 18813 root 168 20 362M 50720K sleep 81:48 4.54 4.53 bptm
2 ? 676 root 154 20 295M 21808K sleep 14:41 2.11 2.10 bptm
0 ? 3916 root -16 20 41752K 17040K run 6456:25 1.87 1.87 midaemon
1 ? 2900 root 178 20 250M 10164K run 14:59 1.12 1.11 bptm
2 ? 2901 root 154 20 251M 10016K sleep 15:22 1.11 1.11 bptm
3 ? 2824 root 154 20 250M 8848K sleep 14:52 1.08 1.08 bptm
2 ? 2941 root 154 20 251M 9704K sleep 14:59 1.04 1.04 bptm
2 ? 2892 root 154 20 250M 9548K sleep 14:45 1.03 1.03 bptm
2 ? 7836 root 128 20 72K 64K zomb 0:17 1.01 1.01 bptm
2 ? 8111 root 154 20 250M 9600K sleep 14:49 1.00 1.00 bptm
0 ? 29822 root 154 20 251M 9688K sleep 15:50 1.00 1.00 bptm
3 ? 2937 root 154 20 251M 9640K sleep 15:07 0.97 0.97 bptm
2 ? 4386 root 168 20 15712K 888K sleep 1714:21 0.93 0.93 utild
1 ? 18876 root 154 20 319M 39084K sleep 15:10 0.71 0.71 bptm
2 ? 2986 root 152 20 44308K 832K run 3578:05 0.69 0.69 cimprovagt
2 ? 8393 root 168 20 251M 10548K sleep 22:13 0.57 0.57 bptm
2 ? 29818 root 154 20 251M 9700K sleep 12:13 0.55 0.55 bptm
3 ? 4757 root 154 20 251M 9724K sleep 10:15 0.53 0.53 bptm
3 ? 2133 root 154 20 251M 10412K sleep 19:58 0.50 0.50 bptm
2 ? 2140 root 154 20 251M 10488K sleep 19:31 0.50 0.50 bptm
1 ? 15739 root 154 20 251M 9720K sleep 6:53 0.48 0.48 bptm
3 ? 4842 root 154 20 251M 9648K sleep 7:31 0.48 0.48 bptm
3 ? 2102 root 154 20 251M 10424K sleep 20:32 0.46 0.46 bptm
3 ? 4940 root 154 20 250M 9680K sleep 10:23 0.46 0.46 bptm
1 ? 9105 root 168 20 251M 11148K sleep 24:59 0.45 0.45 bptm
2 ? 11952 root 154 20 251M 9932K sleep 7:06 0.45 0.45 bptm
04-14-2015 12:28 AM
can restarting netbackup & media manager service help ?
04-14-2015 12:30 AM
Shared memory needed on a media server is SIZE_DATA_BUFFERS * NUMBER_DATA_BUFFERS * number of tape drives * MPX
So, at some point, the total number of streams that shared memory on this media server can accommodate, has increased to more that it can handle.
TNs regarding shared memory:
Recommended NetBackup *NIX semaphore tuning values (Linux/Solaris/HP-UX/AIX)
http://www.symantec.com/docs/TECH203066
Message queue and shared memory parameters on HP-UX
http://www.symantec.com/docs/HOWTO99880
04-14-2015 12:40 AM
Shared memory usage can be checked with 'ipcs' command.
Something else on this server may me using shared memory - not only NBU media manager.
If this is master/media server, you will need even more than recommended values.
04-14-2015 12:41 AM
values of semaphores are as follows:
Tunable semmni
Description Maximum number of semaphore sets on the system
Module pm_usync
Current Value 2048 [Default]
Value at Next Boot 2048 [Default]
Value at Last Boot 2048
Default Value 2048
Constraints semmni >= 2
semmni <= 32767
semmni <= semmns
Can Change At Next Boot Only
Tunable semmns
Description Maximum number of semaphores
Module pm_usync
Current Value 307200
Value at Next Boot 307200
Value at Last Boot 307200
Default Value 4096
Constraints semmns >= 2
semmns <= 335534080
semmns >= semmni
semmns >= semume
Can Change At Next Boot Only
Tunable semmsl
Description Maximum number of semaphores in a set
Module pm_usync
Current Value 2048 [Default]
Value at Next Boot 2048 [Default]
Value at Last Boot 2048
Default Value 2048
Constraints semmsl >= 1
semmsl <= 10240
Can Change Immediately or at Next Boot
04-14-2015 12:52 AM
Look for shmmax.
See http://www.symantec.com/docs/HOWTO99880 and http://www.symantec.com/docs/HOWTO99781
To change values: http://www.symantec.com/docs/HOWTO99881
04-14-2015 04:46 AM
Thanks everyone, issue resolved with reboot of media server