Netbackup Issue status code - 36

[Sameer_Nirmal]

Hi Guys,

I have a Netbackup HP-UX 11.11 4.5FP_5 client being backed by Netbackup 5.1
Windows Server. The backup of one of the logical volumes is failing with status code 36 - falied trying to allocate memory.

The physical memory is 1GB free Swap is 3 GB free. I suspect the error is being caused on account of volume of data being backed up. The data is image file in the form of jpg in directories for each day. I checked the directory for a day which holds around 38000 *.jpg files. The file name length in total is 41 characters.
Is there any patch for this issue?

The "bpbkar" debug log shows
bpfsmap: ERR - alloc_bitmap: can't allocate bitmap
bpfsmap: FTL - bpfsmap: pass 1 failed

Thanks in Advance
Sam

[zippy]

Sam,

This error is a Kernel error. Tune your kernel, like you tuna fish!

First Check this link

http://seer.support.veritas.com/docs/243461.htm

if its still fails then.

swapinfo -tam

Then add more physical memory, two gigs physical, then 4 gigs primary swap.

swapinfo -tam

output should be

root@master_backup_server-/ # swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve -493 -493
memory 2983 2113 870 71%
total 7079 2606 4473 37% - 0 -


Status Code: 36 Top

Message: failed trying to allocate memory

Explanation: Allocation of system memory failed. This error occurs when there is insufficient system memory available. This could be caused by the system being overloaded with too many processes and there is not enough physical and virtual memory.

Recommended Action: Free up memory by terminating unneeded processes that consume a lot of memory. Add more swap space or physical memory.


Peace....

JD

[Sameer_Nirmal]

Hi James,

I appreciate your reply.

I do see differences with kernel parameters on my hp-ux client against the recommended in the document. I guess I can simply change them as mentioned.

Since at client end "bpbkar" process runs during backup, I am not sure which kernel parameters are more relavent to this process. We are using flash backup.

Here is the more messages from debug.

12:21:48.418 <4> bpfsmap: INF - vx_snap: snap mount cmd /sbin/mount -F vxfs -o snapof=/dev/vgnfs1/lvol1 /dev/vg00/lvo
l10 /usr/openv/netbackup/BPFSMAP_TMPDIR/vxsnap/bpfsmap.26808
12:23:20.507 <4> bpfsmap: INF - vx_get_fsdata: lastblock=-1325924352

12:23:20.507 <4> bpfsmap: INF - vx_get_fsdata: ninode = 7639808
12:23:20.507 <4> bpfsmap: INF - vx_get1inode: ino = 65 inodebuf = 0
12:23:20.509 <4> bpfsmap: INF - getmem: inadequate memory to allocate 371130368, process unlocked to allow swap usage
- expect slower execution
12:23:20.509 <16> bpfsmap: ERR - alloc_bitmap: can't allocate bitmap
12:23:20.509 <32> bpfsmap: FTL - bpfsmap: pass 1 failed
12:23:20.509 <4> bpfsmap: INF - close_snapdisk: /sbin/umount /usr/openv/netbackup/BPFSMAP_TMPDIR/vxsnap/bpfsmap.26808
; rmdir /usr/openv/netbackup/BPFSMAP_TMPDIR/vxsnap/bpfsmap.26808
12:23:21.394 <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 36: failed trying to allocate memory
12:23:21.395 <4> bpbkar Exit: INF - EXIT STATUS 36: failed trying to allocate memory


As indicated, it requires 371130368 bytes for allocation. I am not sure if it from shared memory requirement or total virtual memory required by the process.
As mentioned before I have 2GB physical ram 4GB swap. Out of it, arounf 1GB physical ram is free and 3.9 GB swap is free.

Any thoughts?

Thanks
Sam

[zippy · Accepted solution]

Samurai,

This is a doc sent to me some time ago by an HP dude. I apply this to all my servers then add what each application needs, in your case its Veritas Netbackup, Apply these changes to your server in addition to what Veritas says to do, you need to apply ALL the ones that Veritas says to do. These changes plus Veritas changes are Shared memory changes plus others, they will help whith your problem.

1 gig should be enough but I like to have at least 2 gigs. Buy it when you can.

I do "snaps" all the time and dont get this error, the last time I saw this type of error was on a D350 with 500 megs of ram running Verita 3.2 :)

* Configurable IPC Message Parameters:
mesg Enable/disable IPC messages (Series 700 only)
Minimum: 0 (Exclude System V IPC message parameters from
kernel)
Maximum: 1 (Include System V IPC message parameters in kernel)
Default: 1
msgmap message free-space map size
Minimum: 3
Maximum: Memory limited
Default: msgtql+2
msgmax maximum message size
Minimum: 0
Maximum: 65535 bytes
Default: 8192 bytes
msgmnb maximum bytes in message queue
Minimum: 0
Maximum: 65535 bytes
Default: 16384 bytes
msgmni maximum message queues on system
Minimum: 1
Maximum: Memory limited
Default: 50
msgseg number of segments in message queue
Minimum: 1
Maximum: 32767
Default: 2048
msgssz message segment size
Minimum: 1
Maximum: Memory limited
Default: 8 bytes
msgtql maximum total messages on system
Minimum: 1
Maximum: Memory limited
Default: 40

* Configurable IPC Semaphore Parameters:
sema Enable/disable semaphores (Series 700 only)
Minimum: 0 (exclude System V IPC semaphore code from kernel)
Maximum: 1 (include System V IPC semaphore code in kernel)
Default: 1
semaem Semaphore value-change limit
Minimum: 0
Maximum: semvmx or 32767, whichever is smaller
Default: 16384
semmap Size of free-semaphore resource map
Minimum: 4
Maximum: Memory limited
Default: semmni+2
semmni Maximum semaphores system-wide
Minimum: 2
Maximum: Memory limited
Default: 64
semmns Maximum user-accessible semaphores system-wide
Minimum: 2
Maximum: Memory limited
Default: 128
semmnu Maximum undos per semaphore
Minimum: 1
Maximum: nproc-4
Default: 30
semume Maximum semaphore undos per process
Minimum: 1
Maximum: semmns
Default: 10
semvmx Maximum allowed semaphore value
Minimum: 1
Maximum: 65535
Default: 32767

* Configurable IPC Shared Memory Parameters:
shmem Enable/disable shared memory (Series 700 only)
Minimum: 0 (exclude System V IPC shared memory code from
kernel)
Maximum: 1 (include System V IPC shared memory code in kernel)
Default: 1
shmmax Maximum shared memory segment size
Minimum: 2 Kbytes
Maximum: 1792 Mbytes
Default: 0x04000000 (64 Mbytes)
shmmni Maximum segments on system
Minimum: 3
Maximum: (memory limited)
Default: 200 identifiers
shmseg Maximum segments per process
Minimum: 1
Maximum: shmmni
Default: 120

[Sameer_Nirmal]

James,

Great response..

I would tune the kernel parameter and see if the error goes away.