bpdbm double free or corruption

[Adalberto_Blanc]

Hello everybody, 

I am getting this on my laboratory.  Master server installed on redhat 6.5 and Veritas Netbackup 7.7.2.

Does anybody faced this?

*** glibc detected *** /usr/openv/netbackup/bin/bpdbm: double free or corruption (out): 0x0000000000df5ba0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3ee6276166]
/lib64/libc.so.6[0x3ee6278c93]
/usr/openv/lib/libvxcPBXST.so(pbxFree+0x2f)[0x7f7e01de192f]
/usr/openv/lib/libnbbaseST.so(vnet_pbxFree+0xf)[0x7f7e06585f2d]
/usr/openv/lib/libnbbaseST.so(daemon_shutdown_listeners+0x1cf)[0x7f7e06528cd2]
/usr/openv/netbackup/bin/bpdbm[0x47dce1]
/usr/openv/netbackup/bin/bpdbm[0x481030]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3ee621ed1d]
/usr/openv/netbackup/bin/bpdbm(_ZN2v629TAO_ThruPOA_Object_Proxy_Impl5_is_aEPNS_12CORBA_ObjectEPKc+0xd1)[0x47a109]
======= Memory map: ========
00400000-00970000 r-xp 00000000 fd:00 1985073                            /usr/openv/netbackup/bin/bpdbm
00b6f000-00bb0000 rw-p 0056f000 fd:00 1985073                            /usr/openv/netbackup/bin/bpdbm
00bb0000-00bb9000 rw-p 00000000 00:00 0
00d0f000-00e08000 rw-p 00000000 00:00 0
00e08000-00e21000 rw-p 00000000 00:00 0
3ee5a00000-3ee5a20000 r-xp 00000000 fd:00 917917                         /lib64/ld-2.12.so (deleted)
3ee5c1f000-3ee5c20000 r--p 0001f000 fd:00 917917                         /lib64/ld-2.12.so (deleted)
3ee5c20000-3ee5c21000 rw-p 00020000 fd:00 917917                         /lib64/ld-2.12.so (deleted)
3ee5c21000-3ee5c22000 rw-p 00000000 00:00 0
3ee5e00000-3ee5e02000 r-xp 00000000 fd:00 917925                         /lib64/libdl-2.12.so (deleted)
3ee5e02000-3ee6002000 ---p 00002000 fd:00 917925                         /lib64/libdl-2.12.so (deleted)
3ee6002000-3ee6003000 r--p 00002000 fd:00 917925                         /lib64/libdl-2.12.so (deleted)
3ee6003000-3ee6004000 rw-p 00003000 fd:00 917925                         /lib64/libdl-2.12.so (deleted)
3ee6200000-3ee638b000 r-xp 00000000 fd:00 917918                         /lib64/libc-2.12.so (deleted)
3ee638b000-3ee658a000 ---p 0018b000 fd:00 917918                         /lib64/libc-2.12.so (deleted)
3ee658a000-3ee658e000 r--p 0018a000 fd:00 917918                         /lib64/libc-2.12.so (deleted)
3ee658e000-3ee658f000 rw-p 0018e000 fd:00 917918                         /lib64/libc-2.12.so (deleted)
3ee658f000-3ee6594000 rw-p 00000000 00:00 0
3ee6600000-3ee6617000 r-xp 00000000 fd:00 917923                         /lib64/libpthread-2.12.so (deleted)
3ee6617000-3ee6817000 ---p 00017000 fd:00 917923                         /lib64/libpthread-2.12.so (deleted)
3ee6817000-3ee6818000 r--p 00017000 fd:00 917923                         /lib64/libpthread-2.12.so (deleted)
3ee6818000-3ee6819000 rw-p 00018000 fd:00 917923                         /lib64/libpthread-2.12.so (deleted)
3ee6819000-3ee681d000 rw-p 00000000 00:00 0
3ee6a00000-3ee6a07000 r-xp 00000000 fd:00 917930                         /lib64/librt-2.12.so (deleted)
3ee6a07000-3ee6c06000 ---p 00007000 fd:00 917930                         /lib64/librt-2.12.so (deleted)
3ee6c06000-3ee6c07000 r--p 00006000 fd:00 917930                         /lib64/librt-2.12.so (deleted)
3ee6c07000-3ee6c08000 rw-p 00007000 fd:00 917930                         /lib64/librt-2.12.so (deleted)
3ee7200000-3ee7283000 r-xp 00000000 fd:00 917919                         /lib64/libm-2.12.so (deleted)
3ee7283000-3ee7482000 ---p 00083000 fd:00 917919                         /lib64/libm-2.12.so (deleted)
3ee7482000-3ee7483000 r--p 00082000 fd:00 917919                         /lib64/libm-2.12.so (deleted)
3ee7483000-3ee7484000 rw-p 00083000 fd:00 917919                         /lib64/libm-2.12.so (deleted)
3ee8600000-3ee8671000 r-xp 00000000 fd:00 917926                         /lib64/libfreebl3.so (deleted)
3ee8671000-3ee8870000 ---p 00071000 fd:00 917926                         /lib64/libfreebl3.so (deleted)
3ee8870000-3ee8872000 r--p 00070000 fd:00 917926                         /lib64/libfreebl3.so (deleted)
3ee8872000-3ee8873000 rw-p 00072000 fd:00 917926                         /lib64/libfreebl3.so (deleted)
3ee8873000-3ee8877000 rw-p 00000000 00:00 0
3ee8a00000-3ee8a07000 r-xp 00000000 fd:00 917927                         /lib64/libcrypt-2.12.so (deleted)
3ee8a07000-3ee8c07000 ---p 00007000 fd:00 917927                         /lib64/libcrypt-2.12.so (deleted)
3ee8c07000-3ee8c08000 r--p 00007000 fd:00 917927                         /lib64/libcrypt-2.12.so (deleted)
3ee8c08000-3ee8c09000 rw-p 00008000 fd:00 917927                         /lib64/libcrypt-2.12.so (deleted)
3ee8c09000-3ee8c37000 rw-p 00000000 00:00 0
3ee9600000-3ee9616000 r-xp 00000000 fd:00 917922                         /lib64/libgcc_s-4.4.7-20120601.so.1;5728a50b (deleted)
3ee9616000-3ee9815000 ---p 00016000 fd:00 917922                         /lib64/libgcc_s-4.4.7-20120601.so.1;5728a50b (deleted)
3ee9815000-3ee9816000 rw-p 00015000 fd:00 917922                         /lib64/libgcc_s-4.4.7-20120601.so.1;5728a50b (deleted)
3ee9a00000-3ee9ae8000 r-xp 00000000 fd:00 1970268                        /usr/lib64/libstdc++.so.6.0.13 (deleted)
3ee9ae8000-3ee9ce8000 ---p 000e8000 fd:00 1970268                        /usr/lib64/libstdc++.so.6.0.13 (deleted)
3ee9ce8000-3ee9cef000 r--p 000e8000 fd:00 1970268                        /usr/lib64/libstdc++.so.6.0.13 (deleted)
3ee9cef000-3ee9cf1000 rw-p 000ef000 fd:00 1970268                        /usr/lib64/libstdc++.so.6.0.13 (deleted)
3ee9cf1000-3ee9d06000 rw-p 00000000 00:00 0
3eece00000-3eece17000 r-xp 00000000 fd:00 917924                         /lib64/libaudit.so.1.0.0 (deleted)
3eece17000-3eed016000 ---p 00017000 fd:00 917924                         /lib64/libaudit.so.1.0.0 (deleted)
3eed016000-3eed017000 r--p 00016000 fd:00 917924                         /lib64/libaudit.so.1.0.0 (deleted)
3eed017000-3eed01c000 rw-p 00017000 fd:00 917924                         /lib64/libaudit.so.1.0.0 (deleted)
7f7df9001000-7f7df900d000 r-xp 00000000 fd:00 917534                     /lib64/libnss_files-2.12.so (deleted)
7f7df900d000-7f7df920d000 ---p 0000c000 fd:00 917534                     /lib64/libnss_files-2.12.so (deleted)
7f7df920d000-7f7df920e000 r--p 0000c000 fd:00 917534                     /lib64/libnss_files-2.12.so (deleted)
7f7df920e000-7f7df920f000 rw-p 0000d000 fd:00 917534                     /lib64/libnss_files-2.12.so (deleted)
7f7df920f000-7f7dff0a0000 r--p 00000000 fd:00 1968408                    /usr/lib/locale/locale-archive (deleted)
7f7dff0a0000-7f7dff100000 rw-p 00000000 00:00 0
7f7dff100000-7f7dff116000 r-xp 00000000 fd:00 917754                     /lib64/libnsl-2.12.so (deleted)
7f7dff116000-7f7dff315000 ---p 00016000 fd:00 917754                     /lib64/libnsl-2.12.so (deleted)
7f7dff315000-7f7dff316000 r--p 00015000 fd:00 917754                     /lib64/libnsl-2.12.so (deleted)
7f7dff316000-7f7dff317000 rw-p 00016000 fd:00 917754                     /lib64/libnsl-2.12.so (deleted)
7f7dff317000-7f7dff31b000 rw-p 00000000 00:00 0
7f7dff31b000-7f7dff327000 r-xp 00000000 fd:00 917928                     /lib64/libpam.so.0.82.2 (deleted)
7f7dff327000-7f7dff527000 ---p 0000c000 fd:00 917928                     /lib64/libpam.so.0.82.2 (deleted)
7f7dff527000-7f7dff528000 r--p 0000c000 fd:00 917928                     /lib64/libpam.so.0.82.2 (deleted)
7f7dff528000-7f7dff529000 rw-p 0000d000 fd:00 917928                     /lib64/libpam.so.0.82.2 (deleted)
7f7dff529000-7f7dff52d000 rw-p 00000000 00:00 0
7f7dff52d000-7f7dff538000 r-xp 00000000 fd:00 1985285                    /usr/openv/lib/libjanssonST.so
7f7dff538000-7f7dff737000 ---p 0000b000 fd:00 1985285                    /usr/openv/lib/libjanssonST.so
7f7dff737000-7f7dff738000 rw-p 0000a000 fd:00 1985285                    /usr/openv/lib/libjanssonST.so
7f7dff738000-7f7dff91a000 r-xp 00000000 fd:00 1985597                    /usr/openv/lib/libnbsslST.so
7f7dff91a000-7f7dffb19000 ---p 001e2000 fd:00 1985597                    /usr/openv/lib/libnbsslST.so
7f7dffb19000-7f7dffb49000 rw-p 001e1000 fd:00 1985597                    /usr/openv/lib/libnbsslST.so
7f7dffb49000-7f7dffb4e000 rw-p 00000000 00:00 0
7f7dffb4e000-7f7e00cbe000 r--p 00000000 fd:00 1985601                    /usr/openv/lib/libvxicudataST.so.6
7f7e00cbe000-7f7e00ebd000 ---p 01170000 fd:00 1985601                    /usr/openv/lib/libvxicudataST.so.6
7f7e00ebd000-7f7e00ebe000 rw-p 0116f000 fd:00 1985601                    /usr/openv/lib/libvxicudataST.so.6
7f7e00ebe000-7f7e0100f000 r-xp 00000000 fd:00 1985604                    /usr/openv/lib/libvxicuucST.so.6
7f7e0100f000-7f7e0120f000 ---p 00151000 fd:00 1985604                    /usr/openv/lib/libvxicuucST.so.6
7f7e0120f000-7f7e01221000 rw-p 00151000 fd:00 1985604                    /usr/openv/lib/libvxicuucST.so.6
7f7e01221000-7f7e01224000 rw-p 00000000 00:00 0
7f7e01224000-7f7e013f6000 r-xp 00000000 fd:00 1985602                    /usr/openv/lib/libvxicui18nST.so.6
7f7e013f6000-7f7e015f5000 ---p 001d2000 fd:00 1985602                    /usr/openv/lib/libvxicui18nST.so.6
7f7e015f5000-7f7e01602000 rw-p 001d1000 fd:00 1985602                    /usr/openv/lib/libvxicui18nST.so.6
7f7e01602000-7f7e01604000 rw-p 00000000 00:00 0
 

[mph999]

Can't find any matchs for this error agaisnt bpdbm.

Searching for 'double free or corruption (out)' gives a few results with multiple causes :

OS bugs (Found one on RedHat)

Version mis-match of NBU binaries (eg system has binaries from multiple different NBU versions)

System tuning

Given that no one else seems to have reported this against bpdbm (and eberyone who uses NBU uses bpdbm ) I suspect that this is an environmental issue.  Lab machines have a hard life, so I'd suggest a clean install of the OS if possible, followed by a clean install of NBU.

[Nicolai]

Run the Veritas SORT collector to check if OS is Netbackup ready

https://sort.veritas.com/data_collectors

The SORT data collector provide a text file with findings, like this one :

https://sort.veritas.com/sample/install_and_upgrade_nbu

[nbutech]

This looks like to be from OS side...

One quick test what you can do is copy/replace the bpdbm from your test/lab environment for the same NBU version to the one where you can facing issue..then recycle nbu services and see if you still see the problem...