Troubles with NetBackup 7.5.0.7 on RHEL 6.4 and 6.5

[trv]

Is anybody running NBU 7.5.0.7 on RHEL 6.4 or newer? We did try to upgrade one of our master servers from 7.5.0.6 to 7.5.0.7 and we are now experiencing core dumps of almost all NetBackup binaries (bpps is working ...) as soon as we launch them. For example:

...
Dec 27 09:44:24 foobar kernel: bpclntcmd[3832] general protection ip:3132c7b72c sp:7ffffa9b87b8 error:0 in libc-2.12.so[3132c00000+18a000]
Dec 27 09:44:25 foobar kernel: bmrd[3841] general protection ip:3132c7b72c sp:7fff0b9f3128 error:0 in libc-2.12.so[3132c00000+18a000]
Dec 27 09:44:25 foobar kernel: bmrsetupmaster[3842] general protection ip:3132c7b72c sp:7fff450154e8 error:0 in libc-2.12.so[3132c00000+18a000]
Dec 27 09:47:14 foobar kernel: bpminlicense[5142] general protection ip:3132c7b72c sp:7fffcdc05e78 error:0 in libc-2.12.so[3132c00000+18a000]
...

This particular master is a preproduction instance and I have case opened, but I would like to know if there is / was somebody else affected.

[Will_Restore]

Looks like the real trouble is with libc-2.12.so

Can you install updated package?

[trv]

We are using latest rhel 6.4 glibc package available - glibc-2.12-1.107.el6_4.5.x86_64. Segfaults seems to be glibc related indeed, so I did try to yum update to the rhel 6 head (rhel 6.5 with glibc-2.12-1.132.el6.x86_64 at this moment), but it didn't stop the coredumps.

[Will_Restore]

I see that is latest according to RedHat.  Interesting since "The current stable version of glibc is 2.18" per http://www.gnu.org/software/libc/

Wonder what Symantec support has to say about this issue (core dumps of almost all NetBackup binaries)

[Marianne]

Best to get support calls logged with Symantec and Redhat. ASAP..

[trv]

We have calls logged, but no progress so far. NBU 7.5.0.6 binaries are working fine on the same system ... seems more like Symantecs problem than Redhat related to me. Anyway I am going to install fresh rhel 6.5 VM and see if I can reproduce crashes will all new 7.5 + 7.5.0.7 installation.

[CRZ]

I found one case with similar symptoms....but I think it may be yours.  :)  Could you please PM me with your case ID (or post it here)?

In the meantime, maybe we can help a little here as well. 

Can you please post the contents of /etc/redhat-release (and/or `uname -a`) as well as `ls -l` and (I think) `cksum` output against your 7.5.0.7 downloads?  Both the client pack and the server pack if possible.  We can see if anything looks weird.

Since I'm asking for stuff, let's look at one of your segfaulting binaries, too.  Could I trouble you for output of THESE commands?  (I picked bpclntcmd because you listed it first.)

# file /usr/openv/netbackup/bin/bpclntcmd
# strings /usr/openv/netbackup/bin/bpclntcmd | grep 7.5.0
# ldd /usr/openv/netbackup/bin/bpclntcmd

[trv]

OS release:

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

# uname -a
Linux foobar 2.6.32-431.1.2.el6.x86_64 #1 SMP Sun Nov 24 09:37:37 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

Checksums:

# md5sum NB_7.5.0.7.linuxR_x86.tar NB_CLT_7.5.0.7.tar
6eec1ca54077d0fb8305fb1bf966affd NB_7.5.0.7.linuxR_x86.tar
156b658aa98332995f7c3aecc1b6a7e0 NB_CLT_7.5.0.7.tar

# sha1sum NB_7.5.0.7.linuxR_x86.tar NB_CLT_7.5.0.7.tar
b214a785edb80fd79b4514cf26e3ad13bcfe99f5 NB_7.5.0.7.linuxR_x86.tar
765d9f19344fb43306a1cd8898b258771a5ab9d1 NB_CLT_7.5.0.7.tar

Cmd output:

# file /usr/openv/netbackup/bin/bpclntcmd
/usr/openv/netbackup/bin/bpclntcmd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

# strings /usr/openv/netbackup/bin/bpclntcmd | grep 7.5.0
@(#) NetBackup_7.5.0.7

# ldd /usr/openv/netbackup/bin/bpclntcmd
linux-vdso.so.1 => (0x00007fff805d4000)
libnbcclST.so => /usr/openv/lib/libnbcclST.so (0x00007ff32b72e000)
libnbclientST.so => /usr/openv/lib/libnbclientST.so (0x00007ff32b4d1000)
libnbbaseST.so => /usr/openv/lib/libnbbaseST.so (0x00007ff32b16f000)
libvrtsLogReadST.so.3 => /usr/openv/lib/libvrtsLogReadST.so.3 (0x00007ff32af5d000)
libvxexticuST.so.3 => /usr/openv/lib/libvxexticuST.so.3 (0x00007ff32ad54000)
libvrtsLogFormatMsgST.so.3 => /usr/openv/lib/libvrtsLogFormatMsgST.so.3 (0x00007ff32ab4b000)
libvrtsLogSLST.so.3 => /usr/openv/lib/libvrtsLogSLST.so.3 (0x00007ff32a943000)
libvxulST.so.3 => /usr/openv/lib/libvxulST.so.3 (0x00007ff32a71d000)
libvxlisST.so.3 => /usr/openv/lib/libvxlisST.so.3 (0x00007ff32a515000)
libvxustdioST.so => /usr/openv/lib/libvxustdioST.so (0x00007ff32a303000)
libvxicui18nST.so => /usr/openv/lib/libvxicui18nST.so (0x00007ff32a027000)
libvxicuucST.so => /usr/openv/lib/libvxicuucST.so (0x00007ff329d6f000)
libvxicudataST.so => /usr/openv/lib/libvxicudataST.so (0x00007ff32936a000)
libvxcPBXST.so => /usr/openv/lib/libvxcPBXST.so (0x00007ff329163000)
libvxACEST.so.6 => /usr/openv/lib/libvxACEST.so.6 (0x00007ff328e43000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003b79800000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003b79000000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003b77400000)
libacl.so.1 => /lib64/libacl.so.1 (0x0000003b7c800000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003b7a400000)
libm.so.6 => /lib64/libm.so.6 (0x0000003b77c00000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff328c20000)
libc.so.6 => /lib64/libc.so.6 (0x0000003b77000000)
/lib64/ld-linux-x86-64.so.2 (0x0000003b76c00000)
libattr.so.1 => /lib64/libattr.so.1 (0x0000003b7b800000)

Seems okish to me, but:

# /usr/openv/netbackup/bin/bpclntcmd -is_local_host foobar
Segmentation fault (core dumped)

# strace /usr/openv/netbackup/bin/bpclntcmd -is_local_host foobar 2>&1 | tail -10
munmap(0x7fc46bc22000, 4096)            = 0
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET6, sin6_port=htons(1556), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
getsockname(3, {sa_family=AF_INET6, sin6_port=htons(60965), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(1556), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
getsockname(3, {sa_family=AF_INET6, sin6_port=htons(40306), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
close(3)                                = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV (core dumped) +++
 

GDB backtrace:

(gdb) bt
#0  0x0000003b7707b81c in free () from /lib64/libc.so.6
#1  0x00007fc46dfe3dfe in nbconf_free_preferred_network_list () from /usr/openv/lib/libnbbaseST.so
#2  0x00007fc46dfe151f in copy_preferred_network_list () from /usr/openv/lib/libnbbaseST.so
#3  0x00007fc46dfe21c4 in nbconf_get_bpinfo () from /usr/openv/lib/libnbbaseST.so
#4  0x00007fc46dfe3afc in nbconf_get_info () from /usr/openv/lib/libnbbaseST.so
#5  0x00007fc46dfe3bf9 in nbconf_get_info_ref_count () from /usr/openv/lib/libnbbaseST.so
#6  0x00007fc46dfe9431 in get_config_values () from /usr/openv/lib/libnbbaseST.so
#7  0x00007fc46dfeed15 in vnet_cached_getaddrinfo_and_update () from /usr/openv/lib/libnbbaseST.so
#8  0x00007fc46dff103f in vnet_cached_getaddrinfo () from /usr/openv/lib/libnbbaseST.so
#9  0x00007fc46dff54b0 in vnet_is_local_host () from /usr/openv/lib/libnbbaseST.so
#10 0x0000000000407301 in main ()
 

[CRZ]

Last stupid question from me (probably): Do you have a proper entry for localhost in your hosts file?  IPv4 or IPv6?  (I would think you would have to, or 7.5.0.6 also wouldn't work... but doesn't hurt to ask - MAYBE something is different with IPv6 in 7.5.0.7?)

I would have recommended escalating your case to backline, but I think that's already happened and Engineering is looking at it.  We may have to wait and see what they say as we've rapidly risen way above my head.  ;)

[trv · Accepted solution]

This issue was solved by support earlier for us, but the technote was finally published:

(NEW) During an installation of the NetBackup 7.5.0.7 maintenance release, clustered master server NetBackup processes core dump or application fault.
 http://www.symantec.com/docs/TECH213788

[Marianne]

You never said anything about having a clustered master server??

[trv]

Well I didn't think it was relevant ... but it was.

[Marianne]

You have for some reason selected "I do not need a solution (just sharing information)" when you posted...

I have changed this and marked your post as solution as this will help other users with similar issues.