Forum Discussion

Chris_W's avatar
Chris_W
Level 4
7 years ago

Netbackup Appliance 3.1.1 (8.1.1.) crashing

After upgrading Netbackup Appliance 5230 from 3.1 to 3.1.1 (Netbackup 8.1.1) we have had frequent unexpected reboots.

The OS reboots and there is nothing in the Netbackup logs explaining what went wrong. Also there are no core dumps created. The only bit of detail is this strange behaviour in the /var/log/messages, below, note the break sequence ^@^@ - looks like the OS runs out of kernel(?) memory even though the available memory is almost 50% - by checking sar reports eg.

 

sar -r -f /var/log/sa/sa{dayofthemonth}

 

After weeks of struggling with support we eventually found an engineer that dug up a known bug! for instability in Netbackup 8.1.1. The EEB is ET3942191 but please note that 8.1.1 already comes with this EEB installed but Veritas have silently being updating it and not telling anyone - nowhere to be found in late breaking news - shocker I know... :smileyindifferent:

If I remember correctly 8.1.1 comes with version 1 or 2 of this EEB. My engineer gave me version 9 of it. Once it was installed the rebooting went away for a month.

A new case was logged when the issue re-occured, I then after being told by support there is nothing they can do for me, I asked them about ET3942191 because maybe it was updated again, and low and behold it was! I was given version 14 to install and its been stable since - holding thumbs. Patch installed was NBAPP_EEB_ET3942191-3.1.1.0-14.x86_64-180814140122.rpm.

So if you have weird crashing on your Netbackup Appliance running version 3.1.1 ask for ET3942191 it may change your life and award you extra sleep at night - might also cure baldness.

I am hoping this tyrade of mine will help someone that is experiencing the same thing.

May 28 16:50:09 netbackup-appl-01 systemd: Started Serial Getty on ttyS0.
May 28 16:50:09 netbackup-appl-01 systemd: Starting Serial Getty on ttyS0...
May 28 16:51:09 netbackup-appl-01 systemd: serial-getty@ttyS0.service holdoff time over, scheduling restart.
May 28 16:51:09 netbackup-appl-01 systemd: Started Serial Getty on ttyS0.
May 28 16:51:09 netbackup-appl-01 systemd: Starting Serial Getty on ttyS0...
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May 28 16:54:10 netbackup-appl-01 kernel: Initializing cgroup subsys cpuset
May 28 16:54:10 netbackup-appl-01 kernel: Initializing cgroup subsys cpu
May 28 16:54:10 netbackup-appl-01 kernel: Initializing cgroup subsys cpuacct
May 28 16:54:10 netbackup-appl-01 kernel: Linux version 3.10.0-693.17.1.el7.x86_64 (mockbuild@x86-041.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Sun Jan 14 10:36:03 EST 2018
May 28 16:54:10 netbackup-appl-01 kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=/dev/mapper/system-root ro nomodeset rdblacklist=qla2xxx rd.lvm.lv=system/root rd.lvm.lv=system/swap nodmraid LANG=en_US.UTF-8 crashkernel=256M loglevel=1 pcie_aspm=off modprobe.blacklist=qla2xxx,ahci,isci audit=1
May 28 16:54:10 netbackup-appl-01 kernel: e820: BIOS-provided physical RAM map:
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000008bbff] usable
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x000000000008bc00-0x000000000009ffff] reserved
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bad06fff] usable
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bad07000-0x00000000baf83fff] reserved
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000baf84000-0x00000000bafb5fff] usable
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bafb6000-0x00000000bafcafff] reserved
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bafcb000-0x00000000bb3d3fff] usable
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bb3d4000-0x00000000bdd2efff] reserved
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bdd2f000-0x00000000bddccfff] ACPI NVS
May 28 16:54:10 netbackup-appl-01 kernel: BIOS-e820: [mem 0x00000000bddcd000-0x00000000bdea0fff] ACPI data

 

  •  LOL - might also cure baldness.

    I need that patch even thou I don't have a appliance :-)