linux agent crashes right after starting

[CrackedJack]

The linux admin at work updated a server that has been running the linux agent (ralus) without any problems but since the upgrade I cannot get the agent to start. It crashes seconds after starting.

I'm running BE 2010R3.

The linux server is running Red Hat Enterprise Linux 5.8, kernel 2.6.39 (previously is was 2.6.32 but I don't know what version of linux... maybe 5.6 or 5.7?). It's 64bit .

I extracted the SP2 update from the BE installer and tried that on the Linux server but it made no difference. As a workaround I'm tring to use a remote mount point on a Linux server where the agent is still working but that's proving to be problematic (when it does work it reports a failure for the job and when restoring data BE claims it's corrupt. The data I can get off the tape may not be useable for what I need)

Below is the log when I start be on the server with the --log-console option. If anyone can help me get the agent started I would appreciate it.

18e396e0 Mon Mar 12 09:47:37 2012 : Starting BE Remote Agent
18e396e0 Mon Mar 12 09:47:37 2012 : Requested no generation of log file
18e396e0 Mon Mar 12 09:47:37 2012 : No configuration file specified.  Using default.
18e396e0 Mon Mar 12 09:47:37 2012 : Log to console: enabled
18e396e0 Mon Mar 12 09:47:37 2012 : Successfully set the supplementary groups of the process
18e396e0 Mon Mar 12 09:47:37 2012 : Initialized locks for SSL callbacks
18e396e0 Mon Mar 12 09:47:37 2012 : Starting NDMP processor
18e396e0 Mon Mar 12 09:47:37 2012 : NDMPDMainThreadFunc spawned: grpid=1, tid=1099725120
418c7940 Mon Mar 12 09:47:37 2012 : FS_InitFileSys
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsnt5.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedssql2.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsxchg.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsxese.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsmbox.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedspush.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsnote.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsmdoc.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedssps2.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedssps3.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsupfs.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsshadow.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsoffhost.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   loaded libbedsvx.so
418c7940 Mon Mar 12 09:47:37 2012 :   loaded libbedsrman.so
418c7940 Mon Mar 12 09:47:37 2012 :   loaded libbedssms.so
418c7940 Mon Mar 12 09:47:37 2012 :   loaded libbedssmsp.so
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsra.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsdb2.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 :   loaded libbedsedir.so
418c7940 Mon Mar 12 09:47:37 2012 :   libbedsvmesx.so could not be loaded: 0x       2 (2)
418c7940 Mon Mar 12 09:47:37 2012 : Initializing FSs
418c7940 Mon Mar 12 09:47:37 2012 : FS 1 failed to initialize: 0xE000FE46
418c7940 Mon Mar 12 09:47:37 2012 : Function called: RMAN_InitFileSys
418c7940 Mon Mar 12 09:47:37 2012 : Using 'UTF-8' Encoding.
418c7940 Mon Mar 12 09:47:37 2012 : Using vfm path /opt/VRTSralus/VRTSvxms from config.
418c7940 Mon Mar 12 09:47:37 2012 : Sucessfully set VFM_PRIVATE_ROOT env to /opt/VRTSralus/VRTSvxms.
418c7940 Mon Mar 12 09:47:37 2012 : VFM_PRIVATE_ROOT was set with value /opt/VRTSralus/VRTSvxms
418c7940 Mon Mar 12 09:47:37 2012 :      VXMS Initialization OK.
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <ext3> mounted at </>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <proc> mounted at </proc>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <sysfs> mounted at </sys>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <devpts> mounted at </dev/pts>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <ext3> mounted at </var>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <ext3> mounted at </u01>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <ext3> mounted at </boot>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <tmpfs> mounted at </dev/shm>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <binfmt_misc> mounted at </proc/sys/fs/binfmt_mis       c>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <rpc_pipefs> mounted at </var/lib/nfs/rpc_pipefs>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <oracleasmfs> mounted at </dev/oracleasm>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <ext3> mounted at </revlx_ext>
418c7940 Mon Mar 12 09:47:37 2012 : Detected Mounted Filesystem: type <ext3> mounted at </revlx>
418c7940 Mon Mar 12 09:47:37 2012 : INFORMATIONAL: Zero value found for 'DisableRMAL' from ralus.cfg, allowing RMAL to        initialize
418c7940 Mon Mar 12 09:47:37 2012 : Successfully resolved the "ndmp" service to port: 10000 (host order)
418c7940 Mon Mar 12 09:47:37 2012 : BETCPListener successfully installed a signal handler for SIGTERM
418c7940 Mon Mar 12 09:47:37 2012 : BETCPListener::BETCPListener: This system appears to be a Dual IP system
418c7940 Mon Mar 12 09:47:37 2012 : BETCPListener::BETCPListener: Successfully set the IPV6_V6ONLY option, this listener may behave as Dual Stack listener
418c7940 Mon Mar 12 09:47:37 2012 : Started NDMP Listener on port 10000
428e8940 Mon Mar 12 09:47:47 2012 : NrdsAdvertiserThread: advertisement cycle started.
428e8940 Mon Mar 12 09:47:47 2012 : RMAN_EnumSelfDLE: AgentConfig GetOracleDBNames returned error. If Oracle Agent is installed, please run AgentConfig.
428e8940 Mon Mar 12 09:47:47 2012 : NrdsAdvertiserThread: EnumSelfDLE for file system 14 returned 0(0x0) and 0 DLEs
GetIfAddrs(LINUX): failed err = 11
GetAdaptersAddresses: error = 1, ret=-1
428e8940 Mon Mar 12 09:47:47 2012 : VX_RemoveDLE: DestroyDLE()
428e8940 Mon Mar 12 09:47:47 2012 : NrdsAdvertiserThread: EnumSelfDLE for file system 22 returned -1(0xFFFFFFFF) and 0 DLEs
428e8940 Mon Mar 12 09:47:47 2012 : NrdsAdvertiserThread: Security is enabled!!!
428e8940 Mon Mar 12 09:47:47 2012 : This instance of BETCPListener was not requested to install a signal handler and hence will not install one!
GetIfAddrs(LINUX): failed err = 11
Segmentation fault (core dumped)

[CrackedJack]

I forgot to mention that I have uninstalled/reinstalled the agent but it doesn't help. The uninstall/install goes fine, no errors. Just getting the agent to actually start is the problem.

Thanks

[pkh · Accepted solution]

RHEL 5.8 is not supported by either BE 2010 or BE 2012.  They only support up till RHEL 5.7.  See the SCL's below

BE 2010|2010 R2|2010 R3 Software (SCL)

BE 2012 SCL

[tjmcgrew]

I've managed to track down the change in the kernel which broke the agent. It's fairly trivial to patch, and I have the agent working with kernel 3.3.2 on Arch by patching their default kernel.

Here is the commit which broke things: http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commitdiff;h=41c31f318a5209922d0...

I'm going to do some further testing to see which of those changes are necessary to revert, but I know reverting them all does fix the problem. There may be some minor side effects of making this change however, detailed here: https://bugzilla.kernel.org/show_bug.cgi?id=33992

[philweb]

Hi there,

Had the same problem. Instead of patching the kernel, i decided to patch BackupExec. Since kernel patching is a bit drastic.

Basically you just have to flip 2 bits. I consider this modification as safe, as the failing function call had no effect on older linux kernel versions either.

I documented my solution here:

http://blog.redweb.at/2012/08/howto-backupexec-2012-linux-agent-and-kernel-3-0-debian/

Maybe someone at symantec reads this and considers removal or conditional call of the problematic call in the next agent release.

cheers,
Phil