RALUS hangs when browsing directory

[steve_williams1]

Howdy folks,

I've worked my way through numerous threads and tried several suggested solutions but still cannot make any headway on this issue. Hope someone can point me in a new direction.

I've got BUExec 2010 (ver. 13.0 rev. 2896) running on a Windows Server 2008 x32 machine and I'm trying to backup a RHEL 5 x64 machine. I've been running this particular setup for several years now without any trouble until about a month ago - it may have begun after I rebooted my RHEL machine (first reboot in 8 months).

When I try to browse the RHEL directories from BUExec, the system hangs. I can see the machine, I can connect, it asks me for credentials (root) which it accepts and then shows [ROOT] to the right of the RHEL computer icon, but when I try to expand the directory it hangs.

Here's what I've tried...

I have SE Linux enabled but currently suppressed.

I've disabled IPv6 on both the Windows and RHEL machines.

I've checked my credentials several times.

I've uninstalled RALUS, deleted the jobs and selection list, restarted BUExec services, reinstalled RALUS - still can't browse remote directories.

I've collected a BUExec log file on the RHEL machine when trying to access the directories. Evidently, it seems to connect ok. For some reason it seems to be trying to use the wrong set of credentials, BASIC/BUMonkey - the user on the Windows Server - it should be using ROOT. Then starts spewing about 2.5 millions lines of connection errors which is what is hanging it before it finally bails. I am at a loss at when to try next. Below is the beginng of the log file - I've clipped it when it begins the repetitive errors.

Any help would be appreciated.

----------------------------------------------------

: Starting BE Remote Agent
: Requested no generation of log file
: No configuration file specified.  Using default.
: Log to console: enabled
: Successfully set the supplementary groups of the process
: Initialized locks for SSL callbacks
: Starting NDMP processor
: NDMPDMainThreadFunc spawned: grpid=1, tid=-514971328
: FS_InitFileSys
:   libbedsnt5.so could not be loaded: 0x       2 (2)
:   libbedssql2.so could not be loaded: 0x       2 (2)
:   libbedsxchg.so could not be loaded: 0x       2 (2)
:   libbedsxese.so could not be loaded: 0x       2 (2)
:   libbedsmbox.so could not be loaded: 0x       2 (2)
:   libbedspush.so could not be loaded: 0x       2 (2)
:   libbedsnote.so could not be loaded: 0x       2 (2)
:   libbedsmdoc.so could not be loaded: 0x       2 (2)
:   libbedssps2.so could not be loaded: 0x       2 (2)
:   libbedssps3.so could not be loaded: 0x       2 (2)
:   libbedsupfs.so could not be loaded: 0x       2 (2)
:   libbedsshadow.so could not be loaded: 0x       2 (2)
:   libbedsoffhost.so could not be loaded: 0x       2 (2)
:   loaded libbedsvx.so
:   loaded libbedsrman.so
:   loaded libbedssms.so
:   loaded libbedssmsp.so
:   libbedsra.so could not be loaded: 0x       2 (2)
:   libbedsdb2.so could not be loaded: 0x       2 (2)
:   loaded libbedsedir.so
:   libbedsvmesx.so could not be loaded: 0x       2 (2)
: Initializing FSs
: FS 1 failed to initialize: 0xE000FE46
: Function called: RMAN_InitFileSys
: Using 'UTF-8' Encoding.
: Using vfm path /opt/VRTSralus/VRTSvxms from config.
: Sucessfully set VFM_PRIVATE_ROOT env to /opt/VRTSralus/VRTSvxms.
: VFM_PRIVATE_ROOT was set with value /opt/VRTSralus/VRTSvxms
:      VXMS Initialization OK.
: Detected Mounted Filesystem: type <ext3> mounted at </>
: Detected Mounted Filesystem: type <proc> mounted at </proc>
: Detected Mounted Filesystem: type <sysfs> mounted at </sys>
: Detected Mounted Filesystem: type <devpts> mounted at </dev/pts>
: Detected Mounted Filesystem: type <ext3> mounted at </pub>
: Detected Mounted Filesystem: type <ext3> mounted at </usr/local>
: Detected Mounted Filesystem: type <ext3> mounted at </home>
: Detected Mounted Filesystem: type <ext3> mounted at </var>
: Detected Mounted Filesystem: type <ext3> mounted at </data>
: Detected Mounted Filesystem: type <tmpfs> mounted at </dev/shm>
: Detected Mounted Filesystem: type <binfmt_misc> mounted at </proc/sys/fs/binfmt_misc>
: Detected Mounted Filesystem: type <rpc_pipefs> mounted at </var/lib/nfs/rpc_pipefs>
: Detected Mounted Filesystem: type <afs> mounted at </afs>
: INFORMATIONAL: Zero value found for 'DisableRMAL' from ralus.cfg, allowing RMAL to initialize
: ndmpRun
: Successfully resolved the "ndmp" service to port: 10000 (host order)
: NrdsWatchThread spawned: grpid=1, tid=-433452736
: NrdsAdvertiserThread spawned: grpid=1, tid=-422962880
: NrdsAdvertiserThread spawned: grpid=1, tid=-412473024
: NDMP Daemon Running..
: BETCPListener successfully installed a signal handler for SIGTERM
: Could not create an IPv6 socket at this time, reason:An error occurred during a socket creation operation: Error Code: 97, System Error Message: Address family not supported by protocol
: SocketService::IsSystemDualIP: No support found for IPv6 currently
: Started NDMP Listener on port 10000
: NrdsAdvertiserThread: advertisement cycle started.
: RMAN_EnumSelfDLE: AgentConfig GetOracleDBNames returned error. If Oracle Agent is installed, please run AgentConfig.
: NrdsAdvertiserThread: EnumSelfDLE for file system 14 returned 0(0x0) and 0 DLEs
: BENetConfigEx: Successfully refreshed adapter information.
: NrdsAdvertiserThread: EnumSelfDLE for file system 22 returned 0(0x0) and 1 DLEs
: NrdsAdvertiserThread: Nrds Message Len : 129.
: VX_RemoveDLE: DestroyDLE()
: This instance of BETCPListener was not requested to install a signal handler and hence will not install one!
: ConnectToServerEndPoint: dest=192.168.1.2, service=6101
: CreateConnection type=0 on socket 5 via BESocket OK
: NrdsAdvertiserThread: send of Tango.zo.ncsu.edu type 22 subtype 0 to target=192.168.1.2 port=6101 succeeded
: This instance of BETCPListener was not requested to install a signal handler and hence will not install one!
: ConnectToServerEndPoint: dest=glaurung.basic.local, service=6101
: CreateConnection type=0 on socket 5 via BESocket OK
: NrdsAdvertiserThread: send of Tango.zo.ncsu.edu type 22 subtype 0 to target=glaurung.basic.local port=6101 succeeded
: NrdsAdvertiserThread: advertisement cycle complete.  Waiting 240 minutes before advertising again.
: NrdsAdvertiserThread: negative (purge) advertisement cycle started.
: NrdsAdvertiserThread: no purge is pending.
: NrdsAdvertiserThread: negative (purge) advertisement cycle complete.  Waiting 240 minutes before advertising again.
: Control connection accepted : connection established between end-points 192.168.1.17:10000 and 192.168.1.2:55637
: ndmpCreateConnection
: conHandlerFunc for conn'n=483047792 spawned: tm=0x1cc770c0, grpid=2, tid=-401983168
: ndmpdConnectionHandler: received connection(5).
: ndmpSendRequest: message:0x502
: sent          seq#: 1 msg: 0x502
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetServerInfo:
: ndmpSendReply: message:0x108
: ndmpFreeMessage: message:0x108
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetServerInfo:
: ndmpSendReply: message:0x108
: ndmpFreeMessage: message:0x108
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConnectOpen: protocol_version:3.
: ndmpSendReply: message:0x900
: ndmpFreeMessage: message:0x900
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetAuthAttr:
: ndmpSendReply: message:0x103
: ndmpFreeMessage: message:0x103
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdPKCSSession: mode:start.
: ndmpSendReply: message:0xf33b
: ndmpFreeMessage: message:0xf33b
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConnectClientAuth: auth_type:BEWS2.
: LogonUser failed for user: BASIC\BUMonkey because LogonUser: Password is Null for user
: LogonUser failed for user: BASIC\BUMonkey
: ndmpSendReply: message:0x901
: ndmpFreeMessage: message:0x901
: ndmpProcessRequests
: ndmp_recv_msg
: ndmp_readit: Caught message on closed connection. Socket 0x5 len 0x0
: ndmp_readit: ErrorCode :: 2 : 0x       2 (2)
: ndmp_process_messages: detected eof
: ndmpFreeMessage: message:0x0
: FreeFormatEnv( cur_fmt=0 )
: FreeFormatEnv( cur_fmt=0 )
: ndmpDestroyConnection
: Control connection accepted : connection established between end-points 192.168.1.17:10000 and 192.168.1.2:55638
: ndmpCreateConnection
: conHandlerFunc for conn'n=483048096 spawned: tm=0x1cc770c0, grpid=2, tid=-401983168
: ndmpdConnectionHandler: received connection(5).
: ndmpSendRequest: message:0x502
: sent          seq#: 1 msg: 0x502
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetServerInfo:
: ndmpSendReply: message:0x108
: ndmpFreeMessage: message:0x108
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetServerInfo:
: ndmpSendReply: message:0x108
: ndmpFreeMessage: message:0x108
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConnectOpen: protocol_version:3.
: ndmpSendReply: message:0x900
: ndmpFreeMessage: message:0x900
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetAuthAttr:
: ndmpSendReply: message:0x103
: ndmpFreeMessage: message:0x103
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdPKCSSession: mode:start.
: ndmpSendReply: message:0xf33b
: ndmpFreeMessage: message:0xf33b
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConnectClientAuth: auth_type:BEWS2.
: ndmpSendReply: message:0x901
: ndmpFreeMessage: message:0x901
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdPKCSSessionData:
: ndmpSendReply: message:0xf33c
: ndmpFreeMessage: message:0xf33c
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdPKCSSessionData:
: ndmpSendReply: message:0xf33c
: ndmpFreeMessage: message:0xf33c
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetHostInfo:
: BENetConfigEx: Successfully refreshed adapter information.
: ndmpSendReply: message:0x100
: ndmpFreeMessage: message:0x100
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetServerInfo:
: ndmpSendReply: message:0x108
: ndmpFreeMessage: message:0x108
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdConfigGetHostInfo:
: BENetConfigEx: Successfully refreshed adapter information.
: ndmpSendReply: message:0x100
: ndmpFreeMessage: message:0x100
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdPropSet:
: ndmpdPropSet properties before:
: ndmpdPropSet properties after:
: NDMP_PROP_PEER_PRODUCT_TYPE=BEWS
: NDMP_PROP_PEER_OS_TYPE=NT
: NDMP_PROP_PEER_PROTOCOL_VERSION=6.4
: UTILITY_PARTITION_OEMIDS=
: SPS_SERVER_LIST=
: PDI_BKUP_TEMP_LOCATION=C:\TEMP
: ndmpSendReply: message:0xf270
: ndmpFreeMessage: message:0xf270
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdBEWSBrowseScanBegin:
: ndmpdBEWSBrowseScanBegin: Created dataSetHandle
: ndmpdBEWSBrowseScanBegin: Created dataSetHandle->browseObject
: ndmpdBEWSBrowseScanBegin: malloc'd uchar_ary
: ndmpdBEWSBrowseScanBegin: MapDsTypeToAuthBits complete
: ndmpdBEWSBrowseScanBegin: DeasciifyAndDecrypt complete
: ndmpdBEWSBrowseScanBegin: copied uname and pwd
: DS_OpenSelectionService
: RMAN_FindDrives - Oracle RMAN Agent not authorized.
: VX_FindDrives: Creating DLE: [ROOT]
: FS_ResolveDevName: [\]
: FS_BlowOutMachine: entered -->\\Tango.zo.ncsu.edu<--
: ******************************************************************************
: Initial DLE Tree
: -->[ROOT]<-- address 0x00001CD407A0 type 22 subtype 0 feature bits 0x03020601 0x00000000 dataId 49 bedsId 49 bedsVer 0
: End of List
: FS_ResolveDevName: [\]
: FS_BlowOutMachine: FS_ResolveDevName returns -->\\Tango.zo.ncsu.edu<-- as the machineName
: FS_BlowOutMachine: processing as local machine
: FS_BlowOutMachine: failed to find machine DLE in DLE tree, attempting DLE create
: FS_CreateTempDLE
: ====>VX_CreateTempDLE
: VX_CreateTempDLE: Creating DLE: \\Tango.zo.ncsu.edu
: FS_BlowOutMachine: trying to attach to newly created DLE
: ====>VX_AttachToDLE
: Resetting hard link info
: Hard link info has been reset
: VX_AttachToDLE: initializing mount list
: VX_AttachToDLE: create mount list returned 0
: Adding any excluded filesystems
: Excluding filesystem type devpts
:             AddToExcludedDirsList: Directory /dev/pts added to exclude list
: Excluding filesystem type proc
:             AddToExcludedDirsList: Directory /proc added to exclude list
: Excluding filesystem type sysfs
:             AddToExcludedDirsList: Directory /sys added to exclude list
: Excluding filesystem type rpc_pipefs
:             AddToExcludedDirsList: Directory /var/lib/nfs/rpc_pipefs added to exclude list
: Excluding filesystem type tmpfs
:             AddToExcludedDirsList: Directory /dev/shm added to exclude list
: Excluding filesystem type ftp
: Excluding filesystem type ntfs
:     BlowupChildren: Did not find DLE \\Tango.zo.ncsu.edu\[ROOT] in list. Creating new.
: RMAN_SurrogateCalling - Oracle RMAN Agent not authorized.
: VX_SurrogateCalling: Tango.zo.ncsu.edu
: VX_RemoveDLE: DestroyDLE()
: Attach to: \\Tango.zo.ncsu.edu
: FS_BlowOutMachine: attached successfully
: FS_BlowOutMachine: found -->\\Tango.zo.ncsu.edu<-- in DLE tree
: FS_BlowOutMachine: detaching from newly created DLE
: Detach from: \\Tango.zo.ncsu.edu
: Resetting hard link info
: Hard link info has been reset
: ******************************************************************************
: Modified DLE Tree
: -->[ROOT]<-- address 0x00001CD407A0 type 22 subtype 0 feature bits 0x03020601 0x00000000 dataId 49 bedsId 49 bedsVer 0
: -->\\Tango.zo.ncsu.edu<-- address 0x00001CD413D0 type 22 subtype 0 feature bits 0x03020603 0x00004002 dataId 49 bedsId 49 bedsVer 0
:     -->\\Tango.zo.ncsu.edu\[ROOT]<-- address 0x00001CD574F0 type 22 subtype 0 feature bits 0x03020601 0x00000000 dataId 49 bedsId 49 bedsVer 0
: End of List
: FS_BlowOutMachine: returning 0
: ====>VX_AttachToDLE
: Resetting hard link info
: Hard link info has been reset
: VX_AttachToDLE: initializing mount list
: VX_AttachToDLE: create mount list returned 0
: Adding any excluded filesystems
: Excluding filesystem type devpts
: Excluding filesystem type proc
: Excluding filesystem type sysfs
: Excluding filesystem type rpc_pipefs
: Excluding filesystem type tmpfs
: Excluding filesystem type ftp
: Excluding filesystem type ntfs
: Attach to: \\Tango.zo.ncsu.edu
: ndmpSendReply: message:0xf315
: ndmpFreeMessage: message:0xf315
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdBEWSBrowseScanEnd:
: ndmpdBEWSBrowseScanEnd: setting dataSetHandle
: ndmpdBEWSBrowseScanEnd: dataSetHandle complete
: ndmpdBEWSBrowseScanEnd: making the handle available
: ndmpdBEWSBrowseScanEnd: freeing the dataSetHandle
: ndmpdBEWSBrowseScanEnd: ERROR: NDMP_NO_ERR
: ndmpSendReply: message:0xf317
: ndmpFreeMessage: message:0xf317
: ndmpProcessRequests
: ndmp_recv_msg
: ndmpdBEWSBrowseScanBegin:
: ndmpdBEWSBrowseScanBegin: Created dataSetHandle
: ndmpdBEWSBrowseScanBegin: Created dataSetHandle->browseObject
: ndmpdBEWSBrowseScanBegin: malloc'd uchar_ary
: ndmpdBEWSBrowseScanBegin: MapDsTypeToAuthBits complete
: ndmpdBEWSBrowseScanBegin: DeasciifyAndDecrypt complete
: ndmpdBEWSBrowseScanBegin: copied uname and pwd
: FS_ResolveDevName: [\]
: FS_BlowOutMachine: entered -->\\Tango.zo.ncsu.edu\[ROOT]<--
: ******************************************************************************
: Initial DLE Tree
: -->[ROOT]<-- address 0x00001CD407A0 type 22 subtype 0 feature bits 0x03020601 0x00000000 dataId 49 bedsId 49 bedsVer 0
: -->\\Tango.zo.ncsu.edu<-- address 0x00001CD413D0 type 22 subtype 0 feature bits 0x03020603 0x00004002 dataId 49 bedsId 49 bedsVer 0
:     -->\\Tango.zo.ncsu.edu\[ROOT]<-- address 0x00001CD574F0 type 22 subtype 0 feature bits 0x03020601 0x00000000 dataId 49 bedsId 49 bedsVer 0
: End of List
: FS_ResolveDevName: [\]
: FS_BlowOutMachine: FS_ResolveDevName returns -->Tango.zo.ncsu.edu<-- as the machineName
: FS_BlowOutMachine: processing as local machine
: FS_BlowOutMachine: returning 0
: ====>VX_AttachToDLE
: Resetting hard link info
: Hard link info has been reset
: VX_AttachToDLE: initializing mount list
: VX_AttachToDLE: create mount list returned 0
: Adding any excluded filesystems
: Excluding filesystem type devpts
: Excluding filesystem type proc
: Excluding filesystem type sysfs
: Excluding filesystem type rpc_pipefs
: Excluding filesystem type tmpfs
: Excluding filesystem type ftp
: Excluding filesystem type ntfs
: Attach to: \\Tango.zo.ncsu.edu\[ROOT]
: ====> VX_ChangeDir
: ====> VXX_ChangeDir : apath : , asize : 1
: ====>ChangeDir : psize : 1
:         vx_stat_by_name: Getting stat of /
:         vx_open_by_name: Opening object name : /, openflags : B
:         vx_open_by_name: new handle =0x1cd91330
:         vx_stat_by_name: Getting stat of /var
: ====> VX_CloseDir
: VX_SetupWorkPath: dss_cur_dir is not valid
: VX_SetupWorkPath: req size is 119
: VX_SetupWorkPath: req size after strsize(sname) is 119
: VX_SetupWorkPath: path is /. copying cur dir and sname
: VX_SetupWorkPath: path with cur dir is /
:         vx_stat_by_name: Getting stat of /
: ====> VX_FindFirst
: last_slash called with non absolute path name '*.*'
:         vx_stat_by_name: Getting stat of /
:         vx_open_by_name: Opening object name : /, openflags : B
:         vx_open_by_name: new handle =0x1cd91330
:         vx_stat_by_name: Getting stat of /var
: VX_ModPathDDB (get_it): full_name_ptr : var, size : 4
: VX_ModPathDDB (get_it): wide name : var, size : 16
: ====> VX_FindNext
:         vx_stat_by_name: Getting stat of /etc
: VX_ModPathDDB (get_it): full_name_ptr : etc, size : 4
: VX_ModPathDDB (get_it): wide name : etc, size : 16

[VJware]

LogonUser failed for user: BASIC\BUMonkey because LogonUser: Password is Null for user
: LogonUser failed for user: BASIC\BUMonkey

Are you using the true "Root" user or a different account ? And is the root account added as a logon account in Backup Exec - Network - Logon accounts ?

Try to add the Linux server via user-defined selections in the backup job. And check if you are able to select & expand the machine without any error.

Lastly, when you try to select the linux machine, is the beremote process still running ?

[steve_williams1]

Yes, true "Root" on linux box.

Yes, "Root" is added on the BEX server as a logon account for that particular linux box.

Same result when linux box added as user-defined selection - still seems to take the Root user account and password, shows [ROOT], but hangs when trying to expand.

I'm not clear on what "beromote process" you're refering to in your last line. Yes, the RALUS agent is running on the linux box. Yes, all BEX services are running on the BEX server.

[VJware]

What is the exact version of Linux & what's the kernel version as well ?

[steve_williams1]

Red Hat Enterprise Linux Server release 5.9 (Tikanga)

Linux 2.6.18-348.6.1.el5 x86_64
 

[VJware · Accepted solution]

BE 2010 supports upto RHEL 5.7 & 5.9 isn't supported.

Did the backup failure coincide with the upgrade of linux ?

EDIT **

Try this KB ~ http://www.symantec.com/business/support/index?page=content&id=TECH205146

Consider updating the kernel.

[steve_williams1]

That KB is exactly what's going on. It did seem to stop working after a reboot - probably when that kernal got updated. I'll see about backing it down to the last working one. Thanks!

[steve_williams1]

Backed the kernel down and I'm able to browse remotely again. Thanks.

Is this something that will never get fixed in ver. 2010?

[VJware]

BE 2010 actually doesn't support RHEL 5.9 & I doubt this will get fixed then.

[cdims]

The same for the BE 2012. It doesn't support RHEL 5.9. Is it going to be fixed in the near future?