Backup of Linux Server using Backup Exec Remote Agent RALUS Extremely Slow

[ntripp]

In the last 2 weeks backups have gone from running at 1,000MB/min to 30MB/min.  This is causing the jobs to take too long to complete.  I'm trying to find the cause of this. 

We're running:

Distributor ID: Ubuntu
Description:    Ubuntu 11.10
Release:        11.10
Codename:       oneiric
 

VERITAS_Backup_Exec_File_Version=13.0.5204.0
 

Backup Exec Server is 2010r3

sudo /opt/VRTSralus/bin/beremote --log-console

8e629720 Fri Jul  6 12:01:27 2012 : Starting BE Remote Agent
8e629720 Fri Jul  6 12:01:27 2012 : Requested no generation of log file
8e629720 Fri Jul  6 12:01:27 2012 : No configuration file specified.  Using default.
8e629720 Fri Jul  6 12:01:27 2012 : Log to console: enabled
8e629720 Fri Jul  6 12:01:27 2012 : Successfully set the supplementary groups of the process
8e629720 Fri Jul  6 12:01:27 2012 : Initialized locks for SSL callbacks
8e629720 Fri Jul  6 12:01:27 2012 : Starting NDMP processor
8e629720 Fri Jul  6 12:01:27 2012 : NDMPDMainThreadFunc spawned: grpid=1, tid=-2016676096
87cbf700 Fri Jul  6 12:01:27 2012 : FS_InitFileSys
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsnt5.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedssql2.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsxchg.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsxese.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsmbox.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedspush.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsnote.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsmdoc.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedssps2.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedssps3.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsupfs.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsshadow.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsoffhost.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   loaded libbedsvx.so
87cbf700 Fri Jul  6 12:01:27 2012 :   loaded libbedsrman.so
87cbf700 Fri Jul  6 12:01:27 2012 :   loaded libbedssms.so
87cbf700 Fri Jul  6 12:01:27 2012 :   loaded libbedssmsp.so
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsra.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsdb2.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 :   loaded libbedsedir.so
87cbf700 Fri Jul  6 12:01:27 2012 :   libbedsvmesx.so could not be loaded: 0x       2 (2)
87cbf700 Fri Jul  6 12:01:27 2012 : Initializing FSs
87cbf700 Fri Jul  6 12:01:27 2012 : FS 1 failed to initialize: 0xE000FE46
87cbf700 Fri Jul  6 12:01:27 2012 : Function called: RMAN_InitFileSys
87cbf700 Fri Jul  6 12:01:27 2012 : Using 'UTF-8' Encoding.
87cbf700 Fri Jul  6 12:01:27 2012 : Using vfm path /opt/VRTSralus/VRTSvxms from config.
87cbf700 Fri Jul  6 12:01:27 2012 : Sucessfully set VFM_PRIVATE_ROOT env to /opt/VRTSralus/VRTSvxms.
87cbf700 Fri Jul  6 12:01:27 2012 : VFM_PRIVATE_ROOT was set with value /opt/VRTSralus/VRTSvxms
87cbf700 Fri Jul  6 12:01:27 2012 :      VXMS Initialization OK.
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <xfs> mounted at </>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <proc> mounted at </proc>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <sysfs> mounted at </sys>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <fusectl> mounted at </sys/fs/fuse/connections>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <debugfs> mounted at </sys/kernel/debug>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <securityfs> mounted at </sys/kernel/security>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <devtmpfs> mounted at </dev>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <devpts> mounted at </dev/pts>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <tmpfs> mounted at </run>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <tmpfs> mounted at </run/lock>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <tmpfs> mounted at </run/shm>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <xfs> mounted at </pg_data>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <ext3> mounted at </boot>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <nfs> mounted at </var/lib/lps>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <nfs> mounted at </var/lib/backups>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <ext3> mounted at </pg_xlog>
87cbf700 Fri Jul  6 12:01:27 2012 : Detected Mounted Filesystem: type <nfsd> mounted at </proc/fs/nfsd>
87cbf700 Fri Jul  6 12:01:27 2012 : INFORMATIONAL: Zero value found for 'DisableRMAL' from ralus.cfg, allowing RMAL to initialize
87cbf700 Fri Jul  6 12:01:27 2012 : Successfully resolved the "ndmp" service to port: 12000 (host order)
87cbf700 Fri Jul  6 12:01:27 2012 : BETCPListener successfully installed a signal handler for SIGTERM
87cbf700 Fri Jul  6 12:01:27 2012 : BETCPListener::BETCPListener: This system appears to be a Dual IP system
87cbf700 Fri Jul  6 12:01:27 2012 : BETCPListener::BETCPListener: Successfully set the IPV6_V6ONLY option, this listener may behave as Dual Stack listener
87cbf700 Fri Jul  6 12:01:27 2012 : BETCPListener::Bind: Could not bind to port 12000 on IPv6, An error occurred during a socket bind operation: Error Code: 98, System Error Message: Address already in use
87cbf700 Fri Jul  6 12:01:27 2012 : BETCPListener::Bind: Could not bind to port 12000 on IPv4, An error occurred during a socket bind operation: Error Code: 98, System Error Message: Address already in use
87cbf700 Fri Jul  6 12:01:27 2012 : Could not start NDMP Listener on port 12000: An error occurred during a socket creation operation: Error Code: 98, System Error Message: Address already in use
87cbf700 Fri Jul  6 12:01:27 2012 : ndmpRun: exiting

I saw errors about the NDMP listening port and earlier in troubleshooting process and changed to 12000 and restarted the service.  iotop running during backup shows the beremote is the hihgest user of io and is only using at most 600kbps.

I don't see this being a network mismatch or other issue with the server since connection to nfs and other are fine.  All the troubleshooting documents I could find focused more on the install and getting it to run.  It's running, but too slow to effectively backup the server.

[ntripp]

Update:

I tried running the install patch to SP2 but that failed..

sudo sh ./installraluspatch.sh
************************************************************************
*        SYMANTEC BACKUP EXEC FOR WINDOWS RALUS4.4 PATCH PROGRAM       *
************************************************************************
* Copyright (c) 2010 Symantec Corporation. All rights reserved         *
*                                                                      *
* Symantec, the Symantec Logo and all other Symantec product names     *
* and slogans are trademarks or registered trademarks of Symantec      *
* Corporation.  Symantec and the Symantec Logo Reg. U.S. Pat. &        *
* Tm. Off. Other product names and/or slogans mentioned herein may be  *
* trademarks or registered trademarks of their respective companies.   *
*                                                                      *
************************************************************************
RALUS is not installed on this system...exiting
 

[pkh]

Ubuntu 11.10 is not supported by BE 2010.  See the SCL below

BE 2010|2010 R2|2010 R3 Software (SCL)

[ntripp]

Thanks, but I'm already aware of that.  I'm actually looking for a solution for finding out why the performance has dropped off so drastically.

I've gone through the Linux agent best practices.  Tried updating the client and scoured these forums and the Ubuntu forums for more documentation / diagnostics.  I've turned on ndmp logging level 30 and enabled debug logging on the media server.

Any ideas / similar experience about the performance issue / error installing the SP are welcome.

[jayaramk19831]

Hi ntripp

The "system address already in use error" says that port 12000 is already being used by another process running on the machine.

To find out which process can be using port 12000 , run "netstat -anp | grep 12000" and see if you get any output. If yes, then you can deduce the process which is currently using port 12000 and might want to turn it off because beremote will not start if the port specified for the ndmp connection is already in use. Alternately you could check if another beremote process is already running before starting beremote process under --log-console option.

Regarding slow backups. I think the reason could be congestion on the network. However, you might want to run the backups at different times to check if the speeds improve at some time or not.

Regards,

Jayaram

[ntripp]

Thanks for the reply.  I think the conflict on the port you are referring to is because of ipv6 being enabled.  Tha backups started completing normal speed again so the issue appears to be temporary and may have been related to an NTP bug that was detected as causing some CPU spikes.

I had started running the detailed debugging but issue has not returned.

[jayaramk19831]

Good to know that the speed is back to normal. One question I wanted to get clear is that did you have to disable IPv6 to get the speed back? Please report if your queries have been solved so that we are clear that you do not have any other issues other than the initial ones you specified and which have been solved for now.

Thanks,

Jayaram

[ntripp]

actually, ipv6 is still enabled.

I initially though that the error you pointed out when it was still running on the default port was the cause.  But after changing the port i got the same error.

On closer inspection log shows it is successful binding first time and then there is a second entry for the ipv6 address.  My next step was going to be to disable ipv6 but by then the backups had run and full speed so I followed the 'if it ain't broke don't fix it' rule.