RALUS Backup Exec 2010 R2 Linux agent, so slow, so leaky

[Alec_K]

Just before Christmas we upgraded to 2010 R2, this is working well for the Windows boxes we back up with Job Rates of up to 4245MB/min

The problem I have is the the Linux remote agent is an order of magnitude slower than this with the best I've seen being 285MB/min

In both Win and Linux cases we are using 1GB network cards through the same switch infrastructure to the same HP auto loader using Ultrium 4 tapes.

This occurs with 2 different Linuxes red hat 9 and Ubuntu 6.06

We are also seeing the beremote agent leak masses of memory on both Linuxes.

It took down a home directory server by consuming all the memory before it had backed up the 170 GB it was supposed to

By adding an extra 2GB swap file we got this backup to complete (beremote grew to 1.9GB at the end) and by setting up a cron job we get the memory back by regularly restarting beremote as it doesn't release the memory even when the backup completes.

On my Ubuntu box it grew to 4.9GB but I didn't get the chance to see how big it would finally get as it had been running for over 16 hours and had only backed up 168GB of my 300 GB build box so I cancelled the job.

This was after adding a 4GB swap file as before it always fell over at about 130GB after running out of memory.

There was a case raised against 12.5 (http://www.symantec.com/docs/TECH51566) that was supposedly fixed by a hotfix has anyone else seen this memory leak behaviour on the 2010 R2 version, My agent is I belive the latest version from ralus4164SP1.tar.gz dated 23/12/2010

[adk@darwin /opt/VRTSralus/bin]$ strings beremote | grep Version
_Z14ndmpSetVersionPvt
_Z14ndmpGetVersionPv
VERITAS_Backup_Exec_File_Version=13.0.4164.0

Is it just me or are there a whole bunch of frustrated Linux users out there ?

Alec

[pkh · Accepted solution]

Both your Linux flavours are not supported.  See the SCL below

ftp://exftpp.symantec.com/pub/support/products/Backup_Exec_for_WindowsNT/329254.pdf

[Alec_K]

From kern.log at the time my backup failed:

Mar  6 22:24:23 darwin kernel: [4699294.490940] Free swap  = 0kB
Mar  6 22:24:23 darwin kernel: [4699294.490944] Total swap = 6146184kB
Mar  6 22:24:23 darwin kernel: [4699294.490947] Free swap:            0kB
Mar  6 22:24:23 darwin kernel: [4699294.497880] 262080 pages of RAM
Mar  6 22:24:23 darwin kernel: [4699294.497887] 5001 reserved pages
Mar  6 22:24:23 darwin kernel: [4699294.497891] 71901 pages shared
Mar  6 22:24:23 darwin kernel: [4699294.497893] 7920 pages swap cached
Mar  6 22:24:23 darwin kernel: [4699294.498064] Out of Memory: Killed process 6029 (beremote).

Had tried giving the box 6GBytes of swap (it has 1 GB RAM) still not enough, this was after 21 hours and it had only backed up 240GB

As I said, so slow, so leaky !

[Ben_Collver]

It is less severe than your experience, but after upgrading from 2010 to 2010 R2, we began to see similar behavior on a RHEL5 mail server that uses Maildir.  The RALUS service continually grows until it consumes all memory and disrupts service.  Support advised me to periodically restart the RALUS service, and so far that has prevented further downtime.

[avalanche]

However my backups are not slow and speed seems to depend on file count in my case (~12 million small files takes some time to backup). Although my memory gets wasted and NEVER returns unless i restart the service, when i do it goes back to ~200M (was 5100M).

This happens on several Debian 5 machines (which IS supported by Symantec)

Here's an extract from  /var/VRTSralus/ralus.ver:
ralus=4164.111.146017
mdm=MDM_v0.0.5071
vxms=VxMS_v4.4-036

This issue started after I upgraded the agent to 4164.5 and 4164.111.146017. I've actually had a rather nasty crash due to this issue, after maybe a week or two beremote ate all available memory and the linux kernel started killing processes to free up memory (leaving beremote intact for some reason) resulting in a nasty a really kernel panic, luckily there where no fs corruptions or other errors.

I guess a possible workaround could be to use an earlier version of ralus, i can't really spot any differences between them, however it feels kind of strange running old agents on a rather new system.

The best part is that this issue does npt exist on Debian 6 which isn't supported.

In short, you're not alone. I'm really frustrated by this aswell. I've searched the forums and googled like a mad man to bring any light on this issue and i've just come up with one issue on solaris which vaguely resemble this one (http://www.symantec.com/business/support/index?page=content&id=TECH140784) it has a rather nasty workaround (restart the service every now and then) and this forum post: https://www-secure.symantec.com/connect/forums/linux-agent-crashes-memory-error which i haven't tried.

Best regards,
Andreas
 

[avalanche]

As i wrote before:

I guess a possible workaround could be to use an earlier version of ralus, i can't really spot any differences between them, however it feels kind of strange running old agents on a rather new system.

I removed the new "fancy" agent (4164.111.146017) and installed 2896.9 instead (Debian package name: VRTSralus-13.0.2896-0.x86_64.deb). Last night's incremental backups ran fine without eating all my memory, however, I'm not 100% sure this fixed the issue because it was just an incremental backup, the issue is easier to spot after a full backup.

/var/VRTSralus/ralus.ver:

ralus=2896.9
mdm=MDM_v0.0.4284.1
vxms=VxMS_v4.4-035

[avalanche]

 2896.9 did the trick, no more memory leakage. Not sure about the speeds though, I have to make a few more measures.

[Kerzi_Peterson]

I have the same exact problem on Novell OES2 SP3.  This same volume was on another server and backup speeds were fine.  I moved it to this new server and they start off fine, but the memory is quickly consumed, and backup speeds go under 100MB/MIN.    I am trying to search for 2896.9 right now for suse, I hope this fixes my problem.