Out of Memory

[Rich_Lichvar]

We keep getting a job that "fails" due to an "out of memory" error. The job backs up all but one of our remote machines, the media server, Exchange 2003 (running on the media server), and a couple of SQL Server 2000 databases on remote machines. The backup job actually does run to full completion. A second job that runs after the first (and backs up the other remote machine) finishes normally every single time.The interesting thing is, if we re-start all the BENT services, we don't get the error! Any thoughts?

[Christiane_Pete]

Hi Rich,

It all depend on the exact "out of memory" error and what you are backing up when getting it.  Are you running the job with Advanced Open File Option (AOFO) selected? Are you running your jobs with encryption - and if so, is it 128 or 256 bit encryption?

Also, what revision and hotfixes do you have installed on the media server  (Help | About | Installed hotfixes), and are your remote servers to the save level? 

Please let me know.

Regards,

Christiane

[Rich_Lichvar]

Here is the exact message from the job log:

Backup- HERCULES
V-79-57344-34110 - AOFO: Initialization failure on: "Shadow?Copy?Components". Advanced Open File Opt...
Snapshot provider error (0x8007000E): Ran out of memory
Check the Windows Event Viewer for details.

The event view shows a volsnap error with ID 21.

Media Server is Rev. 6235, hotfixes 6, 9, 10, 11, 13, 15, 16. And no, AOFO was not chosen.

It's really crazy because some time it happens and some times it doesn't. Usually, as I said, if I restart all the BE services, it doesn't happen on the next backup. Last night was an exception to that rule. I restarted the services and the job still "failed" with that error.

I'm trying tonight's run after using the BE Wizard to select the AOFO method to see if that works.

BTW, BE is installed on the D: drive on that machine. The C: drive only has a little over 10% free space on it. The machine itself is running 2 GB of RAM.

[Christiane_Pete]

Hi Rich,

Thank you for the information.  Please see below technote:

Backup jobs fail with "out of memory" error or memory and handle leaks appear when creating snapshots with Microsoft Volume Shadow Copy Service providers.

http://seer.entsupport.symantec.com/docs/273562.htm

Even through the technote is seen as being valid for Backup Exec for Windows Servers (BEWS) v10x, the suggested patches are fixes concerning the OS of the server, so BEWS11d should also be covered. 

I would recommend that you verify whether the suggested patches are part of a service pack, and if so, install the service pack instead.

Also, I entered the event id "21" with source "volsnap" in www.eventid.net, and found the following:

http://www.eventid.net/display.asp?eventid=21&eventno=5418&source=VolSnap&phase=1

The above link is referring to a couple of different Microsoft patches, so I would recommend you also check that out as well.

Hopefully, the above will assist you in eliminating the "Out of memory" error.

Regards,

Christiane

[Coen_2]

Goodmorning,

We have the same problem @ multiple customers ( 8 so far).
It seems that it all started after the installation of Service Pack 2 for 2003 server (SBS...?)
What also seems to be od is that this problem only occurs on small business server 2003 and not on the normal 2003 server.

the problem occurs on the version 10d of backup exec with al patches and updates loaded, but also on the latest 11d version.

so far whe have tried a couple of things but nothing seems to work, if someone knows how to fix this problem please let us know !

[Steven_Hunter_2]

I've also got the same issue.

   V-79-57344-34110 - AOFO: Initialization failure on: "[system name]C:". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS). Snapshot provider reports "Out of memory".


The System log shows this error:

   The flush and hold operation for volume C: was aborted because of low available system memory.

   For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

   Event ID 21


I'm running the latest version of 11D  6235 on a fully updated 2003 Ent. SP1. I've looking into MS's solutions and defragged and checked the memory usage (it has 8GB) and changed the boot.ini
I've updated from 10D to 11D since this problem first arose, and I've been consistently update the remote agents with every update. None of my other servers have this issue, and that includes another web server.

The error occurs on Full and Differential backups.

Message Edited by Steven Hunter on 06-03-200704:14 PM

[8i5_2]

I've been getting these errors since upgrading several Exchange 2003 boxes from 2000 to 2003 (sp2). Contacted Veritas/Symantec (no help). Contacted MS and they suggested the very latest vss hotfix (kb931312) but that did not help. Still waiting on an answer. Unfortunately this issue is very intermittent, it can fail on one backup and then succeed the following run.

[Rich_Lichvar]

Ours, too, is intermittant. For example, we've now run 5 straight jobs w/o the problem; however, this has happened before and, wham, it will pop up again.

[thegoolsby]

Hello,

The condition which causes the out of memory errors are due to failures with Microsoft's Volume Shadow Copy Services, also known as VSS.

Backup Exec relies totally on these VSS writers to function correctly. When these shadow copy backups are performed, they are done entirely by VSS. If the writers fail, then Backup Exec consequently will fail, and report that VSS has failed.

Microsoft has the following knowledge base article on what can trigger a VSS failure which generates the Event ID 21 from VOLSNAP:

Event ID 21 is logged when you create a shadow copy backup on a Windows Server 2003-based computer
http://support.microsoft.com/kb/907875

As per Microsoft, "This problem occurs because the available system memory is low. The low system memory is caused by disk fragmentation." In some cases, a hard drive defrag has been known to resolve the issue, at least temporarily.

There are also various hotfixes which Microsoft has to resolve failures with VSS.

Since VSS failing in all of these cases, there is no fix or modification that can be made within Backup Exec to resolve the issue, as Backup Exec is not the cause of the failure.

As for the failures which occur with VSS and Exchange, whenever the Advanced Open File Option is selected in a backup job which contains the Exchange Information Store, it causes the Remote Agent to call on VSS to perform an open file, or ‘snapshot’ backup of Exchange. These snapshot backups can only be performed by VSS. Deselecting AOFO from your Exchange Backups will cause the Remote Agent to perform the backup rather than using VSS to perform the snapshot backup.

Hopefully this will help provide guidance toward a resolution for all.

Collin Goolsby
Technical Support Engineer
Symantec Corporation
www.symantec.com

[8i5_2]

I have been pointed to this article http://support.microsoft.com/kb/907875 umpteen times. Unfortunately if you read it you'll notice it does not apply to sp2 of w2k3. The hotfix I mentioned in my earlier post is the most up to date (as of 4th June 2007) hotfix and includes all previous vss hotfixes.
I am beginning to suspect this is simply a memory issue. It may be that the backup exec remote agent is consuming excessive memory but will have to carry out additional tests to find out if this is the case.
We are not using AOFO to backup exchange - we are using the full Veritas Exchange agent.

[thegoolsby]

As per the tech note, the problem occurs because the available system memory is low, so the issue is definitely a memory issue. You will also see the following on that document:

"Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section. This problem was first corrected in Windows Server 2003 Service Pack 2"

Unfortunately, since this is not our application which is failing, I not could fairly explain why applying SP2 did not resolve the problem, nor would I be qualified to do so.  

If you would like our assistance in troubleshooting the memory usage of our Remote Agent service, then I would need you to call in for support so we can assist you with that, if you feel it is needed.

[Rich_Lichvar]

I believe we have found at least a partial, if not complete, answer to our problem: disk space on the root/C: drive. We were below 5% free space on the drive. Even though BE was installed on another drive (which also had plenty of free disk space, like 40+ GB), apparently the lack of disk space on the C: drive caused the intermittant out of memory problems. Once we cleaned-up this drive to where we had about 15% free space, then we have no longer had the problem.