Forum Discussion

dhill82's avatar
dhill82
Level 3
12 years ago

Exchange 2010 "Out of page buffers"

We have been running Backup Exec 2012 with all patches as of today (8/8/2012) to back up our Exchange 2010 (also fully up to date) for a few months.  Full backups always complete successfully, then Incrementals will run for 2-3 days just fine.  After a few days, Incrementals start to fail with the error "V-79-57344-759 - Unable to complete the operation. The following error was returned when opening the Exchange Database file: '-1014 The database is out of page buffers. '".  After this happens, the Incrementals fail with the error "The last backup job was not run by Backup Exec" until the weekly Full backup runs.  This happens every week.

The Backup Exec temp files are not be scanned by the antivirus, and I can't find anything on the Exchange server that would run at these times that would use extra resources during the backup windows.  Circular loggin is not enabled.  I tried changing the registry entries mentioned in the MS KB article that discusses Exchange running out of page buffers (don't remember the KB# off the top of my head).

I've also tried modifiying the backup windows (changing the day the Full runs, changing the times, etc).  Still happens within a few days.  Is there anything that could be causing this, or settings I can change to avoid this going forward? 

 

2012
Agent for Microsoft Exchange Server
Backup and Recovery
Backup Exec
Troubleshooting
  • IS2's avatar
    IS2
    12 years ago

    Update - but not necessarily good news.

    For the past week I did the following with the indicated results:

    • Separate the System and Information Store jobs into two jobs - GOOD
    • Use “Differential” for the Information Store - Didn't make a difference and actually seemed to cause move issues when GRT was turned on.
    • Use “Incremental” for the System - GOOD
    • Do NOT allow the “Differential” and “Incremental” to run at the same time - MAKES SENSE
    • Do NOT turn on Snapshotting (Advanced File Open) for the Information Store job - UNNECESSARY - In reality, BE uses Microsoft VSS automatically on Exchange backups.
    • Turn on GRT for both the System and Information Store jobs - BAD - on the Infostore backups, using GRT created different issues.

    The Symantec Tech I talked to recommended that we simply get away from backing up the Infostore using Dedup. While not necessarily documented, using Dedup for Exchange, due to the overhead required, isn't an efficient way to go.

    So, what I am now doing, and it appears to work fine, is the following:

    • System and Infostore jobs are separated
    • System - Full with incrementals using GRT and DEDUP storage
    • Infostore - Full with incrementals using GRT and DISK storage

    The obvious downside is the need to use a lot more disk. I like to keep four weeks of fulls and two weeks of incrementals. I won't be able to do that with this technique. So, I've also added a "Duplicate to Tape" step on the Infostore so that I can have access to older copies of the Infostore.

    Here are some of the salient points of my discusison with Symantec Technical Support:

    • Because of the dynamic nature of the Exchange database, Dedup storage is not the recommended storage media for Exchange databases. Even though my results were pretty good using Dedup, apparently the BE resources necessary to determine what needs to be written to Dedup are substantial. And, the further away you get from the full backup, the greater the resource requirement becomes - which may be why the incremental jobs die after a few days.
    • One possible solution (which I am not going to try) is to run full backups every night using GRT and Dedup. I may try this at some point, but I'm going to give the Disk backup a shot first.
    • Using Disk instead of Dedup will provide for much faster recovery time for an individual mailbox or mailbox items.
    • Be sure that your Exchange database maintenance jobs do not run at the same time the backup is running.  This is documented in the Best Practices Guide.

    Finally, just in case you haven't seen this link, it's worth checking out. It doesn't really help this situation (unless the documenation has been updated), but there are some good things to consider.

    http://www.symantec.com/business/support/index?page=content&id=HOWTO74626

    I will update this post next weekend after I've had a chance to let a full week go by using the solution outlined above.

31 Replies

Sort By
Show More
  • IS2's avatar
    IS2
    Level 3
    12 years ago

    I did check for the update and there was nothing available for our installation - but thank you for the info...

    An update... The change to "server-side" dedup did not appear to work. However, it wasn't a perfect test because we had two issues with the Exchange backup this week.

    What I am going to do this week is move the Exchange backup from dedup storage to disk storage. Given that we didn't have this situation prior to moving to dedup, I want to prove that it works in exactly the same environment with disk storage. If it does, I'll move it back to dedup and see what happens.

    I'll check back by the end of the week to update our status.

  • IS2's avatar
    IS2
    Level 3
    12 years ago

    Changing the storage media from dedup to disk worked like a charm.  We had no issues this week (first time in months).

    I'm moving back to dedup this week but changing the schedule so that the Exchange backup runs on its own.  I currently allow four concurrent jobs on the dedup storage media an only one on the disk media.  I'm wondering if the Exchange job exposes a resource issue that I don't have otherwise.

  • IS2's avatar
    IS2
    Level 3
    12 years ago

    Not surprisingly, the issue is back...  The full backup on Friday night ran fine as did the incrementals on Sunday and Monday.  The Tuesday night incremental however failed with the "Out of page buffers" error again.  Needless to say, this is frustrating.

    My next test is to run the full backups using dedup and the incrementals using disk storage.

  • dhill82's avatar
    dhill82
    Level 3
    12 years ago

    Been meaning to update this post.  I still have the issue, even after the latest patches.  Haven't tried moving off of dedup, but I have a feeling that if I do I'll have the same results as IS@ESPC. 

  • GarfieldMaximus's avatar
    GarfieldMaximus
    12 years ago

    Hi ,

    I do have the exact same problem as you guys. But I'm wondering ... would it not be possible to solve this problem without changing the storage type. (we also have dedup disks right now) Maybe someone from Symantec could help us ? I don't know if they read the forum themselves ?

    Thanks for the topic , it has been very interesting so far.

  • IS2's avatar
    IS2
    Level 3
    12 years ago

    The change to a combination of Dedup (full) and Disk (incremental) did not solve the issue. I had my typical Wednesday issue this morning. For now, I'll move back to all disk.

    I will however call Symantec today and see what they can do. So far, they have not had a solution for the issue. I hope we now have enough information to help them resolve the problem.

    I'll post how the discussion went.

  • auth1299's avatar
    auth1299
    Level 2
    12 years ago

    I'm having the exact same issue. I am also running a similar environment. I have exchange 2010 Version: 14.02.0318.001 running on windows 2008 R2 SP1 64 bit as a virtual machine on xen server.

    My backup exec 2010 is runing on windows 2008 R2 SP1 64 bit on a stand alone dell server with 8gb of ram and the virtual disk is set to automatic.

    This issue also started around september. Maybe a windows patch broke something. I'm getting the exact same error as mentioned above. Sometimes the backup runs fine but most of the time I get the error:

     

    Backup- \\ExchangeDAG.company.local\Microsoft Information Store\company
V-79-57344-759 - Unable to complete the operation for the selected resource using the specified options.  The following error was returned when opening the Exchange Database file:  '-1014 The database is out of page buffers. '

     

  • auth1299's avatar
    auth1299
    Level 2
    12 years ago

    I'm backing up using  (FULL - Using archive bit (reset archive bit)) every backup. My exchange DB is about 60GB.

  • IS2's avatar
    IS2
    Level 3
    12 years ago

    Sorry about the delay in posting. I am working with with Symantec Tech Support. They started debug and have collected two days worth of logs. A couple of interesting items to report... When we started using the Troubleshooting tool, it found that the ASR Writer was in a failed stated. After a bit more research, we found that this seems to occur only when the Symantec Backup process begins. On servers where I do not run BE2012, I do not have this issue. The only way to get the ASR Write into a 'stable' state is to restart the server.

    If you want to run the troubleshooting tool, go to "Backup Exec button" --> Technical Support --> Run the support tool..."

    Also, changing the Exchange Server backup to NOT use GRT seems to have worked. It's already Thursday and the Exchange job has not failed. This is not a good long-term solution, but it's better (for me) to not have the job fail.

    The Tech Support person did have me change my default snapshot technology option on the backup job (Advanced File Open) from "Automatic" to "Microsoft VSS..." then pick "System - ". I'll post the results next week.

    Finally, I am also beginning to have an issue where periodically BE2012 reports that one of my Windows Agents is not the right version. When I look at the versions, all is fine... I have to reinstall the Agent to resolve this issue. Is that something any of you are also dealing with?

  • dhill82's avatar
    dhill82
    Level 3
    12 years ago

    Thanks for the update.  Nothing I've done has worked, and setting up off-host backup failed miserably.  Not sure what else to do, so hopefully Symantec can get this straightened out.