Exchange 2010 "Out of page buffers"

[dhill82]

We have been running Backup Exec 2012 with all patches as of today (8/8/2012) to back up our Exchange 2010 (also fully up to date) for a few months.  Full backups always complete successfully, then Incrementals will run for 2-3 days just fine.  After a few days, Incrementals start to fail with the error "V-79-57344-759 - Unable to complete the operation. The following error was returned when opening the Exchange Database file: '-1014 The database is out of page buffers. '".  After this happens, the Incrementals fail with the error "The last backup job was not run by Backup Exec" until the weekly Full backup runs.  This happens every week.

The Backup Exec temp files are not be scanned by the antivirus, and I can't find anything on the Exchange server that would run at these times that would use extra resources during the backup windows.  Circular loggin is not enabled.  I tried changing the registry entries mentioned in the MS KB article that discusses Exchange running out of page buffers (don't remember the KB# off the top of my head).

I've also tried modifiying the backup windows (changing the day the Full runs, changing the times, etc).  Still happens within a few days.  Is there anything that could be causing this, or settings I can change to avoid this going forward? 

[dhill82]

Thanks for the update.  Nothing I've done has worked, and setting up off-host backup failed miserably.  Not sure what else to do, so hopefully Symantec can get this straightened out.

[IS2]

Symantec is escalating the issue on Monday.  In the meantime, I was successful this week by simply turning off the GRT option.  This won't work for me long-term, but at least I have a backup of the Exchange Server in case something goes wrong.

[IS2]

Update:  Symantec Tier 3 support has suggested the following (which I will try for a week).

• Separate the System and Information Store jobs into two jobs
• Use “Differential” for the Information Store
• Use “Incremental” for the System
• Do NOT allow the “Differential” and “Incremental” to run at the same time
• Do NOT turn on Snapshotting (Advanced File Open) for the Information Store job
• Turn on GRT for both the System and Information Store jobs

I’ll give this a shot and let you all know how it works.  Obviously, we won’t know much until the end of next week.

[Larry_Hyman]

Having the exact same problem. Fulls work great and Inc seems to not be able to handle the load. I also have a case open with Sym. My tech should talk to your tech..     Case 420-132-318 Thx, Larry

[dhill82]

I just made the changes that IS@ESPC received from Symantec as well.  I set my full backup to run tonight, so we'll see what happens in a few days.  I also added 4GB of RAM to my Exchange VM.  Probably won't make a difference with the backup, but it's possible.

[RSpicer]

Just checking in to see if the problem still exists for dhill82 and all. 

I have the exact same symptoms but with Exchange 2007.  Breaking out the File backup to its own job and adding the VFF timeout entry (up to the max of 600) to the registry on the Backup Exec server seemed to help but not cure the problem.  I no longer have to restart the services after adding the VFF entry when the job fails. 

Key : HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VirtFile

Value: VFF Extended Timeout

Type: DWORD

The Maximum value is 600 seconds.

Start out at 240 seconds at first. If error still occurs then 480, before going to 600.

Restart all the Backup Exec Services for the changes to take effect.

[mbsweep16]

Just got this error on my Exchange Inc today for the first time. Similar setup. Exchange 2010 on Hyper-V.  BE 2012 on 3600 R2 Appliance (installed 2 months ago). Hyper-V agent backup with GRT runs on Fri night, then Exchange agent Full on Sat night and Incrementals Sun-Thur night. Haven't had any issues until last nights' backup. Read the TECH articles referenced earlier. Will try the fixes IS@ESPC mentions.

Have had sporadic problems with different backups ever since install. Open case for SharePoint failures. Have been nothing but frustrated with BE and Symantec for the last 2 months. We've been using BE for years and have never had the issues we've experienced with 2012 and this appliance. Don't know who to blame it on. Did Symantec release a product that wasn't ready for prime time or have M$ and others made changes/updates that make it difficult to backup their products? Would love to be able to go even one day without worrying about backups. I'll get down off my soap box now...

[IS2 · Accepted solution]

Update - but not necessarily good news.

For the past week I did the following with the indicated results:

• Separate the System and Information Store jobs into two jobs - GOOD
• Use “Differential” for the Information Store - Didn't make a difference and actually seemed to cause move issues when GRT was turned on.
• Use “Incremental” for the System - GOOD
• Do NOT allow the “Differential” and “Incremental” to run at the same time - MAKES SENSE
• Do NOT turn on Snapshotting (Advanced File Open) for the Information Store job - UNNECESSARY - In reality, BE uses Microsoft VSS automatically on Exchange backups.
• Turn on GRT for both the System and Information Store jobs - BAD - on the Infostore backups, using GRT created different issues.

The Symantec Tech I talked to recommended that we simply get away from backing up the Infostore using Dedup. While not necessarily documented, using Dedup for Exchange, due to the overhead required, isn't an efficient way to go.

So, what I am now doing, and it appears to work fine, is the following:

• System and Infostore jobs are separated
• System - Full with incrementals using GRT and DEDUP storage
• Infostore - Full with incrementals using GRT and DISK storage

The obvious downside is the need to use a lot more disk. I like to keep four weeks of fulls and two weeks of incrementals. I won't be able to do that with this technique. So, I've also added a "Duplicate to Tape" step on the Infostore so that I can have access to older copies of the Infostore.

Here are some of the salient points of my discusison with Symantec Technical Support:

• Because of the dynamic nature of the Exchange database, Dedup storage is not the recommended storage media for Exchange databases. Even though my results were pretty good using Dedup, apparently the BE resources necessary to determine what needs to be written to Dedup are substantial. And, the further away you get from the full backup, the greater the resource requirement becomes - which may be why the incremental jobs die after a few days.
• One possible solution (which I am not going to try) is to run full backups every night using GRT and Dedup. I may try this at some point, but I'm going to give the Disk backup a shot first.
• Using Disk instead of Dedup will provide for much faster recovery time for an individual mailbox or mailbox items.
• Be sure that your Exchange database maintenance jobs do not run at the same time the backup is running.  This is documented in the Best Practices Guide.

Finally, just in case you haven't seen this link, it's worth checking out. It doesn't really help this situation (unless the documenation has been updated), but there are some good things to consider.

http://www.symantec.com/business/support/index?page=content&id=HOWTO74626

I will update this post next weekend after I've had a chance to let a full week go by using the solution outlined above.

[dhill82]

I've had good luck so far with the changes that IS@ESPC mentioned on the 17th.  I had one backup fail, but the others ran fine afterwards.  Before when one failed, they would continue to fail until the next Full backup.  I have noticed that since the 25th my logs haven't been getting deleted as they should be, but that's probably an unrelated issue.  SQL logs on other servers are deleting as normal, it's just Exchange that's having the problem.

[IS2]

I should have mentioned that the 'differential' will not delete log files - by design.  Since the differential contains everything since the last 'full', it needs to keep all the logs in case of a restore.

Glad to hear the 'differential' is working for you.

[dhill82]

I didn't think of that.  Of course, now that I said something yesterday, last night's backup failed with the error "V-79-57344-759 - Unable to complete the operation for the following reason: VFF Open Failure. This can be caused by low memory or disk resources."  Ugh...

[IS2]

Well, after a week of using the following configuration, I have had no problems.  See previous post from 10/28 for more details on why this works.

• System and Infostore jobs are separated
• System - Full with incrementals using GRT and DEDUP storage
• Infostore - Full with incrementals using GRT and DISK storage