Backup Exec 2010 R3 - Final error: 0xe00002f7 - Cannot extract mailbox messages from the Exchange backup.

[GMAN70]

We have a Windows 2008 R2 (64 Bit) Backup Exec 2010 R3 Servers (Ver 13.0 Rev 5204) that protects 2 Exchange Mailbox Servers using Backups to Tape (via an Autoloader). Both Exchange Servers are running Windows 2008 R2 (64 Bit) and Exchange 2010 SP2. Server 1 has a single Mailbox DBF (302 GB) and a Public Folder DBF (27 GB), Server 2 has 2 Mailbox DBF's (DBF1 = 456Gb, DBF2 = 30 GB) and a Public Folder DBF (27 GB). Both Exchange Servers backup the server drives first then the Mailbox DBF's, then the Public Folder DBF and System State etc.

The Backup Server fails to backup DBF2 on Server 2 on a random bassis (not every backup) with the error. However the public Folder is successfully backed up immediately after the second DBF that fails.

Completed status: Failed
Final error: 0xe00002f7 - Cannot extract mailbox messages from the Exchange backup. Review the job log for more information.
Final error category: Resource Errors

Backup- \\Server2\Microsoft Information Store\Mailbox Database 2V-79-57344-759 - Unable to complete the operation for the selected resource using the specified options for the following reason: VFF Open Failure.  This can be caused by low memory or disk resources.

Looking at the Exchange Server Logs there is no difference for Log entries for when the Backup Works on the DBF2 and when it doesn't VSS Opens, Backup Completesd and VSS Closes.

Searched the Symantec Site and found the following articles;

http://www.symantec.com/connect/forums/v-79-57344-759-unable-complete-operation-selected-resource-us...

Which references http://www.symantec.com/business/support/index?page=content&id=TECH127758.

I've implemented the VFF Extended Timeout Reg Entry on the Exchange Server, the is a second entry for Session Manager\Memroy Management but this appears to be a 32 Bit OS requirement not 64 Bit so I've not implemented.

 
http://www.symantec.com/connect/forums/exchange-database-backup-failing-time-time

Which references re-registering VSS (but as its working for the larger DBF and the PF DBF aferwards I don't believe the issue is VSS)

http://www.symantec.com/business/support/index?page=content&id=TECH205240

Which references apply SP4 to the Media Server and Remote Agent - which we did and it made no difference so we downgraded to SP3 (to keep the estate consistent - this is the only backup issue we have)

If we disable GRT on he Backup Job all DBF's are backed up without any issues but we can't selective restore items from a Backup.

We have also re-created the Backup Job (from scratch to remove potential job corruption) and had Debug logs running but they didn't really provide any useful infomation.

http://www.symantec.com/business/support/index?page=content&id=TECH128187

Which references setting OnHostTemp Reg Key (which I haven't set as the errors aren't exactly the same, the dirves are all local and largest drive is local therefore used by default). Also haven't had to set anything on Server 1

[citybett1]

I am having the same problem as described above.  I get the same error when backing up an Exchange 2010 server with 5 databases (4 mailbox databases, 1 public folder database).   Regardless of whether I'm backing up to disk or to tape, randomly one of the mailbox databases will fail with that error.  The public folder database has never failed, but it's much smaller than the mailbox databases.  All of the mailbox databases are in the ballpark of 20GB each.

I've implemented the registry entry for the "VFF Extended Timeout" referenced in the following article on both the BackupExec server and on the Exchange server, and restarted the Backup Exec services on both, but that did not resolve the problem.

http://www.symantec.com/business/support/index?page=content&id=TECH127758

This morning I have also added the "Detach Timeout" registry entry on the BackupExec server referenced in the following article to see if it helps (I haven't tested extensively yet since adding it and restarting the services).

http://www.symantec.com/business/support/index?page=content&id=TECH180039

My environment is Windows 2008 R2 64-bit Backup Exec 2010 R3 SP4 on the tape backup server.  Windows 2008 R2 64-bit on the Exchange 2010 server.  Plenty of RAM, virtual memory allowed is very large, more than 100GB free diskspace on all drives.

This has been happening randomly on my backups for nearly 2 months now.  Looking back through the logs, it looks like the errors started occurring shortly after applying Exchange 2010 SP3 Rollup5 that was released nearly the end of February.  No idea if this is the cause, but I thought I'd mention it.

http://support.microsoft.com/?kbid=2917508

[lmosla]

Hello,

Have you tried recreating the job? I see you have been researching the errors pretty thoroughly but here is another link that might help:  http://www.symantec.com/docs/TECH187210

[citybett1]

I haven't tried recreating the jobs, there are multiple jobs that all fail randomly the same way so I'm guessing it's not the problem.  These backups ran correctly for months and started failing around the beginning of March, which as I mentioned is when I installed Exchange 2010 SP3 Rollup5 on the Exchange server.

Adding the "Detach Timeout" registry entry referenced in http://www.symantec.com/business/support/index?page=content&id=TECH180039 hasn't fixed it either. 

Last night I had the system run an Exchange with GRT backup to disk, and it failed with the same error on mailbox database #4.  A few hours later the system ran an Exchange with GRT backup to tape, and it failed with the error on mailbox database #3.    The previous night, no errors.

[lmosla]

Do you have antivirus exclusions?

the processes to exclude from being scanned include:

C:\Program Files\Symantec\Backup Exec\beremote.exe
C:\Program Files\Symantec\Backup Exec\beserver.exe
C:\Program Files\Symantec\Backup Exec\bengine.exe
C:\Program Files\Symantec\Backup Exec\benetns.exe
C:\Program Files\Symantec\Backup Exec\pvlsvr.exe
C:\Program Files\Symantec\Backup Exec\BkUpexec.exe
C:\Program Files\Symantec\Backup Exec\RAWS\beremote.exe  
and the Backup Exec temp directory C:\Temp\Backup Exec on the Exchange server 

In Windows Events check for errors that occur during the time of the failed backups.

Are these backup jobs in a policy.

Try recreating one or two of the jobs that fail most frequently and keep the Exchange backup seperate from other types of backups (flat files, SQL etc...)

[citybett1]

Thanks for the suggestions, there's no antivirus software on either of the servers to conflict with.  No errors in the Windows Event viewer other than the single Backup Exec entry in the Application log with the job failure message.

Not sure what you mean about the backup jobs in a policy.  They're very simple jobs created by simply creating a Backup job with the wizard and choosing the Exchange Server and enabling the GRT backup option.  AOFO is turned off, and no other data from anything else is getting backed up along with these jobs.

I also added the "OnHostTemp" registry key pointing to C:\Temp on both servers referenced in the following article.  It has not resolved the issue either.

http://www.symantec.com/business/support/index?page=content&id=TECH128187

I will try creating a completely new job and see what happens.......

[citybett1]

Another person in the following thread is having the same problem, so we're not the only ones.
 
https://www-secure.symantec.com/connect/forums/exchange-2010-unable-complete-operation-selected-reso...
 
I added the "VFF Extended Timeout", "Detach Timeout", and "OnHostTemp" registry keys to both the Backup Exec server and the Exchange server.  None of it fixed the problem.
 
I ended up opening a support case with Symantec and it's still being worked on.  The support tech ultimately couldn't resolve the issue either, so we ran the debug tool in Backup Exec, had the jobs fail and they handed all of the data off to the next level of support.  And that's where my problem currently sits.

[bnich309]

Curious if you received a solution for this issue from Symantec support or if you're still working on it? We're experiencing the same error to a tee, but the jobs fails randomly, sometimes runs fine for a week or two and will then fail one or two nights in one week.

We're also running BE 2010 R3 SP4 but Exchange 2010 SP2 Rollup 2 (v14.02.0298.004); our BE and EX servers are both x64 running Server 2008 R2 SP1. So I don't know this issue is limited to EX2010 SP3 RU5.

I've tried all the solutions I can find online, except recreating the jobs, and was about to create a case but found your article, any insight you can provide is much appreciated! Did recreating the jobs resolve the issue for anyone?

[citybett1]

No solution yet specifically.  Symantec's second level of support had to get their next higher up level of support involved and they're still working on it. 

In the meantime, I'm working around it by simply running individual backup jobs of the database that fails and having it append to the backup.   So if last night's backup has Database #4 as the one that fails with the error, when I come in the next morning I run a manual backup job of only Database #4 and have it append to the media used for last night's backup. 

[citybett1]

Well, here's the end of my story on this problem. Symantec's developers ended up blaming the problem on the server running out of memory, and suggested I double the RAM from 8GB to 16GB. Which doesn't make any sense to me, since the server always has free memory available to use, and PLENTY of virtual memory.

Rebooting the server periodically resolves the issue for awhile.

[Nogatri]

Hi

Same problem on my environment Exch2010 Server / Windows 2008 64bits with 2 mails databases + Public Folder database.

We get randomly the error:

Etat de fin : Echec
Erreur finale : 0xe00002f7

V-79-57344-759 - Impossible de terminer l'opération pour les raisons suivantes : Echec d'ouverture d...

1. We have tried to increase the VFF Timeout as proposed for this issue by Sym:

http://www.symantec.com/business/support/index?page=content&id=TECH127758&actp=search&viewlocale=en_...

2. We have updated Backup Exec with SP4  and updated the BE Agent too:

http://www.symantec.com/business/support/index?page=content&id=TECH205240&actp=search&viewlocale=en_...

Same problem...

We have noticed that when BE displays "backup fails 0xe00002f7" with 0% finished status, the backup job has been correctly done when we look at the BE log. We have the good amount of data and files. And the Exchange server receives correctly the OK from BE at the end of the backup job, in order to truncate logs as well.

What's the trouble with this Symantec software??? We are tired of spending hours searching information and fixes for this error.

Anyone has another suggestion to solve this issue? Removing BE for another backup tool?

Thanks

[Vlad_Kap]

Hi,

We have the same problem. We have just update Microsoft Exchange from 2007 to 2010 SP3 Rollup Update 6. Rendomly we get this errors.

For backup we Use Backup Exec 2010 R3 SP4 which installed on Windows server 2008 R2 Standart Edition.

Have anybody found solution? What Symantec support say?

[Nogatri]

Hi,

We have found a workaround : we have scheduled a daily reboot on the Backup Exec Server.

It's not glorious...