Exchange 2010 - Unable to complete the operation for the selected resource using the specified options for the following reason: VFF Open Failure. This can be caused by low memory or disk resources.

[KirkBoyd]

Hi All,

We are currently running Symantec Backup Exec 2010 R3 (SP3) on our media server (Windows Server 2008 R2) and attempting to backup an Exchange 2010 environment.

Our normal backup schedule includes backing up our Exchange 2010 Mailbox server on a daily basis however over the last couple of weeks we have been receiving the following issue:

Job ended: Monday, April 28, 2014 at 1:08:00 AM
Completed status: Failed
Final error: 0xe00002f7 - Cannot extract mailbox messages from the Exchange backup. Review the job log for more information.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-759

Backup- \\EXCHANGETEST.com\Microsoft Information Store\Public Folders EXCHANGETEST

V-79-57344-759 - Unable to complete the operation for the selected resource using the specified options for the following reason: VFF Open Failure.  This can be caused by low memory or disk resources.

This incident does not occur every night but it has been happening sporadically over the last 6 weeks.

Has anyone seen this issue before?

Thanks in advance,

Kirk

[RahulG]

refer http://www.symantec.com/business/support/index?page=content&id=TECH66427&actp=search&viewlocale=en_US&searchid=1398692384945

If youa re abcking upto tape refer 

http://www.symantec.com/business/support/index?page=content&id=TECH127758&actp=search&viewlocale=en_US&searchid=1398692337210

[KirkBoyd]

Thanks for the prompt replay RahulG but we have tried the suggestions in this document:

We have changed the registry key (below) to 600.  Issue still occurs.

Key : HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VirtFile

Value: VFF Extended Timeout

Articles regarding the Windows Server 2003 are irrelevant as we are running Windows Server 2008 r2 x64.

[ArulPrasad]

is this backup to tape....?

If either it is to disk or Tape , if AV is Present ensure on access scan has a exclusion on the Disk location/ path of the b2d or Temp staging location (i.e c:\temp )

[KirkBoyd]

Thanks Arul. However we do run AV on both the media and remote server and we exclude the temp staging location

This backup is running to tape.

Forgot to mention that when we untick the "Use Backup Exec Granular Recovery Technology (GRT ) option on the job it completes successfully.

[ArulPrasad]

Yup that is true... The issue happens during the GRT processing... so if this is to tape the the GRT processing is done on the exchange server... , also try setting the registry to a different path than the one exist...

HKLM\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Exchange

String Value, "OnHostTemp".

set the value of OnHostTemp to "C:\temp"

ref:http://www.symantec.com/business/support/index?page=content&id=TECH164075

[KirkBoyd]

Thanks Arul.

What is the default staging area on the exchange server?  Is it C:\TEMP?

We currently don't have these registry settings on our exchange servers however I can add that entry to see if it helps.

The problems is difficult to diagnose correctly as the error is intermittent.  The backups completed successfully last night for instance without any changes / modifications. However it could fail again on Thursday / Friday which is the infuriating thing.

[VJware]

When the GRT Exchange backup runs to tape, the staging area is chosen on the Exchange server and typically the volume with the largest amount of free space is chosen for staging. Creating the OnHostTemp registry key on the Exchange server kinda forces the staging to the specified location only.

[citybett1]

I am having the same problem as described above.  I get the same error when backing up an Exchange 2010 server with 5 databases (4 mailbox databases, 1 public folder database).   Regardless of whether I'm backing up to disk or to tape, randomly one of the mailbox databases will fail with that error.  The public folder database has never failed, but it's much smaller than the mailbox databases.  All of the mailbox databases are in the ballpark of 20GB each.

I've implemented the registry entry for the "VFF Extended Timeout" referenced in the following article on both the BackupExec server and on the Exchange server, and restarted the Backup Exec services on both, but that did not resolve the problem.

http://www.symantec.com/business/support/index?page=content&id=TECH127758

This morning I have also added the "Detach Timeout" registry entry on the BackupExec server referenced in the following article to see if it helps (I haven't tested extensively yet since adding it and restarting the services).

http://www.symantec.com/business/support/index?page=content&id=TECH180039

My environment is Windows 2008 R2 64-bit Backup Exec 2010 R3 SP4 on the tape backup server.  Windows 2008 R2 64-bit on the Exchange 2010 server.  Plenty of RAM, virtual memory allowed is very large, more than 100GB free diskspace on all drives.

This has been happening randomly on my backups for nearly 2 months now.  Looking back through the logs, it looks like the errors started occurring shortly after applying Exchange 2010 SP3 Rollup5 that was released nearly the end of February.  No idea if this is the cause, but I thought I'd mention it.

http://support.microsoft.com/?kbid=2917508

[citybett1]

Followup to my previous post.  Adding the "Detach Timeout" registry entry referenced in http://www.symantec.com/business/support/index?page=content&id=TECH180039 hasn't fixed it either.

Last night I had the system run an Exchange with GRT backup to disk, and it failed with the same error on mailbox database #4. A few hours later the system ran an Exchange with GRT backup to tape, and it failed with the error on mailbox database #3. The previous night, no errors.

[KirkBoyd]

The "Detach Timeout" registry entry has not fixed it for me either.  The backup still fails intermittently.

It's becoming frustrating.

[citybett1]

Another person in the following thread is having the same problem, so we're not the only ones.

https://www-secure.symantec.com/connect/forums/backup-exec-2010-r3-final-error-0xe00002f7-cannot-ext...

I added the "VFF Extended Timeout", "Detach Timeout", and "OnHostTemp" registry keys to both the Backup Exec server and the Exchange server.  None of it fixed the problem.

I ended up opening a support case with Symantec and it's still being worked on.  The support tech ultimately couldn't resolve the issue either, so we ran the debug tool in Backup Exec, had the jobs fail and they handed all of the data off to the next level of support.  And that's where my problem currently sits.

[KirkBoyd]

How did you get on with the support case?  Any luck?

I'm still getting the intermittent errors on these jobs.

[citybett1]

No resolution yet.  The next level up of tech support is still working on it.  Based on my experience so far though, I'm guessing that rebooting the Exchange Server will "fix" the problem for you at least for awhile.

Prior to opening the case, I was having the random failures on GRT processing for both backups to disk and backups to tape.  As mentioned above in this thread, backups to tape do the GRT processing on the Exchange Server, and backups to disk do the GRT processing on the BackupExec server.

Anyway, to make a long story short, I ended up rebooting the BackupExec server, and so far since the reboot the backups to disk haven't failed.  But I have no idea if the problem is going to return in the future or not.

Tech support's analysis of the debug logs from the failures seems to indicate that the GRT processing fails because it can't find it's temp files that it created for some reason, which has everyone scratching their heads on why a reboot would fix the problem, but it seems to have fixed it (unknown if the problem is going to return) for the disk backups by rebooting the BackupExec server.   Antivirus isn't the culprit in my case on why the GRT processing can't find it's temp files.   If you think antivirus might be the culprit in your case, add the OnHostTemp registry key to force a specific location to be used for the temp files, and then exclude that location from your antivirus.

I'm sure I'll ultimately end up rebooting the Exchange server, but I haven't yet to allow Symantec to continue work on it since backup to tape still fails.

[KirkBoyd]

This absolutely sucks...  I'm going to try and implement the OnHostTemp registry key and see if it helps.

Restarting the Exchange Servers fixes the problem for about a day or two.

Any luck at your end?

[citybett1]

Nothing new on my end since the last post.

[KirkBoyd]

Any more news ?

The "OnHost" registry change has not really fixed the issue.  I'm running out of ideas now.

[citybett1]

Nothing new really.  The symantec tech got back in touch with me to collect additional information from the servers, but that's about it.

I'm working around the problem by manually running additional backups of the database that failed and having it append the backups to the tape over and over again until the backup succeeds.  Last night, database #4 failed with the error, so this morning I ran a manual backup of just database #4 and had it append to the tape.  Running the manual backup of the single database eventually succeeds.

[citybett1]

Just a followup.   No solution yet, Symantec's second level of support had to get their next higher up level of support involved and they're still working on it.

In the meantime, I'm still working around the problem by doing a manual backup of whichever database fails, as described in the post above.

[KirkBoyd]

Thanks.  We were still getting inconsistent failures so we created seperate DB backup jobs as a workaround.

Let me know if you get a result.