Backup Exec 2010 R2 SP1 - Exchange 2007 Incremental Backup Failure

[SYMAJB]

I have BEX 2010 R2 SP1 installed on W2K8R2, and am running backup to disk - then duplicating to Tape.

I am backing up SQL 2005, Exchange 2007 and regular application/file servers.  We have servers at 2003, 2008 and 2008R2.

We are running jobs based on a policy.  Jobs within the policy are a monthly full, weekly full and daily differential. Within each of the jobs we had Exchange (2 servers) set to do a FULL backup everytime (irrelevant of the overall job being differential or full).

I switched the exchange tab of the daily job within the policy to incremental - flush logs,and the job immediately started failing.

I am getting various errors:

Final error: 0xe000035e - The Backup Exec Agent for Microsoft Exchange was not used to create the last full backup of this database.  You must use the Exchange Agent to run a full backup before you run a differential or incremental backup.
Final error category: Resource Errors
Unable to complete the operation for the selected resource using the specified options.  The following error was returned when opening the Exchange Database file:  '-528 The current log file is missing. '


Or

Unable to complete the operation for the selected resource using the specified options.  The following error was returned when opening the Exchange Database file:  '-515 The timestamp in the next log does not match the expected timestamp. '


I changed the daily job to run incremental (as opposed to differential) on job and incremental on Exchange tab - and performed a RUN NOW on the weekly job (FULL).  Everything worked OK.  I then ran the daily (incremental) and all worked OK also.  That night when the job kicked of on a scheduled basis we had the same failures again.  If we change the exchange tab to be FULL - all works OK.

I have two Exchange 2007 servers (agents are up to date with server on each), and one works fine wilst one fails.  The working server is not yet live and has minimal traffic, the failing exchange server is live and has much mail every day.

I have seen various knowledgebase articles around this subject but am now unsure as to how to proceed

Any input appreciated.

AJ  
 

Backup Exec Server 2010 R2 SP1 installed with Hotfix 150095, Hotfix 144101, Hotfix 148347, Hotfix 153090.

[DominikG]

Try the following:

- Configure your Policy the way you want it to be: daily inc, weekly full.

- Make sure the selection list only contains the exchange ressources (information store) no disk/folder/file selections!

- Delete the association with the selection list ("delete jobs using this policy") and re-create the jobs.

- Please also make sure, weekly full runs at first (by rule or schedule)

Please also check, that no exchange maintenance runs at the time, the backup runs and please also make sure, that no other exchange backup runs between your backups of this policy.

[SYMAJB]

When you say make sure the list only contains the information store (no disks/folders etc.) - do you mean for the Exchange servers themselves ?  I obviously have many other servers in the selection list (non-exhange) which do contain folders etc. 

At present I have only the information stores for the exchange servers in the selection list, plus all the other selections for the other servers.

I did recreate the selection list in it's entirety this morning, delete the jobs and recreate the jobs.  I have not run the backup to test yet.  Tonight will be a scheduled FULL, with the next incremental setto be Monday evening.

Tanks for your input.

AJ

[DominikG]

Normally you do not need to split file/folder selections from Exchange data. On the other hand Symantec always recommends to do it like this. Especially if you encounter some problems backing up Exchange. ;)

Let's see what happens with you newly created backup...

[pkh]

The reasosn why you seperate your file backups from your Exchange backups is that it is not recommended that you turn on AOFO when backing up Exchange

[SYMAJB]

OK - here is an update and some more info.

Rather than wait for the Monday night incremenal run I was pro-active following the comments received and did the following:

I removed the Exchange Servers (2) from the standard backup policy

I created a new policy for the Exchange servers alone, 2 jobs, FULL on Saturday and INCR on Tue-Fri.

In the new policy I turned off AOFO on all jobs, had FULL for the job and FULL - Flush Logs on the Weekly (Saturday) Job, and INCR for the job and INCR - Flush Logs on the Daily (Tue-FRI) Job.

I ran the Weekly job and all ran OK - approx 126GB for the database and logs, and the logs flushed. (after the job - based on policy rules - a copy to tape runs for the backup image)

I then ran the daily job and all ran OK - approx 96MB for the logs, and the logs flushed. (after the job - based on policy rules - a copy to tape runs for the backup image)

All good so far.

So, the scheduled daily ran last night at 11pm, and we had the same failure again.

One more piece of info which I previously ommitted, the Exchange servers are VM's - and we also run Veeam to back these up on a daily basis.  Veeam comes in at 7pm daily, takes a VMWare snapshot of the VM's, backs them up and subsequently releases the snapshots.  The Veeam backups (incremental on a daily basis with a Synthetic FULL on a Friday) are running clean - always. 

My thinking at the moment is that Veeam is interfering with this in some way.  Until we recently switched the Backup Exec backups of Exchange to Incremental (it used to be FULL - Flush logs) - Backup Exec and Veeam coexisted perfectly - with both of them completing backups successfully.  But as Veeam is only operating on a VMWare snapshot, how can this be upsetting Backup Exec.  The Backup Exec error does however always indicate that we must use Backup Exec agent to perform a FULL backup prior to running an incremental - so he 'knows' that something else has been in there.

What is actually happenning in the Backup Exec job is that the job comes in at 11pm, he does a consistency check which is OK, then backs up the logs OK.  It is the next step when he tries to fush the logs where the error occurs.  The following are the entries from the Exchange Server Application event log from the start of the backup to the end.

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0) has prepared for backup successfully

Information Store (1688) Shadow copy instance 23 starting. This will be an Incremental shadow copy.

Exchange VSS Writer (instance 23) has successfully prepared the database engine for an incremental or differential backup of storage group 'First Storage Group'. The following 2 databases are mounted (though they will not be backed up):

Mailbox Database

Public Folders

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0:23) has prepared for Snapshot successfully.

Information Store (1688) Shadow copy instance 23 freeze started.

MSExchangeIS (1688) First Storage Group: Shadow copy instance 23 freeze started.

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0:23) has frozen the storage group(s) successfully.

Information Store (1688) Shadow copy instance 23 freeze ended.

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0:23) has thawed the storage group(s) successfully.

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0:23) has processed the post-snapshot event successfully.

Instance 1: A physical consistency check has been successfully initiated. The transaction log files in '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy45\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\' with a base name of 'E00' will be validated. No databases will be validated.

beremote (5844) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy45\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\E00): The database engine (8.03.0137.0000) started a new instance (0).

beremote (5844) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy45\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\E00): The database engine stopped the instance (0).

Instance 1: Transaction log files in '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy45\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\' with a base name of 'E00' have been successfully validated. No errors were detected.

Instance 1: The physical consistency check has completed successfully. All files were validated and no errors were detected.

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0:23) has unsuccessfully completed the backup of storage group 'First Storage Group'. No log files have been truncated for this storage group.

Information Store (1688) Shadow copy instance 23 aborted.

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0:23) has processed the backup completion event successfully.

Exchange VSS Writer (instance 3cacd899-f5ae-4ed1-b416-ca04dd54bcc0:23) has processed the backup shutdown event successfully.

The VSS service is shutting down due to idle timeout.

To try and bottom this one out I will test the following today:

Run weekly exchange backup job (FULL)

Run Daily exchange backup job (INCR) x 2

Assuming both the above run clean I will then perform a Veeam backup of the exchange server

Run Daily exchange backup job (INCR)

If Veeam is the issue then the last INCR job should fail with the same error.

After taking a look at the above does anyone have any input / comments ??

Thanks,

AJ

[DominikG · Accepted solution]

Veeam should be the reason for your errors.

I think it is releated to Veeam triggering the VSS Writer of Exchange, which leads to Backup Exec thinking, that it did not do the last backup of exchange. So the sequence numbers mix up. After A veeam backup there should be always a full backup of Exchange with BEwhich the incrementals can build up on.

Ever thought of using Agent for VMware? Can do your Exchange and VM-Backup in one single Job. single-pass backup, multiple restore options (files, folders, exchange data, complete VM)...

[SYMAJB]

With the help of the input above I resolved this as follows:

Veeam backup is taken once per week, immediately prior to the BEX FULL backup.  For the remainder of the week BEX takes incremental backups and all is well (as long as Veeam keeps his fingers out of things).  In summary, after any Veeam backup the next BEX must be a full, as Veeam has interfered with the VSS side of things - and BEX will not successfully conduct an incremental until a BEX FULL has been run.