Exchange 2010 SP1 and NBU 7.0 logs not truncating

[jlpeppers]

Has anybody successfully backed up an Exchange 2010 SP1 environemnt with a DAG and/or single database such as a public folder database with version 7.0?  We are having an issue where log files don't always truncate and when they do it still keeps several days to 12 hours worth.  We cannot figure out the pattern or what it is doing.  We tried backing up the passive copies, the active copies, nothing.  We did have to add the template for the Microsoft Database Availability group in the directive set but that setting is not maintained in the policy.  It keeps reverting back to Microsoft Exchange_200x. 

[RiaanBadenhorst]

Hi,

So does the backup complete with 0, or 1?

Can you post the bpbkar log?

[Zahid_Haseeb]

You may be in the following circumstances:

Netbackup 7.0 for Exchange Server administration guide.

http://www.symantec.com/business/support/index?page=content&id=TECH127056

Page # 67

Full backup:

This schedule type backs up the Exchange Server database and associated transaction

logs. Exchange truncates all committed transaction logs after NetBackup notifies it that

the backup succeeded. In replicated environments, the truncation is scheduled and does not occur immediately.

By default, transaction logs are not truncated for Instant Recovery backups.You can enable the truncation of logs for this type of backup or you can perform a backup to a storage

unit.

[jlpeppers]

The backups complete with a status of 0.

[jlpeppers]

It is unlikely that we would have 5 days of uncomitted transaction logs. 

[Marianne]

Have another look at this extract from the manual:

" Exchange truncates all committed transaction logs after NetBackup notifies it that the backup succeeded. "

Also on page 181 of the NBU for Exchange Admin Guide:

Exchange Server transaction log truncation error

The Exchange server deletes transaction logs after a successful backup (for full and differential backup types). If the Exchange server encounters any errors during the deletion process, it logs this information in the application event log.
Since the actual backup was successful, NetBackup exits with a status 0 (successful backup). Refer to the Microsoft Exchange Server documentation for information on any errors that are encountered with the transaction logs.

[RiaanBadenhorst]

I'm not too familiar with Exchange 2010 but from what I've read on the net, mostly MS forums is that there are a lot of dependencies involved, like the replication of the logs to other servers in the DAG, the replay of said logs etc.

Are all your servers "in sync" from this point of view?

I'll see if i can find any of the post relating to that.

[jlpeppers]

All servers are in sync.

[jlpeppers]

This is the weird part, there are no errors in the event log.  I actually get event ID 2046 that log truncation has been requested.

[jlpeppers]

I also notice that in the bpbkar log file it refers to the databases as Microsoft Information store instead of Microsoft Database Availability Group.

[RiaanBadenhorst]

Hi,

I found this on MSDN.

http://msdn.microsoft.com/en-us/library/bb204080%28EXCHG.140%29.aspx

Not saying its an MS issue, just highighting this because maybe the two applications are talking to each other that well.

Take note of this section.

Interactions Between the Writers, VSS, and VSS Requestors

The high-level interaction between the VSS, the Exchange Writer, and Exchange Server 2010 during backup operations is as follows:

1. The backup program (or agent) runs a scheduled job.
2. The VSS requestor in the backup/restore application sends a command to the VSS to take a shadow copy of the selected Exchange Server 2010 databases.
3. The VSS communicates with the Exchange Writer to prepare for a snapshot backup. Exchange Server 2010 prohibits administrative actions against the databases, checks volume dependencies, and suspends all write operations to selected instance of the database and transaction log files while allowing read-only access.
4. The VSS communicates with the appropriate storage provider to create a shadow copy of the storage volume that contains the Exchange Server 2010 databases.
5. The VSS releases Exchange Server 2010 to resume ordinary operations.
6. The VSS requestor verifies the physical consistency of the backup set prior to signaling the backup was successful. Exchange Server 2010 truncates the transaction logs (if the database is part of a DAG, log truncation is replicated among all the copies) and records the time of the last backup for the database.

So from your logs we see that NetBackup thinks the backup is successful, but are there any indications on the exchange server that it has completed a successful backup so it can proceed to truncate the logs? An event viewer entry maybe, or the status of DAG/Store usually shows the last succesful backup time.

[Zahid_Haseeb]

would you share policy ?

bppllist POLICY_NAME -U

[JP97]

We are attempting to backup Exchange 2010 from a passive copy.  We determined that log truncation occurrs if you backup using the active copy.  When backing up via the passive copy, there are errors about 10 minutes after the completion of the backup in the truncation log about an RPC failure with the VSS writer.  Microsoft is in agreement that this is their issue.

Has anyone solved this yet?  We have MS engineering working the issue with us now.

[Flappyhead]

We are having the same issue where a passive copy backup does not truncate the logs! Please let us know if you find a resolution.

[RiaanBadenhorst]

Hi,

How many DB's, how many servers? Did you check the application event viewer on the passive nodes. I'm sure there will be a lot of errors in there.

Does your backups complete with a status ZERO?

Does the backed up status of the database reflect the time that NetBackup completed the backup?

What is your file selection in the policy?

[JP97]

There are two potential things going on...

1.  If you're just doing testing and not seeing log truncation, you're probably dealing with a "feature" in exchange 2010.  In 2010 there is a log truncation checkpoint.  By default, Exchange 2010 keeps 100 logs.  You can view the log truncation checkpoint by reading the header information of any of the databases.  You can configure this checkpoint (not recommended by Microsoft, but it's possible).  MS is currently writing a white-paper/blog on the log truncation checkpoint and it should be available soon.

2.  (our issue)  We have 9 servers, 6 in one site and 3 in our DR site.  All servers are members of the DAG.  We also have two network interfaces, one for MAPI traffic and one for replication (REPL).  Our MAPI traffic goes through a Big IP F5 for load balancing.  If we run backups from the DR site (our preferred config) then we get mixed results on the log truncation.  Sometimes it works and sometimes it doesn't.  It's completely random.  We can see on the server with the active DB that it receives the log truncation request, but 10 minutes later (yep that's a timeout value), the application logs on the server report that the backup was unsuccessful.  Also, netbackup always reports 100% success.  If we backup in our primary site from the passive copies there, we get 100% success.  There is an option (not sure if it's in Exchange or the cluster config) to force log truncation information to select the REPL network.  By Microsoft's own admission, it doesn't mean that it will actually select the REPL network.  We proved this to be true through network traces (we even rebooted the systems to see if the option had to be forced to take effect through a reboot).  We finally put entries in the hosts files of each of the mailbox servers to only know about each other through the REPL network IP addresses.  Once we did this, we achieved 100% success.  Microsoft is allowing us to do this for now.  They say the hosts file entries may have unintended consequences, but they can't give us an explanation of what those consequences are.  Microsoft has two network traces from us, one that failed and one that worked after we added the hosts file entries.  They are also building a dev environment that matches ours and are attempting to reproduce the issue.