Highlighted

BE2012 - Exchange 2010 consistency fails, not with Server Backup

After an infrastructure upgrade replacing basic switches without management capabilities (replaced 100Mbit -> 1000Mbit) on May 1st, I've had issues with backing up our Exchange 2010 server (Windows 2008 R2). It started with a failure on the full backup (incrementals after failed fulls shouldn't be considered) and it got worse over time. Backups before May 1st have 99% been successful, after I cancelled the running job on that day due to the upgrade, it started failing with about 3 returning errors. These machines (backup and mail) haven't had a brown-out during this time and kept working.

 

This is the summary of the last month (dates in dd/mm/yyyy format)

Naamloos.png

Errors and their failures:

E0000397 (BLUE):

Backup --> Cannot extract mailbox messages from the Exchange backup. Review the job log for more information.

Backup --> Unable to complete the operation. The following error was returned when opening the Exchange Database file: '-1022 There is a disk IO error. '

Verify --> Data for \\MAILSRV\Microsoft Information Store\MAILDB\DLLs is corrupt.

E00002F7 (GREEN):

Backup --> \\MAILSRV\Microsoft Information Store\MAILDB Unable to complete the operation. The following error was returned when opening the Exchange Database file: '-543 The required log files for recovery are missing. '

E00084F9 (RED):

Verify --> A selection on device \\MAILSRV\Microsoft Information Store\MAILDB was skipped because of previous errors with the job.

E000035E (YELLOW):

Backup --> The Backup Exec Agent for Microsoft Exchange was not used to create the last full backup of this database. You must use the Exchange Agent to run a full backup before you run a differential or incremental backup.

Also accompanied by the E000035E error usually (not always):

Backup --> Unable to complete the operation. The following error was returned when opening the Exchange Database file: '-543 The required log files for recovery are missing. '

This one happened once with the 12/05 incremental backup:

Backup --> Unable to complete the operation. The following error was returned when opening the Exchange Database file: '-1003 An invalid API parameter is being used. '


 

Information:

We have 5 mail databases all on the same server and all of them have are reported by BE2012 as being a cause. So the MAILDB is short for a random or multiple databases.

We EXCLUSIVELY use Backup Exec 2012 for this server and no other mailservers are in our network.

Exchange 2010 isn't updated to SP1 or later, same for Server 2008 R2. The Backup Exec 2012 version is the latest, with SP4 and updated agents.

Backups run through Windows Server Backup succeed without fail onto a network drive. No errors with consistency checks,

 

Mail server Event Log:

When Backup Exec fails the backup, I can sometimes find these errors in Windows Event Log:

Storage Group Consistency Check - Event ID: 403
Instance 1: The physical consistency check successfully validated 18 out of 1052928 pages of database '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\MAILDB\MAILDB.edb'. Because some database pages were either not validated or failed validation, the consistency check has been considered unsuccessful.

Out of the 6 occurances of this error, 4 show "18 out of xxxxx pages" on any of the databases, two occurances showed much later pages. The first Event ID 403 error did occur at the moment I canceled the backup on May 1st but it also happened another time at 2 AM for example, when no one would have canceled it.

 

Errors that seem relevant:

ESE Event ID 2007: Information Store (3676) Shadow copy instance 7 aborted.

The "instance" has counted up to 168 before starting at 1 after rebooting.

 

MSExchangeIS Event ID 9782: Exchange VSS Writer (instance 295a99e2-7875-4bd1-b3c3-58dc18a23f67:7) has completed the backup of database 'MAILDB' with errors. The backup did not complete successfully, and no log files were truncated for this database.

These last two are usually in tandem.

 


 

Steps I've taken in Backup Exec:

- Reinstalled agent through Backup Exec (says correct latest version is installed)

- disabled/enabled AOFO

- disabled/enabled consistency check option

- disabled/enabled Verify as a seperate job

- restarted

 

Steps I've taken in Mail server:

- Disk Check with sector test

- checked "vssadmin list writers" for any unstable or failed writers (none were)

- latest non-SP1 updates are installed regularly (SP1 is scheduled in the near future if this is resolved)

- restarted

 

Since I'm not a skilled or experienced Exchange admin, I've not gone much further on the Exchange server.

1 Solution

Accepted Solutions
Accepted Solution!

I found the problem: Volume

I found the problem: Volume Shadow Copy was disabled for the storage on the Exchange server. Enabling it for the size of the largest database with some room, fixed the errors displaying and backups now run succesfully.

2 Replies

So it's a while later. The

So it's a while later. The Exchange server has been updated to the latest SP3 with the latest Rollup, with the latest patches. Everything went smoothly for a while, no errors at all. Until this month every backup with Backup Exec is throwing out errors again. It started after doing the first incremental.

I got the E00084F9 error e few times, along with the E000FF12 error which is new:

 

A communications failure has occurred with an Exchange Store resource. Final error category: Resource Errors

Backup- \\MAILSRV\Microsoft Information Store\one of the databases
V-79-57344-65298 - The Exchange Store service is not responding. Backup set canceled.

V-79-57344-65072 - An error occurred while connecting to the Backup Exec Remote Agent on the Backup Exec server. Ensure that the job options are correct, and then submit the job again. The backup set was not created.

 

Again, no other time EXCEPT when making backups with Backup Exec, does the server exhibit database inconsitencies. Not during normal operation, not during Windows Server Backup, not during the update from RTM to SP3.

 

Accepted Solution!

I found the problem: Volume

I found the problem: Volume Shadow Copy was disabled for the storage on the Exchange server. Enabling it for the size of the largest database with some room, fixed the errors displaying and backups now run succesfully.