Exchange Backups keep failing - VSS Writers

[Remko_de_Koning]

Dear Experts,

I am having a backup problem on my Exchange 2007 SP2 box that is givin me quite some headaches. It has been troubling me for some time now and I decided it is about time now to call in the Experts

Configuration:

• Exchange 2007 SP2 Standard Edition
• Windows Server 2003 R2 x64
• 4 storage groups
• 3 storage groups with 1 database
• 1 storage group with 2 databases (this will be also one in the future)
• for 3 storage groups LCR enabled. The 4 storage group I cannot enable LCR yet as it has 2 databases
• D-volume with active databases (1 TB)
• E-volume for logfiles storage
• F-Volume for LCR storage and LCR logfiles. (1.4 TB)
• Backup Exec 12.0 - Old version...Will be upgraded soon to Backup Exec 2010 R3. Waiting for financial approval to replace the current server

I have my Exchange backup split into two Policies. Each Policy based on GFS principle. Backups go to disk. Each month backup also gets duplicated to tape.

The first policy makes a backup of the entire server including system state except the databases. The other policy backups only the storage groups. Weekly a full. During the week incrementals.

I have the checkmark on the "Advanced Open File" tab called "Process logical volumes for backup one at a time" unchecked and forced the Microsoft Software Shadow Copy provider as the standard snapshot provider. ( I want to make sure the database and logfiles are in sync at the time of backup)

For Exchange I have the GRT option enabled and I am using the Active Database for backup. Using the LCR database for backup only gives more problems.

Problem description:

Full backups run without a problem after a fresh restart of the Exchange Server. Subsequent backups tend to fail. Sometimes it works for a few days but most of the times my backups fail the second time.

Error:

Backup Exec Log file

V-79-57344-34110 - AOFO: Initialization failure on: "\\SERVER\Microsoft Information Store\Archive St...
Snapshot provider error (0xE000FED1): A failure occurred querying the Writer status.
 

Event viewer Exchange

not much information other than the preparation for backup was successfull

Exchange VSS Writer (instance 2eb46463-b16b-4018-8d0c-fc2c3bca37e4) has prepared for backup successfully.

The Microsoft Exchange VSS writer has successfully collected the metadata document in preparation for backup. 

VSS errors:

X:\>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {a5d350c8-a001-4f5f-a358-4ca45f9a54b1}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {e27fb699-8d0e-4fb6-86de-3dbdb93a2934}
   State: [1] Stable
   Last error: Retryable error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {4ac5263c-62a5-41e6-b485-e87c95215e19}
   State: [1] Stable
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {cb6c7c2b-afbb-4933-a048-457d5cc547e4}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {add8df76-95aa-4a07-a6c0-d94e3ebeba04}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {7bd7f0f0-6707-4d75-ac92-79d8826dc454}
   State: [7] Failed
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {69fa3053-3ffb-4bd0-be5c-2188cfa4b1a7}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {880a9b3d-fec0-4b1f-b091-ed2fdcb25b5b}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {f33af3fb-54f4-4998-bfbe-267b8cb89b05}
   State: [1] Stable
   Last error: No error

I have read many posts on this forum regarding this issue. Many (older) post refer to a VSS hotfix: Update for Windows Server 2003 (KB940349)
This one is already applied.
I guess I have tried any other combination as well to create a successfull backup

Question:

Will an upgrade to Backup Exec 2010 R3 fix my problem? What is the best approach in my situation to make a successfull error free backup of Exchange? What am I not seeing?

Thanks for your time reading this post and much appreciated if you could give me any feedback on how to troubleshoot this problem.

Remko

Update1:
I did found some more entries earlier in the Windows eventviewer. Not sure if it is related. I do not understand why there is an entry about streaming backups. I thought only NTBackup uses streaming backups and not even remote Streaming Backups.

Information Store (2184) A remote (TCP/IP) based streaming backup was attempted. This functionality has been disabled by default settings.

There server mentioned in this event is my Backup Exec Media server though.

[ArulPrasad · Accepted solution]

Hi ,

That Was Reall Good Explanation about the Environment.

First Thing Solution for the problem:

1. Recommended Procedure To uncheck AOFO From You Database backup (Exchange , Sharepoint, SQL, EV)

2. It is Good to have the latest and Greatest version.

3. Streaming Backup is Officialy supported till Exchange 2003. From exchange 2007 Streaming backup is replaced by snapshot backup (wich is Why the AOFo needs to be u8nchecked as the exchange writer is Automated and called for backup and Force using the VSS is not a good idea)

Please let Know how it goes.

Regards

Arul

[I_T_Guy]

This looks more like a VSS Writer error as visible in the output you have pasted. This also explains why your backup runs fine after a reboot as a reboot will bring all the VSS writers back in stable state.

Are there any other shadow copy providers on the system ? can you provide the output od vssadmin list providers ?

also if possible try enabling the SGMON debugging and see if it catches any obvious errors

http://www.symantec.com/business/support/index?page=content&id=TECH52476

Last but not the least, its always good to have the most recent version.

[pkh]

Try re-registering your VSS components.

http://www.symantec.com/docs/TECH70486

[Remko_de_Koning]

Thanks all for your replies and suggestions. I will give these a try and post my findings once I have tested this.

Your help is much appreciated!

Remko

[Remko_de_Koning]

Backups are running again. Rebooted the Exchange server to reset all VSS writers. Disabled the AOFO on my backup jobs.
Will have to wait now to see if any subsequent backups fail as before.

I found this article that explains VSS writers and Advanced open file option in more detail

https://www-secure.symantec.com/connect/articles/advanced-open-file-option

What is still puzzling me is why my backups are failing when the AOFO is enabled.

Anyway, it will take some time before I can confirm disabling the AOFO has fixed the problem. I hope I learn more after the weekend.

Enjoy your weekend.. Thanks so far.

Remko

[CraigV]

...the thing is that you shouldn't be using AOFO with DB backups, and it's mentioned in the Admin Guide (although not many people know this). It does cause issues with backups, such as logs in Exchange failing to clear, SQL redirected restores not being possible etc. It has to do with the way that AOFO snapshots the DB vs. the way VSS is used as far as I remember.

If you're backing up DBs and data in 1 job, I'd suggest splitting them off...

[Remko_de_Koning]

Thanks CraigV,

This explains the failures I have been having I guess. I did attend the Backup Exec 2010 R3 course but I can't remember this was mentioned. But anyway.. Good news.. So Far So Good. I am having successfull backups of my databases today and so far my third backup is going steady. Will see how this goes over the weekend.

I am happy

Also.. I did split my Exchange backup jobs into various backup jobs. Each database has its own job and there is one job for the entire server except the databases.

This one has the AOFO enabled. I guess I cannot do without this as this is a file level backup. I hope this will not break my VSS writers again :(

Remko