Backup Exec 12 problems backing up Exchange 2007 SP1 with SCR

[Luke_Cassar]

Hi,
I have just installed backup exec 12 on a server, ran live update until nothing was left to isntall, installed the agent on my exchange server which is 2007 SP1 with SCR setup. I then tried to run the exchange enabled backup with GRT + 'backup passive then try active' in the LCR/CCR option and while it does report a success, the job log has an error as follows:

Backup- \\mailserver\Microsoft Information Store\DEVELOPMENT
Writer Name: Microsoft Exchange Writer, Writer ID: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Failed during backup complete operation (12).

I have also noticed that while we have 5 items under 'Microsoft Information Store' on my latest attempt to backup, only 3 of them recieved the above (and below) errors.

If I look at the Exchange logs from when this happened, I see the following errors:

ID: 2038
Source: MSExchangeRepl
Category: Exchange VSS Writer
The Microsoft Exchange VSS Writer (instance f0b6a572-34a0-43ca-9dcb-c91cd8d1ea9b) backup failed for storage group '2a6762fe-b6ca-4d7a-85fb-1ef22de1e601'. No log files were truncated.

ID: 2034
Source: MSExchangeRepl
Category: Exchange VSS Writer
The Microsoft Exchange Replication Service VSS writer (instance f0b6a572-34a0-43ca-9dcb-c91cd8d1ea9b) failed with error code FFFFFFFC when processing the backup completion event.

ID: 914
Source: ESE Backup
Category: General
Information Store (2348) The surrogate backup by MAILSERVER has stopped with error 0xFFFFFFFF.

ID: 215
Source: ESE
Category: Logging/Recovery
MSExchangeIS (2348) DEVELOPMENT: The backup has been stopped because it was halted by the client or the connection with the client failed.

Does anyone know what might be causing this?

Thanks very much!

[Dev_T]

Hi,

Please follow the below Symantec Article.

http://support.veritas.com/docs/297185

If the Exchange server is hosted on Windows Server 2003, I would request you to install the VSS rollup patch on all the Nodes of the Cluster

http://support.microsoft.com/kb/940349 (Reboot is required)

Dev T
MCSA,MCTS,MCT
VERITAS Certified Professional


 

[Luke_Cassar]

Thanks,
I am going to give the rollup a try even though it is not specifically related to the issue we are having, but then there is not much on the Internet which is related to our particular problem.

Will schedule this install and let you know how it goes.

Thanks for your advice!

[Abhijit_Soman]

Hi Luke

Select only CCR option while exchange backup with GRT enabled

[Luke_Cassar]

Dev T,
Thanks for that.. the rollup seems to have done the trick for VSS as I was able to run a successful backup of the whole information store this morning after applying and rebooting the server.

Cheers,
Luke

[Luke_Cassar]

The problem returned.. I was able to run an initial backup of the system after applying the patch, but after that, I scheduled the evening backup and it failed with:

Writer Name: Microsoft Exchange Writer, Writer ID: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Failed during backup complete operation (12).

The event viewer of the Exchange server (source) is showing the same lot of errors as before as well.

Once again I am at my wits end as to why this is happening.

Any further ideas that might help troubleshoot and resolve this issue?

Thanks very much for all your input!

[RahulG]

Split yur backup Jobs into two as it is not recommended to backup system state along with exchange .

[RahulG]

reboot is the temprory fix as it brings all the vss writer to the stable state .

[Dev_T]

The VSS Writers on the Server are not stable. Run the following command in the command prompt and make sure all the Writers are stable  VSSADMIN LIST WRITERS.

This is the issue with with the Writer and not with Backup Exec, you can try rebooting the server and performing backups again or you can try re-registering the writers http://support.microsoft.com/kb/940184

Let me know the results...

[Luke_Cassar]

I tried installing the BETest utility from Microsoft to run some diagnostic backups. After rebooting the server all of the Writers were OK, then I ran this utility without switches to get the writerID's from the file it produces.

However while this program was running, it crashed the exchange writers with the following sort of information (this is from BETest).

status After Prepare Backup (11 writers)
Status for writer Microsoft Exchange Writer: STABLE(0x00000000)
Status for writer Microsoft Exchange Writer:
FAILED_AT_PREPARE_BACKUP(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)

status After Do Snapshot (11 writers)
Status for writer Microsoft Exchange Writer:
WAIT_FOR_BACKUP_COMPLETE(0x00000000)
Status for writer Microsoft Exchange Writer:
FAILED_AT_PREPARE_BACKUP(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)

status After Backup Complete (11 writers)
Status for writer Microsoft Exchange Writer:
FAILED_AT_BACKUP_COMPLETE(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)
Status for writer Microsoft Exchange Writer:
FAILED_AT_PREPARE_BACKUP(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)

status After PreRestore (11 writers)
Status for writer Microsoft Exchange Writer:
FAILED_AT_PRE_RESTORE(0x800423f4 - VSS_E_WRITERERROR_NONRETRYABLE)
Status for writer Microsoft Exchange Writer:
FAILED_AT_PRE_RESTORE(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)

status After PostRestore (11 writers)
Status for writer Microsoft Exchange Writer:
FAILED_AT_POST_RESTORE(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)
Status for writer Microsoft Exchange Writer:
FAILED_AT_POST_RESTORE(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)

Does this help anyone figure out what the problem might be?

Thanks very much,
Luke

[Luke_Cassar]

Anyway, Have started a case with MS because Symantec say it is not an issue with their software which sounds reasonable.

MS have suggested as a first step that we install KB903234 and then Security Rollup 8 for Exchange 2007.

I will give these a go and let you know the outcome (hopefully it wont be 'me looking for a new job')

[Luke_Cassar]

The first patch I noticed was old, so I never installed this.

I did install Rollup 8 even though the documentation mentions nothing about anything to do with fixing any VSS issues.

Rebooted the server to put the writers back to stable, installed rollup 8, rebooted again and ran BETest. This caused the writers to fail immediatley.

The next instructions from MS were to recreate the Volume Shadow Copy service again which I did and again, no difference.

The next suggestion was to stop a couple of services and re-register a LOT of DLL files. At this point I had to draw the line because there is a LOT of commands and no real explaination from the MS Tech about what I am doing or why.. it seemed like a cut and paste series of generic 'try this and hopefully it will fix it' steps.

HOWEVER.. I have seen a few people report that they can backup sucessfully if they chose the active copy rather then the passive copy of a database (I see that we are also running LCR).

A final reboot of the server to put the writers back to stable, changed all backup jobs to backup the active copy only..... and now its worked for 3 runs of GRT based store backups. I was able to restore a few individual mail items as well, so this is fantastic!

I am going to leave it at this now because I dont have a lot of confidence in the MS troubleshooting steps anymore. The main thing is we seem to be getting complete backups and that GRT is working, so we'll see how this goes.

Hopefully someone else may find this information useful.

[Luke_Cassar · Accepted solution]

Just heard back from MS saying this is a Symantec issue, which I dont believe. MS are stating that the backups from the passive copy are not supported, but the documentation for Exchange clearly states that it is supported.

Anyway, found this article from Symantec http://seer.entsupport.symantec.com/docs/305815.htm which at least proves that something is happening to try and resolve an issue.

Suffcie to say I am not paying for that support case.

Thought I would update incase someone else out there is desperate for an answer as I was when I started this thread. The solution is to backup active copy only for now. If you ever select passive copy, the writers will crash and the server will need a reboot (or restart the IS).

Cheers!

[kkate]

I thinks this was the fast reply and solution :)


Abhijit Soman 3 weeks 1 day ago
Hi Luke Select only CCR
Hi Luke

Select only CCR option while exchange backup with GRT enabled

[Jyrki]

Randomly backup goes well, about 50/50 (after fail passive node restart).

Environment:
Windows 2008 SP2 64 bit
Exchange 2007 SP1 Update Rollup 9, CCR Cluster.

Backup Exec 12 SP3 (HF 324963 and 323893), backup to disk.

Have exactly same error messages than Luke, plus also this at job history log:
Resource name : \\Mailserver\Microsoft Information Store\First Storage Group
Error : e0008445 - The media operation was terminated by the user

Maybe I must wait fix from MS and backup active copy that time? Seen also that backup to tape works better than to disk.
Anybody knows timetable for fix mentioned this article: http://seer.entsupport.symantec.com/docs/305815.htm?

Rgds, Jyrki

[Jyrki]

Hi,

I am not sure if problem start after this hotfix: http://seer.entsupport.symantec.com/docs/324963.htm?
Track down job history log and seen that problems start next day after installation.
I try to uninstall it and test more...

Rgds, Jyrki

[Luke_Cassar]

Hi Jyrki,
There is no fix from Microsoft. Rollup 9 for Exchange 2007 does not include any VSS patches that I could see from the MS website.. I still installed it, but I dont think it helped.

The real solution for me at the moment was to reboot the server and get the Exchange writers in a stable state, then make sure you NEVER try to backup the passive copy in backup exec again. If you do, you will crash the writers and need to reboot the Exchange server to put them back into a stable state.

I know that after a reboot, we can take ONE backup of the passive copy only, but after that all attempts to backup active or passive will fail until the server is rebooted again.

Is that the same experience for you as well?

Cheers!

[Jyrki]

Hi Luke,

yesterday i uninstall that Symantec hofix 324963. Do you have it also installed?

Last night backup from passive copy goes well, but as you mentioned maybe only one backup success after restart. Let’s look forward what happened next and next night.

Btw, backup from passive copy works well before June, something happen after that. Problems was before Windows 2008 SP2 also.

I know also Symantec have lot of big bugs their product these days, for example these already fixed problems: BE DLO with Office 2007 SP2, SEP firewall bug at MR4 MP1 and W2008 x64 file share/server hang.

Rgds, Jyrki

[Luke_Cassar]

Well we are on Windows 2003 Server SP2. I started this job in June and prior to that, the company I am working for were just using NTBackup for Exchange, so I cant say if the issue existed before or not.

I did not install that Symantec hotfix, I basically ran live update and installed the patches that were available, then put the software into production and immediatley hit these VSS issues.

Let me know how you go with your backup from passive copies next time they run though!

Cheers.

[Jyrki]

Hi Luke,

yes, passive copy backup goes well two times after hotfix uninstall. No server restart between backup.
Can you check if live update install patch 324963, you can find that info from BE console: Help, About Symantec Backup Exec... and Installed updates.

Rgds, Jyrki