Problem with VSS ASR Writer

[Borealis]

Hi all,

I'm using Backup Exec 2012 fully patched on Windows 2008 R2 standard  SP1

Very often I have a situation that many backup jobs failed for various reasons. The weird thing is that the very same jobs run perfectly for many days (with the same selections for the same remote servers) until I get to the point with massive failures.

After that, (because I know now what is the cause most of the times) I have to check the VSS writers status where I'm always getting the ASR writer in failed state or waiting for completion. This is a really frustrating situation because I can't be rely to the continuity of backup operations.

Finally, what is the reason that the ASR writer in getting into unstable status so often?

Is there any other workaround to bring it back to a stable status without a server restart?

Is it a Symantec or a Microsoft issue and is there any released to public patch that I can use?

Please help guys!

Honestly, it is not a solution to have to restart a server (Media, Exchange, etc)  because every now an then a writer fails.

Many thx in advance

[Earl_H]

I am interested in a response to this issue too.  Sorry I don't have an answer.

We are having this issue on one of our Windows 2008 R2 servers.  The problem started around the end of February 2013 and we continue to have problems.  We are seeing the error E00084F8.  The Backup Exec Remote Agent service stops when this error occurs, even though I am backing up the local server.  The error points to "network connection issues".  Through a lot of research, I have come to believe it is related to VSS, as that is the only portion of the backup that is failing.

I think the errors are related to this:

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {8493094c-e3e3-43a8-9e78-2771a9ab70ef}
   State: [7] Failed
   Last error: Timed out
 

I see this when I type "vssadmin list writers" at a command prompt.

I am wondering if it could be related to a Windows update.

I can clear the error by following this Tech Document, but the problem comes back.  Also, not all of the DLLs listed will register.

http://www.symantec.com/business/support/index?page=content&id=TECH64330&actp=search&viewlocale=en_U...

[Donald_Eady]

Writer failure is an issue that is often troubleshot with M.S. Symantec suggests rebooting the machine with the failing writers. This will often restore them to a state 1 stable state. However if the writers continuously fail they may need to be looked by M.S. to appropriately diagnose and remedy the failure. This may include reinstalling/registering the VSS DLL's. 

[pkh]

This may include reinstalling/registering the VSS DLL's. 

This should not be done with a Server 2008 machine.

[Donald_Eady]

That may very well be the case PKH.. thats why i suggested M.S. be involved as they can more appropriately diagnose and remedy. Please note that i did not suggest doing so rather that M.S. may end up suggesting this as a course of action. Thank you much for the input 

[pkh]

My comment is from a MS TN.

http://support.microsoft.com/kb/940184

in particular, take note of this

This article is not for use with Windows Vista, with Windows Server 2008, or with later Windows operating systems. Starting with Windows Vista and with Windows Server 2008, Windows component installation is manifest-based. Trying to manually register specific components, as described in the following steps, can have unexpected results that may require you to reinstall Windows to resolve.

[Mange_24S]

Have seen this problem before when an installed application on the server registered some paths/files that was related to the ASR in System State. App was uninstalled but not unregistered from the ASR-writer. When ASR-writer was told that a snapshot will be taken, it couldnt find the paths and failed.

Could see what paths/files was involved in this issue when debugging the remote agent on the server.

[Borealis]

I'll try what Mange_24S suggest.

If I find something I will post back. It looks like that this is a dead end issue. Now, almost every day I have a failed or unstable ASR writer in the Media Server and when this happens the majority of the jobs are failed. When I make a restart of the Media Server most of the jobs complete succesfully. Looks like something (probably a job) brings the ASR writer into the failed or unstable state. But there so many jobs that I have to hold and test one by one to verify if this is the case or not.

[Mange_24S]

Try to isolate the problem by doing a one-time backup of just the system state of that server. Then you can start the sgmon during that time. Just to make sure you dont have other unrelated things in the logs.

System state includes a lot of different services depending of what roles/features has been added to windows. Could be either a DLL that needs to be re-registered or a previous feature/role/app that has been registered with system state dependencies but is no longer available.

[Borealis]

Thank you Mange_24S for your guidance

I will try to isolate the problem with the One-Time Backup option but:

About sgmon. Our Media Server is a Backup Exec 2012 SP1a fully patched. Does it supported on this version because according to the following technote

http://www.symantec.com/business/support/index?page=content&id=TECH4986

BE 2012 it is not listed inside the related products.Do you know if  there is any other Symantec based monitoring solution? Even If I use it there're a lot of option ans setting inside there. Can someone suggest which option(s) do I have to select to collect more specific infos?

About the DLL case. The Media Server has been installed on a Windows 2008 R2 Server and as pkh stated in a previous comment it is not recommended according to the following MS link

http://support.microsoft.com/kb/940184

[Mange_24S]

Just run the SGMON, works fine for 2012. Seems like that is an old article from 2009 that havent been updated with the new versions. 

You mention the mediaserver, what server is backed up when you get the ASR writer error? That server is the one with the problem. If 2008, the re-registering is not recommended - for 2003 I think many have tried and succeeded.

[Mange_24S]

Forgot to mention, please try and just open sgmon and turn on some debugging during a backup and you will se a live feed of whats happening. Then you can turn on log to file and start your isolated backup of system state on the failing server.

[Mange_24S]

Better KB for SGMON on 2012:

http://www.symantec.com/business/support/index?page=content&id=HOWTO11932

[Borealis]

Thx Mange_24S

I will try the SGMON according to KB. I don't really know which server job is causing the ASR witer problem to the media server. I guess I have to use SGMON with every Server System State backup involved in every job!

Thx again for your help

[Pablo010885]

Hey,

do you have an update for us?

thank you

[Colin_Weaver]

I have just made an amendment in the Environment section of TECH64330 to warn abouth not re-registering DLLs on Windows 2008 and later and added the link to the Microsoft KB.

This amendment should be public within 24hours.

[Borealis · Accepted solution]

Hello Pablo010885 and the rest

Find below a Solution to the problem:

Just follow the steps described in:

http://www.symantec.com/business/support/index?page=content&id=TECH159538

and

http://www.symantec.com/business/support/index?page=content&id=TECH190079

Cheers

PS. Please, post back here if you have any positive development applying the above solution

[Borealis]

Since the day I follow the solution described in TECH159538 and 190079, I never had again a Failed ASR writer status. For this reason and since no one else placed a negative feedback of this solution I will mark it as solution.

Thank you all