Hangs at 5% - vss down

[PCIP]

I'm running a trial version of BESR 8.02 and only one of my servers is having an issue. It stops at 5%. No matter if the destination is local on that server or onto the remote backup server. The rest of the servers are fine.

It's a fileserver/SQL server and after it is stuck the vss service stops working and BESR will hang for days. (Have tried) I have to restart the server to free it up again.

If we run it in "files and folders"  mode it can back up the content. However we would like to have an image for faster restore.

Any insight?

[Yury_Urbanovich]

We have same issue with BESR 8.02. After manually restart all BESR Services (Backup Exec System Recovery, Symantec SymSnap VSS Provider, SymSnapService) and manually start missed job it seems that Recovery point is created. But in some cases above procedure does not help, so we need to restart server.

[Andreas_Horlach]

Is there an error generated anywhere during the freeze, either in the event log or anything on the screen?

[PCIP]

When we first installed BESR 8.02 we saw the 8193 error:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 8193
Date:  6/26/2008
Time:  2:35:53 PM
User:  N/A
Computer: SERVER
Description:
Volume Shadow Copy Service error: Unexpected error calling routine IMultiInterfaceEventControl::GetSubscriptions.  hr = 0x80040206.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

and the 12291:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12291
Date:  6/26/2008
Time:  2:35:53 PM
User:  N/A
Computer: SERVER
Description:
Volume Shadow Copy Service error: Error on creating/using the COM+ Writers publisher interface: IVssWriter::RequestWriterInfo, GatherWriterMetadata [0x8000ffff].

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

Then we uninstalled the remote agent for BE10d that we previously used to backup our servers and when we tried again yesterday to make it work we first saw the 12303:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12302
Date:  7/1/2008
Time:  11:34:41 AM
User:  N/A
Computer: SERVER
Description:
Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers.  Please check to see that the Event Service and Volume Shadow Copy Service are operating properly.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:


that went away and we alternately saw the 12292 and the 20:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12292
Date:  7/1/2008
Time:  2:32:08 PM
User:  N/A
Computer: SERVER
Description:
Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {5fc845eb-f299-40f0-9725-34045f7ea8ba} [0x80040154].

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 20
Date:  7/1/2008
Time:  2:33:24 PM
User:  N/A
Computer: SERVER
Description:
Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered.  This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider.  The error returned from CoCreateInstance on class with CLSID {5fc845eb-f299-40f0-9725-34045f7ea8ba} and Name HWPRV is [0x80040154].

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

and one 12302:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12293
Date:  7/1/2008
Time:  3:01:08 PM
User:  N/A
Computer: SERVER
Description:
Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {262b716e-bb23-41b5-aaef-e2c15e767167}. Routine details Error calling Query(). [0x80004004] [hr = 0x80004004].

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

Thanks for looking into this. It's really driving us to the edge.

Pcip

Message Edited by PCIP on 07-02-2008 11:32 AM

[PCIP]

-one of the time we reinstalled the BESR 8.02 remote agent we got the following error:

Event Type: Information
Event Source: MsiInstaller
Event Category: None
Event ID: 1025
Date:  7/1/2008
Time:  1:03:41 PM
User:  DOMAIN\administrator
Computer: SERVER
Description:
Product: Symantec Backup Exec System Recovery. The file C:\WINDOWS\winsxs\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.42_x-ww_0de06acd\msvcr80.dll is being held in use by the following process Name: sqlservr , Id 2248.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

the same dll were also being held by several other processes. However that cleared after a restart.
One time we saw this error, but that seem to have cleared again:

Event Type: Information
Event Source: SymSnapService
Event Category: None
Event ID: 0
Date:  7/1/2008
Time:  3:43:01 PM
User:  N/A
Computer: SERVER
Description:
The description for Event ID ( 0 ) in Source ( SymSnapService ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Service started.

Pcip

[diliddo]

I've got a similar issue here:
We have 2 identical Windows 2008 (64 bit) servers setup exactly the same time the same way.

One runs BESR 8.02 fine with no delays in creating Recovery Points or completing Backup Jobs.

The other bottlenecks at 5% for 10-15 minutes (after that it eventually creates the Recovery Point and runs the backup).

 

Here are the errors in the Application Event Log:

 

------------------------------------------

Log Name:      Application

Source:        VSS

Date:          8/6/2008 3:10:56 PM

Event ID:      12290

Task Category: None

Level:         Warning

Keywords:      Classic

User:          N/A

Computer:      test.irm.ad.fau.edu

Description:

Volume Shadow Copy Service warning: ASR writer Error 0x80070001.  hr = 0x00000000.

 

Operation:

   PrepareForBackup event

   PrepareForBackup event

 

Context:

   Execution Context: ASR Writer

   Execution Context: Writer

   Writer Class Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

   Writer Name: ASR Writer

   Writer Instance ID: {dcfc0879-8bdf-473e-823c-f4e0cf6fad13}

  

--------------------------------------

Log Name:      Application

Source:        VSS

Date:          8/6/2008 3:14:57 PM

Event ID:      24

Task Category: None

Level:         Warning

Keywords:      Classic

User:          N/A

Computer:      test.irm.ad.fau.edu

Description:

Volume Shadow Copy Service Warning: The Volume Shadow Copy Service is shutting down  and is experiencing delay while waiting for in-progress calls to complete.

--------------------------------------

[PCIP]

I actually ended up contacting Microsoft on the VSS writers as even the NT backup util stopped working even after uninstalling everything Symantec.

The one server had both a SQL 2000 and 2005 express and someone had created several aliases for the sql servers some time back when we were facing connectivity problems.

After removing the aliases and applying all patches the servers are backing up through BESR 8.02 almost smooth. "Just" a dropped connection and a missed backup now and then ? :|

So applying all update patches and making sure the VSS writers work smooth on their own seem to be the solution in my case.

Thanks for chiming in.

PCIP

[diliddo]

Glad to hear that worked out for you.

My issue still persists unfortunately.

Since posting, I wiped the problematic server and layed down a clean install of Windows Server 2008.

The server is still hanging at 5%, just before creating the Recovery Point.

I also ran extensive diganostics on all of the hardware and came up clean.

For comparison, I ran the same diagnostics on my "good" server. 

Both reports were identical, verbatim.

I'm really at loss here.

[Andreas_Horlach]

diliddo -  Changes have been made to the VSS Provider in the next service pack (8.03) scheduled to be released in the next couple of days. I would like you to try this when it releases. Until then, ensure that your server has the latest SP from Microsoft for all applications.

[diliddo]

Sounds good Andreas.  Please let me know when it is available.

Joe

[Andreas_Horlach]

BESR LiveUpdate should grab the 8.03 update by Monday (8/18) or later. I don't have all of the details yet, but I do know this fix will be there.

[diliddo]

Ok.  I'll check for it then and let you know how it goes.

[skirvmi]

Any update on the new version?  I tried live update and the file connect site and both are v8.02.

[Andreas_Horlach]

We're still waiting for the LiveUpdate team to verify and upload the build. I thought it would happen on Monday, but we are looking at anytime now this week. 

[skirvmi]

Any update from the Live Update team?  Sorry to be a pest, but we need this to backup our SQL servers.

[diliddo]

Bad news folks... I ran LiveUpdate, and updated to v8.0.3.28325.

Now instead of bottlenecking at 5% and eventually creating a Recovery Point, it just hangs indefnitely without ever passing the 5% mark.

The update seems to have made matters worse...

[599116367]

where did you get the update to 8.0.3?  I run liveupdate and stay at 8.0.2.26324.

[diliddo]

I got the 8.0.3 update by running LiveUpdate.  Not sure why you aren't getting it.

On a side note, I restored my system from a Recovery Point created before installing the update.
I'm back to version 8.0.2.26324 now.
My backups are still hanging at 5% for about 10 minutes, but they eventually pass the bottleneck and finish creating Recovery Points again.

[Andreas_Horlach]

I'd like a few more people to try this and report back their results with 8.03. If you can't get it through LiveUpdate, you can download it from our FTP site. Email me for the credentials.

Joe - what is the chance of getting an image file from you for internal testing? I can provide a secure FTP for upload. Otherwise, after the 8.03 upgrade, and the system hangs at 5%, I would like to see the log files from BESR.