VMWare Host backups hanging on tracking.log

[O_Jay]

Hi,

I am running Backup Exec 2014 V-Ray Edition. When I backup several of the guests on my ESXi host the backup hangs for ages backing up \System Volume Information\tracking.log. The byte count doesn't increase and the job rate doesn't change.

I would be happy to just deselect the whole folder but you don't have that option at the host level.

Is there any way I can prevent this from happening?

[lmosla]

Hi O_Jay,

Are you getting any errors in Windows Events on the Media server or/and Remote servers?  Check the VSS writers to see if they are stable.

If this is a large job try breaking a smaller job off of it and seeing if that helps.

[O_Jay]

Hi Imosla,

There are no errors in the event logs. How do I "check the VSS writers to see if they are stable"?

How will breaking the job up help? Its not that big to begin with.

EDIT: There is an informational message on the media server and the target event logs which says:

"The VSS service is shutting down due to idle timeout."

[pkh]

To check on your VSS writers, start a command prompt and issue

vssadmin list writers

If there are any errors, then you need to reboot the server to clear the error.

[O_Jay]

Most said [1] Stable. A few said [5] Waiting for completion

Rebooted and now they all say [1] Stable

[pkh]

Now you can try your job again.

[O_Jay]

No change it is still hanging on the tracking.log

[VJware]

Are the VSS writers for the affected VM still in a non-stable state ?

Filter the events on this VM for any errors/warnings during the time of the backup & you may find some relevant events.

[O_Jay]

They have stayed in stable since the reboot. The job has been hanging now for over an hour.

In between heaps of "The VSS service is shutting down due to idle timeout" I can see the following:

Event ID: 0

Source: BeVssProvider

Description:

The description for Event ID 0 from source BeVssProvider cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Service stopped

[pkh]

Push out the remote agent again to the VM's

[VJware]

And for the time being, you could deselect the affected VM from the backup selections, thereby allowing the backup job to complete for the remaining VMs and then focus troubleshooting the issue with this VM.
 

[O_Jay]

Will do. I will report back again tomorrow after I have done some more testing. Thanks for the help so far.

[O_Jay]

No difference, same hanging on that VM I redeployed RAWS to with System Volume Information\tracking.log. The event logs are only showing the messages I already mentioned. There are at least four different VMs on this host that are having the issue. Sometimes some will work and then one will hang.

I have checked that the VSS writers are all in a stable state.

I tried creating a global exception for the System Volume Information folder but I don't think it worked.

I might try removing the entire host from BE and re-add.

[DKK]

Hello O_Jay,

i have the same problem with Backup Exec 2014 V-Ray Edition than you. The Backup job is hanging on serverel virtual servers from vmware always in the Folder System Volume Information. Together with the symantec support i have tried the following:

1) Uninstalled BE Remote Agent

2) Redeployed BE Remote Agent

3) Check all VSS writers -> all ok

4) Installed all Windows Updates

5) Rebooted Server

6) Check event logs from VM's and BE Server

The problem is not resolved, at the moment i tried to defrag the virtual machine. (http://www.symantec.com/connect/forums/backup-hanging-remoteserverd-system-volume-information)

[O_Jay]

Hi DKK,

Glad I am not alone :)

I am trying my weekend backup again this weekend  after completely removing and re-adding the host to BE but I don't hold out much hope for this.

I will open a case next week with support and hopefully between the two of us we can find a solution.

[O_Jay · Accepted solution]

OK... since completely removing and re-adding the virtual host to BE I have been able to get successful backup all week.

Maybe worth a try for you too DKK.

I am however getting two of the guests not able to do GRT enabled backups even though the credentials verify successfully and I am able to browse both servers in the selection of the indivual servers on the Backup and Restore servers view.

[DKK · Accepted solution]

Hi O_Jay,

my backup job runs now successfully without hanging, i have take many steps to reach this result:

1) run: dism.exe /online /cleanup-image /spsuperseded

2) uninstall BE Agent | reinstall Agent (push) on all servers where the job has hung up

3) defrag of all partitions on all servers

4) check all event logs to critical errors

5) Boot from Windows Server Disk and run CHKDSK with options /F and /R for all disks  -> some errors have been fixed

6) changed order of the virtual server in the backup job

7) Install all available Windows Updates for the VM‘s

8) Set the backup exec user rights to Folder "System Volume Information" on all servers where the job has hung up

9) In the backup job settings disable GRT for SQL and Sharepoint

Unfortunately I do not know exactly what has solved the problem.

[O_Jay · Accepted solution]

One of the recent ones was actually GRT on SQl database I didn't need so I turned GRT off and the other I tinkered with the credentials until it was happy. I got my first successful (i.e. not hung or "Completed with Exceptions") backup last night. So for now this is resolved.

Like DKK though I don't really know exactly how I fixed it but removing and re-adding the hostto the Backup Exec server seemed to be the clincher in my case.