Backup Exec 2012 error- Remote Agent crash during VMware backup

[steven_man]

When I backup VM using Backup Exec, the following errors was found in Event Viewer. The Backup Exec Remote Agent service was stopped.

Event ID 1000

Faulting application name: beremote.exe, version: 14.0.1798.0, time stamp: 0x4f1e02d7

Faulting module name: vddkWrapper.dll, version: 14.0.1798.111, time stamp: 0x50be7e05

Exception code: 0xc0000005

Fault offset: 0x0000000000019f7a

Faulting process id: 0x1704

Faulting application start time: 0x01cddd446eb13e76

Faulting application path: C:\Program Files\Symantec\Backup Exec\beremote.exe

Faulting module path: C:\Program Files\Symantec\Backup Exec\vddkWrapper.dll

Report Id: fac63bb8-4942-11e2-8cdb-001b7892bb88

I found one KB about Backup Exec 2010 Remote Agent crashes during VMware backups.

http://www.symantec.com/business/support//index?page=content&pmv=print&impressions=&viewlocale=&id=TECH200972

Can I follow the above KB to solve Backup Exec 2012?

[Backup_Exec1]

Hi

Yes you can try the keys mention in the article only you may expereience a bit performance issue

Thanks

[Nitin_P]

Hi steven_man

This issue is resolved in the latest hotfix no : 199866.

Please run a live update on BE media server and update respective remote agents as well. Please reboot if notified [ generally it shouldnt request for a reboot ] .

Ref : http://www.symantec.com/docs/TECH199866

After this hotfix installation and the updates , please disable following keys if available. 

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

Please change the Value to 1 .

Note : Do not create keys if not available , disable only those which are available.

[steven_man]

Nitin_P,

I read the URL you mention. But I cannot find the change about registry.

http://www.symantec.com/docs/TECH199866

Can I provide the document about this?

Copy to Backup_Exec,

After I install hotfix 199866, do I need to change the registry accordin to the following URL.

Set the following registry keys to a value of "0"

1. HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Reads
2. HKEY_LOCAL_MACHIEN\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Writes

http://www.symantec.com/business/support//index?page=content&pmv=print&impressions=&viewlocale=&id=TECH200972

[Nitin_P]

steven_man , 

I have referred the document you provided.

This change in the registry was involved only as a work around. But Now , since the hotfix is released , you can enable the keys back again. However , the keys mentioned below might conflict. Therefore , please disable the keys mentioned below : 

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

Please change the Value to 1 .

Note : Do not create keys if not available , disable only those which are available.

[Nabil_polska]

1. Stop the Backup Exec services on the Backup Exec sever
2. Set the following registry keys to a value of "0"
   1. HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Reads
   2. HKEY_LOCAL_MACHIEN\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Writes
3. Start the Backup Exec Services

[Nitin_P]

Nabil polska , 

Since Steven already have installed the hotfix. " Buffer read / writes "  are not required to be disabled , they can be enabled back again using the value 1. 

Note : Disabling buffer read / writes reduces performance of VMWARE backups. Therefore we need to enable these keys back again .

However , due to this Hotfix installation , following keys needs to be disabled - 

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

Please change the Value to 1 .

Note : Do not create keys if not available , disable only those which are available.

Steven , please provide us the update after disabling these keys.

[steven_man]

In our environment, I have 3 Backup Exec 2012 Server (Windows 2008 R2 with SP1).  And I ran Symantec LiveUpdate to install all hotfixs for Backup Exec 2012.

One Backup Exec 2012 (Server 1) encounter the issue of Remote Agent service stopped when backup up VMware VM. Other two Backup Exec servers (Server 2 and Server 3) are normal.

After change the following registry, the service still failed when backup VM for a while (about 2 hrs).

Key_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

Please change the Value to 1 .

When I run Window Update, I found that Server 1 have not installed any Window hotfixs. After I install all hotfix from Windows Update, the Remote Agent service seem become normal. 

Do anyone know any Symantec document mention about Microsoft hotfix is neccessr for Backup Exec 2012?

Also, after I modifed the two registry above, the backup time and backup size is larger than before in Server 2 and Server 3. Is it normal?

And I found that the backup size of incremental backup in backup job often larger than the backup size of Full backup in backup job. So strange! The incremental backup size should be <=  The Full backup size.

Can anyone know explain what the backup size means in backup log?

[Nitin_P]

Steven_man, 

While taking backup of VMs , if VMWARE API's have any issues to collect change block tracking information , it will backup the entire VMDK and not just the used space.  This may lead to find increase in the time and the size of backup

To get this corrected, we need to reset Change Block Tracking [ CBT ] for VMs.

Please refer : http://www.symantec.com/docs/TECH197311

[steven_man]

Nitin_P,

Key_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

Please change the Value to 1 .

Do I still need to reset Change Block Tracking [ CBT ] for VMs if I disable to use CBT in registry above?

Please refer : http://www.symantec.com/docs/TECH197311

[Nitin_P]

Hi Steven_Man ,

Yes , we need do do this. The issue that we have about backup size increase , it should be taken care due to reset operation for CBT.

Please update me about the progress after resetting CBT for VM.

[Colin_Weaver]

Please BE aware

"NTFS Used Sector tracking" and VMWare "Change Block Tracking" (CBT) are two completely different concepts/processes. Disabling the used sector tracking reverts the system to using CBT, this however can still affect performance and amount of data backed up.

Also we are aware of a condition where the latest hotfixes for both BE 2010 R3 and BE 2012 can still see problems with the stability of the beremote.exe process during VMware backups unless either the buffers are disabled or the used sector tracking is disabled. These new, post hotfix, conditions are currently being investigated.

[Nitin_P]

Hi Colin Weaver,

So , in this situation , will it be ok for us to go back to the position to disable buffer reads / writers for now  , it will stop crashing beremote.

[steven_man]

VM version of some VMs in my environment are 4, some are version 7 and 8. After upgrade some VMs, the issue of Remote Agent service seem to be solved after upgrade of VM to version 8. The service never stop after upgrade some VMs to version 8.

Because my environment have many VMs, I need some time to upgrade VM. Keep monitoring the backup result.

[steven_man]

After upgrade all VMs to version 8, the issue of remote agent service does not happen again.

No change on the following registry.

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

The value is still 0.

[steven_man]

But the followng three errors sometimes found.

V-79-57344-38725 - One or more virtual disks contain volumes that do not support Granular Recovery Technology (GRT). Granular restores cannot be performed for these volumes.

V-79-57344-38724 - Directory

 C:\XXX does not support Granular Recovery Technology (GRT).  Granular restores cannot be performed for items in this directory.

The job failed with the following error: Incremental\\Differential backup methods are not supported for this VM. The backup job failed, since the option 'to fall back to a full backup' was not selected. 

Why?