Solution for VSS exceptions with VMware guests / VM tools
Hi everyone, We have been experiencing VSS issues with VMware guests in regards to the installed Backup Exec Agent and a previously installed VMware Tools VSS option. Uninstalling the VMware Tools VSS option in various ways including restarts did not fix our issues. If you search the internet for solutions, you find many attempts but no real solution or explanation. One of our admins spend several hours with the Veritas support without a solution, he was about to escalate the issue with them, when we found the root cause and could actually fix our issues. First the steps to solve this: Uninstall the VMware Tools VSS option (no restart will be requiered) Make sure VMware VSS service was deleted If this is not the case, you might need to do so manually and remove additional DLL's etc. as well as restart the system, but this is independent from this solution You might have already done steps 1 and 2 but you still get VSS exceptions from the backup that says you have more than one VSS agent installed: V-79-8192-38331 - The backup has detected that both the VMware VSS Provider and the Symantec VSS Provider have been installed on the virtual machine 'hostname'. However, only one VSS Provider can be used on a virutal machine. You must uninstall the VMware VSS Provider. Now you wonder what causes this and you get stuck You could uninstall the Veritas/Symantec Backup Exec Agent and only back the system up per VMDK You would lose the GRT / granular backup / restore capabilities Check your registry for the following reg key HKLM\System\CurrentControlSet\Services\BeVssProviderConflict If this key exists, but your VMware VSS provider is uninstalled, you need to follow up with step 5 Open a notepad as administrator Open this file in the notepad C:\ProgramData\VMware\VMware Tools\manifest.txt Search for the following two entries: Vcbprovider_2003.installed vcbprovider.installed Make sure both of them are set to FALSE, most likely one of them is TRUE Run a test backup This test backup now should not show the exception anymore The registry key should vanish (refresh/press F5) without you taking action So what happened? You uninstalled the VMware Tools VSS provider, but this manifest file did not get updated. We actually could see that it sometimes does get updated and sometimes does not. This seems to be some kind of issue with the VMware Tools uninstalled/installer. But why this manifest.txt file? As we found out, there scripts that get executed by Symantec/Veritas Backup Exec before the backup. You might find them in two locations, and it seems to depend a bit on the Windows version which script is executed (at which location). You could edit them both and just undo the checks in the scripts, but this wouldn't be correct. It is more correct to update the manifest.txt file. If you want to, you can check the date/time of the manifest.txt file before you change it - you might see it was not updated while you uninstalled the VMware Tools VSS provider (assuming you did only do this and not do additional installs/uninstalls within the VMware Tools / please note as well that this only is true when you still experienced those issues). Now, back to those scripts, you find them here: C:\Windows C:\Program Files\Symantec\Backup Exec\RAWS\VSS Provider The name of the script that matters: pre-freeze-script.bat This script checks several DLLs, registry entries, paths and on Windows 2008 and newer the ProgramData-Path for this specific manifest.txt file and the two entries mentioned. Once you uninstall the VMware VSS provider, and the file did not get updated, you might see this issue and wonder how to solve it. The solution is to simply update it to mirror the uninstallation of the VMware VSS provider (vcbprovider). We double checked this with several installations and could see if the file actually gets updated, those two values are set to FALSE, if it doesn't, at least one of the values remains true, what causes the pre-freeze-script.bat to write the registry key mentioned earlier and therefor causing the issue in the backup - exceptions. If you still have the same issues after updating the manifest.txt, simply check all the DLL's that are mentioned in the script and make sure they don't exist. You might also consider to manually delete the registry-key (it seems to be just a dumy-key) to make sure there is no issue that prevents the script from deleting it. Make sure it does not re-appear after a backup! Otherwise you might still have some DLLs left on your system that cause the script to re-create the registry key. Hope this helps a few of you out there. This was an ongoing issue for a while and I came accross those issues many times ever since Windows 2008. This applies to Windows 2008 R2, Windows 2012, Windows 2012 R2 and pretty sure to Windows 2016 as well. It helped us getting rid of those issues completely and actually not even needing a single restart of the guest VM to solve the issues (removing the VMware VSS provide did not need a restart). Regards Florian RossmarkSolved14KViews2likes10CommentsEvent 1023 Windows cannot load the extensible counter DLL Backup Exec.
I am getting the following events from our newly build server 2012 r2 server with BE 15. I am not sure what is causing this issue, but it appears to be related to BE and performance counters. Log Name: Application Source: Microsoft-Windows-Perflib Date: 4/24/2015 11:02:17 AM Event ID: 1008 Task Category: None Level: Error Keywords: Classic User: N/A Computer: XXXXXXX Description: The Open Procedure for service "BITS" in DLL "C:\Windows\System32\bitsperf.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Perflib" Guid="{13B197BD-7CEE-4B4E-8DD0-59314CE374CE}" EventSourceName="Perflib" /> <EventID Qualifiers="49152">1008</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-04-24T18:02:17.000000000Z" /> <EventRecordID>3405</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>XXXXXXX</Computer> <Security /> </System> <UserData> <EventXML xmlns="Perflib"> <param1>BITS</param1> <param2>C:\Windows\System32\bitsperf.dll</param2> <binaryDataSize>4</binaryDataSize> <binaryData>02000000</binaryData> </EventXML> </UserData> </Event> Log Name: Application Source: Microsoft-Windows-Perflib Date: 4/24/2015 10:53:43 AM Event ID: 1023 Task Category: None Level: Error Keywords: Classic User: N/A Computer: XXXXX Description: Windows cannot load the extensible counter DLL Backup Exec. The first four bytes (DWORD) of the Data section contains the Windows error code. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Perflib" Guid="{13B197BD-7CEE-4B4E-8DD0-59314CE374CE}" EventSourceName="Perflib" /> <EventID Qualifiers="49152">1023</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-04-24T17:53:43.000000000Z" /> <EventRecordID>3403</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>XXXXXX</Computer> <Security /> </System> <UserData> <EventXML xmlns="Perflib"> <param1>Backup Exec</param1> <binaryDataSize>4</binaryDataSize> <binaryData>7E000000</binaryData> </EventXML> </UserData> </Event>12KViews0likes4CommentsBE2014 DLM not delete already expired backup sets
Hello, I have upgraded a BE2012 SP4 installation to BE2014. The backups go to a DataDomain DD160 using as a OpenStorage device. There a lot of backup sets which are already have been expired. The DLM of BE2014 seems not deleting already expired backup sets (e.g. from BE2012). For new expired backup sets the deletion works. How can I delete the >1000 expired backup sets?11KViews3likes44CommentsV-79-57344-38366 Backup Exec 20.6 12TB VMDK
Hey all, I've finally got all my backups working correctly EXCEPT a legacy drive inside a VMware guest server. I had excluded this single VMDK from the nightly backup job and have set up a single one-time backup of this drive as a seperate job. However I cannot get the one-time job to succeed. The job runs for the entirety of the 11.7TB of data inside that 12TB drive but results in the error I included below. The job takes about 2 days to complete unsuccessfully. Currently the job is setup to use the same storage space as the pre-mentioned nightly backup job of this guest VM's other drives. So I know this isn't a permissions issue. I'm assuming as well, since I simply copied the nightly job and then unselected the drives from that job and added the single 12TB drive that the configuration is likely not the issue. Backup- VMVCB::\\<vmguest>.example.com\VCGuestVm\(DC)ha-datacenter(DC)\vm\[110] VMGUSET-79-57344-38366 - VDDK-Warn: VDDK_PhoneHome: HTTP response error in VddkVacCurlHttpsPost with http status 400 at 1058. VDDK-Warn: VDDK_PhoneHome: VddkVacPushData : Failed to post data to VC/PH at line 1022. VDDK-Warn: VDDK_PhoneHome: VddkVacPushVendorProductData : Failed to push vendor product data at line 1326. VDDK-Warn: VDDK_PhoneHome: VddkVacWriteMember : Empty field to serialize for key edition at line 188. VDDK-Warn: VDDK_PhoneHome: VddkVacPushProductInstanceData : Failed to push product instance data at line 1219. VDDK-Warn: VDDK_PhoneHome: VddkVacPushEnvironmentData : Failed to push env data at line 1136. VDDK-Warn: VDDK_PhoneHome: VddkVacPushSessionData : Failed to push session data at line 1709. VDDK-Warn: VixDiskLib: VixDiskLibIsLegacyConnParams: the instance of VixDiskLibConnectParams is NOT allocated by VixDiskLib_AllocateConnectParams. The new features in 6.7 or later are not supported. VDDK-Warn: HostAgent is not a VirtualCenter. Error 20 at 3710. VDDK-Warn: VDDK_PhoneHome: Invaild transport mode used at line 271 VDDK-Warn: VixDiskLibProvider::NameBasedProvider::Get: Unable to open disk \\UNC\PATH\TO\STORAGE\IMG000027\GUESTVM_1-000002.vmdk, openflags = 4 - VixError 0x3e86. VDDK-Warn: DiskLibProvider_GetDisk: Open failed - index 0. VDDK-Warn: ERROR 2 opening disk 0. VDDK-Warn: VixDiskLibProvider::NameBasedProvider::Get: Unable to open disk 1:\IMG000027\GUESTVM_1-000002.vmdk, openflags = 4 - VixError 0x3e86. The line "The new features in 6.7 or later are not supported.", seems of note as the GUESTVM I'm trying to backup is on a ESXi Server 6.7. But again, that same GUESTVM does backup correctly if I exclude that one 12TB vmdk. My backup exec server is on an older ESXi 5.5 host running a Windows Server 2012R2 guest. The GUESTVM I'm backing up is a Windows Server 2016 install. I notice that in the \\UNC\PATH\TO\STORAGE location when the job starts, it creates 8 GUESTVM_1-000002 vmdk that look to split. The first VMDK reaches 1.98TB and then it starts filling the second. This one ends up around 700GB when the job ends. Please help!8.2KViews0likes7CommentsBackupExec 2014 - No writable idle devices available
Hello, I'm needing some help with what to me is a strange issue. I have several jobs which were queued but not running even though they should have completed. I found thatthe spoold service was stuck or hung in a starting condition.After searching the forums before I tried anything, I re-started the server and this time the spoold service is no longer hung. All services are running correctly. Now I'm getting a Job Status of Ready; No writable idle devices are available. Also when I go to the storage tab I find that everything is online except Deduplication disk storage 0001 of the Robotic library says it is offline. I am using BE2014 v14.1 rev 1786 64-bit on a Dell PowerEdge R320 Server running Windows 2012 R2 Server Standard (full installation) 64-bit. I appreciate any help that I can get for this issue. I'm not sure what other information anyone may need. I'll be happy to try andsupply any information needed. Thanks in advance for your assistance!Solved7.9KViews0likes10CommentsBackup fail - e00084ed - A hardware error occurred
I am having trouble in backing up my servers to tape. I am able to back up to disk. However when I try to backup the system state of my server to tape the backups fail with the error e00084ed - A hardware error occurred. However when I try to backup a single folder of size 3MB to tape I can backup. Attached find the screenshot. Any ideas?Solved7.1KViews1like15CommentsNDMP with NetApp cDOT (Cluster-Mode)
Has anyone had success in BackupExec 2012 or 2014 in backing up via NDMP a NetApp filer running cDOT (Cluster-Mode) 8.2 or higher? We have an open case with support when running 2014+cDOT 8.2.1 where we cannot browse any snapshots and/or files within our volumes when connecting to our filers, something we were able to do with 7-Mode. Thanks, RS6.2KViews2likes24CommentsVDS Basic Provider 1
I have 3-node Windows Server 2012 cluster of Hyper-V virtual machines. Some virtual machines arebackuped by means of Backup Exec 2014 system. Sometimes during this backup I see next error in the Event Log: Source: VDS Basic Provider ID: 1 There was an unknown error. Error state number: 48F@01000003 (this nessage translated from Russian). What is the problem?Solved6.1KViews2likes5Commentsbackup exec 15 backup sets not deleted.
Hi Running Backup Exec 15 on a Windows server 2012 R2 host for Hyper-V. I use5 indetical usb disk's that i swap daily. Connected as drive X:. I have made a backup store call'd "Daily" that point's to X:\bedata. Backup job has set that the backup set is write protected 3 days. I have scheduled a Inventory run befor the backup job. The problem is when i swap usb disk the system see the "new" disk, and it can see if there is space available for backup. But it does not see the backup set's on the disk that is expired. If i do a catalog on present usb disk, it see's the expired backup set and is cleaning up as expected. But i cant schedule a catalog. Therefore my disk is runnig full and the backup i fails. What ohter options do i have to get i done automatic?Solved5.8KViews0likes5Comments