Solution for VSS exceptions with VMware guests / VM tools
Hi everyone, We have been experiencing VSS issues with VMware guests in regards to the installed Backup Exec Agent and a previously installed VMware Tools VSS option. Uninstalling the VMware Tools VSS option in various ways including restarts did not fix our issues. If you search the internet for solutions, you find many attempts but no real solution or explanation. One of our admins spend several hours with the Veritas support without a solution, he was about to escalate the issue with them, when we found the root cause and could actually fix our issues. First the steps to solve this: Uninstall the VMware Tools VSS option (no restart will be requiered) Make sure VMware VSS service was deleted If this is not the case, you might need to do so manually and remove additional DLL's etc. as well as restart the system, but this is independent from this solution You might have already done steps 1 and 2 but you still get VSS exceptions from the backup that says you have more than one VSS agent installed: V-79-8192-38331 - The backup has detected that both the VMware VSS Provider and the Symantec VSS Provider have been installed on the virtual machine 'hostname'. However, only one VSS Provider can be used on a virutal machine. You must uninstall the VMware VSS Provider. Now you wonder what causes this and you get stuck You could uninstall the Veritas/Symantec Backup Exec Agent and only back the system up per VMDK You would lose the GRT / granular backup / restore capabilities Check your registry for the following reg key HKLM\System\CurrentControlSet\Services\BeVssProviderConflict If this key exists, but your VMware VSS provider is uninstalled, you need to follow up with step 5 Open a notepad as administrator Open this file in the notepad C:\ProgramData\VMware\VMware Tools\manifest.txt Search for the following two entries: Vcbprovider_2003.installed vcbprovider.installed Make sure both of them are set to FALSE, most likely one of them is TRUE Run a test backup This test backup now should not show the exception anymore The registry key should vanish (refresh/press F5) without you taking action So what happened? You uninstalled the VMware Tools VSS provider, but this manifest file did not get updated. We actually could see that it sometimes does get updated and sometimes does not. This seems to be some kind of issue with the VMware Tools uninstalled/installer. But why this manifest.txt file? As we found out, there scripts that get executed by Symantec/Veritas Backup Exec before the backup. You might find them in two locations, and it seems to depend a bit on the Windows version which script is executed (at which location). You could edit them both and just undo the checks in the scripts, but this wouldn't be correct. It is more correct to update the manifest.txt file. If you want to, you can check the date/time of the manifest.txt file before you change it - you might see it was not updated while you uninstalled the VMware Tools VSS provider (assuming you did only do this and not do additional installs/uninstalls within the VMware Tools / please note as well that this only is true when you still experienced those issues). Now, back to those scripts, you find them here: C:\Windows C:\Program Files\Symantec\Backup Exec\RAWS\VSS Provider The name of the script that matters: pre-freeze-script.bat This script checks several DLLs, registry entries, paths and on Windows 2008 and newer the ProgramData-Path for this specific manifest.txt file and the two entries mentioned. Once you uninstall the VMware VSS provider, and the file did not get updated, you might see this issue and wonder how to solve it. The solution is to simply update it to mirror the uninstallation of the VMware VSS provider (vcbprovider). We double checked this with several installations and could see if the file actually gets updated, those two values are set to FALSE, if it doesn't, at least one of the values remains true, what causes the pre-freeze-script.bat to write the registry key mentioned earlier and therefor causing the issue in the backup - exceptions. If you still have the same issues after updating the manifest.txt, simply check all the DLL's that are mentioned in the script and make sure they don't exist. You might also consider to manually delete the registry-key (it seems to be just a dumy-key) to make sure there is no issue that prevents the script from deleting it. Make sure it does not re-appear after a backup! Otherwise you might still have some DLLs left on your system that cause the script to re-create the registry key. Hope this helps a few of you out there. This was an ongoing issue for a while and I came accross those issues many times ever since Windows 2008. This applies to Windows 2008 R2, Windows 2012, Windows 2012 R2 and pretty sure to Windows 2016 as well. It helped us getting rid of those issues completely and actually not even needing a single restart of the guest VM to solve the issues (removing the VMware VSS provide did not need a restart). Regards Florian RossmarkSolved14KViews2likes10CommentsBackup Exec hangs while backing up VMware VMs (snapshot)
We are having this issue for weeks/months now : while processing a backup-to-disk job using Backup Exec 2012 SP1a VMware infrastructure agent the backup process hangs at some point. We are running a VMware 5 (Build 768111) environment with a fibre channel SAN which is being backed up by a Windows 2008 R2 BE2012 SP1a (Verion 14.0 Rev. 1798 64 Bit) VM. At some point in the backup process - and different VMs - I can see that there is no progress anymore for hours or days in my vSphere client (percentage of the job freezes) and in BE the MB/min indicator decreases and no changes to the number of backed up Bytes. This is only when using backup-to-disk, which actually is a (VMFS 5) partition within the BE virtual machine (drive letter V: size 1,5TB). Last time it stopped at 95% of our backend Exchange 2010 server (running, approx. 300GB), last time before it stopped while backing up a powered off Windows Server 2003 test system (approx. 30GB). The only thing to do is hit cancel at the BE machine (which actually wouldn't cancel or stop the job), then reboot the BE machine, cancel the VMware job in vSphere client and then "delete" the VMware snapshot to return to merge the changes on the disks. Gladly we back up all data (non-VMs) separately directly to tape (VMware Direct I/O connected SCSI LTO juke box), so I have at least my data at hand, but I don't understand why even a snapshot of a powered off system halts the Symantec VMware job. Snapshots manually initiated in vSphere client have not been an issue yet, disk space is available, too. Have not found any similar problem desciption in Google... Any help really appreaciated!Solved6.3KViews1like15CommentsvSphere 5.1 Support Update
Symantec and VMware are unwaveringly dedicated to quality in our jointly developed solutions. During our testing with the VMware vStorage 5.1 API Symantec discovered issues* that introduce data recovery risks. We raised these concerns with our friends at VMware, who documented them in their knowledge base and in the VMware vSphere 5.1 release notes. VMware expects to release a vStorage API update, and at that time we expect to retest and once again support joint solutions. Read this blog for more details!6.3KViews4likes38CommentsSAN Transport on VMware VSAN
Hi, We recently deployed a new VM infrastructure using Vmware VSAN 6.0 It's running in 10G network and the netbackup 5230 appliance is using the same network for backup. Currently backups are running over LAN method which has some speed limitation on VMware kernel side. I would like to know if VSAN is supported for SAN Transport to maximize the backup speed. Based on the Veritas Education videos online, it is supported but I dont know how to enable it. I cant find any documentation about it as well. https://www.youtube.com/watch?v=-IILCXjhBZc Please help.5.9KViews0likes6CommentsWhy I unable to view the old archived email from my Outlook 2007 SP 2 ?
Hi, When I go to the Outlook to open up my shared mailbox, I got the grey bar which says “Item has been archived by Enterprise Vault”. ? where are the rest of the message goes and how to restore it back from archive ? ThanksSolved5.2KViews3likes8CommentsSnapshot error status 156 using VCB
Enviro: NB6.5.3.1 on dedicated master, media and proxy. Proxy attached to 2 CLARiiONS, one for source vmdks, the other for snapshot cache. VCB1.5 Windows Server 2003 R2 SP2 x64 is the VCBproxy. Attached to array over 2Gb FC. VM Backup : 1 transfer type: 3 vmdk type: 1 Seems like if backing up one VM at a time will work. Running them in "streams" all results in snapshot errors although Idon't see any bottlenecks anywhere. When I setup the VCB policy I enabled 5 streams. When the jobs kick off, only the first one was successful, the others all fail with "- Critical bpbrm(pid=5932) from client qa01: FTL - snapshot creation failed, status 156". 90% of our VM's are Linux so all I care about is fulls I guess. I'm attaching the bpfis log from the proxy, the bpvmutil log has nothing relavent to this issue as the discovery is all running fine, just incredibly slow. Has anyone seen this issue? thanks in advance. There are multiple failing commands and errors but Ido not see how to resolve this based on other posts.Solved5KViews1like42Commentssymantec backupexec 2012 AOF VSS hatası (Advanced open file)
Exchange sp1 den 2 ye geçiş yaptıktan sonra seninde her ay gelip kontrollerinde fark ettin ve gördüğün hataydı bildiğin gibi. Hata: LCR veya CCR Yapısındaki ddatabes lerin bir kısmını başarı ile alıp diğerlerinde VSS hatası vermesiydi. Bu VSS hatası için genel olarak net bir çözüm bulamamıştık, microsoft VSS Backup hatasını kabullenmiş bununla ilgili bir sürü makale ve güncelleme çıkarmış ancak net bir çözüm için Exc SP2 Rollup 4 veya SP3 Geçerseniz düzelir diyordu.. Veritas backup forumlarından yola çıkarak edindiğim bilgiler doğrultusunda şunları yaparak çözdüm. 1. Ekteki Vss-update311216.reg dosyasını her CCR deki nod1 ve nod2 ye yüklenmesi gerekiyor 2. Nod’larda symantec agent kurulurken AOF (Advanced open file) kurulmaması gerekiyor 3. Backup jop ayarlarken AOF disable edilmesi 4. Back up from the active copy only mutlaka seçilmeli. Default gelen önerilen seçilmemeli. Konuyu farklı bir şekilde anlatan symantec formu aşağıdadır. Ama en doğrusu benim izlediğim yoldur. Evyap’ta bu sorun devam ediyor diye biliyorum.. http://www.symantec.com/business/support/index?page=content&id=TECH616075KViews0likes9Commentse000959a - An attempt to take a snapshot of a virtual machine failed because it was unable to quiesce an application
When doing backups of VM servers I get the following error and some of my backups don't comlete. e000959a - An attempt to take a snapshot of a virtual machine failed because it was unable to quiesce an application This happens with both Full backups and Differentials. I am running Backup Exec 2012 fully patched. The VMWare environment is version 5. Any thoughts or ideas are appreciated.5KViews2likes8CommentsFailed to get VM Server Info List
Hi, I am trying to set up new clients in the VCB policy on a 6.5.3 server. The proxy is dedicated and is running 6.5.3 as well. The Virtual Center Server has over 75 running VM's it knows about, yet when I try to Browse Virutal Machines in Net Backup, all Iget is the "fileread failed (13)" error. I created the bpvmutil logs and setup the verbose logging levels. Seems as though from the command line Ican login to the VCS fine but when Iquery names, it only returns one junky little test VM and none of the others: C:\Program Files (x86)\VMware\VMware Consolidated Backup Framework>vcbVmName.exe -h virmgmt101-dev -u sa_vmbackup -p****** -s any [2009-04-21 07:00:57.739 'App' 5996 info] Current working directory: C:\Program Files (x86)\VMware\VMware Consolidated Backup Framework [2009-04-21 07:00:57.739 'BaseLibs' 5996 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 4 numThreadsPerCore 1. [2009-04-21 07:00:57.739 'BaseLibs' 5996 info] HOSTINFO: This machine has 1 physical CPUS, 4 total cores, and 4 logical CPUs. [2009-04-21 07:00:57.739 'BaseLibs' 5996 info] Using system libcrypto, version 90709F [2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyCertAgainstSystemStore: Subject mismatch: VMware vs virmgmt101-dev [2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyCertAgainstSystemStore: The remote host certificate has these prob lems: * The host name used for the connection does not match the subject name on the host certificate * A certificate in the host's chain is based on an untrusted root. [2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyIsEnabled: failed to read registry value. Assuming verification is disabled. LastError = 0 [2009-04-21 07:01:13.066 'BaseLibs' 5996 warning] SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so c onnection will proceed despite the error [2009-04-21 07:01:15.316 'vcbVmName' 5996 warning] IP address not set. Found VM: moref:vm-23253 name:test4 uuid:50165156-871d-1e74-52e2-18d82d7e8584 ipaddr: I'll attach the relavent log file as Iam stumped and we don't know where to look for the answers. The one thing that might be biting us is the VCBproxy / master/ media servers are completely firewalled off from the ESX/VCS servers. We have port 443 open from the VCB proxy to the VCS, and the logs show the communication is working. Just a sanity check, but do the master and media server need to talk to the VMware servers or does everything pass thru the proxy as I think? thanks in advance!Solved4.9KViews1like9Comments