Solution for VSS exceptions with VMware guests / VM tools
Hi everyone, We have been experiencing VSS issues with VMware guests in regards to the installed Backup Exec Agent and a previously installed VMware Tools VSS option. Uninstalling the VMware Tools VSS option in various ways including restarts did not fix our issues. If you search the internet for solutions, you find many attempts but no real solution or explanation. One of our admins spend several hours with the Veritas support without a solution, he was about to escalate the issue with them, when we found the root cause and could actually fix our issues. First the steps to solve this: Uninstall the VMware Tools VSS option (no restart will be requiered) Make sure VMware VSS service was deleted If this is not the case, you might need to do so manually and remove additional DLL's etc. as well as restart the system, but this is independent from this solution You might have already done steps 1 and 2 but you still get VSS exceptions from the backup that says you have more than one VSS agent installed: V-79-8192-38331 - The backup has detected that both the VMware VSS Provider and the Symantec VSS Provider have been installed on the virtual machine 'hostname'. However, only one VSS Provider can be used on a virutal machine. You must uninstall the VMware VSS Provider. Now you wonder what causes this and you get stuck You could uninstall the Veritas/Symantec Backup Exec Agent and only back the system up per VMDK You would lose the GRT / granular backup / restore capabilities Check your registry for the following reg key HKLM\System\CurrentControlSet\Services\BeVssProviderConflict If this key exists, but your VMware VSS provider is uninstalled, you need to follow up with step 5 Open a notepad as administrator Open this file in the notepad C:\ProgramData\VMware\VMware Tools\manifest.txt Search for the following two entries: Vcbprovider_2003.installed vcbprovider.installed Make sure both of them are set to FALSE, most likely one of them is TRUE Run a test backup This test backup now should not show the exception anymore The registry key should vanish (refresh/press F5) without you taking action So what happened? You uninstalled the VMware Tools VSS provider, but this manifest file did not get updated. We actually could see that it sometimes does get updated and sometimes does not. This seems to be some kind of issue with the VMware Tools uninstalled/installer. But why this manifest.txt file? As we found out, there scripts that get executed by Symantec/Veritas Backup Exec before the backup. You might find them in two locations, and it seems to depend a bit on the Windows version which script is executed (at which location). You could edit them both and just undo the checks in the scripts, but this wouldn't be correct. It is more correct to update the manifest.txt file. If you want to, you can check the date/time of the manifest.txt file before you change it - you might see it was not updated while you uninstalled the VMware Tools VSS provider (assuming you did only do this and not do additional installs/uninstalls within the VMware Tools / please note as well that this only is true when you still experienced those issues). Now, back to those scripts, you find them here: C:\Windows C:\Program Files\Symantec\Backup Exec\RAWS\VSS Provider The name of the script that matters: pre-freeze-script.bat This script checks several DLLs, registry entries, paths and on Windows 2008 and newer the ProgramData-Path for this specific manifest.txt file and the two entries mentioned. Once you uninstall the VMware VSS provider, and the file did not get updated, you might see this issue and wonder how to solve it. The solution is to simply update it to mirror the uninstallation of the VMware VSS provider (vcbprovider). We double checked this with several installations and could see if the file actually gets updated, those two values are set to FALSE, if it doesn't, at least one of the values remains true, what causes the pre-freeze-script.bat to write the registry key mentioned earlier and therefor causing the issue in the backup - exceptions. If you still have the same issues after updating the manifest.txt, simply check all the DLL's that are mentioned in the script and make sure they don't exist. You might also consider to manually delete the registry-key (it seems to be just a dumy-key) to make sure there is no issue that prevents the script from deleting it. Make sure it does not re-appear after a backup! Otherwise you might still have some DLLs left on your system that cause the script to re-create the registry key. Hope this helps a few of you out there. This was an ongoing issue for a while and I came accross those issues many times ever since Windows 2008. This applies to Windows 2008 R2, Windows 2012, Windows 2012 R2 and pretty sure to Windows 2016 as well. It helped us getting rid of those issues completely and actually not even needing a single restart of the guest VM to solve the issues (removing the VMware VSS provide did not need a restart). Regards Florian RossmarkSolved14KViews2likes10CommentsProblem with VSS on a physical Windows Server
Hi, I am adding a physical Windows Server 2016machine as a first client in a new Windows Policy (This is a new NetBackup installation, version 8.1.1). The backups fail with status 69, the log says (some host names edited throughout): [...] 07.06.201812:43:59-Infobpbkar(pid=13064)willattempttousechangejournaldatafor<C:\> 07.06.201812:43:59-Infobpbkar(pid=13064)notusingchangejournaldatafor<SystemState:\>:notsupportedfornon-localvolumes/filesystems 07.06.201812:43:59-Infobpbkar(pid=13064)notusingchangejournaldatafor<_BACKUP_SPECIAL_OBJECTSAFTERSystemState:>:notsupportedfornon-localvolumes/filesystems 07.06.201812:43:59-Infobpbkar(pid=13064)notusingchangejournaldatafor<EFISystemPartition:\>:notsupportedfornon-localvolumes/filesystems 07.06.201812:44:03-Errorbpbrm(pid=62449)fromclient<client>:ERR-UnabletobackupSystemStateorShadowCopy.PleasecheckthestateofVSSandassociatedWriters.ERR-UnabletobackupSystemStateorShadowCopy.PleasecheckthestateofVSSandassociatedWriters.INF-Estimate:-1-1 07.06.201812:44:09-Infobpbkar(pid=13064)notusingchangejournaldatafor<C:\>:noprevioustracklog 07.06.201812:44:51-Infobpbkar(pid=13064)5000entriessenttobpdbm 07.06.201812:45:11-Infobpbkar(pid=13064)10000entriessenttobpdbm 07.06.201812:45:36-Infobpbkar(pid=13064)15000entriessenttobpdbm 07.06.201812:45:54-Infobpbkar(pid=13064)20000entriessenttobpdbm 07.06.201812:46:26-Infobpbkar(pid=13064)25000entriessenttobpdbm 07.06.201812:46:44-Errorbpbrm(pid=62449)fromclient<client>:ERR-UnabletobackupSystemStateorShadowCopy.PleasecheckthestateofVSSandassociatedWriters.JBD-acceleratorsent6763415552bytesoutof6732860928bytestoserver,optimization0.0% 07.06.201812:46:46-Errorbptm(pid=62640)mediamanagerterminatedbyparentprocess 07.06.201812:46:51-Info<appliance>(pid=62640)StorageServer=PureDisk:<appliance>;Report=PDDOStats(multi-threadedstreamused)for(<appliance>):scanned:6575591KB,CRsent:18475KB,CRsentoverFC:0KB,dedup:99.7%,cachedisabled 07.06.201812:46:52-Errorbpbrm(pid=62449)couldnotsendserverstatusmessagetoclient 07.06.201812:46:53-Infobpbkar(pid=13064)done.status:69:invalidfilelistspecification 07.06.201812:46:40-endwriting;writetime:0:02:54 invalidfilelistspecification (69) 07.06.201812:46:55-Infobpbrm(pid=63398)Startingdeletesnapshotprocessing 07.06.201812:46:56-Infobpfis(pid=11140)Backupstarted 07.06.201812:46:56-Warningbpbrm(pid=63398)fromclient<client>:cannotopenC:\ProgramFiles\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.<client>_1528368217.1.0 07.06.201812:46:56-Infobpfis(pid=11140)done.status:4207 07.06.201812:46:56-Infobpfis(pid=11140)done.status:4207:Couldnotfetchsnapshotmetadataorstatefiles - "vssadmin list writers" on the client shows "No errors" and "stable" for all writers - The backup stores about 6GB (of about 56GB on the disk) of data, so at least something must be working - A VSS backup using Windows Backup succeeded - I have tried all available options for the VSS provider type, the error is always the same - During the (NetBackup) backup, "vssadmin list shadows" shows an active shadow copy that did not exists before the backup: C:\Windows\system32>vssadmin list shadows vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2013 Microsoft Corp. Contents of shadow copy set ID: {306403af-d5a9-4d24-b3c2-4914a0a1a5a8} Contained 1 shadow copies at creation time: 6/7/2018 12:43:54 PM Shadow Copy ID: {e07da3a5-2dba-4544-b3f4-9893df0759d8} Original Volume: (C:)\\?\Volume{aa396902-6d6d-48a3-a039-67d87383d939}\ Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy10 Originating Machine:<client> Service Machine:<client> Provider: 'Microsoft Software Shadow Copy provider 1.0' Type: ApplicationRollback Attributes: Persistent, No auto release, Differential Any ideas anybody?Solved3.8KViews0likes4CommentsMS-Windows policy and VSS question
Hi everyone, Just a quick question regarding an MS-Windows policy and VSS. Does the policy backup from the VSS snapshot or does it only use VSS for open files? If the backup takes 7 hours, is there a chance of backing up changes files during that 7 hours? Or will the backup only run from the VSS snapshot taken when the backup starts? We need to backup a file/folder structure at a specific time on a physical machine. Any advise would be much appreciated! Thanks in advance!Solved1.5KViews0likes4CommentsBackup Exec VSS Writer
Good day, We are running Backup Exec on Windows Server 2008 R2 Standard. The backups keep failing when running a job for the Hyper-V Host server (Windows Server 2016 Standard) with 3x VM'sand generates the errors below. The backup job will run once after the Host server has been restarted but after that it keeps failing. When running the VSSADMIN LIST WRITER command, no errors are generated on the Hyper-V host or the clients. Please may this get your full attention as these are critical servers that need to be backed up. Error log found on backup Exec (Backup Server) Job Log: BEX_HEBS_08814.xml<noscript> 30 October 2018 11:20:01 AM - V-79-40960-38521 - Snapshot Technology: Initialization failure on: "VRTSRV::\\****.he.local\Hyper-V?Virtual?Machine\****". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS). A virtual machine snapshot could not be created. Use the Event Viewer on the Hyper-V host to identify the cause. If the virtual machine is not properly configured, it may be necessary to use the Event Viewer in the virtual machine to identify the cause. The following volumes are dependent on resource: "C:" . The snapshot technology used by VSS for volume C: - Microsoft Software Shadow Copy provider 1.0 (Version 1.0.0.7). Please see Acronis VSS Doctor report below (Errors were generated on Hyper-V Host server): 2018-10-30 11:23:23 GMT+02:00 AcronisVSSDoctor-net45, 1.0.39.0, v4.0.30319 Microsoft Windows NT 6.2.9200.0, 4.0.30319.42000 =========================================================== VSS Services Configuration and Status Status: OK Description: All VSS-related services are configured correctly Services: Name: swprv Title: Microsoft Software Shadow Copy Provider Status: Running ReferenceStatus: NeedToFixStatus: False StartMode: Manual ReferenceStartMode: Manual NeedToFixStartMode: False IsOk: True ExpectationDescription: Exception: Name: VSS Title: Volume Shadow Copy Status: Running ReferenceStatus: NeedToFixStatus: False StartMode: Manual ReferenceStartMode: Manual NeedToFixStartMode: False IsOk: True ExpectationDescription: Exception: Name: EventSystem Title: COM+ Event System Status: Running ReferenceStatus: Running NeedToFixStatus: False StartMode: Auto ReferenceStartMode: Auto NeedToFixStartMode: False IsOk: True ExpectationDescription: Exception: =========================================================== VSS Providers Configuration Status: OK Description: Only known VSS provider(s) found Providers: Id: 89300202-3cec-4981-9171-19f59559e0f2 Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{89300202-3cec-4981-9171-19f59559e0f2} Name: Microsoft File Share Shadow Copy provider IsDefault: True Id: b5946137-7b9f-4925-af80-51abd60b20d5 Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{b5946137-7b9f-4925-af80-51abd60b20d5} Name: Microsoft Software Shadow Copy provider 1.0 IsDefault: True =========================================================== VSS Access Control Configuration Status: OK Description: Known VSS access control entries are configured correctly AccessControls: UserName: NT Authority\NetworkService EntryExists: True UserExists: True IsActive: True IsOk: True StatusDescription: =========================================================== VSS Components Availability Status: OK Description: VSS components are accessible =========================================================== VSS Writers Status Status: OK Description: All VSS writers are working correctly Writers: Name: Task Scheduler Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124} State: [1] Stable IsOk: True Error: No error Name: VSS Metadata Store Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06} State: [1] Stable IsOk: True Error: No error Name: Performance Counters Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2} State: [1] Stable IsOk: True Error: No error Name: System Writer Id: {e8132975-6f93-4464-a53e-1050253ae220} State: [1] Stable IsOk: True Error: No error Name: ASR Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4} State: [1] Stable IsOk: True Error: No error Name: Shadow Copy Optimization Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} State: [5] Waiting for completion IsOk: True Error: No error Name: FRS Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29} State: [1] Stable IsOk: True Error: No error Name: WMI Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} State: [1] Stable IsOk: True Error: No error Name: MSSearch Service Writer Id: {cd3f2362-8bef-46c7-9181-d62844cdc0b2} State: [5] Waiting for completion IsOk: True Error: No error Name: Registry Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485} State: [1] Stable IsOk: True Error: No error Name: NTDS Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757} State: [1] Stable IsOk: True Error: No error =========================================================== VSS Shadow Storages Status Status: OK Description: All VSS shadow storages are configured correctly Storages: ForVolume: C:\ OnVolume: C:\ Used: 96 MB Allocated: 50 GB Maximum: UNBOUNDED Minimum: 320 MB VolumeSize: 3721 GB Available: 3218 GB TargetMaximum: 372 GB MaximumGreaterThenMinimum: True MaximumIsEnough: True MaximumIsNotReached: True AvailableIsEnoughForMaximum: True IsOk: True Description: =========================================================== Disk Free Space Available Status: OK Description: All disks have enough free space Volumes: Name: C:\ DeviceId: \\?\Volume{1c4947d2-e53e-e711-a993-6679abe43d90}\ Size: 3721 GB Available: 3218 GB Minimum: 320 MB IsOk: True Description: IsMounted: True Name: E:\ DeviceId: \\?\Volume{0005f107-0000-0000-0000-100000000000}\ Size: 1862 GB Available: 1520 GB Minimum: 320 MB IsOk: True Description: IsMounted: True =========================================================== Disk I/O Load Status: OK Description: All disks are currently not overloaded Disks: Name: 0 C: DiskBytesPersec: 437574 DiskReadBytesPersec: 173144 DiskReadsPersec: 9 DiskTransfersPersec: 36 DiskWriteBytesPersec: 264429 DiskWritesPersec: 27 PercentDiskReadTime: 6 PercentDiskTime: 28 PercentDiskWriteTime: 23 PercentIdleTime: 74 IsOk: True Name: 1 E: DiskBytesPersec: 0 DiskReadBytesPersec: 0 DiskReadsPersec: 0 DiskTransfersPersec: 0 DiskWriteBytesPersec: 0 DiskWritesPersec: 0 PercentDiskReadTime: 0 PercentDiskTime: 0 PercentDiskWriteTime: 0 PercentIdleTime: 97 IsOk: True =========================================================== NTFS Volume Access Control Configuration Status: OK Description: All volumes have correct access control configuration Volumes: Name: C:\ SystemHasAccess: True SystemSid: S-1-5-18 IsOk: True Description: "SYSTEM" account has "Full Control" access Name: E:\ SystemHasAccess: True SystemSid: S-1-5-18 IsOk: True Description: "SYSTEM" account has "Full Control" access =========================================================== Event Log Status: Warn Description: Some VSS-related errors or warnings occurred during the previous day Events: Timestamp: 2018/10/30 10:31:50 AM Type: Warning Source: VSS Message: A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur. . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {51b65980-2447-4878-9507-c4a25021a881} Command Line: C:\Windows\system32\vmms.exe Process ID: 4032 InstanceId: 8229 SearchLink: https://www.google.com/#newwindow=1&q=VSS+event+id+8229 Timestamp: 2018/10/30 10:18:48 AM Type: Warning Source: VSS Message: A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur. . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {51b65980-2447-4878-9507-c4a25021a881} Command Line: C:\Windows\system32\vmms.exe Process ID: 4032 InstanceId: 8229 SearchLink: https://www.google.com/#newwindow=1&q=VSS+event+id+8229 Timestamp: 2018/10/29 11:04:06 PM Type: Warning Source: VSS Message: A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error. If the backup process is retried, the error may not reoccur. . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. Operation: PrepareForSnapshot Event Context: Execution Context: Writer Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de} Writer Name: Microsoft Hyper-V VSS Writer Writer Instance ID: {51b65980-2447-4878-9507-c4a25021a881} Command Line: C:\Windows\system32\vmms.exe Process ID: 4032 InstanceId: 8229 SearchLink: https://www.google.com/#newwindow=1&q=VSS+event+id+82291.4KViews0likes0CommentsPlease update Article ID:100023134
I had an issue where my VSS Microsoft Exchange Writer was missing from theVSSADMIN.EXE LIST WRITERScommand. When viewing article ID:100023134, sectionVSSADMIN Commands, it reccomended restarting the Exchange Information Store service. This did not work. According to microsoft documentationstarting in Exchange 2013 the VSS writer was moved tothe Exchange Replication service. Therefore, please updateArticle ID:100023134 to reccomend restarting theExchange Replication service in addition to restarting the Exchange Information Store service.Solved1.4KViews0likes3CommentsHow unix backup works?
Hi, Can anybody please explain how unix file system backup works ? We have vss in windows filesystem backup which is needed as there are many files being used at the time of backup, what is the similar technology in unix? And for a unix incrimental backup, is netbackup using atime or mtime to backup changed files?927Views0likes2CommentsSystem State Components are missing from Backups
We have discovered some of the System State Components are missing from our Backups. While we are trying to restore the server's system state, the recovery is failing. System state backup along with Filesystem backups are getting completed without any issues. These components also remain unavailable while performing windows native backup. This issue is under investigation with Microsoft. At the same time, we would like to have one script created which will be used as bpstart_notify batch script which will check the following components before the backup gets triggered. If the components are missing, the backup will not get triggered. ASR - Automated system recovery Background intelligent transfer server COM + class registration database Performance counter registry system files task scheduler vss express writer store windows management instrumentation Can anyone help me with such a script? Or if there is an alternate option for this. Thanks in advance.844Views0likes2Comments