cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup Accelerator Error

rVikas
Level 5
Partner

Hi Team, 

I had enabled accelerator for few windows file server backups.

Backup used to be success when server size was small however from last few weeks, these servers size increased.

 

Below is specification:

Server has 4 drives :  out of which 2 drives size is about 2.8 TB, and 2 TB respectively.

Backup frequently fails with error 42 and 13 :

Error for code 13 :  not using change journal data for <F:\>: unable to initialize change journal usage <reason=record for requested start usn has been dropped from the change database (too many changes)>

Error for code 42: ERR - Snapshot Error while enumerating: GLOBALROOT\Device\HarddiskVolumeShadowCopy105\Profiles\

 

Seems this file servers experience high rate of change.

 

Kindly let me know how can i avoid these failures.

Do i need to disable accelerator then perform backup.

 

 

 

17 REPLIES 17

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Try with out "allow multiple datastreams" selected. so there will be only 1 job. Also, what the deduplication setting for the client, media server or client-side? Client-side sometimes causes status 13.
 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

This looks like Windows 'Change Journal' issue and not really Acceleraror.

Please see if this section: “Accelerator and the Windows change journal” on page 694 of NetBackup Administrator's Guide, Volume I  helps.
 

rVikas
Level 5
Partner

Hello Marianne,

Sorry for checking late, as i was out of offive for few days.

I tried backup without accelerator, though backup speed was fast, still received error with code 24/13 and backup failed.

Now a days, i am experienceing backkup failure for all file servers with drive size around TBs failing woth code 13 and 24.

Not able to figure out cause of issue as other file servers with less drives size is working fine.

On both the servers, i have installed Netbackup 7.6.0.1 client version and storage target is DataDomain.

Multistreaming is allowed.

Backup runs fine for some time a then after wrting around 1TB or more it fails it incomplete with code 13/24.

Kindly assist me.

 

sdo
Moderator
Moderator
Partner    VIP    Certified

It would seem to me that your file servers have a high rate of change.  This is borne out by the NTFS Change Journal running out of room to track all file changes:

Error for code 13 :  not using change journal data for <F:\>:
unable to initialize change journal usage
<reason=record for requested start usn has been dropped
from the change database (too many changes)>

Can I ask whether the NTFS volumes are quite full too?

If so, then there's probably not a lot you can do from NetBackup side.  Before doing anythign else, I would suggest monitoring both free disk space and VSS disk space consumption and which volumes (drive letters) are being used by which other volumes (drive letters) as VSS delta snap space... on the nodes with failing backup... do this during the actual backups, because this may give you the evidence that you need to present to management to prove that you may need to look at providing some additional storage space, not for files, and not for allplcations - but purely sitting there unused (most of the time) to provide nothing other than delta snap space for VSS.  i.e. you may need to give more space to VSS and to manually configure VSS so as not to leave VSS to make its own mind up about what it should do.

Remember:  NetBackup uses bpfis to call VSS and if VSS fails, then it is most definitely not a NetBackup software issue per-se - in fact, it is a OS and/or server administration and/or storage administration issue.  If VSS cannot do its job, then NetBackup Client cannot do its job.

.

Have you considered FlashBackup instead?  FlashBackup is a backup method which is sometimes a useful alternative (to walking the file system) for volumes which are large and quite full and contain millions of folders and files. 

rVikas
Level 5
Partner

Hi Sdo

While using accelerator i also received same error " 

<reason=record for requested start usn has been dropped
from the change database (too many changes)>

i disabled "use change journal option" and tried backing up without accelerator, still seeing failure with code 13/42.

I could not understand troubleshooting steps given by you. kindly simplify.

 

 

WVT
Level 4
Partner Accredited Certified
Your change journal is wrapping. do this fsutil usn queryjournal F: post the output

sdo
Moderator
Moderator
Partner    VIP    Certified

Also, please show us output from the problem client:

wmic logicaldisk list brief

vssadmin list providers

vssadmin list volumes

vssadmin list writers

vssadmin list shadowstorage

vssadmin list shadows

rVikas
Level 5
Partner

sure ..i will provide the output

one more behaviour , the clients are configured as cluster say client1A and client1B and are windows2012 servers

have configured both the clients in backup.

 

while backing up one client , its total size for one drive is 5 TB used space is 2 TB. however my backup stream for that drive is running more than3.5 TB dont know how..

 

have anyone came across this issue.

sdo
Moderator
Moderator
Partner    VIP    Certified

1) Is NTFS native de-duplication being used within the NTFS volumes?

see:

http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx

.

2) Is Enterprise Vault configured to archive the contents of the file system?  Maybe something is missing from your server configuration which is causing stubbed/offline files to be recalled?

.

3) Or maybe you have a different archiving solution, which is causing files to be fully recalled/retrieved from the archive store, i.e. the 'offline-ness' or 'stubby-ness' of the stubbed files is being ignored when NetBackup walks down through the file system looking for files to backup.

e.g.:

https://www-secure.symantec.com/connect/forums/ignoring-stub-files

 

rVikas
Level 5
Partner

we have DataDomain system as target device.

 

We dont have archviing solution , its a standard file system backup.

sdo
Moderator
Moderator
Partner    VIP    Certified

1) To be clear, I wasn't asking about your de-dupe target.  I was aksing whether you have a de-duped source.  I'll restate the question:  Is Windows NTFS native internal file-system de-duplication being used on the NTFS volumes/drive-letters?

rVikas
Level 5
Partner

Hello Sdo,

below is requested output:

 

C:\Windows\system32>fsutil usn queryjournal F:
Usn Journal ID   : 0x01d0fc9fe1f625cc
First Usn        : 0x000000243f4c0000
Next Usn         : 0x000000245f4eb730
Lowest Valid Usn : 0x0000000000000000
Max Usn          : 0x7fffffffffff0000
Maximum Size     : 0x0000000020000000
Allocation Delta : 0x0000000000040000
Minimum record version supported : 2
Maximum record version supported : 4
Write range tracking: Disabled


C:\Windows\system32>wmic logicaldisk list brief
DeviceID  DriveType  FreeSpace      ProviderName  Size           VolumeName

C:        3          388435955712                 449694396416   System

E:        3          7337857327104                8589798273024  PDW203PAFS002-V
ol01
F:        3          2383314305024                4397910192128  PDW203PAFS004-V
ol01
G:        3          5477908062208                5497421819904  PDW203PAFS002-V
ol02
Q:        3          241692672                    771616768      QuorumW

 

C:\Windows\system32>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line t
(C) Copyright 2001-2013 Microsoft Corp.

Provider name: 'Microsoft CSV Shadow Copy Helper Provider'
   Provider type: Software
   Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d}
   Version: 1.0.0.1

Provider name: 'Microsoft iSCSI Target VSS Hardware Provider'
   Provider type: Hardware
   Provider Id: {3f900f90-00e9-440e-873a-96ca5eb079e5}
   Version: 6.3.9600.16384

Provider name: 'Microsoft CSV Shadow Copy Provider'
   Provider type: Software
   Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
   Version: 1.0.0.1

Provider name: 'Microsoft File Share Shadow Copy provider'
   Provider type: Fileshare
   Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
   Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7


C:\Windows\system32>vssadmin list volumes
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Volume path: \\?\Volume{0425fd85-79b8-11e4-80e5-806e6f6e6963}\
    Volume name: \\?\Volume{0425fd85-79b8-11e4-80e5-806e6f6e6963}\
Volume path: E:\
    Volume name: \\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
Volume path: Q:\
    Volume name: \\?\Volume{a201ae40-434a-456f-bf5a-48c24bfd308e}\
Volume path: G:\
    Volume name: \\?\Volume{fbca6175-df00-4ba9-8e99-58ac71cd16ac}\
Volume path: F:\
    Volume name: \\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
Volume path: C:\
    Volume name: \\?\Volume{0425fd86-79b8-11e4-80e5-806e6f6e6963}\

 

C:\Windows\system32>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association
   For volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
   Shadow Copy Storage volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb98
39}\
   Used Shadow Copy Storage space: 15.9 GB (0%)
   Allocated Shadow Copy Storage space: 18.8 GB (0%)
   Maximum Shadow Copy Storage space: 500 GB (6%)

Shadow Copy Storage association
   For volume: (Q:)\\?\Volume{a201ae40-434a-456f-bf5a-48c24bfd308e}\
   Shadow Copy Storage volume: (Q:)\\?\Volume{a201ae40-434a-456f-bf5a-48c24bfd30
8e}\
   Used Shadow Copy Storage space: 24.0 MB (3%)
   Allocated Shadow Copy Storage space: 87.4 MB (11%)
   Maximum Shadow Copy Storage space: UNBOUNDED (100%)

Shadow Copy Storage association
   For volume: (G:)\\?\Volume{fbca6175-df00-4ba9-8e99-58ac71cd16ac}\
   Shadow Copy Storage volume: (G:)\\?\Volume{fbca6175-df00-4ba9-8e99-58ac71cd16
ac}\
   Used Shadow Copy Storage space: 13.2 GB (0%)
   Allocated Shadow Copy Storage space: 16.1 GB (0%)
   Maximum Shadow Copy Storage space: 512 GB (10%)

Shadow Copy Storage association
   For volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
   Shadow Copy Storage volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0
cd}\
   Used Shadow Copy Storage space: 221 GB (5%)
   Allocated Shadow Copy Storage space: 223 GB (5%)
   Maximum Shadow Copy Storage space: 750 GB (18%)

Shadow Copy Storage association
   For volume: (C:)\\?\Volume{0425fd86-79b8-11e4-80e5-806e6f6e6963}\
   Shadow Copy Storage volume: (C:)\\?\Volume{0425fd86-79b8-11e4-80e5-806e6f6e69
63}\
   Used Shadow Copy Storage space: 5.43 GB (1%)
   Allocated Shadow Copy Storage space: 8.25 GB (1%)
   Maximum Shadow Copy Storage space: UNBOUNDED (4102062249%)

C:\Windows\system32>vssadmin list shadows
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy688
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential


Contents of shadow copy set ID: {83a4671a-b4d7-4e5c-82ee-48c174971e66}
   Contained 1 shadow copies at creation time: 10/23/2015 7:00:13 AM
      Shadow Copy ID: {fff7f210-0099-4b58-a17a-e1fee9ab2e66}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy708
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {3a458d64-583e-4180-9eae-4deb5e70ccfa}
   Contained 1 shadow copies at creation time: 10/23/2015 12:00:12 PM
      Shadow Copy ID: {0cf2f7ad-1b03-4b3e-a656-7a0c0295c3a7}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy743
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {9dd5fcbf-9545-4752-a175-4b842ec30c76}
   Contained 5 shadow copies at creation time: 10/24/2015 10:53:31 PM
      Shadow Copy ID: {af5b2d92-a68c-45b9-81da-fc2a064e8cc0}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy780
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ApplicationRollback
         Attributes: Persistent, No auto release, Differential

      Shadow Copy ID: {d42cec66-27a3-42c1-aa11-d606e817514c}
         Original Volume: (Q:)\\?\Volume{a201ae40-434a-456f-bf5a-48c24bfd308e}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy777
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ApplicationRollback
         Attributes: Persistent, No auto release, Differential

      Shadow Copy ID: {c67571c0-23ad-4d72-91be-70da34a55166}
         Original Volume: (G:)\\?\Volume{fbca6175-df00-4ba9-8e99-58ac71cd16ac}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy778
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ApplicationRollback
         Attributes: Persistent, No auto release, Differential

      Shadow Copy ID: {668a14fc-ed5b-4c69-ab70-8aa65a624e32}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy779
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ApplicationRollback
         Attributes: Persistent, No auto release, Differential

      Shadow Copy ID: {86de38c4-185a-414a-87c6-2ea0de550549}
         Original Volume: (C:)\\?\Volume{0425fd86-79b8-11e4-80e5-806e6f6e6963}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy776
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ApplicationRollback
         Attributes: Persistent, No auto release, Differential

Contents of shadow copy set ID: {48a8e552-4879-466c-886c-62cc25da5f2d}
   Contained 1 shadow copies at creation time: 10/26/2015 7:00:11 AM
      Shadow Copy ID: {9df667a5-6b05-4b91-951a-79b8585a61f8}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy813
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {f6958a24-55e8-4dc9-88f8-488b7954d8c4}
   Contained 1 shadow copies at creation time: 10/26/2015 12:00:11 PM
      Shadow Copy ID: {f885326f-393e-4565-91ad-52134e01f899}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy815
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {6a6174e8-4147-4c7b-b848-082193696334}
   Contained 1 shadow copies at creation time: 10/27/2015 7:00:12 AM
      Shadow Copy ID: {5a3a1c75-151f-461e-b89a-135ea6475366}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy841
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {d36d74bf-e9e2-4b03-965d-c43d7e8477e8}
   Contained 1 shadow copies at creation time: 10/27/2015 12:00:56 PM
      Shadow Copy ID: {b039443a-b3d4-48f1-af39-71ac51126aec}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy844
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {8e55400f-588d-4d3b-8274-b053c72456c4}
   Contained 1 shadow copies at creation time: 10/28/2015 7:01:07 AM
      Shadow Copy ID: {c4f07ce7-ba93-46fd-b4a5-ab999e4b8b50}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy869
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {e339aa74-3722-4ff0-9930-4f31964adda7}
   Contained 1 shadow copies at creation time: 10/28/2015 12:00:12 PM
      Shadow Copy ID: {6bae1004-a411-44e3-b839-055c8c31acf0}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy870
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {33c751a7-6d88-4073-970d-5c700c5babb7}
   Contained 1 shadow copies at creation time: 10/29/2015 7:00:58 AM
      Shadow Copy ID: {3e67c370-f55f-424f-8b69-44e619b85640}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy891
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {42d19586-45a0-4d42-b81d-cc5b2bed190f}
   Contained 1 shadow copies at creation time: 10/29/2015 12:00:12 PM
      Shadow Copy ID: {42dc1d58-63d8-4fc1-8656-186a67c2218d}
         Original Volume: (E:)\\?\Volume{9e366a4b-2674-4bd4-ba1a-c4eb96cb9839}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy892
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {504e3f06-f462-489d-a55f-33a4c73b6cf7}
   Contained 6 shadow copies at creation time: 9/25/2015 9:01:51 PM
      Shadow Copy ID: {c9309c30-4eb4-4f8d-9d17-448398e2f9e2}
         Original Volume: (G:)\\?\Volume{fbca6175-df00-4ba9-8e99-58ac71cd16ac}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy103
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ApplicationRollback
         Attributes: Persistent, No auto release, Differential

Contents of shadow copy set ID: {9aaf28af-aa33-48ea-b9cd-21d922add4b0}
   Contained 5 shadow copies at creation time: 9/29/2015 4:32:18 AM
      Shadow Copy ID: {4ae9d35e-7cf2-43ff-846f-3e67dd4f0f87}
         Original Volume: (G:)\\?\Volume{fbca6175-df00-4ba9-8e99-58ac71cd16ac}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy116
         Originating Machine: PDW203PAFS001A.ca.bestbuy.com
         Service Machine: PDW203PAFS001A.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ApplicationRollback
         Attributes: Persistent, No auto release, Differential

Contents of shadow copy set ID: {aab6b3a0-7eba-44da-bfa3-ebc58a894c29}
   Contained 1 shadow copies at creation time: 10/26/2015 7:00:37 AM
      Shadow Copy ID: {926e618e-74ad-4c47-8ab9-f49191f97f1c}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy814
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {20464ab3-3498-4753-9dae-9fbfca8582ae}
   Contained 1 shadow copies at creation time: 10/26/2015 12:00:34 PM
      Shadow Copy ID: {74eaa624-71a9-4f34-a7e0-918e611b3e4f}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy816
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {bc444d19-a4ae-4db9-81da-2ff36751a779}
   Contained 1 shadow copies at creation time: 10/27/2015 7:00:35 AM
      Shadow Copy ID: {aeb3a3df-4c3e-4a74-905e-652e2ab5d084}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy842
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {57552ba6-0e5c-4e53-a7c6-418fd6c04850}
   Contained 1 shadow copies at creation time: 10/27/2015 12:00:06 PM
      Shadow Copy ID: {674a5014-90e9-4516-9eb2-3f78bae886e9}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy843
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {eb515261-d819-4e00-a2b0-f7c5070e8913}
   Contained 1 shadow copies at creation time: 10/28/2015 7:00:13 AM
      Shadow Copy ID: {65d2d62d-4cb6-43c3-a6e8-1ea26d23fe41}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy868
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {f3bb0b77-b473-45e5-97f4-82b84a651f0c}
   Contained 1 shadow copies at creation time: 10/28/2015 12:00:41 PM
      Shadow Copy ID: {98112b68-efa4-4897-94a6-202f7f227228}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy871
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {f97c9d17-765c-472d-9187-b746adfecaa5}
   Contained 1 shadow copies at creation time: 10/29/2015 7:00:09 AM
      Shadow Copy ID: {df1a7346-7a88-464d-8306-b81aec9a76e4}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy890
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential

Contents of shadow copy set ID: {2eface3b-36b0-4642-bb6b-4865c37246c8}
   Contained 1 shadow copies at creation time: 10/29/2015 12:00:44 PM
      Shadow Copy ID: {8df573ee-2d76-4140-b710-8ce5584d7e6f}
         Original Volume: (F:)\\?\Volume{72ba8ebd-03c9-4b7f-b132-c5b1e09ce0cd}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy893
         Originating Machine: PDW203PAFS001B.ca.bestbuy.com
         Service Machine: PDW203PAFS001B.ca.bestbuy.com
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessible
         Attributes: Persistent, Client-accessible, No auto release, No writers,
 Differential


C:\Windows\system32>.
'.' is not recognized as an internal or external command,
operable program or batch file.

sdo
Moderator
Moderator
Partner    VIP    Certified

My initial reaction is that this is a fairly complex situation:

- Clustered, using iSCSI.

- There appears to be some 'server admin' or 'application/service admins' scheduled VSS snapshots in place.

- Oddly the Quorum drive also has VSS configured - and I wouldn't expect to see this.

- The NetBackup Client at 7.6.0.1 is not good.

- The actual VSS output above is messed up - clearly you have not pasted everything, or you have pasted in between other text - thus it is unreliable.

- The USN of the F: drive is clearly wrapping:

First Usn        : 0x000000243f4c0000
Next Usn         : 0x000000245f4eb730

Difference:      : 0x000000002002b730

Maximum Size     : 0x0000000020000000

.

1) Were the vssadmin commands run whilst the backup was running?

2) Why is the NetBackup Client at v7.6.0.1 and not v7.6.0.4?

.

I think you're going to need to open a Veritas Support case to get to the bottom of this one.

.

Somethings to ask internally, i.e. discuss with your server admins - you shouldn't need to discuss these with Veritas Support - but finding out the answers to these will help you understand the nature and usage of the data:

- If you've disabled Accelerator and no longer want to use this feature... then how to disable MS Change Journal tracking?  (..unless perhaps NTFS Change Journal is required for another reason.)

- Can you upgrade the NetBackup Client to v7.6.0.4?

- How many resources groups exist on the cluster?  Are all of E: F: G: all in the same resource group?

- This probably isn't really relevant, but might have some clue as the server's intended behaviour... why is the C: drive so large?

- Why are the VSS snapshots of the G: drive timed differently to those of the E: and F: drives?

- Each of the large drives are using themselves for their shadow space.  This can lead to horrible performance and timeouts.  Why not rotate?  i.e. E use F, F use G, G use E?  Or if the volume of data and churn/change rate is going to increase notably, then perhaps look at providing additional LUNs for volumes to act purely as VSS snap space - e.g. add four additional LUNs or volumes (I,J,K,L), and configure VSS to use I for C, and J for E, and K for F, and L for G - to both ease IO load and spread IO load both during normal use and during backups.

sdo
Moderator
Moderator
Partner    VIP    Certified

You can expand the size of the change journal, see here:

https://technet.microsoft.com/en-gb/library/cc788042.aspx

.

Just out of curiosity (and it may help others):

3) What OS is the problem client running?

4) Which drive is NetBackup Client installed upon?

5) What is the client buffer size?   (found in:  client properties, windows client, client settings, buffer size)

.

But if it was me, I'd open an official support case.

rVikas
Level 5
Partner

Hello sdo,

Even sysadmins in our team are not able to figure out what causing failure as the server is not namaged by them and they are not supposed to access any drive as it is used to store users data.

Servers are running on  - Windows 2012

Two servers are there say client1A and client1B and are configured as cluster.

Netbackup version 7.6.0.1 is installed

Both the servers have : Communication buffer size = 128 kb / Raw partition read buffer size - 32 kb

I tried backup even withput accelerator , it is failing.

I have logged a case with symantec support.

 

 

 

 

rVikas
Level 5
Partner

On both nodes say client1A and client1B, in event viewer, i could see below VSS error:

Error on Cleint1A

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {d46bf321-fdba-4a35-8ec3-454df03bc86a}
   Writer Name: Sync Share Service Writer
   Writer Instance ID: {07bd3903-3982-4e3a-a1d6-8a8df0455dbf}


Error on Client1B:

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {d46bf321-fdba-4a35-8ec3-454df03bc86a}
   Writer Name: Sync Share Service Writer
   Writer Instance ID: {53f6f316-8bb9-4c06-8fa1-d1750ed50f53}

 

Also i had checked with server owner regarding configuration:

The cluster name is Client1 being hosted on servers Cleint1A and Client2B. There are 4 cluster resources on the cluster.

 

rVikas
Level 5
Partner

Hello Sdo,

One query, does successfull backup delete a VSS snapshot.