Shadow Copy Components backup of DFSR data fails on Windows 2008 R2 Netbackup Client.

[AB_Chaudhry]

Environment Details:

Master Server: AIX 6.1, NetBackup 7.5.0.5

Client: Windows 2008 R2, Clustered, NetBackup 7.5.0.5

Issue:

Backup of DFSR data from Windows 2008 R2 client failed with error 69.

"Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - EXIT STATUS 69: invalid filelist specification"

"vssadmin list writers" from client report all writers as "Stable" and "No Error".

No application events related to VSS on Cleint

Logs from Client report that:

"WRN - Can't open file: Shadow copy components ..... Win32 3: The system can not find the path specified."

The backup has been tested with dedicated policy and only DFSR folder to avoid multiple streams but result was same failure.

The issue is more related to VSS and Microsoft, but any ideas to proceed will be helpful and appreciated.  

[Dyneshia]

What is in the backup selection for the policy ?

netbackup\bin\admincmd\bppllist <policy name> -L
 

[Mark_Solutions]

I realise that you say this is dfsr related but the error often relates to a service or other Shadow Copy Component that is in the registry but does not actually exist

It may be worth increasing the logging on the client to get a full bpbkar log to see exactly where it fails

Is there anything else on the event logs at around the time of the failure?

As Dyneshia asks - what exactly are you backing up here?

[D_Flood]

Are you trying to use client side dedup?  If so then try the backup with "disable client side dedup" checked.  DFS and Client Side Dedup just don't get along when you have large amounts of data.

[AB_Chaudhry]

Hi Dyneshi,

BPPLLIST output is provided below. Backup selection is Shadow Copy Components only. 

Policy Name:       FilePrint-dedup-Shadow-Test

Options:           0x0

template:          FALSE

audit_reason:         ?

Names:             (none)

Policy Type:       MS-Windows (13)

Active:            yes

Effective date:    04/24/2013 10:50:38

Client Compress:   no

Follow NFS Mnts:   no

Backup netwrk drvs:no

Collect TIR info:      no

Mult. Data Stream: no

Perform Snapshot Backup:   no

Snapshot Method:           (none)

Snapshot Method Arguments: (none)

Perform Offhost Backup:    no

Backup Copy:               0

Use Data Mover:            no

Data Mover Type:           -1

Use Alternate Client:      no

Alternate Client Name:     (none)

Use Virtual Machine:      0

Hyper-V Server Name:     (none)

Enable Instant Recovery:   no

Policy Priority:   0

Max Jobs/Policy:   Unlimited

Disaster Recovery: 0

Collect BMR Info:  no

Keyword:           (none specified)

Data Classification:       -

Residence is Storage Lifecycle Policy:    no

Client Encrypt:    no

Checkpoint:        no

Residence:         kii-Eii-LTO4

Volume Pool:       Production_Pool

Server Group:      *ANY*

Granular Restore Info:  no

Exchange Source attributes:              no

Exchange 2010 Preferred Server: (none defined)

Application Discovery:      no

Discovery Lifetime:      28800 seconds

ASC Application and attributes: (none defined)

Generation:      23

Ignore Client Direct:  no

Enable Metadata Indexing:  no

Index server name:  NULL

Use Accelerator:  no

Client/HW/OS/Pri/DMI:  dii00730-fpvp11 Windows-x64 Windows2008 0 1 0 0 ?

Include:           Shadow Copy Components:\

Schedule:              Full

  Type:                FULL (0)

  Frequency:           7 day(s) (604800 seconds)

  Maximum MPX:         1

  Synthetic:           0

  Checksum Change Detection: 0

  PFI Recovery:        0

  Retention Level:     3 (1 month)

  u-wind/o/d:          0 0

  Incr Type:           DELTA (0)

  Alt Read Host:       (none defined)

  Max Frag Size:       0 MB

  Number Copies:       1

  Fail on Error:       0

  Residence:           (specific storage unit not required)

  Volume Pool:         (same as policy volume pool)

  Server Group:        (same as specified for policy)

  Residence is Storage Lifecycle Policy:         0

  Schedule indexing:     0

  Daily Windows:

   Day         Open       Close       W-Open     W-Close

   Sunday      000:00:00  000:00:00

   Monday      000:00:00  000:00:00

   Tuesday     000:00:00  000:00:00

   Wednesday   000:00:00  000:00:00

   Thursday    000:00:00  000:00:00

   Friday      000:00:00  000:00:00

   Saturday    000:00:00  000:00:00

[AB_Chaudhry]

Hi Mark,

Actually we need to get backup of DFSR data but we are facing problem with Shadow Copy Components across the server for everything. So somehow I agree with you that the suspect is Shadow Copy Components backup. 
Logs from server shows following message for all files and folders and logs are huge in size:

"WRN - Can't open file: Shadow copy components ...FULL PATH OF FILE OR FOLDER.. Win32 3: The system can not find the path specified."

Event log is not reporting any thing.

Policy details are pasted above as discussed with Mr. Dyneshia.

[AB_Chaudhry]

Hi D.Flood.

Currently we are testing without dedupe and we are unable to get Shadow Copy Components from system. In future we will try to get it dedupe if we get successful backup.

[AB_Chaudhry]

Hi Everyone,

We have following two VSS providers available on system:

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

Provider name: 'Data ONTAP VSS Hardware Provider'
   Provider type: Hardware
   Provider Id: {ddd3d232-a96f-4ac5-8f7b-250fd91fd102}
   Version: 6. 4. 1. 5319

I can change the VSS provider with following command from Auto to System or Hardware

w2koption -backup -snapshotprovidertype <0, 1, 2, 3>

but how can I verify or check wich VSS Provider is currently selected for backups of netbackup?

w2koption -display

shows only options available. does not show selected option.

[Mark_Solutions]

I would change it to system and try that - Auto and Hardware will use the ONTAP provider which not setup will cause a failure

Try system to at least see what happens

The only other way around this is to use bpstart_notify to take down the DFSR and then back the files up via their normal drive letter, then bpend_notify to fire it back up once the backup has finished

[D_Flood]

Good luck!  After months of testing with Support I've come to the decision that Client-Side and DFS don't mix when you have large amounts of DFS data. 

Support did suggest that I could turn DFS off during the backups (then the files get backed up as normal, not as Shadow Copies) but my Windows Admin won't let me do that and there are all the dire warnings in that particular KB about log files wrapping around and changes getting lost.

I've got one DFS backup that VSS takes 10+ hours to make the copy (approx 150G of data) and there's absolutely no way to tell client-side to wait that long.  So I have to run those backups without client-side and send them to a local Media Server Dedup Pool rather than over the WAN.

So I'm in the process of convincing the Power$$$$ that be that I need to set up MSDP's at each of our remote sites so I can back up the data that they're asking me to back up.