Shadow copy backup fails (156) on both nodes of a 2008 R2 SQL cluster

[Alun]

I have a policy that backs up both nodes of a SQL cluster in an active / active configuration, all of the visible drives on the active nodes are backed up successfully but the "Shadow Copy Components" job fails on each node with a snapshot error (156).

Here is the sanitised output from the job log and I've attached the output from the bpbkar log on the client in full.

16/10/2014 03:47:54 - Info nbjm(pid=2492) starting backup job (jobid=278247) for client SERVERNAMEHERE, policy POLICYNAMEHERE, schedule Diff-Daily
16/10/2014 03:47:54 - estimated 0 Kbytes needed
16/10/2014 03:47:54 - Info nbjm(pid=2492) started backup (backupid=SERVERNAMEHERE_1413427674) job for client SERVERNAMEHERE, policy POLICYNAMEHERE, schedule Diff-Daily on storage unit DISKSTORAGEUNITHERE
16/10/2014 03:47:56 - started process bpbrm (1572)
16/10/2014 03:47:57 - Info bpbrm(pid=1572) SERVERNAMEHERE is the host to backup data from
16/10/2014 03:47:57 - Info bpbrm(pid=1572) reading file list from client
16/10/2014 03:47:58 - connecting
16/10/2014 03:48:01 - Info bpbrm(pid=1572) starting bpbkar32 on client
16/10/2014 03:48:01 - connected; connect time: 00:00:03
16/10/2014 03:48:04 - Info bpbkar32(pid=13136) Backup started
16/10/2014 03:48:04 - Info bptm(pid=12036) start
16/10/2014 03:48:04 - Info bptm(pid=12036) using 262144 data buffer size
16/10/2014 03:48:04 - Info bptm(pid=12036) setting receive network buffer to 1049600 bytes
16/10/2014 03:48:04 - Info bptm(pid=12036) using 256 data buffers
16/10/2014 03:48:08 - Info bptm(pid=12036) start backup
16/10/2014 03:48:09 - Info bptm(pid=12036) backup child process is pid 12112.5528
16/10/2014 03:48:09 - Info bptm(pid=12112) start
16/10/2014 03:48:09 - begin writing
16/10/2014 03:48:50 - Error bpbrm(pid=1572) from client SERVERNAMEHERE: ERR - Volume resolution error for (System State:\System Files\System Files)
16/10/2014 03:48:58 - Error bpbrm(pid=1572) could not send server status message
16/10/2014 03:49:01 - Info bpbkar32(pid=13136) done. status: 156: snapshot error encountered
16/10/2014 03:49:01 - end writing; write time: 00:00:52
snapshot error encountered(156)

Is there anything in the log that suggests the cause and can you point me towards a solution?

Thanks,

Alun.

[sksujeet]

Which version of NBU you are running?

Attached log shows:
0xE0000352:The specified volume does not exist. Recreate the volume, and then try the job again.

and

ERR - Volume resolution error for (System State:\System Files\System Files)

Check the state of your VSS writers (vssadmin list writers) - if everything is stable its fine, if not reboot the server.

Check this article http://www.symantec.com/docs/TECH176793

[Alun]

Master  / Media servers = 7.5.0.7

Client = 7.1.0.3

The error also occurred when the Master / Media servers were running 7.1.0.3.

As I mentioned above, these are nodes in an active / active SQL cluster.

We see the same issue on all four nodes across two SQL clusters running on Windows Server 2008 R2 SP1, all four also show the 'WMI Writer' with a state of [5] Waiting for completion.

I've just reviewed another four nodes on another two clusters and they're fine, all writers are "stable, no error"

[sksujeet]

Have you tried that TN in prvious thread

I would suggest upgrading the client as first thing on all the cluter nodes as couple of those issue resolved in latest updates.

When was the last time you had these servers rebooted, try to reboot them once and see if it resolves.

[Marianne · Accepted solution]

Use 'w2koption -backup -ignore_unresolved_volumes  ' to add cluster volumes not present on each of the nodes.

See http://www.symantec.com/docs/TECH199089 

[Michael_G_Ander]

In addition to Mariannes advise there is also the EXCLUDEBINARYPATHS registry key if there is services which has the binaries on the shared drives. Have seen this for some SQL monitoring tools installed on clusters