Windows 2008 Server backup fails with snapshot failure (status 156)

[WayneLackey]

I have two Windows 2008 servers that are getting snapshot errors when they try to back up. This problem has persisted for at least the last couple of weeks. Environment is as follows:

Master: SunOS 5.10 UNIX running NBU Server 7.1.0.3

Client (both): Windows Server 2008 running NBU client 7.0

---

Job detail from most recent failure from one of the clients - both exhibit similar errors:

08/14/2012 18:29:31 - Info nbjm (pid=8836) starting backup job (jobid=2471022) for client bkusxtvwsabis01, policy HSPD12-Win-Test, schedule Diff-Incr
08/14/2012 18:29:31 - Info nbjm (pid=8836) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2471022, request id:{841F3EB8-E65F-11E1-9992-1CC1DE1EF662})
08/14/2012 18:29:31 - requesting resource bkshelby-hcart-robot-tld-1
08/14/2012 18:29:31 - requesting resource bkferrari.NBU_CLIENT.MAXJOBS.bkusxtvwsabis01
08/14/2012 18:29:31 - requesting resource bkferrari.NBU_POLICY.MAXJOBS.HSPD12-Win-Test
08/14/2012 18:29:36 - awaiting resource bkshelby-hcart-robot-tld-1. Maximum job count has been reached for the storage unit.
08/14/2012 18:33:31 - granted resource  bkferrari.NBU_CLIENT.MAXJOBS.bkusxtvwsabis01
08/14/2012 18:33:31 - granted resource  bkferrari.NBU_POLICY.MAXJOBS.HSPD12-Win-Test
08/14/2012 18:33:31 - granted resource  HS4044
08/14/2012 18:33:31 - granted resource  HP.ULTRIUM4-SCSI.000
08/14/2012 18:33:31 - granted resource  bkshelby-hcart-robot-tld-1
08/14/2012 18:33:31 - estimated 0 kbytes needed
08/14/2012 18:33:31 - Info nbjm (pid=8836) started backup job for client bkusxtvwsabis01, policy HSPD12-Win-Test, schedule Diff-Incr on storage unit bkshelby-hcart-robot-tld-1
08/14/2012 18:33:31 - started process bpbrm (pid=19262)
08/14/2012 18:33:45 - connecting
08/14/2012 18:33:55 - connected; connect time: 0:00:00
08/14/2012 18:33:58 - mounted HS4044
08/14/2012 18:33:58 - positioning HS4044 to file 17
08/14/2012 18:34:00 - positioned HS4044; position time: 0:00:02
08/14/2012 18:34:00 - begin writing
08/14/2012 18:37:21 - Error bpbrm (pid=19262) from client bkusxtvwsabis01: ERR - Volume resolution error for (System State:\System Files\System Files)
08/14/2012 18:37:23 - Error bptm (pid=19266) media manager terminated by parent process
08/14/2012 18:37:44 - Error bpbrm (pid=19262) could not send server status message
08/14/2012 18:37:51 - end writing; write time: 0:03:51
snapshot error encountered  (156)

---

I did a search for the error regarding the resolution error for the System State, and found http://www.symantec.com/docs/TECH164712 . I have attached the bpbkar log from a few days ago, which seems to jive with what this tech note is talking about. It seems to me like this is a likely candidate for resolution, but I wanted to see if anyone knew of any other possible fix actions, since it is such a pain to get any patching approved in this customer's environment.

Thanks,

Wayne

[Zahid_Haseeb]

I think first you should keep the NBU Server and Client on same Netbackup version level. This prevents many problems

[Marianne]

Are these machines perhaps nodes in a cluster? If so, see http://www.symantec.com/docs/TECH128805

Solution is to upgrade to 7.1, then patch to 7.1.0.3 (TN says 7.1.0.1, but best to install latest patch).

After patch has been installed, the 'w2koption' can be executed to ignore missing volumes.

PS: You forgot to attach bpbkar log.

[WayneLackey]

Ugh, details, details....   See attached bpbkar log...

These servers are not part of a cluster.

Thanks,

Wayne

[Tim_Hansen]

Does your system contain a hidden 100MB system partition at the root of the boot disk (commonly created for recovery purposes by the manufacturer)?

[Tim_Hansen]

Also check the following technotes:

http://www.symantec.com/docs/TECH47808

http://www.symantec.com/docs/TECH70168

[WayneLackey]

No idea, I'll have to check with the server admin to see.

[Marianne]

The errors in bpbkar are similar to thisTN: http://www.symantec.com/docs/TECH164712

Problem is solved with MS hotfix  http://support.microsoft.com/kb/2182466

[Zahid_Haseeb]

@ Wayne I think first you should keep your clients/NBU on same patch level. It will definately resolve the issue as I mentioned above. I hope

[Ramiro-Magan]

Wayne, I had several problems with snapshots in the recent past, was fighting with it for 2 weeks.

I was on 7.1.0.2. You know what did the trick? 7.1.0.4 Master, Media, and Clients.

[bbahnmiller]

Wayne,

  I have to agree with Ramiro. We are running 7.1.0.3 Master/Media servers. I won't even look at a client issue if it is Windows running the 7.1 vanilla install. Patching the client to 7.1.0.3 solved 99% of our 156 Windows errors.

  Least effort, most results.

   Bryan

[WayneLackey · Accepted solution]

Turns out these servers have an application installed on them that put registry entries for Windows drives with slashes instead of backslashes. Once the SA corrected the paths with backslashes, this corrected the problem and I am getting status 0 for these backups now.