HyperV backups failing with error 156

[benjayhal]

Hello All,

NBU version - 7.6.0.3

Hyper-v host - windows 2012 R2

HyperV client VM's - all Windows 2012 Server OS

We are geting regular failures of hyper-V VM backups on netbackup. The only solution usually is to live migrate the failing virtual machine to another OS cluster node or to reboot it.

In both cases, this workaround is not very feasible all the time.

Error message received in Netbackup for failing backup's

17/12/2015 11:32:49 - Info bpfis(pid=19892) done. status: 156: snapshot error encountered       
17/12/2015 11:32:49 - end writing
Status 156
17/12/2015 11:32:49 - end Parent Job; elapsed time: 0:05:21
17/12/2015 11:32:49 - begin Hyper-V, Stop On Error
Status 0
17/12/2015 11:32:49 - end Hyper-V, Stop On Error; elapsed time: 0:00:00
17/12/2015 11:32:49 - begin Hyper-V, Delete Snapshot
17/12/2015 11:32:51 - started process bpbrm (9704)
17/12/2015 11:33:32 - Info bpbrm(pid=9704) Starting delete snapshot processing         
17/12/2015 11:33:36 - Info bpfis(pid=12864) Backup started           
17/12/2015 11:33:36 - Critical bpbrm(pid=9704) from client VM_Name: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.VM_Name_1450351416.1.0      
17/12/2015 11:33:37 - Info bpfis(pid=12864) done. status: 4207          
17/12/2015 11:33:37 - end Hyper-V, Delete Snapshot; elapsed time: 0:00:48
17/12/2015 11:33:37 - Info bpfis(pid=12864) done. status: 4207: Could not fetch snapshot metadata or state files  
17/12/2015 11:33:37 - end writing
Status 4207
17/12/2015 11:33:37 - end operation
17/12/2015 11:33:37 - begin Hyper-V, End Notify Script
17/12/2015 11:33:37 - Info RUNCMD(pid=5184) started            
17/12/2015 11:33:37 - Info RUNCMD(pid=5184) exiting with status: 0         
Status 0
17/12/2015 11:33:37 - end Hyper-V, End Notify Script; elapsed time: 0:00:00
Status 156
17/12/2015 11:33:37 - end operation
snapshot error encountered(156)

This is very random and keeps happening to different VM's.

The CSV on which the VM's reside are on HP EVA san.

There are 2 jobs different jobs in NBU which run the backup's. each backup job has 19 VM's in it.

We have tested with 1 VM per CSV per NBU job, the same issue occurs.

Please let me know what other log files are required to take a look. Please specify if from nbu server, hyper V host or from hyper V client.

Any help in this matter will be very much appreciated.

=========

Thanks for the responses to the post.

The issue seems to be resolved now.

We contacted microsoft and they suggested to install various OS patches.

Mainly, the patches were for ntfs, Volsnap and partmgr alongwith a few others on the hyper Hosts.

On the VM's we updated to the latest patches for the same mentioned above and also latest hyperV integration services..

This did reduce the number of failures but not resolve it completely.

WE also reduced the number of concurrent VM backups from 4 to 2. This has drastically reduced the number of failures

[benjayhal]

Hello All,

Can anyone please check and provide any possible solutions to this.

thanks

[Mark_Solutions · Accepted solution]

There can be lots of reasons .. this link is pretty handy http://www.veritas.com/docs/000067928

bpfis logs on the media server are usually where you start but take  a look at the link

[sdo · Accepted solution]

The NetBackup v7.6 EEB Guide shows, on page 14, that v7.6.1.2 includes a fix for:

ETrack: 3748081

Re: Hyper-V Backups on Windows 2012 R2 can exhaust system memory due to aggressive cache memory management. 

.

FYI - the v7.6 EEB guide is here:

http://www.veritas.com/docs/000003552

.

I've had a search through for other Hyper-V related issues, but any references to other known issues for Hyper-V backups are listed in the sections for other lower versions, and so, the only "known" (at the time) issue which may (or may not) be pertinent to you, because you are running v7.6.0.3, is the issue listed above which is documented as being fixed/resolved in v7.6.1.2.

HTH.