Forum Discussion

NBUserLookup's avatar
NBUserLookup
Level 3
10 years ago

Hyper-V cluster some VM's error with status code 156

We have a Hyper-V cluster with 3 machine nodes, using a CSV connecting to an EMC VNX on two luns.   We have two cluster volumes coming from 2 Luns off the VNX. We have two policies one for vol 3 and one for vol 4 vdh's. Each of the policies limits to 1 job per policy.   All of the vhd's on vol 3 backup with no issues.  Some, but not all of the vhd's on vol 4 fail with a status code of 156.  Of the systems that fail, they reside across all three hosts.

 

The bpfis log shows EXIT STATUS 4207: Could not fetch snapshot metadata or state file.

 

We had this happen a week ago and cleaned it up but it's back now and need to figure out why this keeps happening.  Any suggestions would be greatly appreciated.

7.6
Backup and Recovery
Managing Backup Devices
NetBackup
Virtualization
Windows Server (2003-2008)
  • sdo's avatar
    sdo
    10 years ago

    The latest (version 8) EEB guide for v7.6:

    https://support.symantec.com/en_US/article.DOC6085.html

    ...says the following on page 161, in the v7.6 GA section, and because this section (v7.6 GA) immediately follows the v7.6.0.2 section, I think that it can therefore be construed as being aka v7.6.0.1, and due to the limited text/detail, implies that this issue was resolved in v7.6.0.2:

    3002231 The bpfis process hangs during a Hyper-V backup due to a stale snapshot. 

    ...so, if you have Symantec Support contract, it might be worth placing a call to see whether ETrack 3002231 describes symptoms similar to those that you are seeing.

    FYI - I can't find any references to status 4207 in the EEB guide.

  • mph999's avatar
    mph999
    10 years ago

    EEB 3002231
    HyperV backups hang, bpfis is timing out during bprd_read_text_file

    raised for NBU 7.5.0.1, fixed in
    7.5.0.6, and (7.6) (7.6.1) (7.7)

18 Replies

Sort By
  • GulzarShaikhAUS's avatar
    GulzarShaikhAUS
    Level 6
    10 years ago

    I need following

    Netbackup version

    Hyperv version 

    command  - bppllist <hyper-v-polcy-name> -L

    detailed status of the snapshot job

  • NBUserLookup's avatar
    NBUserLookup
    Level 3
    10 years ago

    Sorry for the late reply, apparently the login for symantec was having problems all day yesterday.

     

    Netbackup version 7.6.0.1

    Hyperv version Windows 2008 R2 SP1

    I don't think it is the policy since some of the jobs launched by the policy work fine and others fail.  The status of the snapshot job is status code of 156.

    To recover I moved all the virtuals off one host rebooted it then moved them off a second host and rebooted it.  The interesting thing is that I did NOT have to move anything off the third host.  Even though some of the ones were sitting on the third host rebooting the first two cleared all the issues.

    What I'm looking to figure out is why this keeps happening.

     

     

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    See page 40 of:  Support for NetBackup 7.x in virtual environments:

    https://support.symantec.com/en_US/article.TECH127089.html

    ...and note the '*' against the versions... and the note re:

    *Backup of virtual machines within Cluster Shared Volumes (CSV) is supported. In Hyper-V Server 2008 R2, CSV has a limitation that requires serialized backups. In Hyper-V Server 2012, CSV supports concurrent backups without that limitation. Consequently, the CSV timeout setting in the NetBackup Hyper-V policy is only used in Hyper-V Server 2008 R2 deployments. 

    ...which means that you can only run one backup at a time in your CSV based Windows 2008 R2 Hyper-V estate.  If you are running multiple concurrent backups - you will have problems.

  • NBUserLookup's avatar
    NBUserLookup
    Level 3
    10 years ago

    We are not running multiple concurrent backups, each policy is limited to 1 job per policy at a time.

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    Even with single stream policies... it is still possible that multiple backup jobs try to run against the same CSV volume.  In which case you'll have problems.  Are you absolutely 100% certain that your combination of "CSV volume PLUS backup policy PLUS client list" leads to single active backup per CSV volume - i.e. that policy "A" is not backing-up a VM in CSV "B" ?

  • NBUserLookup's avatar
    NBUserLookup
    Level 3
    10 years ago

    Correct, when we first started having the problem we created two new policies verifying step by step that all the vhd's on CSV 3 were only being run only against that policy and the same for CSV 4.

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    What mix of O/S versions are the virtual machines?

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    The status codes reference manual for NetBackup 7.6:

    https://support.symantec.com/en_US/article.DOC6471.html

    ...for status 4207 appears to imply flakey comms between the client (i.e. the Hyper-V host) and the master server.  Perhaps run a ping -t test overnight, from the master server, to each of the clients (Hyper-V hosts) and then check the log files in the morning?

  • NBUserLookup's avatar
    NBUserLookup
    Level 3
    10 years ago

    The mix is Linux, Window 2008, Windows 2008 r2 sp1 and Windows 2012 r2.  I've cleaned it up again for right now all of them ran good last night.  I'm just going to monitor it for right now and see if I can see anything is causing it to reoccur.

  • sdo's avatar
    sdo
    Moderator
    10 years ago

    Have you seen this:

    https://support.symantec.com/en_US/article.TECH211023.html

    ...re: NetBackup with Hyper-V fails to perform snapshot / online backup of Windows 2012 virtual client.