Forum Discussion

Pamela_S1's avatar
Pamela_S1
Level 3
12 years ago

VSS & Snapshots on virtual clients with Hyper-V on CSV

Hi everyone,

I need a better understanding on snapshots and writers on a Hyper-V backup scenario (CSV with 4 nodes - single policy). As far as i know.

  1. Netbackup is installed on the Hyper-V Hosts (nodes)
  2. Logs are created on the Hyper-V hosts.
  3. Network traffic is between Master-Media and Hyper-V Host

And according to the 7.5 Hyper-V Admin Guide - Phase 3:

The VSS Hyper-V writer quiesces the Windows virtual machine and creates the snapshot on the host volume. If the Hyper-V writer cannot quiesce the virtual machine, the virtual machine is placed in the Saved state before creation of the snapshot.

But, when you get an snapshot error (156) troubleshooting is on the virtual machine, according to this article:

http://www.symantec.com/business/support/index?page=content&id=HOWTO44941#v18522639

To be specific, i mean:

  1. Enough space on each volume of the virtual machine.
  2. NTFS  shadow storage on each/same volume of the virtual machine.

Those 2 issues must be checked on the virtual machine.

Last week i made a recomendation - (before being included in our Hyper-V backup) - to check if our new virtual machines had those configurations correct in order to avoid any CSV issues due the snapshots failing, and when our IT admin asked "why on the virtual machine, if everything happens in the host? makes no sense and nothing should be checked on the virtual machine, just the Hyper-V Hosts" 

I couldn´t answer that question. I know for sure the technote is valid, because i´ve troubleshooted 156 error many times and always is due to no space on the virtual machine or NTFS shadow storage on different volumes. On the other hand, when backup is running, you can only list writers on the Hyper-V hosts,

I´m pretty sure this is something conceptual related to the VSS writer, and how snapshots are done. Would be nice if someone could explain me what i´m missing to understand.

Thanks a lot!

 Pamela. 

7.5
Agent for Microsoft Hyper-V
Agents
Backup and Recovery
NetBackup
  • RLeon's avatar
    RLeon
    12 years ago

    Whether or not it is a VM, for a Windows OS it is normal and quite typical to set at least 10% or more free space on each volume for VSS snapshot operations. (You don't just set it, the free space has to be there too.)

    Having said that, when your policy type is "Hyper-V" and you encounter 156, it is referring to the Hyper-V host, so you troubleshoot on the Hyper-V host level.
    For other policy types such as "MS-Windows" where you are backing from from inside the VM the traditional way using the Nbu client, a 156 in this case refers to the VM, so you troubleshoot from inside the guest OS.
    If you use a Nbu client to backup from inside the VM, effectively you can pretty much take the whole Hyper-V thing out of the picture and just treat it as a physical server backup, and troubleshoot as such.

    Even if you are backing up the VM at the Hyper-V level, for the guest OS you should not disable the VSS snapshot space altogether (or give it very little volume space, like 1%).
    The reason for this is that the VSS provider inside the guest OS is also called during the process. It needs space to create snapshots too, but just long enough for the Hyper-V host's VSS provider to finish creating its, which is a very shot time.
    So short, in fact, changed blocks (if any) would not have enough time to really fill up the VSS space inside the guest OS, before it is deleted.
    (I know this contradicts with what I said earlier about not having snapshots inside the VMs, but digging deeper now.)

    A very good diagram here that explains this process from a source which should not be named, but applicable to NetBackup's and Backup Exec's Hyper-V level backups just the same, nonetheless.

    In short, have space for VSS snapshots on the Hyper-V host, and have space for VSS snapshots inside each VM, and you should have a lot less 156 problems.

     

    Since we are digging deeper, I should add that when I said:

    This snapshot is stored somewhere on the Hyper-V host

    It is also not always true. An exception to this is when you use alternate client offhost SAN volume level snapshot backups (redirect/copy-on-write or split-mirror type snapshots), using special VSS providers from the SAN vendor on the Hyper-V host, the snapshot is not stored on the Hyper-V host, but in the SAN and gets mounted by the alternate client (usually a Nbu media server) for offhost backups.

    Also, when you refer to the diagram linked, you will see that the VSS writers on the Hyper-V host is also called to quiesce the .vhd files, which they then, in turn, call the VSS writers inside the guest OSes (through the Hyper-V Integration Services)... Then, you know the rest. I wrote something about snapshots in general, it's long and crudely written, so only if interested.

4 Replies

Sort By
  • RLeon's avatar
    RLeon
    Level 6
    12 years ago

    He is right in some way, except that the VMs must all have the Hyper-V Integration Services installed, otherwise the Hyper-V host cannot trigger VSS writers inside them.
    I should add that the VSS writers inside individual VMs should all be checked and verified to be "healthy", because they are needed during backups.

    1. During a backup, the VSS writers inside the VMs quiesce them.
    2. Then the Hyper-V host's VSS provider creates a snapshot of them.
    3. NetBackup backs up this snapshot from the Hyper-V host.

    And now to describe the exact same steps from a different point of view:
    1. To make sure all I/O are flushed and files quiescent and consistent before a backup, the Hyper-V host must do something to pause all these I/O and processing on all these .vhd files.
    It calls the Hyper-V Integration Services, then all I/O and processing on those .vhd files stop, very briefly.
    2. At this very moment, the Hyper-V host creates a snapshot of the CSV volume where these .vhd files are located. This snapshot is stored somewhere on the Hyper-V host (not inside individual volumes of individual VMs). After the snapshot is successfully created, the Hyper-V Integration Services are called again, and all .vhd activities resume.
    3. NetBackup backups this snapshot from the Hyper-V host. After the backup, the snapshot is deleted.

    It is important to point out that:
    a. VSS writers do not create snapshots, the VSS provider does.
    b. Snapshots are not created inside the individual VMs themselves. Their volume space is not used to store the snapshot for the backup. (Your 156 concern.)
    c. How NetBackup backs up Hyper-V VMs has nothing to do with Hyper-V's very own .avhd snapshots. A .avhd snapshot is a snapshot of a single VM. NetBackup uses volume based snapshots. I talked a bit about it here. And here is an excellent article that covers the rest.

  • Pamela_S1's avatar
    Pamela_S1
    Level 3
    12 years ago

    Hi,

    Makes sense and i understand better now that snapshots don´t use any virtual machine space, all happens in the host. But then why troubleshooting of 156 error is about having enough space on the virtual machine?

    What i need to understand is the reason why not having enough space on a volume on a virtual machine or having a bad NTFS configuration can make the snapshot in the host fail.

    Thanks a lot,

     

     

     

  • RLeon's avatar
    RLeon
    Level 6
    12 years ago

    Whether or not it is a VM, for a Windows OS it is normal and quite typical to set at least 10% or more free space on each volume for VSS snapshot operations. (You don't just set it, the free space has to be there too.)

    Having said that, when your policy type is "Hyper-V" and you encounter 156, it is referring to the Hyper-V host, so you troubleshoot on the Hyper-V host level.
    For other policy types such as "MS-Windows" where you are backing from from inside the VM the traditional way using the Nbu client, a 156 in this case refers to the VM, so you troubleshoot from inside the guest OS.
    If you use a Nbu client to backup from inside the VM, effectively you can pretty much take the whole Hyper-V thing out of the picture and just treat it as a physical server backup, and troubleshoot as such.

    Even if you are backing up the VM at the Hyper-V level, for the guest OS you should not disable the VSS snapshot space altogether (or give it very little volume space, like 1%).
    The reason for this is that the VSS provider inside the guest OS is also called during the process. It needs space to create snapshots too, but just long enough for the Hyper-V host's VSS provider to finish creating its, which is a very shot time.
    So short, in fact, changed blocks (if any) would not have enough time to really fill up the VSS space inside the guest OS, before it is deleted.
    (I know this contradicts with what I said earlier about not having snapshots inside the VMs, but digging deeper now.)

    A very good diagram here that explains this process from a source which should not be named, but applicable to NetBackup's and Backup Exec's Hyper-V level backups just the same, nonetheless.

    In short, have space for VSS snapshots on the Hyper-V host, and have space for VSS snapshots inside each VM, and you should have a lot less 156 problems.

     

    Since we are digging deeper, I should add that when I said:

    This snapshot is stored somewhere on the Hyper-V host

    It is also not always true. An exception to this is when you use alternate client offhost SAN volume level snapshot backups (redirect/copy-on-write or split-mirror type snapshots), using special VSS providers from the SAN vendor on the Hyper-V host, the snapshot is not stored on the Hyper-V host, but in the SAN and gets mounted by the alternate client (usually a Nbu media server) for offhost backups.

    Also, when you refer to the diagram linked, you will see that the VSS writers on the Hyper-V host is also called to quiesce the .vhd files, which they then, in turn, call the VSS writers inside the guest OSes (through the Hyper-V Integration Services)... Then, you know the rest. I wrote something about snapshots in general, it's long and crudely written, so only if interested.