VM Incremental issue NBU7

[swest092]

We have 11 VMware virtual file servers that used to have the agents installed on them.  After the upgrade to NBU7 we saw an increased performance while testing with flashbackup-windows so we switched them to use flashbackup-windows instead of Windows-NT.  The full backup jobs run fine on them but the incremental fails with an error 11.  We have tried to just send them to tape and just send them to disk with the same results.  We also used the same process for the other 150 VMs and they all work fine.  Below is the error we receive.

4/12/2010 11:34:08 AM - estimated 0 Kbytes needed
4/12/2010 11:34:09 AM - started process bpbrm (6328)
4/12/2010 11:34:10 AM - connecting
4/12/2010 11:34:11 AM - connected; connect time: 00:00:01
4/12/2010 11:34:19 AM - begin writing
4/12/2010 11:35:18 AM - Error bpbrm(pid=6328) from client stfs1: ERR - Unable to retrieve volumes from virtual machine, error = 4109
4/12/2010 11:35:40 AM - Error bpbrm(pid=6328) from client stfs1: ERR - FIML_start_vm for VMWare on C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.stfs1_1271090017.0.NBU_DATA.xml failed VFM error = 4109
4/12/2010 11:35:48 AM - Error bpbrm(pid=6328) could not send server status message      
4/12/2010 11:35:49 AM - end writing; write time: 00:01:30
system call failed(11)

Anyone have any ideas that I can try out?

[Kenneth_Hansen]

A quick search game me this:
https://www-secure.symantec.com/connect/forums/netbackup-651a-amp-vcb11

Can you tell a bit more of how you set up you backup's.
VCB? Vstorage API? Mapped Full backups? And so on.

Do you get any error displayed in Virtual Center ( Esx and VCenter version?? )?

[Kerry_LeRoy]

I believe that Flashbackup does not allow Incremental backups. It is taking a snapshot of the targetted drive/system and backing up using that snapshot.  We use this on servers that have millions of small files because NetBackup takes too long to parse the files with a normal Windows-NT backup type.

[swest092]

Hi Kenneth,

Thanks for your reply.

We are currently running NBU7 and our VM backups are set to go to disk for the full and cumulative backup.  The 2 schedules are in the same policy.  We have it set to do a mapped full vm backup and are not using VCB.  We are currently running ESX 3.5 with VC 2.5.  The snapshot gets created in VC just fine and after the jobs fails it gets removed so I'm not really sure where the issue is at.  My logic tells me that it is NBU with the issue since we have 150 other VMs that work this way just fine. 

[swest092]

Maybe flashbackup doesn't (not sure cause I never tried it) but flashbackup-windows does.  We have it working on 150 VMs already.

[Giroevolver]

The policy should be on Flashbackup-Windows option for it to work correctly. Are you trying to use the Block level inremental option? If so this is not supported on ESX 3.5 and was only introduced in Vsphere. 

If this does not help can you post exactly what you are using with your 150 other machines that do work as there must be a difference. Oh your not using Raw device mappings in those machines are you? These do not work using a VMware backup host.

[Giroevolver]

Another note below make sure you use on All_Local_Drives selection

Note: If you chose Full VM backup or Mapped Full VM

backup for the Virtual Machine Backup parameter,

ALL_LOCAL_DRIVES is the only entry allowed for backup

selections.

[swest092]

The policy is on flashbackup-windows and we are not doing the block level incrementals.

No, we are not using raw device mappings.

The jobs that are failing are identical, with the exception of scheduled time and retention, to the jobs that are suceeding.  I have deleted the old policies and recreated new ones thinking that might be the issue but it wasn't.  Below is the output from bppllist (policy_name) -U.

Policy Name:       stfs1_vm
 
  Policy Type:         FlashBackup-Windows
  Active:              yes
  Effective date:      07/18/2009 21:13:04
  File Restore Raw:    yes
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           stfs1
  Volume Pool:         NetBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
 
Granular Restore Info:  no
Ignore Client Direct:  no
  HW/OS/Client:  vmx-04        winNetEnterpr stfs1
 
  Include:  ALL_LOCAL_DRIVES
 
  Schedule:          Full
    Type:            Full Backup
    Frequency:       every 7 days
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 3 (1 month)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Monday     21:00:00  -->  Monday     22:00:00
 
  Schedule:          Cumulative-Inc
    Type:            Cumulative Incremental Backup
    Frequency:       every 1 day
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 3 (1 month)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Sunday     21:00:00  -->  Sunday     22:00:00
          Tuesday    21:00:00  -->  Tuesday    22:00:00
          Wednesday  21:00:00  -->  Wednesday  22:00:00
          Thursday   21:00:00  -->  Thursday   22:00:00
          Friday     21:00:00  -->  Friday     22:00:00
          Saturday   21:00:00  -->  Saturday   22:00:00
 
 
Policy Name:       sta1_vm
 
  Policy Type:         FlashBackup-Windows
  Active:              yes
  Effective date:      07/19/2009 08:29:48
  File Restore Raw:    yes
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           sta1
  Volume Pool:         NetBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
 
Granular Restore Info:  no
Ignore Client Direct:  no
  HW/OS/Client:  vmx-04        winNetEnterpr sta1
 
  Include:  ALL_LOCAL_DRIVES
 
  Schedule:          Full
    Type:            Full Backup
    Frequency:       every 7 days
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 1 (2 weeks)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Thursday   23:30:00  -->  Friday     00:30:00
 
  Schedule:          Cumulative-Inc
    Type:            Cumulative Incremental Backup
    Frequency:       every 1 day
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 1 (2 weeks)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Sunday     23:30:00  -->  Monday     00:30:00
          Monday     23:30:00  -->  Tuesday    00:30:00
          Tuesday    23:30:00  -->  Wednesday  00:30:00
          Wednesday  23:30:00  -->  Thursday   00:30:00
          Friday     23:30:00  -->  Saturday   00:30:00
          Saturday   23:30:00  -->  Sunday     00:30:00


Let me know if you see something that I don't see.

[marekkedzierski]

Hi.. If you are running backups with vStorage API with Mapped Full VM enabled incrementals are possible only with ESX 4 because Changed Block Tracking feature was introduced with vSphere 4. So cumulative schedule makes no sense because it's not possible. Did you check size of you cumulative backups ?Is there any difference between size of full backup and cumulative ?

[swest092]

....you can use netbackup 7 with ESX 3.5 and VC 2.5 to leverage VADP.  You only need vsphere if you are doing block level incrementals, which we are not.  There is a size difference between the full and cumulative backups for the VM policies that work.

Here is the post from a symantec employee.

https://www-secure.symantec.com/connect/forums/vmware-ending-availability-vcb

NetBackup 7 supports both vStorage (VADP) and VCB

NetBackup 7, which is now available, supports both approaches: vStorage APIs for Data Protection (VADP) and the former VMware Consolidated Backup (VCB).

You don't actually need "vSphere 4" to leverage vStorage.  The minimum SW revisions are:
If you are using vCenter 2.5 or later, then vStorage can protect ESX servers 3.0.2 or later
If not, you need ESX servers 3.5 or later to leverage vStorage.

If neither of those apply, then VCB can still be used.

NetBackup 7 also supports mixed VCB & vStorage deployments.  This approach recognizes that VMware users do not likely upgrade all ESX servers at once.

For more information:
- Attend a NetBackup for VMware webcast at http://www.netbackupdemo.com
- See the latest user guide: http://seer.entsupport.symantec.com/docs/340112.htm

Hope this helps!
Brian Smith
NetBackup Product Manager

[marekkedzierski]

I know about features of vStorage API and Nb7 (I reported few bugs during beta tests :))
My questions are:
What level of backup is used during cumulative job (full image, file level) ?
Could you write more about VM which generates issues ? (hardware version, operating system, type of disks (ide, scsi))

marekk

[swest092]

The full and cumulative are in the same job as I stated in a previous post.  They are doing a mapped full vm backup.

The VM that has the issues is on the same platform as every other VM we have.
hardware version - Not sure what you want here since every other VM is on the same/identical hardware.
OS - W2K3
Disks - SCSI

[Giroevolver]

Marek Means the VM hardware level. It should be 4 if you are running ESX 3.5 to check look on your virtual center click on the machine and on the summary tab there will be VM Version.

You are correct with the Full Mapped VM option. It is supported as long as you are not using BLIB. I ran a test on my enviroment and it works fine but I am running vsphere so not a fair test.

[swest092]

Thanks for the clarification, Giroevolver.

All of the VMs are version 4.

[marekkedzierski]

... and another thing. If you have ide disk in VM NB can't map files inside VM. There is article about this issue. So check how your virtual hardware is configured.

[swest092]

These are all connected to an HP EVA8400 so the only choice it provides is SCSI.

[marekkedzierski]

I'm talking about type of disk configured in virtual machine properties. What type of virtual controler does problematic machine have ?

[swest092]

They are all SCSI Controllers with LSI Logic as the type.

[marekkedzierski]

hmmm try to swith backup type to nbd in backup policy snapshot properties and try to backup problematic vm.

[swest092]

It gives me the same exact error when I use NBD as the tranfer type.