Netbackup 7.6.1.1 vmware problems with CBT option

[RecioLola]

Master/Media :  Symantec Appliance 5220  version 7.6.1.1 upgraded recently

Client:  Vmware client ( ESX version 5.5 ). Vmware tools installed .
 

Problem description:

We are added recently several vmware clients and all of them have the same problem.

They have accelerator option enabled.

They have 3 schedules defined:

1.- first full backup with forced rescan option enabled ( with frecuency repetition of 15 weeks )

2.- Incremental backups for every day

3.- Full backup every week

First full backup ( with rescan) and following days work properly with optimization near 100% of accelerator ratio.

But suddenly one day the backup shows this message

      "There is no complete backup image match with track journal, a regular full backup will be performed."

       and accelerator ratio is 0%, so the backup is like a full ( so CBT is not working )

We opened a support question and they told us a workaround to solve it, but this process is manual:

1) Disable accelerator in the attributes tab
2) Run a full backup without accelerator
3) Check accelerator and run a full backup with accelerator.
4) Run incremental.

This workaround works, and next backups are doing well again with an accelerator ratio near 100%.

But we had to do two backups full for each vmware server, with much volume of backup.

But after a few weeks the problem is repeated and we got again the same message:

      "There is no complete backup image match with track journal, a regular full backup will be performed." .

We are planning to add hundreds of vmware clients and if we need to apply this workaround every time is

a work very tedius.

Anybody know what is the cause of the problem and why can we to avoid these errors?

Thanks in advance,

regards.

Lola Recio.

Here is a complete trace of the backup:

03/18/2015 00:38:55 - Info nbjm (pid=9084) starting backup job (jobid=373438) for client ow-atp, policy xxxxx.VMware.dylan4.pol, schedule xxxxx.VMware.polincr
03/18/2015 00:38:55 - estimated 67241650 kbytes needed
03/18/2015 00:38:55 - Info nbjm (pid=9084) started backup (backupid=xxxxxx_1426635535) job for client xxxxx, policy xxxxx.VMware.pol, schedule xxxxx.VMware.polincr on storage unit stu_disk_dylan4 using backup host yyyy
03/18/2015 00:38:55 - started process bpbrm (pid=22923)
03/18/2015 00:38:55 - Info bpbrm (pid=22923) xxxxx is the host to backup data from
03/18/2015 00:38:55 - Info bpbrm (pid=22923) reading file list for client
03/18/2015 00:38:56 - Info bpbrm (pid=22923) accelerator enabled
03/18/2015 00:38:56 - Info bpbrm (pid=22923) There is no complete backup image match with track journal, a regular full backup will be performed.
03/18/2015 00:38:56 - Info bpbrm (pid=22923) starting bpbkar on client
03/18/2015 00:38:56 - Info bpbkar (pid=22936) Backup started
03/18/2015 00:38:56 - Info bpbrm (pid=22923) bptm pid: 22938
03/18/2015 00:38:56 - Info bptm (pid=22938) start
03/18/2015 00:38:56 - Info bptm (pid=22938) using 262144 data buffer size
03/18/2015 00:38:56 - Info bptm (pid=22938) using 128 data buffers
03/18/2015 00:38:56 - connecting
03/18/2015 00:38:56 - connected; connect time: 0:00:00
03/18/2015 00:38:57 - Info bptm (pid=22938) start backup
03/18/2015 00:39:00 - begin writing
03/18/2015 00:39:57 - Info bpbkar (pid=22936) INF - Transport Type =  nbd
03/18/2015 01:03:14 - Info bpbkar (pid=22936) accelerator sent 57570284544 bytes out of 57570283008 bytes to server, optimization 0.0%
03/18/2015 01:03:14 - Info bpbkar (pid=22936) bpbkar waited 1 times for empty buffer, delayed 791 times
03/18/2015 01:03:14 - Info bptm (pid=22938) waited for full buffer 65609 times, delayed 92625 times
03/18/2015 01:03:17 - Info bptm (pid=22938) EXITING with status 0 <----------
03/18/2015 01:03:17 - Info dylan4 (pid=22938) StorageServer=PureDisk:yyyy; Report=PDDO Stats (multi-threaded stream used) for (yyyy): scanned: 56226368 KB, CR sent: 16742141 KB, CR sent over FC: 0 KB, dedup: 70.2%, cache disabled
03/18/2015 01:03:17 - Info bpbrm (pid=22923) validating image for client xxxxx
03/18/2015 01:03:18 - Info bpbkar (pid=22936) done. status: 0: the requested operation was successfully completed
03/18/2015 01:03:18 - end writing; write time: 0:24:18
the requested operation was successfully completed  (0)

[Nicolai]

What are the retention periods ?

If retention periods are to short for the full backups, there is no reference for the incremental backup. 

Also be aware of Netbackup will require a new full backup is the Nebackup destination storage unit is changed. This is per design to ensure data integrity.

Can you please clarify if this is VMware backup using Vsphere or file system backups using a agent deployed in each VM.

[Marianne]

Also verify that Incremental schedule is in same policy as the Full.

[RecioLola]

Thanks for the answers!,

Nicolai, Our retention period is of 30 days. We are adding our Vmware clients using vpshere, not file system backups. The vmware clients are added last week so is not possible to need data integrity so soon.

Marianne, All of the scheduled are in the same policy  ( an unique policy with accelerator option enabled):

       They have 3 schedules defined in it:

         1.- first full backup with forced rescan option enabled ( with frecuency repetition of 15 weeks )

         2.- Incremental backups for every day

         3.- Full backup every week

All of the Vmware policies we have added has the same problem. After first full backup with

forced rescan the second incremental is done like a full with the warning in the activity monitor:

  "There is no complete backup image match with track journal, a regular full backup will be performed. " And accelerator ratio 0%.

And we can fix it with this workaroung (  support has told us), but is a manual process and we are

planning to add many vmware hosts policies and is a tedious proccess...

Have you tested this scenario?. Are we doing something wrong?.

Thanks!

Lola

[Nicolai]

This sound very wrong - Marianne and I provided the most universal configuration blunders that could lead to this scenario.

I can't say what wrong right now - need more information.

Can you please post the output from 

bppllist {POLICY_NAME} -U

To be honest I think you should re-open the support case with Symantec and tell them workaround is usable on 100 of servers - be firm about this. If nothing happens - call the duty manger and have him escalate the case.

[sdo]

How many different storage unit targets are there defined withinthe backup policy, and/or the schedules, and/or the SLPs?

[RecioLola]

Hi Nicolai,

Bellow I send you my policy so you can see if we have some mistake for this behaviour.

You are right, we can try to scale this issue at symantec they have to give me some solution about that.

Sdo, my master server is a symantec appliance. Storage unit is a SLP to replicate our backups to another

appliance in another location and this is working properly.

Policy Name:       XXXXX.VMware.pol

  Policy Type:         VMware
  Active:              yes
  Effective date:      01/01/1970 01:00:00
  File Restore Raw:    yes
  Application Consistent:    yes
  Block Incremental:   no
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0                                                                                                                                                       
  Max Jobs/Policy:     Unlimited                                                                                                                                               
  Disaster Recovery:   0                                                                                                                                                       
  Collect BMR info:    no
  Residence:           SLP
  Volume Pool:         NetBackup
  Server Group:        *ANY*
  Keyword:             XXXXX 2428
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    yes
  Application Discovery:      no
  Discovery Lifetime:      0 seconds
  ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
  Enable Metadata Indexing:  no
  Index server name:  NULL
  Use Accelerator:  yes
  HW/OS/Client:   Windows-x64     Windows2003                   XXXXX

  Include:  ALL_LOCAL_DRIVES

  Schedule:              XXXXX.VMware.polfull
    Type:                Full Backup
    Frequency:           every 105 days
     Excluded Dates----------
        No specific exclude dates entered
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         1
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:   0
    Daily Windows:
          Sunday     04:00:00  -->  Monday     03:59:59
          Monday     04:00:00  -->  Tuesday    03:59:59
          Tuesday    04:00:00  -->  Wednesday  03:59:59
          Wednesday  04:00:00  -->  Thursday   03:59:59
          Thursday   04:00:00  -->  Friday     03:59:59
          Friday     04:00:00  -->  Saturday   03:59:59
          Saturday   04:00:00  -->  Sunday     03:59:59

  Schedule:              XXXXX.VMware.polzfull
    Type:                Full Backup
    Calendar sched: Enabled
      Allowed to retry after run day
     Included Dates-----------
        Monday, Week 1
        Monday, Week 2
        Monday, Week 3
        Monday, Week 4
        Monday, Week 5
     Excluded Dates----------
      EXCLUDE DATE 0 - 03/27/2015
      EXCLUDE DATE 1 - 03/28/2015
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         1
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:   0
    Daily Windows:
          Sunday     04:00:00  -->  Monday     03:59:59
          Monday     04:00:00  -->  Tuesday    03:59:59
          Tuesday    04:00:00  -->  Wednesday  03:59:59
          Wednesday  04:00:00  -->  Thursday   03:59:59
          Thursday   04:00:00  -->  Friday     03:59:59
          Friday     04:00:00  -->  Saturday   03:59:59
          Saturday   04:00:00  -->  Sunday     03:59:59

  Schedule:              XXXXX.VMware.polincr
    Type:                Differential Incremental Backup
    Calendar sched: Enabled
      Allowed to retry after run day
     Included Dates-----------
        Sunday, Week 1
        Tuesday, Week 1
        Wednesday, Week 1
        Thursday, Week 1
        Friday, Week 1
        Saturday, Week 1
        Sunday, Week 2
        Tuesday, Week 2
        Wednesday, Week 2
        Thursday, Week 2
        Friday, Week 2
        Saturday, Week 2
        Sunday, Week 3
        Tuesday, Week 3
        Wednesday, Week 3
        Thursday, Week 3
        Friday, Week 3
        Saturday, Week 3
        Sunday, Week 4
        Tuesday, Week 4
        Wednesday, Week 4
        Thursday, Week 4
        Friday, Week 4
        Saturday, Week 4
        Sunday, Week 5
        Tuesday, Week 5
        Wednesday, Week 5
        Thursday, Week 5
        Friday, Week 5
        Saturday, Week 5
     Excluded Dates----------
      EXCLUDE DATE 0 - 03/27/2015
      EXCLUDE DATE 1 - 03/28/2015
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         1
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:   0
    Daily Windows:
          Sunday     04:00:00  -->  Monday     03:59:59
          Monday     04:00:00  -->  Tuesday    03:59:59
          Tuesday    04:00:00  -->  Wednesday  03:59:59
          Wednesday  04:00:00  -->  Thursday   03:59:59
          Thursday   04:00:00  -->  Friday     03:59:59
          Friday     04:00:00  -->  Saturday   03:59:59
          Saturday   04:00:00  -->  Sunday     03:59:59

[Nicolai · Accepted solution]

Symante recommend not mixing frequency and calendar based backup in the same policy. I know a lot of user are saying it work fine. But for the sake of problem resolving can you change the schedule XXXXX.VMware.polfull to calender as well.