Forced Rescan

[Matt_Galvin]

Hi all,

I have a strange issue where our Monday night incremental backups for our Vmware policies run very slowly, this has been going on for some time & support have so far been unable to help, all other nights run as expected.

I have however had some thoughts on what this could be & would like to discuss them with the community. It’s around the Vmware Forced Rescan option. 

On our weekly full backups, taken on a Friday, we had this setting mistakenly on. So every Friday Night we were carrying out a forced rescan for the entirety of our estate. Funnily enough, Friday night backups are generally ok and usually complete sometime on Saturday.

I’d like to understand what effect a ‘forced rescan’ has on the next incremental backup (a Monday night for us) and if this potentially could be much slower than a ‘typical’ incremental due to the extra work involved around the data extents of the unchanged blocks (Blurb from the manual below) as there will obviously be a large number of ‘unchanged blocks’ after a forced rescan and thus extra to work to complete.

We’ve now changed the policies to not carry out a ‘forced rescan’ every Friday but we’ll need to wait a week to find out the result so in the mean time I’d like someone in the know to confirm or deny this :)

Finally, how often are people using a forced rescan for VMware policies? I see the docs refer to 6 months.

Matt.

Blurb

From NBU 7.6 Vmware Admin Guide:

The NetBackup Accelerator creates the backup stream and backup image for each virtual machine as follows:

■ If the virtual machine has no previous backup, NetBackup performs a full backup

and uses VMware Changed Block Tracking to track the data in use for each VMDK.

■ At the next backup, NetBackup identifies data that has changed since the

previous backup. Only changed blocks and the header information are included

in the backup, to create a full virtual disk backup.

■ The backup host sends to the media server a tar backup stream that consists

of the following: The virtual machine's changed blocks, and the previous backup

ID and data extents (block offset and size) of the unchanged blocks.

■ The media server reads the virtual machine's changed blocks, the backup ID,

and information about the data extents of the unchanged blocks. From the

backup ID and data extents, the media server locates the rest of the virtual

machine's data in existing backups.

■ The media server directs the storage server to create a new full image that

consists of the following: The newly changed blocks, and the existing unchanged

blocks that reside on the storage server. The storage server may not write the

existing blocks but rather link them to the image.

[sdo]

As a side point... something that you may be able to do, but this depends the nature of your schedules and their retentions, is to apportion when the forced re-scan will take place, i.e. do different policies (and thefore different clients) at different weekends... 1st, 2nd, 3rd, 4th weekends through the month.  If your schedules do not allow to to implement this easily, then what you could do... is...

i) Have a dummy full schedule, perhaps named "Manual_Rescan", in all policies... which does not have a run window, and so never actually fires.

ii) run a script once a week on Fridays, which will 'turn on' forced rescan on your normal weekly/monthly full schedules for selected policies only

iii) run a sister script once a week on Mondays, to 'turn off' forced rescan on... either.... all schedules which have a run window in all policies... or... re-process a saved list (from script at step ii) to turn-off what was enabled by the "Friday" script

...which would be a kind of poor mans way of manually controlling when rescans take place.

[Matt_Galvin]

Thanks SDO, so do you know if the forced rescans do have a negative impact on the next incremental? Do you think my prognosis is likely / possible?

I like your idea of staggering the forced rescans through the policies and this is something we'll need to consider, again what do people typically do here? 

M

[sdo]

I'm personally not aware of a known issue side-effect of forced rescan causing the next VMware style incremental to suffer.  Is it likely/possible?  You seem to know NetBackup, so I would assume that your analysis is sound, so... is it possible... then I think, yes, unfortunately.  Maybe best to open a support case to confirm whether this is not only known behaviour but also the "expected" behaviour.

Are your subsequent incrementals cummulative, or differential ?

Are you able to specify your NetBackup versions, master, media, backup host, and exact VMware ESXi version (including update AND patch number) ?

I do stage forced re-scan at one site, because I was aware of a hit (well, not so much a hit, but more a case of lack of benefit) during a full backup with forced rescan enabled, but I don't do VM backups, just plain client file system backups, and I have not noticed any equivalent behaviour of slower subsequent file system based differential incremental backups.

[sdo]

And remember... that for accelerator to work, there must always exist, in the backup policy, at least one full schedule with "forced rescan" enabled, hence the dummy "manual_full" schedule with no run window.

[Matt_Galvin]

Hi Sdo,

Things are progressing this side, I'm pushing support for an answer.

Are your subsequent incrementals cummulative, or differential ?

We use differential incrementals.

Are you able to specify your NetBackup versions, master, media, backup host, and exact VMware ESXi version (including update AND patch number) ?

We're currently on 7.6.1.2 & ESX 5.5, I've not got the patch versions to hand but will see if I can get them.

For anyone who's reading / finds this thread there's a really useful doc on the nuts & bolts of Accelerator here:

http://www.veritas.com/community/sites/default/files/NetBackup%207.6%20Blueprints%20-%20Accelerator%20for%20VMware.pdf