How intelligent are Intelligent Policies?

[Scott_123]

VMware Intelligent Policies, or VIP, was introduced in NetBackup 7.1.  VIP allows a NetBackup Administrator to discover new VMware virtual machines and automatically add them to backups.  For more information on VIP, see the Symantec NetBackup for VMware Administrator's Guide or visit the following Symantec blog.

https://www-secure.symantec.com/connect/blogs/netbackup-71-features-vmware-intelligent-policy-vip

Release 7.6 of Symantec NetBackup for VMware Administrator's Guide has no mention of the acronym VIP and instead refers to it as Virtual Machine Intelligent Policy, although other 7.6 documentation still uses the term VIP.

NetBackup 7.6 added Oracle Intelligent Policy.  This feature automatically discovers new Oracle instances and eliminates the need to manually write RMAN scripts, as it generates the scripts dynamically at run-time.  For more information on Oracle Intelligent Policy, see the Symantec NetBackup for Oracle Administrator's Guide or visit the following video on Symantec Connect.

https://www-secure.symantec.com/connect/videos/new-oracle-intelligent-policy-netbackup-76

Intelligent Policies bring new ways to configure NetBackup policies.  They have the potential to make backup administration a little easier, but how intelligent are Intelligent Policies and are they right for your environment?

Whether you want to automatically protect new VMs or not, Virtual Machine Intelligent Policy configured with Resource Limits can have a huge performance impact by load-balancing backups from one or more policies across VMware components with restrictions on how many backups can go active on various resource types.  Limiting to no more than 1 or 2 active backups per Datastore can reduce I/O and keep the Datastore from running out of capacity during the backup's snapshot.  Limiting the ESX Server to no more than 2-4 active backups can also be helpful when using one of the LAN transports.

Unfortunately, VMware Resource Limits do not work without Intelligent Policy.  If VMs are selected manually within a VMware policy, any Resource Limits that are configured are ignored for those backups.  According to Symantec, this was done to allow the customer the flexibility to configure Resource Limits, while also being able to selectively ignore them.  A check box in the VMware tab called Ignore Resource Limits would have worked better.  Even better, put it in Advanced Attributes on the VMware tab and allow Resource Limits to be ignored or even overridden with a different value.  With these options, an administrator could ignore Resource Limits on a per policy basis, regardless of client selection method.

Before continuing, it's important to note that Intelligent Policies behave somewhat similar to backup jobs with Allow Multiple Data Streams enabled, in that NetBackup starts a parent backup job for the Intelligent Policy and the parent job then starts the rest of the jobs for the policy and remains active until all jobs it started are complete.  For a VMware backup, the parent job is for the server listed on the Clients tab under NetBackup host to perform automatic virtual machine selection and the additional jobs for the actual VMs consist of the Snapshot and Backup jobs.  If one or more of the jobs started by the parent fails, the parent will also show a failed status code once it completes.

So, what's wrong with Intelligent Policies?

There are some behaviors related to Intelligent Policy that can be annoying or even unexpected.

• NetBackup 7.6 introduced the ability to restart an individual Intelligent Policy job using Activity Monitor or bpdbjobs, but there's still no way to manually launch a backup on an individual Virtual Machine.  A job can be restarted whether it is in a failed status or completed successfully, but if you want to start a backup of a VM that was just added or for a schedule type that is no longer in the jobs database (visible in Activity Monitor), you're out of luck.

• Restarting the parent backup job for the Intelligent Policy will cause ALL jobs for that policy to be restarted.  Restart the job for a specific VM or database instance if that is what you want.  Restarting a backup job for a specific VM or database instance will automatically start a new parent job for the Intelligent Policy, but it won't start the rest of the jobs as it would if the parent itself was restarted.

• Manually starting a Virtual Machine Intelligent Policy backup (or restarting the parent job as mentioned above) will start a backup job for EVERY VM for that policy, even if there is already an Active or Queued backup job for one or more of the VMs.

• A NetBackup Client can be put into an offline state using Client Attributes found under the Host Properties of the Master Server.  However, putting the Client name of a VM or Oracle database server in an offline state does not stop backups for that Client from Intelligent Policies.

• If the parent job for an Intelligent Policy is active longer than expected (due to a long running or even hung job) or is active for any other reason (for example, a single backup job that was restarted), the NetBackup scheduler will not start another parent job, and therefore will not start any additional VM or database backups, regardless of the frequency and how overdue any of the actual VM or database backups are.

• The Schedule backup attempts value under Global Attributes seems to apply to the parent of the Intelligent Policy as well as the backup jobs it starts.  Imagine this scenario, a backup job from an Intelligent Policy is restarted several times during troubleshooting at 4 PM (i.e. failed jobs from the night before are restarted at 4 PM).  Each restart of the job causes a parent to restart as well.  Schedule backup attempts is set to 2 tries per 12 hours.  The backup window opens at 8 PM, but no backups start because the parent job has already exceeded the configured number of scheduled attempts.  This may cause the backups for the Intelligent Policy to run several hours later than expected... if the backup window is still open!

• The parent job for the Intelligent Policy starts according to the schedules in the policy.  The parent job doesn't actually run with a schedule, it is displayed in Activity Monitor with a Job Schedule of "-".  The individual jobs will then run with the appropriate schedule based on what is due.  For example, an Intelligent Policy is configured to backup some VMs and only has a single schedule, a Full Backup schedule with a 1 week frequency and backup window open daily from 8 PM to 6 AM.  Let's say the Intelligent Policy runs on Monday at 8 PM and all VMs are backed up successfully.  A new VM is added that meets the Query Builder rules for the policy.  Even if the value for Reuse VM selection query results is small enough that the new VM will be discovered before the backup window closes, it will not be backed up until the parent job for the Intelligent Policy is due to run again.  Because the Full Backup schedule completed successfully and has a 1 week frequency, it won't be due again until next Monday, leaving the new VM without it's first backup for an entire week.  This behavior may not be expected, as the backup window was open each day and the new VM was without a backup.

• Now, imagine the same policy as above, but with an Incremental Backup schedule added with a 1 day frequency and similar backup window.  On Tuesday, the parent job will be due for an Incremental.  After the parent job starts, the resulting VM backup jobs will be Incremental backups for the VMs backed up Monday and a Full backup for the newly added VM.  This behavior is probably desired and similar to what a VMware policy would do with manually selected VMs.

Intelligent Policies can be beneficial, especially when used with Resource Limits.  Understanding how Intelligent Policies work is the first step towards determining how and where to use them in your environment.

Improvements have been made in Intelligent Policy since first introduced.  As Symantec brings this model to more applications, hopefully it will continue to mature.