Netbackup VM backups extreme delay in snapshot creation

[klp]

We are backing up a large number of clients using vcb.  This works great, except during the time when we observe this file being updated during peak backupwindow activity:  D:\Program Files\Veritas\NetBackup\online_util\fi_cntl\vcbnames_<vcenter-host-name>. 

I have been told this file dynamically updates approximately every 8 hours.  When the dynamic update kicks in during the backup window there is an extreme impact on backup jobs between when the activity detail indicates the snapshot creation is requested and when VM actually sees the request.  Jobs will eventually complete but the completion rate is so slow that many jobs miss the backup window, failing with status 196.

The condition is resolved by deactivating all the VM policies, cancelling all queued VM backups, allow active backups to complete (at the excrutiating pace) and the timestamp of the vcbnames file to stabilize, then re-activate polices.  Cancelled jobs requeue and complete normally.  This requires nightly monitoring and intervention to assure that all VM backups complete timely.

The vcbnames file is also the source for the client list presented in the Policy window for Add Client/Browse Virtual Machines.  The timestamp in the this window matches the timestamp of the vcbnames file. We have observed that during the update of the file if you attempt to browse virtual machines from the policy add client process, it will fail with a timeout/can't connect on socket.  This is observed also during daytime non-backup window, if it is the 8 hour update time, but it resolves within less than 15 minutes.

I have a Symantec case opened.  The only promising suggestion was contained in technote www.symantec.com/business/support/index?page=content&id=TECH129568.  This did sound like our situation, even though earlier version of netbackup.  Support said it would not be necessary to apply the fix since we were at a later version, but to make the registry setting change.  We did this, rebooted proxy server, no change in behavior. 

Environment info:

Netbackup master
 Solaris 10
 Netbackup 7.1.0.2
Media Server/VM proxy host
 Windows 2008 R2
 Netbackup 7.1.0.2
vSphere - 4.1.0 build 345043
vCenter - 4.1.0 build 345043
 

Suggestions? 

-klp

[pikachu]

Why are you using VCB and not vStorage?

Client list creation takes a long time?

Are you using hostname or display name?

[VirtualED]

VCB is dead and soon will be end of life.  If you go to using the vStorage built into the version of NetBackup you currently have, you will be much better off.  If there a reason you are using VCB?

Because you are using VCB, when your cache file is out dated it is necessary for vcbvmname to be called.  It can also be called in the event that the VM name could not be found at backup time.  We will run vcbvmname to see if this is a new VM.  In a busy backup window, this causes all snapshot to wait until the operation is done.

Going to vStorage will fix all of your issues.

[klp]

I mispoke, used wrong terminology,  We do use vStorage (not vcb)

[klp]

We use display name.

[Mark_Solutions]

Just a thought here and it is not clear from how you have you policy configured .....

If you use the VIP policy type (select clients using a query) the default retention of the information is 8 hours - but this is configurable at the bottom of the clients selection tab.

If you did it this way you could set it to collect the data every 24 hours and trigger it by running the first one at mid day

In that way you should be able to resolve you issue by keeping the update out of the backup window

Hope this helps

[klp]

We don't use the automatic machine selector via a query.  To add new clients we use the Browse Virtual Machines feature to select clients.

[Mark_Solutions]

OK

I was just suggesting that if you did use the query method as it would remove the 8 hour refresh issue that you have so you may want to consider it.

[pikachu]

I am scheduled to be working tomorrow. Open a ticket and ping me the case. I can help over a webex.

[klp]

The already opened case is 415-633-084.  It may be in the process of being reassigned.  I think it will be difficult to create the condition dynamically.  Sometimes I have been able to anticpate when during the backup window the problem will begin, but not always.

[marekkedzierski]

did you try to create snapshot from vcenter ? how long does it take ?

[smakovits]

was this case ever solved?  This appears to be my exact issue.  NBU requestred snapshots on a handful of machines takes 15 minutes.  However, on that same machine if I request create the snapshot from vcenter it is done in under a minute.