Forum Discussion

klp's avatar
klp
Level 3
14 years ago

Netbackup VM backups extreme delay in snapshot creation

We are backing up a large number of clients using vcb.  This works great, except during the time when we observe this file being updated during peak backupwindow activity:  D:\Program Files\Veritas\NetBackup\online_util\fi_cntl\vcbnames_<vcenter-host-name>. 


I have been told this file dynamically updates approximately every 8 hours.  When the dynamic update kicks in during the backup window there is an extreme impact on backup jobs between when the activity detail indicates the snapshot creation is requested and when VM actually sees the request.  Jobs will eventually complete but the completion rate is so slow that many jobs miss the backup window, failing with status 196.

The condition is resolved by deactivating all the VM policies, cancelling all queued VM backups, allow active backups to complete (at the excrutiating pace) and the timestamp of the vcbnames file to stabilize, then re-activate polices.  Cancelled jobs requeue and complete normally.  This requires nightly monitoring and intervention to assure that all VM backups complete timely.

The vcbnames file is also the source for the client list presented in the Policy window for Add Client/Browse Virtual Machines.  The timestamp in the this window matches the timestamp of the vcbnames file. We have observed that during the update of the file if you attempt to browse virtual machines from the policy add client process, it will fail with a timeout/can't connect on socket.  This is observed also during daytime non-backup window, if it is the 8 hour update time, but it resolves within less than 15 minutes.


I have a Symantec case opened.  The only promising suggestion was contained in technote www.symantec.com/business/support/index?page=content&id=TECH129568.  This did sound like our situation, even though earlier version of netbackup.  Support said it would not be necessary to apply the fix since we were at a later version, but to make the registry setting change.  We did this, rebooted proxy server, no change in behavior. 

Environment info:

Netbackup master
 Solaris 10
 Netbackup 7.1.0.2
Media Server/VM proxy host
 Windows 2008 R2
 Netbackup 7.1.0.2
vSphere - 4.1.0 build 345043
vCenter - 4.1.0 build 345043
 

Suggestions? 

-klp

7.1.x and Earlier
Backing Up
Backup and Recovery
NetBackup
Troubleshooting

11 Replies

Sort By
  • pikachu's avatar
    pikachu
    Level 6
    14 years ago

    Why are you using VCB and not vStorage?

     

    Client list creation takes a long time?

    Are you using hostname or display name?

  • VirtualED's avatar
    VirtualED
    Level 5
    14 years ago

    VCB is dead and soon will be end of life.  If you go to using the vStorage built into the version of NetBackup you currently have, you will be much better off.  If there a reason you are using VCB?

    Because you are using VCB, when your cache file is out dated it is necessary for vcbvmname to be called.  It can also be called in the event that the VM name could not be found at backup time.  We will run vcbvmname to see if this is a new VM.  In a busy backup window, this causes all snapshot to wait until the operation is done.

    Going to vStorage will fix all of your issues.

  • klp's avatar
    klp
    Level 3
    14 years ago

    I mispoke, used wrong terminology,  We do use vStorage (not vcb)

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    14 years ago

    Just a thought here and it is not clear from how you have you policy configured .....

    If you use the VIP policy type (select clients using a query) the default retention of the information is 8 hours - but this is configurable at the bottom of the clients selection tab.

    If you did it this way you could set it to collect the data every 24 hours and trigger it by running the first one at mid day

    In that way you should be able to resolve you issue by keeping the update out of the backup window

    Hope this helps

  • klp's avatar
    klp
    Level 3
    14 years ago

    We don't use the automatic machine selector via a query.  To add new clients we use the Browse Virtual Machines feature to select clients.

  • Mark_Solutions's avatar
    Mark_Solutions
    Level 6
    14 years ago

    OK

    I was just suggesting that if you did use the query method as it would remove the 8 hour refresh issue that you have so you may want to consider it.

  • pikachu's avatar
    pikachu
    Level 6
    14 years ago

    I am scheduled to be working tomorrow. Open a ticket and ping me the case. I can help over a webex.

  • klp's avatar
    klp
    Level 3
    14 years ago

    The already opened case is 415-633-084.  It may be in the process of being reassigned.  I think it will be difficult to create the condition dynamically.  Sometimes I have been able to anticpate when during the backup window the problem will begin, but not always.