Regarding VCB backup failure

[RohitNayal]

Hi All,

We are facing the below issue in our VCB backups as only one virtual machine backups are failing with error code 156 in our enviorment.

NBU Version : 6.5.6 (Both Master and Proxy Server)

ESX and V-Center : 4.0

Virtual Machine OS : Red Hat Linux 

Bpfis logs from proxy server :

09:26:47.984 [2508.2052] <2> onlfi_vfms_logf:     [2012-02-02 09:26:46.953 'vcbVmName' 2556 error] Error: Invalid user name or password.

09:26:47.984 [2508.2052] <2> onlfi_vfms_logf: --- End of file C:\Windows\TEMP\VfMS-02508328174788703052000000006-a02052

09:26:47.984 [2508.2052] <2> onlfi_vfms_logf: --- Dumping file C:\Windows\TEMP\VfMS-02508328174788703052000000007-a02052 (stderr):

09:26:47.984 [2508.2052] <2> onlfi_vfms_logf: --- End of file C:\Windows\TEMP\VfMS-02508328174788703052000000007-a02052

09:26:47.984 [2508.2052] <2> onlfi_vfms_logf: do_cmd: Command failed ""C:\Program Files (x86)\VMVCB\VMware Consolidated Backup Framework\vcbVmName.exe"  -h XXXXXX-backup:443 -u "soe-backup" -p ************** -c "C:\Program Files (x86)\VMVCB\VMware Consolidated Backup Framework\vmName.cache" -s "ipaddr:XXXXXXXX-backup" -L 6" <NUL >C:\Windows\TEMP\VfMS-02508328174788703052000000006-a02052 2>C:\Windows\TEMP\VfMS-02508328174788703052000000007-a02052

09:26:49.984 [2508.2052] <4> delete_mount_point: Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_2508

09:26:49.984 [2508.2052] <32> onlfi_freeze_fim_fs: VfMS error 11; see following messages:

09:26:49.984 [2508.2052] <32> onlfi_freeze_fim_fs: Fatal method error was reported

09:26:49.984 [2508.2052] <32> onlfi_freeze_fim_fs: vfm_freeze: method: VMware, type: FIM, function: VMware_freeze

09:26:49.984 [2508.2052] <32> onlfi_freeze_fim_fs: VfMS method error 7; see following message:

09:26:49.984 [2508.2052] <32> onlfi_freeze_fim_fs: VMware_freeze: Unable to locate VM XXXXXXXX-backup

09:26:49.984 [2508.2052] <32> onlfi_freeze: VfMS error 11; see following messages:

09:26:49.984 [2508.2052] <32> onlfi_freeze: Fatal method error was reported

09:26:49.984 [2508.2052] <32> onlfi_freeze: vfm_freeze: method: VMware, type: FIM, function: VMware_freeze

09:26:49.984 [2508.2052] <32> onlfi_freeze: VfMS method error 7; see following message:

09:26:49.984 [2508.2052] <32> onlfi_freeze: VMware_freeze: Unable to locate VM XXXXXXXX-backup

09:26:49.984 [2508.2052] <16> bpfis: FTL - snapshot creation failed, status 156

09:26:49.984 [2508.2052] <4> onlfi_thaw: Thawing ALL_LOCAL_DRIVES using snapshot method VMware.

09:26:49.984 [2508.2052] <8> onlfi_thaw: ALL_LOCAL_DRIVES is not frozen

09:26:49.984 [2508.2052] <8> bpfis: WRN - snapshot delete returned status 20

09:26:49.984 [2508.2052] <4> bpfis: INF - EXIT STATUS 156: snapshot error encountered

09:26:55.765 [2872.2164] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpfis.exe delete -nbu -id XXXXXXXX-backup_1328174762 -bpstart_to 300 -bpend_to 300 -clnt XXXXX -S XXXXXX

09:26:55.765 [2872.2164] <4> bpfis: INF - BACKUP START 2872

09:26:55.765 [2872.2164] <32> fis_rebuild_from_db: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.XXXXXXXX-backup_1328174762.0

09:26:55.765 [2872.2164] <4> bpfis: INF - EXIT STATUS 0: the requested operation was successfully 

Troubleshooting Perform:-

1. Delete the client and add into the polciy again
2. No old snapshot is showing in the snapshot navigator also check from V-center the snapshot of the client is running fine.
3. Add the IP of the failed client into the Hosts file of proxy server

Please let me know if any information is required from my side .

regards 

Rohit

[Mark_Solutions]

I see you have already done the obvious part of adding the IP address but you do need to make sure that it  is the correct IP address against the "real" name of the client - as per:

http://www.symantec.com/docs/TECH56729

However NetBackup does cache IP addresses so this could be causing an issue - you need to wait at least 1 hour before you try the backup again after changing the hosts file.

NetBackup 7.x has a new command of \netbackup\bin\bpclntcmd -clear_host_cache

This will flush the IP addresses and start using the ones from the hosts file.

This tech note may also assist as it covers a few troubleshooting tasks:

http://www.symantec.com/docs/TECH71551

PS - you have until October to move to V 7.x to remain in support - you can then start using the vSphere API backups to make life much simpler for your VM Backups

Hope this helps

[VirtualED]

The error:

Error: Invalid user name or password.

Indicates the credential you have added is incorrect.  Remove the credential you have stored in NetBackup.

Media and Device Management -> Credentials -> Virtual Machine Servers -> Remove the Virtual Center or ESX you have listed.

Then run:

nbemmcmd -deletehost -machinename <VC_name> -machinetype ndmp

[Omar_Villa]

Check your storage haves enough free space to create the snapshot and also your VCB box or stagging disk haves the space to replicate the data.

[Omar_Villa]

Check this in your windows boxes

http://www.symantec.com/business/support/index?page=content&id=TECH75543

[Amarnath_Sathis]

Hi Rohit,

Please find attached documents, this will assist in fixing the issue.