Netbackup + VMware 7.5.0.1: only first client in list backed up, another clients are hanged.

[bat0r]

I have a problem with backup of VmWare VM's w\NBU7.5.0.1.

I've tried new type policy - "VMWARE" as well as "FlashBackup-Windows" and the problem occurs in both cases.

Problem in only first client in backuplist backed up, another clients are hanged.

In the attached screenshot scr-err-1.jpg you can see that VM pk6vcenter02 backed up normal - snapshot created, then VM backued up and snapshot removed (see also scr-vsphere-1-vcenter02-backup-OK.jpg), but pk6vcenter01 hanged and at first only snapshot created (see scr-vsphere-1-vcenter01-backup-HANG.jpg), then in Activity Monitor value for "Files" column grew slowly and stopped at 135203, status of the operation =  "Writing" till now, but value of "Kilobytes" column is constantly empty.

vmware-tools is installed on all VM

logs are in logs.zip

VxMS-logs are in VxMS*.zip

vmcloglevel=5

nbucred user have administrative role.

from BPBRM:

17:51:33.260 [72380.69512] <2> bpbrm main: from client pk6vcenter01: INF - Snapshot BLIB type: OFF

17:55:39.459 [72380.69512] <2> bpbrm main: from client pk6vcenter01: INF - VxMS error - severity 99.
17:55:39.459 [72380.69512] <2> bpbrm main: from client pk6vcenter01: INF - VxMS Error message 1 = fiml_backup fiml_set_fvv_tablemessage 2 = vfm_open_file_name <vix>[SharedStorage (1)] pk6vcenter01/pk6vcenter01-000004.vmdk flags = 3 retflag = 0

BLIB is turn on in policy settings, but in log we can see BLIB type: OFF.

Also I've found <INF> messages VxMS error, but no any <ERR> messages.

from BPFIS:

17:50:18.317 [70864.70256] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpfis.exe create -nbu -owner NBU -id pk6vcenter01_1339768216 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 40 -vm_type 2 -vm_client pk6vcenter01 -clnt pk6backup01 -st FULL -block_incr -prev_bli_incr 1339142478 -last_bli_full 1339142478 -dt 0 -rg other -fim VMware_v2:Virtual_machine_backup=2,nameuse=3,snapact=2,file_system_optimization=1,trantype=san:hotadd:nbd:nbdssl,exclude_swap=1,disable_quiesce=0,drive_selection=0,skipnodisk=0,post_events=1,nameuse=3 -class pk6VMWARE-snapshots-2 -sched Full-VM-Snapshot -fso -S pk6backup01 -jobid 2599
......

17:50:45.711 [70864.70256] <4> bpfis: INF - EXIT STATUS 0: the requested operation was successfully completed

i.e. snapshot has been created successfully.

from BPCD:

seems to be nothing wrong

from BPBKAR:

17:55:39.459: [55072.73084] <2> tar_base::V_vTarMsgW: INF - VxMS error - severity 99.

17:55:39.459: [55072.73084] <2> tar_base::V_vTarMsgW: INF - VxMS Error message 1 = fiml_backup fiml_set_fvv_table

again VxMS error ...

from VxMS-74640-061512.log & VxMS-73084-061512.log:

in VxMS-73084-061512.log for the problem host pk6vcenter01 backup we can see that time difference between piecewise data transfers equals 6 SEC (!!!) whereas it amounted to a few thousandths of a second for the normal host pk6vcenter02 backup (see in VxMS-74640-061512.log).

Begin:vfm_read [20:00:48.0482 : ]
In: nreq=1
vfm_obj_t { orig_name = [SharedStorage (1)] pk6vcenter01/pk6vcenter01-000004.vmdk }

[ I/O Req ]
vfm_io_req_t { offset = 320495616 reqlen = 262144 bufp = 0x00000000067F0000 bufp = 0x00000000067F0000 bytes_trans = 0 bytes_req = 0 error = 0 }
20:00:48.0482 : vixMapRead:.\VixCoordinator.cpp:1088 <> : Obj: [SharedStorage (1)] pk6vcenter01/pk6vcenter01-000004.vmdk

Out:

[ I/O Req ]
vfm_io_req_t { offset = 320495616 reqlen = 262144 bufp = 0x00000000067F0000 bufp = 0x00000000067F0000 bytes_trans = 262144 bytes_req = 0 error = 0 }

End:(0)

Begin:vfm_read [20:00:54.0940 : ]
In: nreq=1
vfm_obj_t { orig_name = [SharedStorage (1)] pk6vcenter01/pk6vcenter01-000004.vmdk }

[ I/O Req ]
vfm_io_req_t { offset = 320757760 reqlen = 262144 bufp = 0x0000000006830000 bufp = 0x0000000006830000 bytes_trans = 0 bytes_req = 0 error = 0 }
20:00:54.0940 : vixMapRead:.\VixCoordinator.cpp:1088 <> : Obj: [SharedStorage (1)] pk6vcenter01/pk6vcenter01-000004.vmdk

[V4]

time to log call ... you have done half of their work. .Might be there would be any etrack which can sort this out.. check with them. if it's one of known issue with 7.5.0.1

[bat0r]

I've upgrade master (vmware off-host backup) server to 7.5.0.3, but the problem is persist.

[V4]

did you logged case with support ?