vmware snapshot problems

[mo_ali]

i have been using netbackup 7 to backup my vm's for almost 7 months now, everything was working fine backup&restores. then all of a sudden my vm backups stopped working, at first i thought it was related to my vcenter running out of disk space due to the database logs, i have solved this problem by backing up my database and shrinking it. but this did not fix the backup process. i have re-checked every thing i could think of

checked zoning

checked mapping of volume containing vm's

checked security settings.

repaired netbackup

re-installed vcenter

i can't even browse my vm's in my policy when i do that it takes for ever and display request has timed out(195), i also loose connection to my vcenter (when i browse for vm's in netbackup) and it displays HTTP error 503 (which forces me to restart service).  

appreciate if anyone could assist with this.

[Tim_Hansen]

If you add an ESX server to Netbackup and let NBU browse the ESX server directly, can you query VM information then?

IMHO the issue is VCenter. Do you run the VCenter application and database on the same server?

[mo_ali]

yes i did and i was getting the same results , but i tried againg with a different esx and it's working now but i still can't get vcenter to work. here is the output from bpfis log.

14:17:50.314 [14456.14616] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpfis.exe verify -nbu -pt 29 -backup_copy 0 -block_incr -fim VMware:nameuse=0,Virtual_machine_backup=2,trantype=0,disable_quiesce=0,file_system_optimization=1,snapact=2,vmmono=0 -vm_client SERVER.COM
14:17:50.314 [14456.14616] <2> bpfis main: received FIM as [116] VMware:nameuse=0,Virtual_machine_backup=2,trantype=0,disable_quiesce=0,file_system_optimization=1,snapact=2,vmmono=0
14:17:50.329 [14456.14616] <4> bpfis: INF - BACKUP START 14456
14:17:50.345 [14456.14616] <2> bpfis main: receive filelist:<ALL_LOCAL_DRIVES>
14:17:50.345 [14456.14616] <2> bpfis main: receive filelist:<CONTINUE>
14:17:50.439 [14456.14616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:17:50.439 [14456.14616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
14:17:50.439 [14456.14616] <2> logconnections: BPRD CONNECT FROM 10.10.68.27.2605 TO 10.10.68.27.13724
14:17:52.595 [14456.14616] <2> get_long: (2) premature end of file (byte 1)
14:17:52.595 [14456.14616] <2> bprd_read_text_file: get_string() failed, Access is denied.  (5), premature end of file encountered
14:17:52.673 [14456.14616] <8> bpfis: WRN - VfMS error 10; see following messages:
14:17:52.673 [14456.14616] <8> bpfis: WRN - Non-fatal method error was reported
14:17:52.673 [14456.14616] <8> bpfis: WRN - vfm_configure_fi_one: method: FlashSnap, type: FIM, function: FlashSnap_init
14:17:52.673 [14456.14616] <8> bpfis: WRN - VfMS method error 3; see following message:
14:17:52.673 [14456.14616] <8> bpfis: WRN - FlashSnap_init: Veritas Volume Manager not installed.
14:17:52.689 [14456.14616] <8> bpfis: WRN - VfMS error 10; see following messages:
14:17:52.689 [14456.14616] <8> bpfis: WRN - Non-fatal method error was reported
14:17:52.689 [14456.14616] <8> bpfis: WRN - vfm_configure_fi_one: method: vxvm, type: FIM, function: vxvm_init
14:17:52.689 [14456.14616] <8> bpfis: WRN - VfMS method error 3; see following message:
14:17:52.689 [14456.14616] <8> bpfis: WRN - vxvm_init: Veritas Volume Manager not installed.
14:17:52.798 [14456.14616] <4> bpfis: INF - FIS_ID=1287404270
14:17:52.798 [14456.14616] <4> bpfis: INF - Testing freeze of ALL_LOCAL_DRIVES using snapshot method VMware.
14:17:52.798 [14456.14616] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_14456
14:17:52.861 [14456.14616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:17:52.861 [14456.14616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
14:17:52.861 [14456.14616] <2> logconnections: BPRD CONNECT FROM 10.10.68.27.2657 TO 10.10.68.27.13724
14:17:55.173 [14456.14616] <4> bpfis: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_14456
14:17:55.173 [14456.14616] <32> bpfis: FTL - VfMS error 11; see following messages:
14:17:55.173 [14456.14616] <32> bpfis: FTL - Fatal method error was reported
14:17:55.173 [14456.14616] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
14:17:55.173 [14456.14616] <32> bpfis: FTL - VfMS method error 7; see following message:
14:17:55.173 [14456.14616] <32> bpfis: FTL - VMware_freeze: Unable to find VCB path
14:17:55.173 [14456.14616] <32> bpfis: FTL - VfMS error 11; see following messages:
14:17:55.173 [14456.14616] <32> bpfis: FTL - Fatal method error was reported
14:17:55.173 [14456.14616] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
14:17:55.173 [14456.14616] <32> bpfis: FTL - VfMS method error 7; see following message:
14:17:55.173 [14456.14616] <32> bpfis: FTL - VMware_freeze: Unable to find VCB path
14:17:55.173 [14456.14616] <16> bpfis: FTL - snapshot verify failed, status 156
14:17:55.173 [14456.14616] <4> bpfis: INF - EXIT STATUS 156: snapshot error encountered
 

[MOHAMED_PATEL]

Had this same situation at another client -

Increased bp_start_notify timeout to 3600 -

checked VM disk space - this was the cause, as the VM was larger than the available disk for snapshots.

Have the VMguys do a manual snap and check - if that is fine, (as a test) do individual backups of just one VM per policy -

[Tim_Hansen]

Set your policy to use "Mapped Full VM backup" instead of "Mapped Full with file level backup using VCB" and retry it.

If you use a proxy, check that vmount2.exe is running (assuming you're still using VCB) and that it's been installed in Program Files instead of Program Files(x86). There appears to be an error locating VCB on the proxy/media server. You can choose to stop using VCB altogether (by changing the backup type to "Mapped Full VM backup") or you can still use VCB, just ensure it's installed and running in the location you intended it. (I strongly recommend scuttling VCB altogether since you're running NBU 7.x already).

Also verify you are attempting to backup a version 7 VM when selecting the Block Level Incremental Backup option in the policy, it will fail if the VM is still version 4.

[S_Williamson]

Check your VMWare LUN. We had a similar issue (was working then all of a sudden getting 156's) and the LUN turned out to have some corruption.

Corrupted looked like this (look at /dev/sdd)

[root@xxxxxxxxx ~]#fdisk -lu

Disk /dev/sdc: 299.4 GB, 299439751168 bytes
255 heads, 63 sectors/track, 36404 cylinders, total 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *          63     2249099     1124518+  83  Linux
/dev/sdc2         2249100     2474009      112455   fc  VMware VMKCORE
/dev/sdc3         2474010   584830259   291178125    5  Extended
/dev/sdc5         2474073   584830259   291178093+  fb  VMware VMFS

Disk /dev/sdd: 2197.9 GB, 2197949513728 bytes
255 heads, 63 sectors/track, 267218 cylinders, total 4292870144 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System


As you can see /dev/sdd shows no partitions or blocks etc. We logged a call to VMWare \
and they repaired the lun so it then looked like

[root@xxxxxxxxx ~]# fdisk -lu

Disk /dev/sdc: 299.4 GB, 299439751168 bytes
255 heads, 63 sectors/track, 36404 cylinders, total 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *          63     2249099     1124518+  83  Linux
/dev/sdc2         2249100     2474009      112455   fc  VMware VMKCORE
/dev/sdc3         2474010   584830259   291178125    5  Extended
/dev/sdc5         2474073   584830259   291178093+  fb  VMware VMFS

Disk /dev/sdd: 2197.9 GB, 2197949513728 bytes
255 heads, 63 sectors/track, 267218 cylinders, total 4292870144 sectors


Then vStorage API was able to talk and backup correctly again.
Units = sectors of 1 * 512 = 512 bytes