cancel
Showing results for 
Search instead for 
Did you mean: 

NBU 7: snapshot creation failed, status 156 ALL_LOCAL_DRIVES is not frozen

pruvn
Level 5
My environment: NBU 7 - windows server 2008 x64
                              vSphere 4.0
Log:

4:22 AM - begin Create Snapshot
5/6/2010 7:24:22 AM - started process bpbrm (6724)
5/6/2010 7:24:25 AM - snapshot backup of client pmssdbs08.production.com.vn using method VMware
5/6/2010 7:24:27 AM - Critical bpbrm(pid=6724) from client pmssdbs08.production.com.vn: FTL - snapshot creation failed, status 156   
5/6/2010 7:24:27 AM - Warning bpbrm(pid=6724) from client pmssdbs08.production.com.vn: WRN - ALL_LOCAL_DRIVES is not frozen    
5/6/2010 7:24:28 AM - end Create Snapshot; elapsed time: 00:00:06
5/6/2010 7:24:28 AM - end writing

bpfis
07:24:22.980 [5496.1616] <4> bpfis: INF - BACKUP START 5496
07:24:22.980 [5496.1616] <2> bpfis main: receive filelist:<NEW_STREAM>
07:24:22.980 [5496.1616] <2> bpfis main: receive filelist:<ALL_LOCAL_DRIVES>
07:24:22.996 [5496.1616] <2> bpfis main: receive filelist:<CONTINUE>
07:24:23.042 [5496.1616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
07:24:23.042 [5496.1616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
07:24:23.042 [5496.1616] <2> logconnections: BPRD CONNECT FROM 128.235.106.171.50388 TO 128.235.106.171.13724
07:24:25.168 [5496.1616] <2> get_long: (2) premature end of file (byte 1)
07:24:25.168 [5496.1616] <2> bprd_read_text_file: get_string() failed, Access is denied.  (5), premature end of file encountered
07:24:25.168 [5496.1616] <8> bpfis: WRN - VfMS error 10; see following messages:
07:24:25.168 [5496.1616] <8> bpfis: WRN - Non-fatal method error was reported
07:24:25.168 [5496.1616] <8> bpfis: WRN - vfm_configure_fi_one: method: FlashSnap, type: FIM, function: FlashSnap_init
07:24:25.168 [5496.1616] <8> bpfis: WRN - VfMS method error 3; see following message:
07:24:25.168 [5496.1616] <8> bpfis: WRN - FlashSnap_init: Veritas Volume Manager not installed.
07:24:25.168 [5496.1616] <8> bpfis: WRN - VfMS error 10; see following messages:
07:24:25.168 [5496.1616] <8> bpfis: WRN - Non-fatal method error was reported
07:24:25.168 [5496.1616] <8> bpfis: WRN - vfm_configure_fi_one: method: vxvm, type: FIM, function: vxvm_init
07:24:25.168 [5496.1616] <8> bpfis: WRN - VfMS method error 3; see following message:
07:24:25.168 [5496.1616] <8> bpfis: WRN - vxvm_init: Veritas Volume Manager not installed.
07:24:25.183 [5496.1616] <4> bpfis: INF - FIS_ID=pmssdbs08.production.com.vn_1273105461
07:24:25.183 [5496.1616] <4> bpfis: INF - Freezing ALL_LOCAL_DRIVES using snapshot method VMware.
07:24:25.183 [5496.1616] <4> bpfis: INF - ACT=pmssdbs08.production.com.vn
07:24:25.183 [5496.1616] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_5496
07:24:25.230 [5496.1616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
07:24:25.230 [5496.1616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
07:24:25.230 [5496.1616] <2> logconnections: BPRD CONNECT FROM 128.235.106.171.50395 TO 128.235.106.171.13724
07:24:27.933 [5496.1616] <4> bpfis: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_5496
07:24:27.933 [5496.1616] <32> bpfis: FTL - VfMS error 11; see following messages:
07:24:27.933 [5496.1616] <32> bpfis: FTL - Fatal method error was reported
07:24:27.933 [5496.1616] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
07:24:27.933 [5496.1616] <32> bpfis: FTL - VfMS method error 7; see following message:
07:24:27.933 [5496.1616] <32> bpfis: FTL - VMware_freeze: Unable to locate VM pmssdbs08.production.com.vn
07:24:27.933 [5496.1616] <32> bpfis: FTL - VfMS error 11; see following messages:
07:24:27.933 [5496.1616] <32> bpfis: FTL - Fatal method error was reported
07:24:27.933 [5496.1616] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
07:24:27.933 [5496.1616] <32> bpfis: FTL - VfMS method error 7; see following message:
07:24:27.933 [5496.1616] <32> bpfis: FTL - VMware_freeze: Unable to locate VM pmssdbs08.production.com.vn
07:24:27.933 [5496.1616] <16> bpfis: FTL - snapshot creation failed, status 156
07:24:27.933 [5496.1616] <4> bpfis: INF - Thawing ALL_LOCAL_DRIVES using snapshot method VMware.
07:24:27.933 [5496.1616] <8> bpfis: WRN - ALL_LOCAL_DRIVES is not frozen
07:24:27.933 [5496.1616] <8> bpfis: WRN - snapshot delete returned status 20
07:24:27.980 [5496.1616] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
12 REPLIES 12

pruvn
Level 5
This error is randomly occur to serveral [1-4] of my 14 VMs.

Thanks for any comment,

Vic

rj_nbu
Level 6
Employee Accredited Certified

HI,

this is the error in the logs

VMware_freeze: Unable to locate VM pmssdbs08.production.com.vn

Delete the VM from the nbu policy, add the client, and make sure that you refresh the cache




pruvn
Level 5
Thanks for comment, I did try to remove and refresh topology & add the VM again ... but the issue is still the same. I have a question, this machine have 2 record in DNS, is it a potential problem ?

Vic

GG0001
Level 3
Vic,

Just to let you know as I think you have the exact same error as me ,I got a hotfix from support that resolved it,its about 35mb if you want it or you can of course call up the support guys and grab it from them : eebinstaller.1976029.1.AMD64 thats the file name if you need to tell them it,works for intel and amd they said.Hope it helps
GG

Chuck_Stevens
Level 6
I'm getting a similar error on a small group of VMWare guest servers.

bpfis log:

14:10:31.622 [8396.7068] <2> logparams: D:\Program Files\Veritas\NetBackup\bin\bpfis.exe create -nbu -owner NBU -id idmz02app39_1273698640 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 29 -vm_type 2 -vm_client idmz02app39 -clnt insseabumed02.pemcoins.net -st FULL -rg root -fim VMware:file_system_optimization=1,snapact=0,Virtual_machine_backup=2,vmmono=0,disable_quiesce=0,nameuse=1,trantype=6,nameuse=1 -class PMIC_VM_Test_Wednesday -sched Full_Weekly -fso -S pccsseabumstr01
14:10:31.622 [8396.7068] <2> bpfis main: received FIM as [126] VMware:file_system_optimization=1,snapact=0,Virtual_machine_backup=2,vmmono=0,disable_quiesce=0,nameuse=1,trantype=6,nameuse=1
14:10:31.622 [8396.7068] <4> bpfis: INF - BACKUP START 8396
14:10:31.622 [8396.7068] <2> bpfis main: receive filelist:<NEW_STREAM>
14:10:31.622 [8396.7068] <2> bpfis main: receive filelist:<ALL_LOCAL_DRIVES>
14:10:31.622 [8396.7068] <2> bpfis main: receive filelist:<CONTINUE>
14:10:31.637 [8396.7068] <2> vnet_connect_to_vnetd_extra: ../../libvlibs/vnet_vnetd.c.188: msg: VNETD CONNECT FROM 172.16.4.99.4663 TO 10.200.18.20.13724 fd = 480
14:10:31.715 [8396.7068] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:10:31.715 [8396.7068] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
14:10:31.918 [8396.7068] <2> logconnections: BPRD CONNECT FROM 172.16.4.99.4663 TO 10.200.18.20.13724
14:10:34.122 [8396.7068] <2> get_long: (2) premature end of file (byte 1)
14:10:34.122 [8396.7068] <2> bprd_read_text_file: get_string() failed, Access is denied.  (5), premature end of file encountered
14:10:34.122 [8396.7068] <8> bpfis: WRN - VfMS error 10; see following messages:
14:10:34.122 [8396.7068] <8> bpfis: WRN - Non-fatal method error was reported
14:10:34.122 [8396.7068] <8> bpfis: WRN - vfm_configure_fi_one: method: FlashSnap, type: FIM, function: FlashSnap_init
14:10:34.122 [8396.7068] <8> bpfis: WRN - VfMS method error 3; see following message:
14:10:34.122 [8396.7068] <8> bpfis: WRN - FlashSnap_init: Veritas Volume Manager not installed.
14:10:34.122 [8396.7068] <8> bpfis: WRN - VfMS error 10; see following messages:
14:10:34.122 [8396.7068] <8> bpfis: WRN - Non-fatal method error was reported
14:10:34.122 [8396.7068] <8> bpfis: WRN - vfm_configure_fi_one: method: vxvm, type: FIM, function: vxvm_init
14:10:34.122 [8396.7068] <8> bpfis: WRN - VfMS method error 3; see following message:
14:10:34.122 [8396.7068] <8> bpfis: WRN - vxvm_init: Veritas Volume Manager not installed.
14:10:34.153 [8396.7068] <4> bpfis: INF - FIS_ID=idmz02app39_1273698640
14:10:34.153 [8396.7068] <4> bpfis: INF - Freezing ALL_LOCAL_DRIVES using snapshot method VMware.
14:10:34.153 [8396.7068] <4> bpfis: INF - ACT=idmz02app39
14:10:34.153 [8396.7068] <4> bpfis: INF - Created mount point D:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_8396
14:10:34.153 [8396.7068] <2> vnet_connect_to_vnetd_extra: ../../libvlibs/vnet_vnetd.c.188: msg: VNETD CONNECT FROM 172.16.4.99.4670 TO 10.200.18.20.13724 fd = 416
14:10:34.231 [8396.7068] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:10:34.231 [8396.7068] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
14:10:34.434 [8396.7068] <2> logconnections: BPRD CONNECT FROM 172.16.4.99.4670 TO 10.200.18.20.13724
14:11:20.435 [8396.7068] <4> bpfis: INF - Deleted mount point D:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_8396
14:11:20.435 [8396.7068] <32> bpfis: FTL - VfMS error 11; see following messages:
14:11:20.435 [8396.7068] <32> bpfis: FTL - Fatal method error was reported
14:11:20.435 [8396.7068] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
14:11:20.435 [8396.7068] <32> bpfis: FTL - VfMS method error 7; see following message:
14:11:20.435 [8396.7068] <32> bpfis: FTL - VMware_freeze: VIXAPI freeze failed with 36
14:11:20.435 [8396.7068] <32> bpfis: FTL - VfMS error 11; see following messages:
14:11:20.435 [8396.7068] <32> bpfis: FTL - Fatal method error was reported
14:11:20.435 [8396.7068] <32> bpfis: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
14:11:20.435 [8396.7068] <32> bpfis: FTL - VfMS method error 7; see following message:
14:11:20.435 [8396.7068] <32> bpfis: FTL - VMware_freeze: VIXAPI freeze failed with 36
14:11:20.435 [8396.7068] <16> bpfis: FTL - snapshot creation failed, status 156
14:11:20.435 [8396.7068] <4> bpfis: INF - Thawing ALL_LOCAL_DRIVES using snapshot method VMware.
14:11:20.435 [8396.7068] <8> bpfis: WRN - ALL_LOCAL_DRIVES is not frozen
14:11:20.435 [8396.7068] <8> bpfis: WRN - snapshot delete returned status 20
14:11:20.435 [8396.7068] <2> vnet_connect_to_vnetd_extra: ../../libvlibs/vnet_vnetd.c.188: msg: VNETD CONNECT FROM 172.16.4.99.4798 TO 10.200.18.20.13724 fd = 480
14:11:20.513 [8396.7068] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2054: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
14:11:20.513 [8396.7068] <2> vnet_vnetd_service_socket: ../../libvlibs/vnet_vnetd.c.2068: service: bprd
14:11:20.700 [8396.7068] <2> logconnections: BPRD CONNECT FROM 172.16.4.99.4798 TO 10.200.18.20.13724
14:11:21.544 [8396.7068] <4> bpfis: INF - EXIT STATUS 156: snapshot error encountered

Reagan
Level 5
Partner Accredited Certified

Symantec support recommended uninstalling and reinstalling VMware tools on the virtual machines.  Tried that and the error went awat for a couple of days and then it came back.  Snapshots from within the virtual machine itself are successful.  If there is a hotfix that resolves this issue, I wish Symantec would release it to the public for download.  The logging could also use improvement as well.

GG0001
Level 3
Heya,

Seems Symantec have published a tecnote on the hotfix that resolved my issue seems to patch the vmwaretools dll's mabye it will fix your problem as well can take a read :

DOCUMENTATION: Issues resolved in the bundle for VMware backups included in Etrack 1976029

this was posted in my thread when I had similar issue.It was posted today as well so brand new.

Chuck_Stevens
Level 6
Hmm. The log entries in that tech note don't match my issue; I'm not seeing any errors in the Windows event logs on the backup host.

I'll try reinstalling the VM tools, see if that helps.

thesanman
Level 6
Beware.  I got this EEB for 156 VMware errors and it plain broke all my VMware backups!  Luckily, it's my test environment.

Look in the bpfis log on your VMware Backup Host; in my case I could see a timeout happening during the VM quiese:

15:25:57.828 [3220.4988] <2> onlfi_vfms_logf: INF - vmwareLogger: WaitForTaskComplete: Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.

This pointed me at VMware tools on the client which appears to be broken!  VSS errors etc so I need to fix that.

IMHO, the 156 error is a general error; you need to dig deeper via the bpfis log file to determine the exact cause.  And yes, make sure your VMware Tools are up to date and working!

pruvn
Level 5
@GG0001: Long time back to this thread, Yeah I saw that you use VCB backup, while I use vStorage backup then your patch might not be applied for my case.

Now I disabled the quiesce snapshot as Symantec support guys suggest. The backup process is now failed with another reason, NBU discover VMs via vCenter sometimes as FQDN, sometimes as NETBIOS name. This is why 1-2 VMs failed each night. In general, these processes are inconsistent !

Vic

marekkedzierski
Level 6
Partner

1. Have you tried to create snapshot of problematic VM from vCenter ?
2. Your Vmware Backup Host should resolve DNS and revDNS of each VM. You can try to inventory VC using Display name as a host name for backup purposes. Then NB will use for examle SAP, SENTINEL, Suse11 as a client name.

pratwani
Level 3
Hi,
I found this error while taking an Off-host backup, using VSS.
"FTL:snapshot creation failed status 156:: WRN - ALL_LOCAL_DRIVES is not frozen"
I have a  Master Server on Win2k8R2(M1) and an alternate client (M2)on Win2k8R2 as well, also a media server running on M1.
The guest OS on the hyperv server(M1) is Shut Down during the backup.
I could not find bpfs log directory inside C:/ProgFiles/Veritas/Netback/log and hence not able to find the root cause of the problem.
I have also tried to increase the limit (no limit) for shadow copies but that has not worked.

I have tried to create a full backup 'without alternate client" as well and that works perfectly.