VMWare Sharepoint Backups

[Spartacus81]

Friends,

NBU Master 7.5.0.6 W2K8 64bit

NBU Media 7.5.0.6 SuSe 10 SP2 Linux 64bit as Proxy Server

Sharepoint FEWS 2010 W2K8 SP 1

Sharepoint SQL Backend  W2K8 SP 2

Sharepoint agent backups are working fine both GRT (ALLWebs) and non-GRT (Microsoft Sharepoint resources:\*)

I am now trying to backup the Sharepoint vm using vmware policy type by selecting the option enable sharepoint recovery directly off the ESX host not the virtual center server. ESX host added by using the root credential into the virtual machine servers.

Backup kicks in but the new job ASC (Application State Capture) failing with error 13 but the subsequent job of the whole vm after the snapshot runs fine.

========================================================

Job log extract:

13/08/2014 12:09:31 - end Application State Capture, Step By Condition; elapsed time: 00:00:00
13/08/2014 12:09:31 - begin Application State Capture, Application State Capture
13/08/2014 12:10:41 - Critical ascc(pid=1440) SharePoint: snapshot processing failed, status 156       
13/08/2014 12:10:41 - Critical ascc(pid=1440) SharePoint: snapshot creation failed - Error occurred while attempting to perform ReadObj on \Metadata., status 13
13/08/2014 12:10:41 - Critical ascc(pid=1440) SharePoint: snapshot creation failed - Error attempting to gather metadata., status 130 
13/08/2014 12:10:42 - Warning ascc(pid=1440) SharePoint: Microsoft SharePoint Resources:\* is not frozen      
Status 13
13/08/2014 12:10:43 - end Application State Capture, Application State Capture; elapsed time: 00:01:12
Status 13
13/08/2014 12:10:43 - end Parent Job; elapsed time: 00:01:12
file read failed(13)

==========================================================

bpfis logs:

==========================================================

11:58:46.486 [10024.2352] <2> onlfi_fi_fs_unlock: INF - buffered logs are flushed
11:58:46.486 [10024.2352] <32> onlfi_freeze: FTL - VfMS error 11; see following messages:
11:58:46.486 [10024.2352] <32> onlfi_freeze: FTL - Fatal method error was reported
11:58:46.486 [10024.2352] <32> onlfi_freeze: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
11:58:46.486 [10024.2352] <32> onlfi_freeze: FTL - VfMS method error 4; see following message:
11:58:46.486 [10024.2352] <32> onlfi_freeze: FTL - VSS_Writer_freeze_commit: snapshot create failed
11:58:46.486 [10024.2352] <2> onlfi_thaw: INF - fim=VSS_Writer
11:58:46.486 [10024.2352] <4> onlfi_thaw: INF - Thawing Microsoft SharePoint Resources:\* using snapshot method VSS_Writer.
11:58:46.486 [10024.2352] <2> do_thaw: INF - do_thaw: parent: thaw flag on
11:58:46.486 [10024.2352] <4> onlfi_thaw: INF - do_thaw return value: 0
11:58:46.502 [10024.2352] <4> delete_mount_point: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_Microsoft SharePoint Resources__10024_1
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=auto
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=Hyper-V
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=Hyper-V_v2
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=VSS
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=OST_FIM
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=RealTime
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=VMware
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=VMware_v2
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=NAS_Snapshot
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - resyncing VSS_Writer object 
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   f_type=0 (F_REG)
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fs_name=Microsoft SharePoint Resources:\*
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fs_type=no_root
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   mnt_device=Microsoft SharePoint Resources:\*
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   raw_device=
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   volume_group=
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_fs_name=C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_Microsoft SharePoint Resources__10024_1
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_host=
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_host_fs_name=
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_mnt_device=
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_raw_device=
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_args= [size = 265] prov_type=1,snap_attr=0,max_snapshots=1,APP_TYPE=FS,BACKUP_TIME=1407931060,POLICY_NAME=unknown,NBU_CLIENT=MTSP02.ix.com,NBU_MASTER=dcgmon.ix.com,SCHED_NAME=unknown,SCHED_TYPE=FULL,POLICY_TYPE=8,PFI_BACKUP=0,NBU_CATALOG=1,GRAN_BACKUP=0,VIRT_BACKUP=1,SPSV2_FE_CLIENT=
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_handle=0000000001FAEF40
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   mnt_flag=0
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fi_flag=1
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   fm_flag=0
11:58:46.502 [10024.2352] <2> log_fs_dev_rec: INF -   cs_flag=0
11:58:46.502 [10024.2352] <2> onlfi_vfms_logf: INF - INF - vss__delete: ***** Delete snapped volumes *****
11:58:46.502 [10024.2352] <2> bpfis main: FTL - snapshot processing failed, status 156
11:58:46.502 [10024.2352] <2> onlfi_vfms_logf: INF - Extended status = 13:1:Error occurred while attempting to perform ReadObj on \Metadata.
11:58:46.502 [10024.2352] <16> bpfis main: FTL - snapshot creation failed - Error occurred while attempting to perform ReadObj on \Metadata., status 13
11:58:46.502 [10024.2352] <2> onlfi_vfms_logf: INF - Extended status = 130:1:Error attempting to gather metadata.
11:58:46.502 [10024.2352] <16> bpfis main: FTL - snapshot creation failed - Error attempting to gather metadata., status 130
11:58:46.502 [10024.2352] <2> onlfi_thaw: INF - fim=VSS_Writer
11:58:46.502 [10024.2352] <4> onlfi_thaw: INF - Thawing Microsoft SharePoint Resources:\* using snapshot method VSS_Writer.
11:58:46.502 [10024.2352] <8> onlfi_thaw: WRN - Microsoft SharePoint Resources:\* is not frozen

======================================================

any suggestion will highly be appreciated. Cheers

[RiaanBadenhorst]

Hello

Just to clarify, you're operating within these limits / confines

The following limitations exist when you configure a VMware policy to protect SharePoint Server:

• VMware incremental backups of SharePoint Server are not supported with this version of NetBackup.
• SQL Servers cannot be clustered.
• The Application State Capture (ASC) job fails and the databases are not protected if you do any of the following:
  • Disable the Virtual Machine quiesce option.
  • Select the Exclude data disks option.

Databases are cataloged and protected only if they exist in a configuration that is supported for VMware backups. As long as there are any databases that can be protected, the ASC job continues. If you select databases for backup that exist on supported and on unsupported disks, the ASC job produces a status 1 (partially successful). The ASC job detects these situations and the job details include the result of the backup operation.

SharePoint Server databases are not cataloged and backed up if they exist on the following:

• Raw device mapping (RDMs). Make sure that the SharePoint virtual machine does not use RDM as storage for databases.
• Virtual Machine Disk (vmdk) volumes that are marked as independent. Make sure that the SharePoint databases are not stored on independent disks.
• Mount point volumes.
• Virtual hard disks (VHDs). If NetBackup detects any database objects on a VHD, the ASC job fails and no SharePoint content is cataloged. This exclusion includes any objects that do not exist on the VHD.
• Any components that reside on a physical machine are not backed up with the VMware backup
• The ASC job detects an excluded Windows boot disk and treats it like an independent disk. The VMware backup cannot exclude for any reason the disk on which NetBackup is installed. For example, do not select the Exclude boot disk option if NetBackup is installed on the boot drive (typically C:).

[Spartacus81]

Thanks Riaan and sorry for the late response.. i was away.

I red that paragraph too in the admin guide and we are all good. I have just installed the Symantec VSS providers and ASC seems to have worked on backend SQL vm but still failing on front end web vm with error 13.. 

26/08/2014 14:23:51 - begin Application State Capture, Application State Capture

26/08/2014 14:24:55 - Critical ascc(pid=9858) SharePoint: snapshot processing failed, status 156       
26/08/2014 14:24:55 - Critical ascc(pid=9858) SharePoint: snapshot creation failed - Error occurred while attempting to perform ReadObj on \Metadata., status 13
26/08/2014 14:24:55 - Critical ascc(pid=9858) SharePoint: snapshot creation failed - Error attempting to gather metadata., status 130 
26/08/2014 14:24:55 - end Application State Capture, Application State Capture; elapsed time: 00:01:04
Status 13
26/08/2014 14:24:55 - end Parent Job; elapsed time: 00:01:04
26/08/2014 14:24:56 - Warning ascc(pid=9858) SharePoint: Microsoft SharePoint Resources:\* is not frozen      
Status 13
file read failed(13)

there appears to be snapshot error.. i have checked the bpfis logs and shows the following.. its definitely not a permission issue as subsequent job where it backs up the whole vm after taking the snapshot finishes okay. any idea?

13:24:48.668 [8588.7796] <32> bpfis: FTL - VfMS error 11; see following messages:
13:24:48.668 [8588.7796] <32> bpfis: FTL - Fatal method error was reported
13:24:48.668 [8588.7796] <32> bpfis: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
13:24:48.668 [8588.7796] <32> bpfis: FTL - VfMS method error 4; see following message:
13:24:48.668 [8588.7796] <32> bpfis: FTL - VSS_Writer_freeze_commit: snapshot create failed
13:24:48.668 [8588.7796] <32> bpfis: FTL - VfMS error 11; see following messages:
13:24:48.668 [8588.7796] <32> bpfis: FTL - Fatal method error was reported
13:24:48.668 [8588.7796] <32> bpfis: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
13:24:48.668 [8588.7796] <32> bpfis: FTL - VfMS method error 4; see following message:
13:24:48.668 [8588.7796] <32> bpfis: FTL - VSS_Writer_freeze_commit: snapshot create failed
13:24:48.668 [8588.7796] <4> bpfis: INF - Thawing Microsoft SharePoint Resources:\* using snapshot method VSS_Writer.

[RiaanBadenhorst]

Hello,

No quick fix, you'll have to check the logs. Try looking at ncfnbcs and event viewer logs in the vm.

[Spartacus81]

ncfnbcs showing these... any interpertation? i can't seem to find anything related to this problem through google..

ProcessDefinition::run,1,(49|i32:8588|A136:bpfis.exe create -vm_pre -fim auto -id MTSP02.ix.com_1409059437 -pt 8 -clnt MTSP02 -catalog -st FULL "Microsoft SharePoint Resources:\*"|)
1,51216,309,366,115,1409059496293,7776,10072,0:,0:,30:ProcessDefinition::waitForDone,1,(46|i32:8588|A136:bpfis.exe create -vm_pre -fim auto -id MTSP02.ix.com_1409059437 -pt 8 -clnt MTSP02 -catalog -st FULL "Microsoft SharePoint Resources:\*"|u32:130|)
0,51216,309,366,116,1409059496293,7776,4600,0:,194:Unblocking for status for command bpfis.exe create -vm_pre -fim auto -id MTSP02.ix.com_1409059437 -pt 8 -clnt MTSP02 -catalog -st FULL "Microsoft SharePoint Resources:\*" (../ProcessMgr.cpp:164),28:ProcessDefinition::getStatus,1
0,51216,309,366,117,1409059496293,7776,4600,0:,126:successfully opened temporary file: C:\Users\NETBAC~1\AppData\Local\Temp\NBU77760000000001F4E7F0.TMP (../LegacyNBUCli.cpp:221),25:LegacyNBUCli::loadResults,1
0,51216,403,366,1,1409059496293,7776,4600,0:,48:unexpected type (../SharepointIteration.cpp:381),32:SharepointIteration::doIteration,1
0,51216,403,366,2,1409059496293,7776,4600,0:,48:unexpected type (../SharepointIteration.cpp:381),32:SharepointIteration::doIteration,1
0,51216,403,366,3,1409059496293,7776,4600,0:,66:converting failure status:  to 13 (../SharepointIteration.cpp:408),32:SharepointIteration::doIteration,1
2,51216,403,366,4,1409059496324,7776,4600,0:,0:,0:,0,(1|)
0,51216,309,366,118,1409059496324,7776,4600,0:,46:Failed to sync buffers (../NcfStreams.cpp:238),45:AceHandleStreamBuffer::~AceHandleStreamBuffer,1

[RiaanBadenhorst]

Seems support will be required.
 

[Spartacus81 · Accepted solution]

Right.. known issue.. upgrading the whole enviornment including both FEWS and BE SQL server to 7.5.0.7 and all worked. 

[CRZ]

Did they happen to give you an Etrack number?  I'd like to see if we have this documented AND make sure this was taken care of in 7.6.0.x as well.

(OR could you please PM me your case ID?)

[Spartacus81]

Only just saw you msg chris as i haven't been on community for a while.. 

No they didnt give me the Etrack number but here is the case ID as requested. Case # 06998672 

you might find this interesting hence sharing it.. the backups seems to have worked after upgrading the platform but i can't see the microsoft sharepoint resources out of the this vmware policy type backup in order to restore the single file.. i can see C: :D: E: (if i select restoe from normal backups), i can see VMDKS (if i select restore from virtual machine backup) but the moment i slelect policy type sharepoint  in order to see Microsoft Sharepoint resources it doesnt even pick that servers backup let alone the visibility of the microsoft sharepoint resources...

so the case is still open and Its still with Symantec BL and they have requested heck lot of logs which are being collected...

have you come across this problem? have you tried to restore a single file out the backup vmware sharepoint backup using the policy type sharepoint in the BAR gui?