Unable to make FlashBackup - Windows work.

[smw6180]

I've researched these forums, read all the documentation and I've either missed something simple or there's just plain no way to do this.

Environment:
Netbackup Master Server: Win2k3 x64 NBU Enterprise 6.5.4
Netbackup Media Server : Win2k3 x64 NBU Enterprise 6.5.4 Solutions Enabler with VSS 7.0.1.0

EMC Symmetrix DMX3


Here's what I'm trying to do:

We have many user filesystem with a lot of small files.  In order to speed up our backups I am attempting to use the FlashBackup - Windows policy.
We have BCVs of our production disk mounted to our Media server.
We have tape drives zoned and using SSO on our Media server.

All checks of Tapes/Zones/Disk are fine.  Indeed, normal WindowsNT policy backups work just fine.

Configuration of FlashBackup policy:
Snapshot Method:  VSS
Provider Type 0
Snapshot Attribute 1
Maximum Snapshots 1
(Note:  I've tried changing to every combination of attributes above, no luck)

Array configuration is grayed out.

Backup Selections:
\\.\Y: (Y: is a BCV)

emcsymfi.conf:
[DEBUG_TRACE_INFO]
"TRACERUNTIMEUPDATE"=dword:00000000
"TRACESTDOUT"=dword:00000000
"TRACELEVEL"=dword:00000001
"TRACEFILE"="emcsymfi.log"
"TRACEDIR"="C:\Program Files\Common Files\Symantec Shared\VxFI\4\Logs\"


Here's the problem:
The backup starts, the BCV is unmasked (but not umounted which I guess doesn't matter if the masking is removed).
The BCV is synched with the primary.
At this point the backup errors out, waits 5 minutes and terminates with a 156, leaves the BCV in a synchronized state, and doesn't remask the BCV.

From my BPFIS.LOG:
10:20:46.088 [9732.8824] <4> delete_mount_point: Deleted mount point E:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img___._Y__9732
10:20:46.213 [9732.8824] <4> bpfis: INF - EXIT STATUS 0: the requested operation was successfully completed
10:21:12.338 [9704.9972] <2> onlfi_vfms_logf: snapshot services: emcsymfi:Fri Oct 30 2009 10:21:12.338000 <Thread id - 9972> ERROR: CLI returned; status=11, 3 lines of stdout
10:21:12.338 [9704.9972] <2> onlfi_vfms_logf: snapshot services: emcsymfi:Fri Oct 30 2009 10:21:12.338000 <Thread id - 9972> >> Missing Symmetrix ID.
10:21:12.338 [9704.9972] <2> onlfi_vfms_logf: snapshot services: emcsymfi:Fri Oct 30 2009 10:21:12.338000 <Thread id - 9972> >>
10:21:12.338 [9704.9972] <2> onlfi_vfms_logf: snapshot services: emcsymfi:Fri Oct 30 2009 10:21:12.338000 <Thread id - 9972> >>
10:21:12.338 [9704.9972] <2> onlfi_vfms_logf: snapshot services: emcsymfi: Caught unhandled execution pipe exception VFI_EXEC_PIPE_CMD_FAILED at "C:\Program Files\EMC\SYMCLI\bin\symdisk.exe"  list  -sid  -output XML , providers\emcsymmetrix\emcsym_plugin.cpp, 150
10:21:12.338 [9704.9972] <2> onlfi_vfms_logf: snapshot services: vxfi_analyze fitype=[vss-autoselect-snapshot]
10:21:12.338 [9704.9972] <2> onlfi_vfms_logf:
10:21:42.853 [9704.9972] <2> onlfi_vfms_logf: snapshot services: vss:Fri Oct 30 2009 10:21:42.853000 <Thread id - 9972> VssNode::make: encountered VFI_EPROVIDERVETO
10:21:42.853 [9704.9972] <2> onlfi_vfms_logf: vfm_freeze_commit: vfm_method_errno=[41]
10:21:42.853 [9704.9972] <2> onlfi_vfms_logf: make error [41] encountered. Sleeping for 300 seconds
10:26:42.845 [9704.9972] <2> onlfi_vfms_logf: Retrying again....
10:26:42.970 [9704.9972] <2> onlfi_vfms_logf: vfm_freeze_commit: vfm_method_errno=[6]
10:26:44.970 [9704.9972] <32> onlfi_fim_split: VfMS error 11; see following messages:
10:26:44.970 [9704.9972] <32> onlfi_fim_split: Fatal method error was reported
10:26:44.970 [9704.9972] <32> onlfi_fim_split: vfm_freeze_commit: method: VSS, type: FIM, function: VSS_make
10:26:44.970 [9704.9972] <32> onlfi_fim_split: VfMS method error 6; see following message:
10:26:44.970 [9704.9972] <32> onlfi_fim_split: snapshot services: snapshot creation failed: provider-private error.
10:26:44.970 [9704.9972] <32> onlfi_freeze: VfMS error 11; see following messages:
10:26:44.970 [9704.9972] <32> onlfi_freeze: Fatal method error was reported
10:26:44.970 [9704.9972] <32> onlfi_freeze: vfm_freeze_commit: method: VSS, type: FIM, function: VSS_make
10:26:44.970 [9704.9972] <32> onlfi_freeze: VfMS method error 6; see following message:
10:26:44.970 [9704.9972] <32> onlfi_freeze: snapshot services: snapshot creation failed: provider-private error.


Now this APPEARS to be failing due to the missing symmetrix ID.  I say appears because I'm not entirely sure that's the issue, but it seems indicated by the error I list.  My question is:  Since the Array configuration area under Snapshot Options is grayed out, do I need to put my symid in emcsymfi.conf?  If so, what is the syntax?  After hours of scanning manuals and websites I haven't found it.

If that's not the case does anyone know if it's even possible to do what I'm attempting to do, and if so, where did I make a mistake in my configurations?

Thanks in advance folks.

 -Steve

[Marianne]

Have you read the Snapshot Client Admin Guide?
ftp://exftpp.symantec.com/pub/support/products/NetBackup_Enterprise_Server/290224.pdf
In particular the section called "Configuration of snapshot methods for disk arrays"

[smw6180]

Yep.  Under the Flashbackup-Windows policy it doesn't give me the option to specify timefinder.  I get auto, Hyper-V, VSP, VSS, vxvm.

We're using VSS (as I said I have solutions enabler with the VSS provider).

Now if I change the policy method to Flashbackup (not 'windows') I get those options for timefinder.  But from what I've read I need the Flashbackup-Windows policy method.  True?

[Marianne]

I'll go re-read the manual again - I have personally configured disk array snapshots on Unix (Oracle and Standard policy), and Flashbackup for Windows with vxvm, but in all honesty not FlashBackup for Windows combined with array snapshots.

I think you might be better off logging a call with Symantec.
The other option is to try normal MS-Windows-NT policy with array snapshot...

[smw6180]

Unfortunately we only have Windows Master/Media servers and moving them to Unix isn't an option.  Also unfortunately the issues with these particular backups is that they're file servers with a lot of small files (tens of thousands), which kills our backup speeds.  So the thought was to use flashbackups in order to be able to get block level throughput, combined with the ability to do file based restores.  So basically using the MS-Windows-NT policy doesn't really do us a lot of good right now.

I appreciate your willingness to look into it for me.  It's possible I missed something and having someone to doublecheck me is nice.

As for logging a call with Symantec, we've had several calls into them for this and other issues, and they haven't been getting back to us.  That's why I'm posting here;  to see if anyone else had had experience with this, or could find something I missed.

Thanks again!
-Steve

[Marianne · Accepted solution]

I've checked the Snapshot Client  compatibility guide - your config is supported:


Next step is to go to Snapshot Client Admin Guide, p. 147:
VSS configuration (Windows)
Follow the steps under the Symmetrix heading - p. 148 - 150.

Once this is working fine, double-check policy config - the client name needs to be the source client (where source volume containing lots of small files reside). Media server name will be specified on Attributes tab when 'Use alternate client' is selected.
In Backup Selection, specify the actual drive letter (source_drive letter) on the source client.

The backup process will call all the vshadow and symsnap commands to snap off the BCV volume on the source client and mount it to a temp folder on the media server.

The images will contain the original client name and original drive letter.


Hope this helps!

[smw6180]

Wow.  See, that's what I mean about a second set of eyes.  I just re-read where you pointed me and I found a definite issue with what I've been doing:

Note: In this 6.5 release, for Symmetrix arrays, NetBackup supports VSS with differential (copy-on-write) backup but not plex-based (clone or mirror) backup.


That seems to state that BCVs just plain won't work, and I'll need to convert them to VDEVs.  I'll test it out here and post back.

Thanks again for all your help!

[smw6180]

Ok, finally had some time to mess around with this.  I think I'm pretty well unable to use flash backups with Symmetrix devices.

 I can't use timefinder mirrors or clones.
 I HAVE to use snap devices.

 The problem is that the snap sessions are dragging down the production box.  Given that the server runs 24/7 with a reasonably full user load I'm not going to be able to find a good time to have the snaps running where they won't impact our production server.

I never did get to testing it with NBU.  The slowdowns the snap sessions were causing sealed the deal.

Thanks again for all your help though!

 -Steve