EV10 DR solution

[KeirL]

Hi

I think this is a topic that has been discussed several times, but I'm still struggling to fully grasp the options to design an EV DR solution. My environment will consist of three VMware vm's - 1 x SQL 2008 vm for the EV databases, 1 x EVserver for Exchange email archiving and 1 x EVserver for Exchange journal archiving. Storage will reside on a NetApp array in the production datacentre and there is a second NetApp in the DR datacentre which is connected via a 1Gb link.

The archives are likely to grow to about 50TB+  Indexes up ro 10TB and SQL DB's 500GB

So what are my best options for a DR solution - I want to achieve a low RPO  - perhaps losing no more than 15mins of data changes

How 'consistent' does this need to be? Can I create the SQL server and storage as VMDK's and also the Index locations as VMDK's and use VMware snapshots with SRM to replicate this to the DR datacentre? If so, how do I then capture the archive partitions? and do snapshots of the SQL databases, Index volumes and Vault stores need to occur simaultaneously?

I want to leverage the NetApp replication capabilities (snap mirror) and VMware SRM\Storage vMotion but need some guidance on how best to set this up and what the realistic capabilities are when it comes to consistent\inconsistent snapshots etc.

thanks in advance

Keir 

[Rob_Wilcox1]

You may want to take away some of the pain of the unknown technologies by using something like EVnearSync from us at QUADROtech. http://www.quadrotech-it.com/products/evnearsync/

[KeirL]

Thanks Rob

So does that give me the consistency I'm looking for?

I looked at it and it seems very good but I think I'm still missing the point on 'consistency' somehow.

for example there is an FAQ that says:

Do I still have to backup the Enterprise Vault Index?

Not necessarily. If you want to replicate the Index, you could use the “Index Replication” 

in EVnearSync and define a recurring schedule (i.e. every 8h). Only the modified data 

will be replicated. Of course you can create snapshots of older indexes on the target.

If I take the option to NOT replicate my indexes how do I then search for emails should my production DC fail and I envoke DR?  and if I decide to replicate the Indexes every 1Hr but my SQL databases and archive items are replicating in near realtime then won't I end up with inconsistent Indexes and have archived items that will not exist within my DR index location?

Thanks again for your input.

Cheers K

[KeirL]

So perhaps I'm being unnecessarily cautious and concerned with keeping the sql databases, EV archives and EV indexes 'in synch' at the DR site.

If I just took crash consistent snapshots at regular intervals but didn't strive to ensure they were all taken at precisely the same point in time... what would be the effect when I tried to start the EV services at the DR site?

Does EV have built in consistency checkers that would allow the services to start but perhaps lose a couple of items that may be orphaned by the non-consistent snapshot approach?

I guess indexes can be rebuilt, but that's a long process that I'd rather avoid.

NetApp seems to be a good and frequent choice as an EV platform - is it a common approach just to use snap mirror to take the snapshots and replicate these on a, say hourly, basis?

Perhaps I could create a script to momentarily put EV in backup mode every hour to take the NetApp snaphot?

thanks again

[Rob_Wilcox1]

hmm I see where you're going with this - have you made any contact with VMWare experts?  They might give you a better idea in that direction.

Also thinking about it perhaps EVnearSync doesn't fit all your requirements here. It's more for storage replication, where the main server stays 'up'.  It's possible to do what you've suggested, but isn't quite as slick as it's going to need to be.

[TheEmptyMind]

We have EV storage and SQL storage on NetApp filers and we do the following for DR:

- Our EV and SQL servers are physical and DR servers are virtual

- We take hourly snapshots on both EV and SQL without keeping EV in read-only mode i.e., no quiescing, and SnapMirror them to DR NetApp filer.  We are aware of a possible data loss by not quiescing, but we are OK with that.  We cannot put EV in read-only mode once an hour unfortunately as this may impact users.

- We put EV in read-only mode once a day and take EV and SQL snapshots.  This is a gold/fallback snapshot for us, should something not work with the hourly snaoshot.

- During DR, EV is supposed to be in read-only mode (unless things go otherwise) and Flexclones for the mirrored volumes are mounted on the EV DR servers (once the volumes are disconnected, any data that got written is lost).  If we were to run EV in read-write mode though, we would have to make sure that the data that gets written to the DR volumes is not lost and we replicate the same back to the primary filers.  SnapMirror is supposed to sync the delta though, so hopefully, this shouldn't take forever to finish.

Let me know if you need more information or if something is unclear.

[KeirL]

Thanks for sharing this - this is exactly the type of info I was hoping for.

So when you take your snapshots for SQL and EV are they done simaultaneously? are they on the same NetApp volume or do you just schedule the snapshots for the same time so they should be 'reasonably' consistent with each other?

Have you had to failback to your gold snapshots or generally are you ok with the non-quiesced ones? Have you had to run any tools to get EV to sort our any inconsistencies itself?

kind regards

Keir

[TheEmptyMind · Accepted solution]

So when you take your snapshots for SQL and EV are they done simaultaneously? are they on the same NetApp volume or do you just schedule the snapshots for the same time so they should be 'reasonably' consistent with each other?

We tried to schedule the EV and SQL snapshots reasonably close in time, but it isn't a guarantee that they would happen simultaneously resulting in consistent data.  We may end up with more data on EV volumes and SQL may not be aware of it or more data in SQL and EV storage may not contain the data i.e., inconsistent.

Have you had to failback to your gold snapshots or generally are you ok with the non-quiesced ones? Have you had to run any tools to get EV to sort our any inconsistencies itself?

We were able to serve our users (recently, during Sandy craziness) with the non-quiesced snapshots for about 3 days in read-only mode just fine.  Should we have got calls from users about shortcuts not opening or other weird behavior, I would have failed over to the gold snapshot and announced a data loss scenario until we failed back to our primary data center or permanently, depending on the recoverability of the primary data center.

As far as fixing inconsistencies, yes, I did have some experience with that.  Our DR SQL replication software was buggy, unfortunately, and we only discovered that during a DR scheduled test (that's what these tests are for, anyway).  We ran EV in read-write mode in DR and failed back the data to our primary data center.  We noticed that SQL was missing data (about 40K rows) and we have more archived data on our EV volumes.  We spoke with Symantec support and they quickly gave us a tool called "DVS Checker", which recreated the missing SQL entries and we are back in game.

Hope this helps!