Should we create automated process for setting up DR process?

[nbmagic]

The management asks us to use as much automated process as possibleto set up Storage LUNs/Volumes/Permissions for DR exercise and real DR.

To automate the process, one option is to scripting as much as possible. The other option is to use a GUI driven software on one platform which will be also depending on the  other software on the othe platform.  Both way should be faster than manual steps. I would say probably save half hour or so.

Should this sound a right to do?

Some here don't think this is not right way to set up DR, because we should not use scripting / software. What if the script or the software got broken?  not mention this software has quite depeendencies, and not easy to configure/ fix.

The automated process may be faster to set up DR, but may also cause troubles should any issues with scripts, or any dependencies of the software got broken. It would require high knowlegeble IT professional to fix. What if they are not available then?  Manual process looks a little bit slower, but clearly defined each step and it's requirement, easy to fix should an error occur as long as documented well.  the DR process  should be designed for IT operator level who has basic computer skills.

What is your opinion? Should  we use either script or software?

Thanks for sharing!

[nbmagic]

I am sorry, but can anybody please share your ideas?

[Marianne]

If you want automated failover without any scripting, invest in decent cluster software. Like Veritas cluster in conjunction with Storage Foundation. Product name now is Infoscale Availability.

[nbmagic]

We don't budget for cluster software, and it may not be needed here.

I am not saying that we don't want to have any script, if should if can. However, what will you feel about the overall concern I put with regards using highly dependent software?

[nbutech]

Hello,

Are you looking from NBU for "automated process as possible to set up Storage LUNs/Volumes/Permissions for DR exercise ?" 

Well then I would not expect NBU to do this .. NBU has tried much automation for restoring the catalog using the DR file. All you need is DR file to recover ... 

Storage LUNs/Volumes/Permissions should come from the SAN side ...?

[Marianne]

If you want automotive failover, then invest in cluster software. I do not recommend scripts. What will the scripts do? How will you script the monitoring? How and when will the scripts decide to failover? How will the scripts cater for hostname that needs to stay the same? If you have answers to all off this, then you have invented new cluster software. And yes. Good luck with maintaining scripts across staff and software changes. You have asked for advice. My advice is cluster software for automatic failover. Else, read through Disaster recovery chapter in Troubleshooting Guide or other options that will require manual intervention in NBU HA Guide.

[RiaanBadenhorst]

A cluster, from Microsoft of Veritas is really just a bunch of scripts (perl in the case of veritas). The difference is, these scripts are very well written and cater for 100s of situations, and all packages together is becomes software.

Your in house scripts wont be as clever but it could automate some tasks. Make sure the scripts are well documented though. If they fail to cater for a situation, someone would need to recitfy it. So then you'd need expertise in which ever field the script is running e.g. starting a database, mapping a lun, etc etc. 

[Michael_G_Ander]

If you do that, remember to test it, test it and then test it again. Havn't done a DR exercise yet, where we didn't find something that needed to be changed to get the recovery to work.

And then when an actually disaster hits, it probably something you havn't planned for

Always remember Brown (Murphy is an optimist) when doing DR planning

[Marc_LHeureux]

I've done DR tests on both ends of the spectrum, all manual/GUI and fully scripted.  I found the best solution was to use the CLI (it's almost always faster than the GUI) and have all the commands pre-planned in a document that can be copied and pasted (and troubleshot) at runtime.  I did not hesitate to put some rudimentary logic in the commands, so long as it fit in 1-3 lines in the document.  One example, after doing a catalog recovery wizard I had to delete all the restored tape drives so my document might have something like this:

# tpconfig -l | awk '$1=="drive"{print $3}' | while read DRIVE; do tpconfig -delete -drive $DRIVE; done

My advice - the better your documentation the smoother your DR will go.  We got so good at it, we had the DR master server ready to perform restores before the network and hosts were up at the DR test location.

1. Plan
2. Document
3. Test
4. Review
5. Repeat

We had a bit of a document template and used an excel/word mail merge to substitute variables depending on which clients were being restored and which media the latest backups were on, so that literally every command was available in the DR doc and in an absolute worst case the project manager could print the doc and an untrained person could type the commands if necessary.  My point is that the better you understand your recovery process the more easy it will be for you to answer your own question.