You've got your backup software installed on all your sites, both in your country, and remotely elsewhere. You get frequent notification of backup statuses, and it's going well. Until you hit a critical problem like your server crashing catastrophically, and you need to restore all your data from tape. No problems, you're sure you're covered. The tape is loaded up, and you start a restore...and then find out you cannot restore your data like you thought you could.
The old adage that you're protected only as well as your previous successful backup holds true...then you ask why you never did a disaster recovery test!!!
We went through this a couple of weeks ago. Not a tes, but for real! During a swop out of HDDs between 2 servers (a quick migration!), 1 drive failed, completely unknown to us. While it was rebuilding onto 1 other drive, this drive was pulled out, causing the RAID5 set to break completely. The server wouldn't boot, and we lost all the data. After reinstalling the server from scratch (SmartStart etc), I was able to restore all the data.
However, the question is: Should we be doing regular disaster recovery, and testing of your backups? A lot of considerations ride on this:
1. Do you have spare hardware to do a test run too? This would be first prize, as it would allow you to get your server up and running using your latest successful backup;
2. Are you confident that without testing your restores, that if the need arose, you could do so?
3. Are you sure all the data you would need to restore is covered in your backups? Are all your servers?
4. Are you using offsite storage that would add to your ability to recover from a natural disaster?
5. How quickly will you recover your data? No sense in telling your company it will take 2 hours, but you're restoring TBs of data that would take 5 times longer.
Testing of backups for disaster recovery is something I have mentioned on a couple of occasions to my colleagues, and although I could use the case I mentioned above, it's not something I have pushed. After that incident, it's a very serious matter that needs to be thrashed out in our technical meetings. We have many sites in Africa and Asia, so we're geographically spread-out, and have vulnerabilities on many fronts as a result.
With the rate at which information changes today, with viruses doing the rounds daily, and possible hardware failure on the back of our minds, testing backups for possible disaster recovery is something the IT professional has to consider. Without it, when the crunch comes, if you're not able to deliver, you're out of a job and your company is dead in the water.
Attention has got to be paid to why backups might be failing, and how to remedy that. Consider what your company/client would want, and see it as a value-add for them. They'll be happy to know that all their data can be recovered (even if you do it pice-by-piece if you don't have disk space for instance), and you;ll be sleeping a lot better at night.
Disaster recovery will be something I look into in the coming months, specifically how best to go about it, what's our client requires of us, and what the best method will be, and I will share my experiences on this board.
I believe that if you're not already considering it, or have considered it and implemented some sort of plan, that when the problem arises, you won't be able to deliver.
