BE 2010 R3 Domino server recovery question

[KevinP1]

I have recently went through a Domino Mail server recovery, that to say the least didn't go too well.  We did get it recovered finally, but it took more steps and more time than I thought it should.  So the questions I have are these:

From a Full backup of the Domino server, including the databases.  Why can't I restore the databases back directly to the server before Domino is loaded and or running?

I can restore them to a non domino server with no domino running and then copy them, but this takes more steps than I would like and increases the time required to recover the serer.

I just tried a test this morning on a small Domino server we uninstalled domino completely, rebooted it and I even uninstalled the BE agent and then reinstalled the agent and rebooted it again.  I can restore the domino program data, but as soon as it hits a databse the job fails out with e000848c - Unable to attach to a resource. 

I take this to mean we need Domino installed and running in order to put the files back on the domino server.  But then why can I redirect the databases and it works to a non domino server?

The agents are all BE 2010 R3 agents

The Domino environment is 8.5.1 or .2

Media server is 2008 R2 with BE 2010 R3 and all patched to current.

Domino Server is 2008 Fully patched.

Thanks for any guidance you can provide on this.  I am just trying to get a good solid DR plan in place for these servers, the big production servers are at best a 20 hour recovery the way we did it the last time I had to recover it.

Sincerely,

Kevin Pulford

[Backup_Exec1]

Hi Please check below link shows the steps involved to recover domino server. http://www.symantec.com/docs/HOWTO12622 Thanks

[pkh]

How does the document answer the user's quextion?

[Backup_Exec1]

Hi As per OP comment he is not able to recover domino on same server were he is testing, so above link ,describes steps involved for doing DR and option configured to restore database,so that he can check and compare if he is missing something when doing restore of database. OP comment.(I just tried a test this morning on a small Domino server we uninstalled domino completely, rebooted it and I even uninstalled the BE agent and then reinstalled the agent and rebooted it again. I can restore the domino program data, but as soon as it hits a databse the job fails out with e000848c - Unable to attach to a resource.)

[pkh]

Read the original post in its entirety and read it CAREFULLY.

[KevinP1]

Thank you all for your postings.

As it turns out, and I can't really say why, the link provided by backup-exec is exactly how I had to restore in the test environment.  So it is at best a two step process. I was hoping it could be worked down so that I could justrestore everything in one job.

Now, the question I can't figure out is this.  When I need to restore a single database for Domino, I can do that no problem to a non-domino server.  And in fact I did restore the entire Domino Mail server a few months back to a redirected location and then had to copy all of that data back to the production server. This added at minimum about 6 hours to the recovery process.

So why can I restore the entire Domino environment databases and all to a non-domino server, but not to a Domino Server without Domino installed on it?  Wouldn't this actulally be a non-domino server until Domino was installed on it? :)

Anyway, there is some additinoal concern about restoreing Domino Databases while Domino is running as it is the only mail server this organization has, and so we were reluctant to have the Domino server service running while there were no mailboxes on it.  We didn't want the mail refused outright during recovery.

Thanks for your help. If you have any thoughts on the above questions please let me know.

Sincerely,

Kevin Pulford

[pkh]

Although the document that he has provided is the way to recover a Domino server, it does not answer your question, did it?   If so, you should not mark it as the solution.  If it is the solution, you would not have any more questions to ask.  You already did what the document stated when you started this discussion.

[KevinP1]

Okay, so the link provided, I have taken as the correct "Symantec" way of doing a Symantec recovery.  

However,  Pkh, your right, the entire question wasn't answered.

The rest of the orginal question about, why a file can not be restored to a server directly  (Not redirected) that is a Domino Database, even though Domino is not loaded or installed on the target machine?  The only similarity of the target restore location and the orginal backup location would be the server name.

If I redirect the Domino databases to another server that also has no Domino installed on it, I can restore every database with no errors. Then copy them to the Domino server I orginally wanted them to be restored too.  Of course this takes extra time and network bandwidth.  For a few gigabytes, no big deal but for half a terrabyte it begins to be a bigger deal.

So while the link did answer part of the question orginally posted, which is what is the proper way to recover a Domino environment.  The bigger question orginally posted is, why does it have to be done that way?  And that one has not been answered just yet.  Now the why question may just be for my own understanding at this point.  But I can't seem to figure out why BE would care what the name of the target server is, to do a direct restore rather than a redirected restore. It seems odd to me is all. And it would be much easier for us to recover a Domino server with one long 12 hour restore of the data, program settings and databases, then come in and load Domino and then start the server as a production server.

The Symantec way requires Domino to be running, which according to my Domino Admin (not me), means that as we are restoring the Domino databases, Domino is bringing them up live before the entire data set has been restored. Which means that, in the instance of the recovered mail server, some people would get mail, and others mail would be returned to sender due to no mail box for a specific user.  In the case of the application server (Web server) recovery the impact would be somewhat less, as it is only an Intranet web server, but again some applications would be working and other would appear gone until that data was restored.  In a client server situation, where the web or mail server was the client to a database back end, I would just say turn off the front end client until the data has all been restored.  But I am told that is not really how Domino works, so it may not be an option here. 

Which is why I would like to lay all of the data back down before I bring Domino up.  And that is the one thing right now I can't do directly to the target server.  Which is what I want to know why I can't?

Hopefully this clears up the second part of the orginal question. I am still looking for suggestions or answers to this piece of the oringal post.

Thanks all

Sincerely,

Kevin Pulford

[pkh]

Since you have not gotten your full answer, marking the link as the solution gives the false impression that it  does answer all your questions.  You should not mark it as the solution.  Leave this discussion open.

[Backup_Exec1]

 Hi

Few things which need to be cleared first

If backup is taken by domino agent ,then as per Symantec Admin Guide and docment domino db can only be redirected to a local domino server to different folder  that means to same domino server but can redirect to different location ,it needs to go a directory under the lotus domino data directory.

So if you are redirecting to other server, that means you are using file redirection option ,that means for doing backup Open file agent that is AOFO is used & not Domino agent. as there is no option reffered in Admin guide on how to redirect domino database to Non domino server.

Moreover the DR link I have given you is manual disaster recovery for domino server & it has be in 2 parts 1 were it will bring the system up & running with C & system state restore & 2nd part will be application restore were we would be restoring the domino database considering domino in place. but please remeber main part is taken care by domino such as assigning IDS or replaying logs etc Backup exec won't do that after restoring domino databases.


Hope that helps

Thanks

[pkh]

I don't think you understand the user's question.

The user is not trying to do a backup of the files which he re-direct to another server, so your 2nd paragraph is not relevant.

Your last paragraph does not address the user's question as to why he cannot restore the databases first and then install Domino.  Logs do not come into the picture because he may not be using logs.

[KevinP1]

I beleive the question may not be understood as well.  I do understand what the admin guide says, and I also understand the material in the link backup-exec provided.

But ..... as a point of fact.

1) I can redirect on restore,a single or all of the Domino Databases to a server that is NOT a Domino server and they will restore intact and as expected.  Then we just have to copy them to the actual Domino server and Domino reads them just fine.

2) If I get a new Domino Server and happen to call it the same name, and Domino isn't loaded on the server yet.  I don't seem be able to directly restore the Domino Datbases to that server.  Why?

To me it falls to reason that if I can redirect a restore to a non Domino server, I should be able to directly restore to a Non-Domino server in the same way. But I definitely can't!  I have tried this several ways and what I am hoping is that I just need to change a setting in the restore job so I can do this.

Otherwise in a full recovery, I am stuck redirecting 500 GB of NSF files to a non-related server and then copying them back accross my LAN to the actual server they will run on.  This just adds time and in a DR scenario this is something we really don't have.

Yes the Domino Datbases are always backed up using the Domino agent for BE.  But I can and as a matter of practice do, restore individual database files (Redirected) when the Domino developers, or an end user has a whoops moment.  They work just fine and I have never lost data using this method since I started more recently with 11d about 4 years ago.  Now I am running 2010 R3 fully patched. It just never dawned on me that if the time came to do a full server recovery I would need to either comprimse the operations of the server in question while the data restores, or restore it all to a different server and take an extra number of hours to copy it to the actual server just to get the server all back together again.

And if I didn't already say so clearly, while the admin guide might say I can only restore a domino database to a domino server.  That is just NOT true.  I do restore only to a non Domino server when I am recovering single files. I let the Domino Admin deal with how to get them back into Domino.

And lastly, in case I haven't covered this.  I can't tell what BE is doing in the background.  I own 3 Domino Agents, but of course when we install the Backup Exec agent on the servers, I make the assumption that if Domino is on the server then the agent is actually being used.  I know that on my Mail server using DAOS, BE seems to know a lot about the DAOS and how to back that up. So I would say I think the Domino Agent is being used to back up the data.  But I can gurantee when I drop single files back I am not putting them in a server that has Domino loaded on it.

So while I understand what the admin guide is saying. I am telling you in practice I have not had the experience of "Can't" restore to a redirected Non Domino Server.  And I do beleive I am using the Domino agent to backup the files.

Also, with regards to logs, my Domino admin is all hung up on Circular logging, so from my experience, BE doesn't really like that method of logging and so we don't back those logs up anyway. Perhaps if I could get him to store linear logs off box backign them up would have some value! :)  So for not pkh is correct, the logs are not in play here and not part of the question or the restore.

So finally, the question is. If I can redirect an entire Domino Data set to a non domino server, then copy it to the Domino server and have it all work.  Why can't I restore the entire Data set directly to a Non-Domino server with the same name as the previous domino server and then load Domino on it?

Thanks very much for all of your comments and suggestions

Sincerely,

Kevin Pulford

[Saurabh_Gupta]

Kelvin, thanks for highlighting the problems you faced with Domino recovery. I am a developer working on Domino Agent and would like to address your concerns.

What I understand from previous posts is that your concerns, when abstracted, are the following four:

1) Can't restore NSFs to a non-Domino server with the same name.

2) Restoring "The Symantec way requires Domino to be running, which ... means that as we are restoring the Domino databases, Domino is bringing them up live before the entire data set has been restored."

3) Circular logging not recommended by Backup Exec

4) "while the admin guide might say I can only restore a domino database to a domino server.  That is just NOT true"

Did I miss something? Now, let me try to answer these, in order:

1) I believe you are doing an in-place restore with no Domino installed and it fails with "e000848c - Unable to attach to a resource". This is expected. If you are doing an in-place restore of databases, Domino server with same name should be installed on the machine with same name. This is required as in-place restore needs to load Domino components in order to make the restored databases consistent. This is true irrespective of type of transaction logging enabled (or not enabled for that matter) on Domino server.

When you say that single database can be restored to a non-Domino server, I feel you are not doing an in-place restore rather it’s a redirected restore. To be sure, I tested just now and observed that in-place restore on a non-Domino server failed with “e000848c - Unable to attach to a resource”. This happened for single database as well entire gamut of Domino databases.

Conversely, a redirected restore of databases is always going to work. (see my answer-4 too).

With these, I hope issue-1 is no longer valid. Of course, you may like to test and confirm. Let me know how it goes.

2) As mentioned in answer-1 above, having Domino installed helps create consistent restore copies of databases. And hence Symantec recommended disaster recovery steps require you to have Domino installed. Note however that, you are not required/recommended to have Domino running. So the concern about incomplete dataset that existed in your or your Domino administrator’s mind, can fade away since the dataset being restored can't be accessed until the Domino is started.

Further, with answer-2, I believe you won’t require a redirected restore of all databases. You can go as the admin guide recommends. If you still want to follow what you do today, then you can do that in only one step – by not doing in-place restore of databases, rather by doing redirected restore to your Domino server itself. However, as you understand redirected restore of a database may yield an inconsistent database copy.

3) Archived style transaction logging is recommended over circular logging by IBM Lotus Domino. One of the reasons for this is, archived style logging is designed with backups in mind. For instance, the archived logs will not be deleted by Domino unless the backup application informs it that they have been backed up. In case of circular or linear logging, there is a risk of logs getting flushed without a backup as Domino doesn’t require the nod of backup application before deleting the logs.

Symantec’s recommendation for archived style transaction logging is in sync with IBM Lotus Domino recommendations.

4) You seem right here. I will check with other folks here and get this piece in Admin Guide revisited:

“Lotus Domino databases can only be redirected to a different directory on the local server from which the database was backed up. If you are restoring a database to a different location, it must reside in or under the Lotus Domino data directory.”

Hope this helps. If you have further questions or suggestions, do write back.

[Russ_Perry]

Important to summarize from all of this... ANY redirection of Lotus Domino agent data becomes a file restore where no API connection to Domino server is attempted.  Whether you're redirecting to another server or redirecting to an alternate folder on the original Domino server. Any 'direct' restore will attempt to contact Domino server to do API restore.

Also note that if Archive Transaction logging is enabled and DAOS is enabled, redirecting the restore of databases will not restore the associated log and DAOS/NLO files and consequently cannot do a point in time restore. 

Russ

[KevinP1]

First off, thank you for your response.  I know this thread is getting a bit old, and I really didn't expect any more comments on it.  I have come to sort of accept the way things are and am trying to work with that.

But I would like to comment on your points.

So this all began when we lost a production Domino server.  I happened to be on vacation far out of town so I couldn't really get a good idea of what was going on back at the office. I was trying to help them get the replaced Domino Server restored to the last backup we had.  What we ended up doing in that case, because time was an issue of course, was we did redirect the entire 500 plus Gig to an alternate location on a different server.  Then copied all of that data one more time to the Domino Server. 

What I was charged with when I got back was to determine what the "rules" are for a restore of the Domino environment.  So along with the Domino administrator we took down his Development box and made a few attempts to see what worked and what didn't work.

This "test" is what my original post was about.

So the way the test was conducted was we dumped the Domino Data and removed the domino program completely from the development server.  Then tried to restore the data. 

Item 1) As you rightly point out this failed. I really never considered the concept of "in-place" versus "redirected" to the same server. So I believe, if I understand the term "in-place" then this first test was an in-place attempt.  This mimicked the restore attempts we tried with the production server when we tried to recover it.  The server had the same name and ip address.  Only Domino was loaded, just not running on the production server at the time.

And you are also correct in saying that when I restore single databases to a redirected location (It is our SOP) that is is not to a Domino server directly.  Generally the Domino Admin wants to look at the file to make sure it is what he needs before placing it into the production environment.

I further agree that item one is no longer an issue. This isn't how I was hoping it would work, but if this is the "rule" then of course I will accept it.

Item 2)  If I understand this correctly, Domino is to be installed but not running.  This did not work in the production environment when we tried restoring.  The Domino Admin did have Domino loaded and configured waiting for the databases and DAOS to be restored. Because he did not want some data being presented before all of it was restored he had the Domino services all stopped.  I still got the same error as if Domino was not installed at all.

So this was in my test and is still very much an issue for me.  I just have never had this work one time.  Now we haven't tried it with 2012 yet, so perhaps the new agent will show something different, but with 2010 R3.2 I could not restore in-place to a Domino server with Domino loaded and configured, but stopped.

The only way I was able to get the Data restored in-place was to have Domino running at the time of restore.

Item 3) Okay so I agree with you on this and have made my comments known to the Domino Admin. He does not know how to run Domino with transaction logging.  He has only run it with Circular logging. I have been tying to get him to switch this for 4 years, but he is not comfortable doing this so we remain using circular logging.  This is out of my hands at this time.

Item 4) I have nothing to add to this comment.

I thank you very kindly for your help an support with this.  I am really trying to get the restore time down from 22 hours to something less.  I think by changing the backup method from one full and 6 incrementals, to just doing a full each night I may have reduced this to 12 hours.  But if I could get the data restored directly to the server in place I think this would cut another 3 hours or so off of my restore time.

Just as a side note. I do not purposely backup the logs. This is my Domino Admins wish, not mine.  What I am finding in my testing of 2012 r1a is something I don't understand completely.  If I don't select the DAOS drive, it still appears I get the DAOS data. If I do select the DAOS drive it appears my data set has nearly doubled so I am assuming I am getting DAOS two times.  So if this is true, does this mean I am also getting the logging even if I don't select it?

Thanks again for all of your help

Sincerely,

Kevn Pulford

[KevinP1]

Thank you for your response.

So the summary makes some sense, I just wasn't sure how BE was telling if the Domino was installed or not. But based on your summary, the agent doesn't even look to see if Domino is there before it makes up its mind to just use Domino calls if it is a direct restore. And if Domino isn't there then it fails outright.

We do use DAOS, but circular logging.  I do not target to backup the circular logs. I do target and backup the DAOS structure though.  My Domino Admin seems less concerned about a point in time recovery as much as just a recovery.  To be honest I think this is his first full recovery of a Domino server so I am not sure what he got was what he expected. 

I have always been told by him that if the circular logs are not lost (they were in the total server crash) that they would make the restore consistent.  Now not having the logs, and a fully crashed server, with my one time per day backup set, I am not really sure how that worked for him.  But he has not said anything to me as far as being missing data or that I need to backup something more or different.

In our after actions we did talk with the VP of technology about using linear or archive transaction logging and even placing them on a different server. But to date this has not been done.  Again, I feel the Domino Admin is not comfortable with this type of logging so we are not using it. My Oracle DBA sorta freaked out when he heard how the Domino Admin was dealing with the logs.  But even that didn't change his mind.

Your summary has proven to be exactly how we have planned to recover in the future as far as in place or direct to the server. This makes my Domino Admin uncomfortable because he does not want the Domino server running for 8 hours with partial data restored and providing refused mail transactions and such while all of the data is getting restored.  But so far this is what we know works.

Thank you again for your response. I really appreciate your help and insight.

Sincerely,

Kevin Pulford

[Saurabh_Gupta]

Kevin, allow me some time to dig into the issue that in-place restore, without Domino running, didn't work for you.

Between, about your other observation that DAOS may be getting backed up twice - Ideally, this should not be the case. When Domino agent is licensed, DAOS automatically gets backed up as part of Domino backup. And for this reason, DAOS gets excluded from drive (or NTFS) backup. Can you confirm that the value of this registry key on Domino server is 0:

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Domino\Enable Offline Backup

[Saurabh_Gupta]

Kevin, on the issue that in-place restore, without Domino running, didn't work - it looks like the Domino Agent needed a reboot after you restored the ID and ini files, and before you started restore of databases. Please try restarting Backup Exec Remote Agent on Domino server, and then try in-place restore without Domino running. Let me know if you still face issues.

You should note however that Domino doesn't allow restore of log.nsf and hence it shall be deselected/excluded when you run restore of databases.

[KevinP1]

Hi,

So here is where my confusion lies.  If I don't have Domino loaded when I am doing an in place restore, but I restore the notes.ini file (All I backup from that folder by the way. No ID files. Those are kept in a different location by the Domino Admin.)

And I restore the notes.ini file, then have to let the job fail completely. Then reboot the BE agent before I can continue, then it is still a two step process.  I am assuming you are talking about the BE domino agent right?

Unfortunately I don't have a system just sitting around waiting for me to try a bunch of things at this time.

I do know that I didn't try to restart the BE agent when it failed after the Notes.ini file was restored.  When the restore job failed, I then loaded the Domino program and restarted the entire box. Then did the full restore.

Again though, while I think this is probably the preferred or stated Symantec method of restoration, it is not a great method from my Domino Admin perspective as Domino, if it is running will try to start and present each of the restored databases as they are restored. When we are talking about a 12 hour restore time, that means there is a lot of files not restored while others are already working.

According to my Domino Admin, there is no good way to have the DBMS portion of Domino running while all other services are down so as not to present these files before the entire restore job is completed.

Just to make sure we are on the same page here. The Production Domino environment that I actually had to recover from a total failure, would not let me restore past the notes.ini file. And I did have Domino loaded but NOT running at that time.

I think the thing that isn't entirely clear to me, is how the Domino agent for BE actually works.  I assume, perhaps incorrectly, that if Domino is loaded and running the agent would make api calls into the Domino environment and provide robust locking and such for each file backed up. However, if Domino isn't running, even if it is loaded, the BE agent wouldn't be able to actually make those calls to Domino since it was not running. This doesn't actually seem to be the way this is working.

The other part of the mystery, and this has always been the case with regards to the BE agent and Domino, is what it backs up and why I even need a selection list?  I say this because, if I select Database files BE recognizes them as .NSF / database files and puts them in a category all their own for restore.  Always has from 11d up until now.  That is fine and I am glad BE understands they are Domino Databases.  But now we enter into some nuances that require more knowledge that just straightforward selection list management.  Such as DAOS.  In 2010, and 12.5 it appeared if I didn't select the DAOS drive and folder structure (NTFS) that I wouldn't get them backed up. And they didn't get recognized as DAOS for restoration unless they were selected.  Now, however, it appears when I select them in 2012 at the NTFS level, they get backed up twice (At least by byte count) even though I have only selected them at the NTFS level.  Further more, if I select the entire folder structure where my databases are located. I don't get the database files, but I get all the index files and template files, etc.  No Databases, and I believe unless I actually go select the database files under Domino Databases, in the selection list, I won't get database files.  but of course if I select database files, I only get database files.

So in some cases I must select folder structure under NTFS to get everything other than Database files, but in the case of DAOS it appears I don't need to select anything at all. So I feel like there is just some inconsistency in how the agent actually works with Domino, and that Symantec is trying to make this all a no brainer for me, yet in doing so I feel more confused.

So as I have said for a while now, I think I understand what Symantec is saying is the true anc correct way to restore fils in a DOmino environment.  ut... it also doesn't make as much sense to me as I would like it too. :)

Thanks for your help with this.I appreciate all of your suggestions.

Sncerely,

Kevin Pulford

[Saurabh_Gupta]

OK, so there are couple of things which need clarification.

Let's begin, yes - the talk is about BE Domino Agent.

Next, please note that Domino Agent makes API calls irrespective of whether Domino is running or down. Of course, Domino must be installed and configured (or loaded in other word).

Recovery process: When I said, it's not a 2 step process, it was in context of your original post - that you restored to a different machine first, and then copied to the Domino machine. Now, let me focus on what Symantec recommends.

1. If needed, you shall first get the machine loaded with OS.
2. Then, install Domino to same path as before. Do not start Domino until step 6.
3. Using BE, create a restore job for restoring only notes.ini and ID files - overwrite if these files already exist.
4. Restart Domino Agent on this Domino machine.
5. Using BE, create another restore job for restoring all databases except log.nsf. BE will be able to do API calls even though Domino is down and the job will succeed.
6. Recovery is now complete. Start Domino for the first time now.

I believe the above summary makes it clear how the things go. I agree, it's not fully automatic process, but this is an optimum one. If these steps are followed, there is no situation that partial data is going online since Domino is down until recovery is complete (that's step 6).

This is for the restore and recovery with Domino Agent. Feel free to try them whenever you have time. If you observe different behavior, do write back or open a support case.

I am taking your other problem in next post, so as to keep things separated.