cancel
Showing results for 
Search instead for 
Did you mean: 

DR Site

Kunalgoel
Level 4

Hello Team,

master server OS: Solaris 10

Master server NBU version : 8.1.1

we are planning to build our DR site. 

1. we will build master server at DR site with same OS and version

2. will install nbu binaries on newly build master server with same version

3. we will configure all policies at DR site as inactive state

4. we will create same db mount points (nbdb, nbemm) at DR site and that mount point will be in real time synch

5. if our primary site goes down then we will re-name the secondary master server (same as primary master server) and run the show.

6. if we need to perform any Test DR restore, in that case also we will re-name the secondary master server (same as primary master server) and perform or we can perform import of that particular image and do test restore.

Is this plan looks good? kindly suggest. 

any other DR plan and suggestions are welcomed

23 REPLIES 23

Kunalgoel
Level 4

Hello,

Should we give same ip or different ip to DR master server?

By above method, if our Prod site goes down then we can perform client restore and run our infra from DR site? 

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

The IP does not really matter but if you can create a dedicated virtual IP which you can bring up in the DR site or Prod site whenever required will help with clients whose DNS cache holds the name for a long time.

You'd need to change the host IP/hostname for the prod Solaris box to "virtualise" NBU's hostname but it's not a big deal tbh.

Have a look at Veritas NetBackup in Highly Available Environments Administrator's Guide and there "Deploying NetBackup master servers with full catalog replication" and specifically "About non-clustered NetBackup master server with catalog replication" - there is a good guide there

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Kunalgoel 

I agree with @Mouse - the IP address does not matter, but the hostname does.

When you install NBU, the hostname MUST be the same as the source master. It cannot be changed afterwards.
But then, if you are planning on importing images, then the name does not really matter.
If you want to do a catalog recovery (much better than import), then the hostname must match.
Bear in mind that import of backups takes as long as the backup did.
What is your RPO and RTO? Can you afford to wait for imports to happen?

So, a DR plan needs to be structured around RPO and RTO.
This will affect your budget for DR.
As per the guide that @Mouse referred to, catalog replication is good where the catalog at the DR site needs to be in sync with the Prod site.

What type of backup storage are you using?
If tape, will you be sending tapes that were written on a daily basis to DR?

Hello,

Got it that Ip doesn't matter but the hostname ..

we have two lun's coming from storage where our database is residing on prod side master server. we are planning to give two LUN's on DR master server with same name that will be in real time synch.

we will configured all policies like prod master server on DR master server but we will keep them inactive.

My query is: If our prod site goes down can we rename out DR master server as prod master server? and can run operation from DR site as our database LUN's are already mounted at DR site and they are real time synch??

why there is a need to change master server name because we cant keep same name under one domain and if we do not change the DR master server name same as prod server name  then backups will not work.

 

is changing DR master server name as prod server name is a challenge here? 

can we put DR master server name same as prod server name but in different domain? will it work here? 

we are not using AIR here or we will not do any catalog recovery. If we will be using these then i know different name of DR master will work.

But in our scenario here, if prod side goes down then we have a NBU database LUN which we is mounting at DR site which is in real synch. we just UP the server and mount the NBU database LUN's and activate the policies and run the show...

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I have prepped a Solaris master server more than once for DR or for migration purposes.
Give it the same hostname with a different IP address, but do not add it to DNS.
You can do that when you need to cut over.

Until then, simply use the IP address if you need to connect to this server.

Once you have installed NBU, the master server hostname cannot be changed. Unless you are prepared to pay expensive consulting fees. https://www.veritas.com/support/en_US/article.100016305

There is also the method that @Mouse mentioned that is documented in NetBackup in Highly Available Environments Administrator's Guide

**** Oh, wait! The 8.1.2 manual no longer contains the section about 'non-clustered NetBackup master server with catalog replication'.
Please read up in 8.0 manual : https://sort.veritas.com/DocPortal/pdf/ka6j00000000A76AAE

Hi @Kunalgoel 

One of the items you haven't mentioned in your DR plan is how you access the backup images at the DR site. Remember if the primary site goes down, you also (presumably) lose access to the media servers as well. 

If you have your images backed up to disk, why don't you look at using AIR to replicate images to the DR site, then the DR site can run all the time and if the primary site does then go down, there is very little to do in regards to making images available for recovery purposes. 

I'm all for keeping things simple and easy - having to remember how to bring up a master server at the DR site in a disaster situation when every one is probably stressed would be something I would look to avoid.

Granted using AIR may require additional storage for your backups, but these days storage is cheap.

David

Hello All,

Instead of replicating all images individualy. Can we just replicate catalog backup at DR site ? is only catalog backup replication is enough to UP the DR server if prod goes down?

how we create all policies at DR site without doing less manual work? Can we  pre-create them as inactive (Can we alleviate most of this work by \netbackup\db\class folder copying).???

Any other method?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Kunalgoel 

We have bee trying to find out from you what type of backup storage you are using.
Disk or tape?

The catalog alone is not sufficient to get up and running.
In a DR situation, you will need to restore first before starting to take backups again. For that, you also need to ensure that the backup storage is available at the remote site. Disk or tape.

You can either restore/recover the replicated catalog backup or replicate the catalogs.
If you replicate all databases and catalogs as per the section in the manual that I shared yesterday, your image catalog and policies will be at the remote site.
Catalog recovery will also restore the policies along with everything backed up by a catalog backup.

PS:
You will need to read up about certificate database restore.
If the DR-master was not installed in DR mode, you will need to perform manual steps to restore security certicificates.

 

 

Hello,

We have bee trying to find out from you what type of backup storage you are using.
Disk or tape? It's Disk (NBU appliance).

The catalog alone is not sufficient to get up and running. : Complete catalog backup replication is not enough to run the show from DR site if prod site goes down?
In a DR situation, you will need to restore first before starting to take backups again. :which restore? Catalog restore? For that, you also need to ensure that the backup storage is available at the remote site. Disk or tape. : Yes, we have 3 appliances at remote site. 

You can either restore/recover the replicated catalog backup or replicate the catalogs. : If catalog backup will be replicated through AIR then in that scenarion there is no need to restore/recover the catalog as it will be impot automatically?right?

If the DR-master was not installed in DR mode, you will need to perform manual steps to restore security certicificates: is there any way to check whether its installed in DR mode? if it's not installed then how we can install it manually? any technote.

Query: If we will replicate the complete catalog backup from Prod to DR site then certificate will not install automatically??

 

 

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

@Kunalgoel  - it really comes down to what you are trying to achieve in the DR scenario. AIR is a great option if you need to restore at the DR site and never run PROD backups from the DR site's master. Because AIR will not replicate backup policies and for all intents and purposes it's a different NBU domain, it just appears to be able to automatically import the replicated images and restore them if required in DR for DR clients (if you want to restore in PROD, for PROD clients, you really need to think how to deliver the data back to PROD) - so it's essentially a data DR tool, not an NBU master infrastructure DR tool. The only proper DR tool for the NBU Master infrastructure, if it's your goal - is the catalog replication, using clustered or non-clustered NBU master, depending on automation requirements.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Kunalgoel 

Maybe we need to take a step back - please explain wat exactly is currently at Prod and DR and what your DR requirements are.

Many user's DR site is meant to take over in in a Disaster situation. Production servers and all of the prod data is gone.
Backed up data now needs to be restored.

This is what I meant here:


@Kunalgoel wrote:


In a DR situation, you will need to restore first before starting to take backups again. :which restore? Catalog restore?


You need the backed up data to start restores to DR servers.

About the Appliances at DR:


@Kunalgoel wrote:

Yes, we have 3 appliances at remote site. 


Please give us more info. Are these appliances media servers under the Prod Master? Or DR Master?
Are all Prod backups written directly to DR Appliances or duplicated/replicated from Prod to DR?

Maybe a diagram of your current environment and planned environment will help us to get a better understanding.

Now, about AIR replication of Catalog backup:


@Kunalgoel wrote:

If catalog backup will be replicated through AIR then in that scenarion there is no need to restore/recover the catalog as it will be impot automatically?right?

 


A catalog backup is just that - a backup of the catalogs. Not a copy of the catalogs.
After A.I.Replication, the backup is imported, not the catalogs.
To be usable, you need to run a Catalog recovery - either back to the original Prod master or to another master at DR with the same hostname as Prod.
If you are using AIR, it means that you have 2 independent master servers with different hostnames. You cannot recover Prod catalog to DR master with different hostname.

But, when I look at your opening post, it looks like your are planning to replicate NBU databases, not the backups (you will need more than just the ones you mentioned, as explained in the HA Guide):
4. we will create same db mount points (nbdb, nbemm) at DR site and that mount point will be in real time synch

About security certificates:


@Kunalgoel wrote:

 

If the DR-master was not installed in DR mode, you will need to perform manual steps to restore security certicificates: is there any way to check whether its installed in DR mode? if it's not installed then how we can install it manually? any technote.

Query: If we will replicate the complete catalog backup from Prod to DR site then certificate will not install automatically??

 


Please read up in NBU Troubleshooting Guide about Disaster Recovery (Chapter 4).
Please take your time and read through the intro topics. Pay special attention to info about Disaster Recovery package.
You will understand when and where this is needed and how to recover certificates using the DR package.

Preparation of DR package is also explained in NBU Admin Guide I where Catalog Backup is discussed.

Suggestion:
Contact your local Veritas office to suggest a knowledgeable partner who will be able to spend some consulting hours with you to put together a workable DR solution.
A real disaster is the wrong time to find out that there are gaps in your DR plan...

Hello,

it really comes down to what you are trying to achieve in the DR scenario: I want that if our product site is down, then we can run the backups of all the clients who are configured on the  product site from DR site, until our product site is live again.So that our backup is not lost till that time period. 

Secondly, When the prod site  goes down, we can access all clients( All the clients are configured on the prod site) backup images from the DR site and if any restore request comes, we can do that from the DR site

Means looking for HA of NBU Master server

 

Hello Marianne,

please explain wat exactly is currently at Prod and DR and what your DR requirements are.:

Prod Site: 

1. Solaris Master server (PROD server name XXXX-bkp)

2. Three NBU appliances act as a media server, which has connectivity to DR server as well

DR Site:

1. Solaris Master server (DR server name XXXX-DR-bkp)

2. Three NBU appliances act as a media server, which has connectivity to PROD server as well

 

Are all Prod backups written directly to DR Appliances or duplicated/replicated from Prod to DR? As of now no, because we are planning to build it. 

First Scenarion, what i was thinking to do...

Here we cannot replicate/duplicate all individual images to DR server.... what i was thinking that  we will just replicate Catalog backup which is happening in prod site to DR site by using AIR (Considering both master server has different names) . will create SLP at prod site with "BACKUP and REPLICATION" operations and second SLP at DR site with "IMPORT" operation.

By doing above can we get this result?if our product site is down, then we can run the backups of all the clients who are configured on the product site from DR site, until our product site is live again.So that our backup is not lost till that time period.
Secondly, When the prod site goes down, we can access all clients( All the clients are configured on the prod site) backup images from the DR site and if any restore request comes, we can do that from the DR site

Second option as per current scenario/architecture

But, when I look at your opening post, it looks like your are planning to replicate NBU databases, not the backups (you will need more than just the ones you mentioned, as explained in the HA Guide):  current scenario is like..Yes, In prod server we have two mount point(LUN's) which is coming from external storage .where our nbdb and nbemm is housed.
And in DR site we have  two mount point (LUN's) with same name nbdb and nbemm coming from external storage. These mount points (both PROD and DR) are in real time synch.
Both prod and DR server have different name (as mentioned in starting) and are in same domain but in different network.

By this current scenario (we can do like this), if our Prod site goes down then at DR site we will mount both mount points (/nbdb and /nbemm) and install nbu binaries again with  same hostname (as prod master) and we will create policies (can we alleviate most of this work by \netbackup\db\class folder copying???) as /nbdb is already mounted or can we just recover configuration files so that all policies will be created automatically??. 

By using above , are we capable to run the show properly like can we access previous backups? can we retore data from previous backup images?

In second option,  we have lots of manual work to do like we need to install binaries and after that we need to create policies by recovering configuration files or by copying \netbackup\db\class folder. so our RPO and RTO will not matched here..

So what i want less manual work and RPO&RTO will not be exceeded.

So i have explained current scenario and the expected one.... 

 

 

 

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

@Kunalgoel wrote:

Hello,

it really comes down to what you are trying to achieve in the DR scenario: I want that if our product site is down, then we can run the backups of all the clients who are configured on the  product site from DR site, until our product site is live again.So that our backup is not lost till that time period. 

Secondly, When the prod site  goes down, we can access all clients( All the clients are configured on the prod site) backup images from the DR site and if any restore request comes, we can do that from the DR site

Means looking for HA of NBU Master server


Ok this suggests me you need two things:

  1. HA for the master server, automated (clustered) or non-automated (non-clustered) with catalog replication. This will address the portability of Master's role between PROD and DR and the ability to perform backups and recoveries when the DR procedure is invoked. AIR will not help in achieving these goals as it does not cover the master role portability function, it's really data-only.
  2. Backup data replication between PROD and DR, this can be achieved by a proper configuration of SLPs  to copy data between PROD and DR MSDPs.

all these procedures are well covered in the guides referenced in this thread.

Hello,

Thank you for explanation

1. HA for the master server, automated (clustered) or non-automated (non-clustered) with catalog replication. This will address the portability of Master's role between PROD and DR and the ability to perform backups and recoveries when the DR procedure is invoked. AIR will not help in achieving these goals as it does not cover the master role portability function, it's really data-only:  As per our current scenario, our NBU database mount point synched daily b/w prod and DR site (it is a rsync) through cron. And our both servers (PROD and DR) are in different network but in same domain..so, if prod master goes down then..

1. we just need to UP the DR server and install master binaries at DR server with same version with same hostname as prod master server. Then we just need to recover configuration files( or any other options to create policies??) so that our all policies will be  created at DR site and we are capable to backup the client data.? right?

Note** Complete Catalog Recovery is not required here as we already synch our netbackup databases mount point b/w prod and DR site.. kindly confirm if this is good to go..

2. Can we eliminate binaries installation step and policy creation step  on DR master server when our PRD master server goes down ( to save time )? Can we build the server and Give the same hostname with a different IP address in advance, means now... and pre-create all policies and make them inactrive, and will not add it to DNS. is this will work? or it will conflict with PRD server name???

3. Backup data replication between PROD and DR, this can be achieved by a proper configuration of SLPs  to copy data between PROD and DR MSDPs. : Yeah, it does not cover master server role portability function. In this, no need to invoked DR procedure. But in this we can perform backup/restore if our prod master goew down. right?.

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

I am afraid you won't be able to build the system by just QnA on the forum as there are a few important fundamentals you need to be aware of, they are quite extensively covered in the guide I've referenced earlier. Some of the key issues I see:

  1. The only supported way for the NBU Master to recover from a failure on a different machine is either running a continuous replication, sync or async that respects the write order fidelity, such as hardware-based replication or something like VVR. Rsync is only good if you shutdown NBU before starting the sync session otherwise you'd be replicating a bunch of non-recoverable garbage.
  2. If you read the NBU HA guide, it's got the setup procedure for the DR site and you have to perform it beforehand, not when the disaster strikes. You won't be able to install NBU and point it to a catalog replica, if you installing a new master in a DR scenario, you need a catalog backup.
  3. On the DNS name, I've already tried to explain that the best way to achieve it is through a manual or automatic (in cluster) assignment of an NBU virtual IP which the Master will belong to, whereas both nodes in PROD and DR will have their independent physical IPs, so it will be safe to move the virtual IP around without worries of IP conflicts.
  4. If you've configured the SLPs correctly and you have data on and connectivity to the respective media servers I see no issues with the recovery of data.

In essence - I suggest to go through the HA guide in details. There are important fundamentals there.

Thank you Mouse for information. It really helps..

You'd need to change the host IP/hostname for the prod Solaris box to "virtualise" NBU's hostname but it's not a big deal tbh.  Suppose my PRD server name is "XYZ", can i use this name as a virtual hostname and give two different names to PRD and DR server like XYZ-1 to PRD server and XYZ-DR to DR server to virualise it??

as my environment was configured/installed as a standalone server. if we want to convert them into cluster, can we?