How to Move Netbackup Appliance N5000 (configured as PDDO) from one Media Server to another Media Server

[Alberto_Colombo]

Hi everybody,

I would like to know the best practice to move a Netbackup Appliance N5000 (configured as PDDO) from one Media Server to another Media Server.

Here is my setup:

One NetBackup Master Server 7.1.0.3 with RHEL 5.6

One NetBackup Media Server 7.1.0.3 with win 2008

One NetBackup Media Server 7.1.0.3 with RHEL 5.6

one N5020 with 1.4.1.1

one N5000 with 1.4.1.1

i've configured the 2 appliance noted above on the production media server (win 2008) using "Disk Storage Server" wizard and "Disk Pool Configuration" Wizard, and also configured several SLP with do optimized deduplication from N5020 to N5000

NOW, i want to move N5000 to our DR Site, so i need to configure N5000 on our RHEL Media Server at our DR Site.

When i try to choose new media server on "Media and Device Management" -> "Credentials" -> "Storage Servers" -> N5000 Storage Server -> "Login Credentials"

It gives the following error:

"Add credentials failed for media server/s:

media_server: Unknown error(-1)"

My questions are:

1) How can add the RHEL Media Server as additional server to have access to the puredisk deduplication pool?

Obviously, i DON'T want to loose any backup image present on N5000

2) To ensure the Optimized duplication keep going on successfully, shall i give RHEL Media Server access to N5020 too?

and, one more question about DR Netbackup Catalog Recovery: 

when i'll add N5000 to our DR Master Netbackup (it's a Virtual Machine, and it's a Clone of our production Netbackup Master Server - in a separated LAN with the same ip address as prod servers), using DR info file and Netbackup Catalog Recovery wizard, can I immediatly start to 

do a catalog recover, right?

thank you in advance, 

Alberto

[Chad_Wansing2]

In my experience, when you get this kind of error, it's either a network problem of the two boxes being able to see each other, name sresolution (both forward and reverse) isn't dead on for each of the boxes from the other, or I see quite often that the media server already has a <storage server name>.cfg file in the /usr/openv/lib/ost-plugins directory which will usually have a wrong pool ID for the appliance which will cause authentication to fail.

-Chad

[Alberto_Colombo]

Hi Chad,

thank you for your reply.

i noticed (after reading what you wrote me) that N5000 could not resolve the linux Media Server (in our DR site). after adding it in the hosts file, i was able to add the additional Media Server  to the Storage Server Configuration (using GUI)

Now i want to try to understand if it would be better for me to give the linux media server full controll on N5000 Storage Server (for optimized duplication), or leave configuration as is, and just use N5000 to backup some production server located in DR site - i'll try to experiment some different config as soon as possible.

can you also tell me if what i've written about the DR Recovery catalog can be right?

thank you very much,

Alberto

[Chad_Wansing2]

So you have two options on how your get the data over to your DR site.  Either way, there has to be a media server that has visibility to both the source and destination storage servers.

Here's a good tech note on setting up PUSH opt dupe:  http://www.symantec.com/business/support/index?page=content&id=HOWTO70619&actp=search&viewlocale=en_...

Here's a good tech note on setting up PULL opt dupe:  http://www.symantec.com/business/support/index?page=content&id=HOWTO70620&actp=search&viewlocale=en_...

What I would personally do (provided there is significantly less production data in your DR site than in your production site), is to set up your DR RHEL server to have visibility to both the 5000 in the DR site and the 5020 in the production site.  As I would expect the DR site to be less busy, I would PULL the opt dupe data from the 5020 down to the DR 5000, and would PUSH the production backups being done in the DR site up to the 5020 in the main data center.  That way your media server that should be the least busy is handling the extra work.  If the media server in the main site are less busy (since there's two of them) I would reverse the process.

Hope this helps!

[Alberto_Colombo]

Hi Chad,

your design is great, and the technotes you linked me are very clear and useful!

i've modified my backup environment to reflect your proposal (using PULL scenario), restarted Netbackup services, and now SLP and backup are running using this new config.

obviously i need to let it flow for at least 24/48hrs to understand if everything is ok.

for example,one possible issue can be due to the fact that we have a tape library (where we keep long retention backup) which is in our PROD environment.

SLP usually is made up of:

1) backup on N5020 (PROD)

2) duplicate on tape library (PROD)

3) optimized duplicate on N5000 (DR)

my concern is that if DR Media Server has to handle the SLP described above, when the backup is to be duplicated on tape, it can be really slow!

how can i manage this?

one last thing, can you point me to a technote which describe how to use the appliance in DR?

thank you very much for your help, i really appreciate it!!!

Alberto

[Chad_Wansing2]

Dedupe's (regardless of the vendor) main weakness is the ability to rehydrate the data.  HOWEVER, NBU can pretty easily work around that, particularly for an environment like yours.  Instead of doing a rehydration out to the tape (causing disk IO for the backup, the rehydration, AND the optimized duplication), what we can do is leverage our old friend inline tape copy to make the tape and dedupe copies simultaneously.

In order to set that up, here's what your SLP should look like:

1)  backup to tape with whatever media pool for whatever retention for the offsite tape copies

2)  backup to dedupe disk with whatever retention you use for your local copy

      i)  duplication to DR site dedupe storage with whatever retention you use for your DR copy

Another good option I've seen a lot of customers use are the new optimized synthetics (although these are only for file system backups today) so that they only ever have to do an incremental but then synthesize the full backups and then the storage has plenty of time to focus on doing rehydration to tape without having to worry about the rehydration stepping on disk IO of the backup.  I've found this to be more fitting for weekly tape creation needs than daily ones though.

Hope this helps!

[Alberto_Colombo]

Hi Chad,

this is also a good idea, and i'm gonna test it today!

about optimized synthetics, is it the new feature in 7.5?

in any case, a lot of our backups are RMAN/Lotus backups, but it's also true that it can be useful. (vmware backup using vStorage API is considered filesystem backups, right?)

thank you!

Alberto

[Alberto_Colombo]

no, inline copy cannot work (at least in our PROD environment), because "Not all storage units are on the same media server".

Actually, tape library is FC connected to our Netbackup Master Server, which is also a Media Server (untill NB 6.5 it was just all in one), so i cannot have an inline backup (on tape library and on N5020) in a single SLP.

It would be possible using tape library in our DR site, but it's not compliant with our backup policy.

###################################

I still cannot get the logic used by Netbackup to choose which Media Server to use for every backup job.

looking at all backups which ran in the last day, i see some jobs (a little % indeed) which have chosen to use (to backup a server in PROD environment on N5020) the DR Media Server!!!

Obviously this is totally wrong, but how can i force these jobs to just use (or wait to use) PROD Media Server?

thank you,

Alberto

[Jeff_Foglietta]

Alberto,

You can absolutely force policies to use specific storage units. A few questions for you first:

Are your policies presently set up to utilize "any available" STU? This could be the reason your backups are written to different STU's

Are you using storage unit groups?

Are you using media server groups?

[Alberto_Colombo]

Hi Jeff,

my problem is that when i use a SLP involving 2 different appliance (which stay in 2 different datacenter),

even if i've configured the PULL design described above (which should force backup in PROD environment use PROD Media Server, and optimized dedup pulled from DR site using DR Media Server), sometimes happens that DR Media Server is used to backup PROD client (on PROD appliance).

In every schedule, a SLP is forced to override policy storage selection

i don't use storage unit groups / media server groups (maybe it could be wise for me to start thinking seriously at them, but i don't want to create too much complexity, so do you think is really worth using them? and how?)

thank you,

Alberto

[Jeff_Foglietta]

Hi Alberto,

This sounds like an issue with your initial backups outside of SLP.When you say "In every schedule, a SLP is forced to override policy storage selection" I assume that all your schedules in all your policies are configured to override the policy STU? I need to ask, when you see a production client backed up by the DR media server, is it the same client(s) everytime? I'm thinking you may be specifically directing those clients to use that STU with the override.

[Jeff_Foglietta]

Hey Alberto,

Sorry for misreading your post. OK So you have your policies set up with SLP's as the policy storage units. Are the prod clients being backed up to the DR site always the same or is this a random issue? Can we see the SLP of one of the policies being backed up to DR?

[Chad_Wansing2]

If you have your DR media server doing backups of prod data to your prod storage, I would take the DR media server out of the prod storage unit.  you may have to go to the "push" opt dupe, but that's not a big deal.

Also, if you configure inline tape copy, it should automatically route the backups through the media server that can see both storage units (provided there is one), just make sure whoever is connected to the tape resource is also authorized to use the dedupe pool.

[Alberto_Colombo]

Hi Jeff,

i've done this change in our backup environment just 2 days ago, so my statistics are quite limited.

i can tell you that yesterday night i've seen policies (with daily schedule) performing the wrong way, while this night that didn't happened

in any case, SLP spread across multiple different schedules (in some cases hundreds of them), so i would expect to see A LOT of jobs using the wrong Media Server to do backups.

Alberto

[Alberto_Colombo]

ok , it just happened:

1) i've some sched which are forced to simply use N5020 (PROD environment - no SLP used in this case)

2) the backup job was using DR Media Server to backup client server (PROD environment)

3) DR Media Server obviously has access to N5020 (PROD environment) and N5000 (DR environment), to be able to PULL Opt Duplication

4) after killing this job, when it restarted it "decided" to use PROD Media Server (which just has access to N5020)

[Alberto_Colombo]

hi Chad,

i won't like to come back to PUSH opt dup, because i've seen that the PULL design improved our environment.

i'm thinking if it's worth add Netbackup Master/Media Server (FC connected to tape library) to N5020/N5000 Storage Credential.

In that case Netbackup Master/Media Server could have access to all 3 storage units, and i should be able to do inline copy.

one side effect is that it will partly compromise the PULL design - and i'm trying to understand if it can be some more side effect... 

[Jeff_Foglietta]

Hey Alberto,

So in the scenerio you just experienced SLP is not used a policy STU and the initial job ran using the DR media server to back up the PROD env. Once you cancelled that job and re-ran it, the job used the PROD media server. Since SLP is out of the equation here I need you to check your storage units. You have the connectivity between the two sites that allows you to use either media server. Seems possible that you may have a storage unit group set up containing both the PROD and DR storage units in a round robin fashion. Could you verify this?

[Alberto_Colombo]

Hi Jeff,

not properly, i have the DR Media Server which can access N5020 (Storage Server in my PROD env) and N5000 (Storage Server in my DR env).

PROD Media Server can just access N5020.

i've done this config following Chad's tip and link (as written above - PULL config):

http://www.symantec.com/business/support/index?page=content&id=HOWTO70620&actp=search&viewlocale=en_...

apart from that, everything else you've written mirror my case.

[Jeff_Foglietta]

Hi Alberto,

Quick question. By any chance do you have media server sharing set up for the disk pool on the 5020 in the PROD environment? Check the storage unit properties for the policies in question and see if you have both the PROD and DR media servers listed as valid for this disk pool. If you do, then either one of these media servers will be used and will display the behavior your experiencing.

[Alberto_Colombo]

hi Jeff,

i had this config:

N5020: PROD Media Server + DR Media Server

N5000: DR Media Server

(that config is needed to perform opt dup). this also mean that DR Media Server uses N5020 to write backup in our PROD env.

i'm thinking to delete DR Media Server from this design (because there's no chance to force it to just do some *specific* backup jobs) and add grant to N5020 and N5000 to both PROD Media server + Master/Media Server (which is the only media server which can access our FC tape library).

in this way, i'm back to original PUSH design, but i hope the load of backup jobs will be better managed by PROD Media Server + Master/Media server.

regards,

Alberto