solution = restore to remote site

[imra2013]

Hi there,

This would be my first post.

I have scenario whereby i need to restore my data to new remote site. Basically need simple solution to complete restore by this weekend - super urgent task.

Primary site setup;

Master server = Windows 2008 Server R2

Media Server = IBM AIX 7.1

Tape library = Quantum scalar i80 with (4 units LTO Ulrium5 tape drives)

Veritas Netbackup version = Symantec Netbackup version 7.1

Remote site setup;

Unix server (IBM server)

Tape drives = IBM LTO Ultrium 5

Appreciated feedback & solution from you all.

Thank you in advance.

[Imra_backup]

Sorry Marianne,

I forgot to attached 'nbemmcmd -getemmserver' output.

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -getemmserver
NBEMMCMD, Version:7.1
These hosts were found in this domain: backupserver, ibmdb3

Checking with the host "backupserver"...
Checking with the host "ibmdb3"...

Server Type    Host Version        Host Name                     EMM Server

MASTER         7.1                 backupserver                  backupserver

MEDIA          7.1                 ibmdb3                        ibmdb3


One or more hosts had mismatched EMM Servers.
The consistency check either failed or was incomplete.
Command did not complete successfully.

[Marianne]

You have installed the Media server as its own EMM server instead of specifying the master as EMM server.

Try this:

Stop NBU on media server
Change 'EMMSERVER = ibmdb3' in bp.conf to 'EMMSERVER = BACKUPSERVER'

Start NBU.

You can also run nbemmcmd -setemmserver on the master.

Run all the commands above and verify output.

PS: WHY do you have Master server name in upper case on the media server? Best to stick to same case.

[Imra_backup]

Hi Marianne,

Here you go, new output after amend bp.conf

Please post output of these commands on master server:
nbemmcmd -listhosts -verbose
nbemmcmd -getemmserver

On media server:
bpps -x
cat /usr/openv/netbackup/bp.conf

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listhosts -verbose
NBEMMCMD, Version:7.1
The following hosts were found:
backupserver
        MachineName = "backupserver"
        FQName = "backupserver"
        MachineDescription = ""
        MachineNbuType = server (6)
backupserver
        ClusterName = ""
        MachineName = "backupserver"
        FQName = "backupserver"
        GlobalDriveSeed = "VEND:#.:PROD:#.:IDX"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = master (3)
        MachineState = active for disk jobs (12)
        NetBackupVersion = 7.1.0.0 (710000)
        OperatingSystem = windows (11)
        ScanAbility = 5
ibmdb3
        ClusterName = ""
        MachineName = "ibmdb3"
        FQName = "ibmdb3"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0
        MachineNbuType = media (1)
        MachineState = active for disk jobs (12)
        MasterServerName = "backupserver"
        NetBackupVersion = 7.1.0.0 (710000)
        OperatingSystem = rs6000 (5)
        ScanAbility = 5
Command completed successfully.

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -getemmserver
NBEMMCMD, Version:7.1
These hosts were found in this domain: backupserver, ibmdb3

Checking with the host "backupserver"...
Checking with the host "ibmdb3"...

Server Type    Host Version        Host Name                     EMM Server

MASTER         7.1                 backupserver                  backupserver

MEDIA          7.1                 ibmdb3                        ibmdb3


One or more hosts had mismatched EMM Servers.
The consistency check either failed or was incomplete.
Command did not complete successfully.

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listhosts verbose
NBEMMCMD, Version:7.1
There is no value associated with the command option
Usage:
nbemmcmd -listhosts [-brief] [-verbose] [-parsable]
    [emmname <string>] [-emmport <unsigned integer]
    [-list_snap_vault_filers -machinename <string>]
    [-list_snap_vault_media_servers -masterserver <string>]
    [-list_sts_hosts -machinename <string>]
    [-list_sts_media_servers -masterserver <string>]
    [-list_app_clusters -masterserver <string>]
    [-list_servers_in_emm_cluster -clustername <string>]
    [-list_servers_in_app_cluster -clustername <string>]
    [-nbservers [-masterserver <string>]]
    [-display_server -machinename <string> -machinetype <api | app_cluster | cli
ent | cluster | disk_array | master | media | ndmp | remote_master | replication
_target | virtual_machine>]
    [-netbackupversion <<level>.<major level>.<minor level>>]]
Command did not complete successfully.

C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -listhosts -verbose
NBEMMCMD, Version:7.1
The following hosts were found:
backupserver
        MachineName = "backupserver"
        FQName = "backupserver"
        MachineDescription = ""
        MachineNbuType = server (6)
backupserver
        ClusterName = ""
        MachineName = "backupserver"
        FQName = "backupserver"
        GlobalDriveSeed = "VEND:#.:PROD:#.:IDX"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = master (3)
        MachineState = active for disk jobs (12)
        NetBackupVersion = 7.1.0.0 (710000)
        OperatingSystem = windows (11)
        ScanAbility = 5
ibmdb3
        ClusterName = ""
        MachineName = "ibmdb3"
        FQName = "ibmdb3"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x17
        MachineNbuType = media (1)
        MachineState = not reachable by master (4)
        MasterServerName = "backupserver"
        NetBackupVersion = 7.1.0.0 (710000)
        OperatingSystem = rs6000 (5)
        ScanAbility = 5
Command completed successfully.


C:\Program Files\Veritas\NetBackup\bin\admincmd>nbemmcmd -getemmserver
NBEMMCMD, Version:7.1
These hosts were found in this domain: backupserver, ibmdb3

Checking with the host "backupserver"...
Checking with the host "ibmdb3"...

Server Type    Host Version        Host Name                     EMM Server

MASTER         7.1                 backupserver                  backupserver

MEDIA          7.1                 ibmdb3                        BACKUPSERVER


Command completed successfully.




root@ibmdb3:>./bpps -x
NB Processes
------------
    root  4194454        1   0 16:30:47      -  0:00 /usr/openv/netbackup/bin/nbsl
    root  6488290        1   0 16:30:46      -  0:00 /usr/openv/netbackup/bin/vnetd -standalone
    root  9175200        1   0 16:30:47      -  0:00 /usr/openv/netbackup/bin/nbrmms
    root 12452044        1   0 16:30:46      -  0:00 /usr/openv/netbackup/bin/bpcd -standalone
    root 12583054  9175200   0 16:31:02      -  0:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -DPSPROXY
    root 12648468        1   0 16:30:49      -  0:00 /usr/openv/netbackup/bin/nbsvcmon
    root 13566092        1   0 16:30:46      -  0:00 /usr/openv/netbackup/bin/bpcompatd


MM Processes
------------
    root 14352394        1   0 16:30:46      -  0:00 vmd


Shared Symantec Processes
-------------------------
    root  7471358        1   0 11:33:52      -  0:00 /opt/VRTSpbx/bin/pbx_exchange

SERVER = BACKUPSERVER
SERVER = ibmdb3
CLIENT_NAME = ibmdb3
CONNECT_OPTIONS = localhost 1 0 2
USE_VXSS = PROHIBITED
VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY
EMMSERVER = BACKUPSERVER
HOST_CACHE_TTL = 3600
VXDBMS_NB_DATA = /usr/openv/db/data
MEDIA_SERVER = ibmdb3


Hope this helps

[Imra_backup]

Hi Marianne,

Everything looks ok now, i can detect tape drive in ibmdb3 (my media server) and have successfully create storage unit. Now, as you know my goal is to restore data that I have backup last week. i have bring the tape here at remote site.

So appreciated your assist from here - to perform restore to ibmdb3(media server) ...thank you in advance

[Imra_backup]

Hi Marianne,

i have start restore but got small error. Appreciated to check below;

1/17/2013 5:31:10 PM - media 0030L5 required
1/17/2013 5:31:18 PM - restoring image ibmdb1_1358031600
1/17/2013 5:31:33 PM - requesting resource 0030L5
1/17/2013 5:31:34 PM - Error nbjm(pid=4164) NBU status: 95, EMM status: Media does not exist    
1/17/2013 5:31:34 PM - Error nbjm(pid=4164) NBU status: 95, EMM status: Media does not exist    
1/17/2013 5:31:35 PM - restored image ibmdb1_1358031600 - (requested media id is not assigned to this host in the EMM database(95)); restore time 00:00:17
1/17/2013 5:31:36 PM - Warning bprd(pid=4496) Restore must be resumed prior to first image expiration on 2/12/2013 3:00:00 PM
1/17/2013 5:31:37 PM - end Restore; elapsed time: 00:00:35
invalid error number(2826)

[Marianne]

You now need to go back to Recovery Without Import doc....

Images have already been copied to DR master, right?

You now need to put the tapes in the robot and Inventory NBU.

Ensure Media Ids are in the same format as production site - If production robot and server was configured to use 1st 6 characters of barcode, ensure same config at DR. Same if last 6 characters are used at Prod. (Lots of posts here and also in Admin Guide - search for Media ID Generation.)

Add Media Host override entries for Prod media servers to force restores to DR media server.

Ensure all DR clients have SERVER entries for media server and that forward and reverse lookup works properly.

Use BAR GUI to restore - remember to select original client as source client and DR client as destination.

Ensure ALL of these logs exist for troubleshooting:

On master: bprd (restart NBU Request Manager service after creating folder)
On media server: bptm and bpbrm
On destination clients: bpcd and tar

Good luck!!

[Marianne]

See my post below about Media Id Generation,

Verify that robot can see all 8 characters of the barcode and not only 1st 6.

[Imra_backup]

Hi Marianne,

Thanks, I overlook on the inventory robot.

When I run inventory robot, i got =cannot connect to robotic software daemon (42)

Check in symantec forum they ask me to delete vmd.lock (in /usr/openv/volmgr/misc) but it didn't work.

Can please assist me this morning...almost there ....

Thank you in advance.

[Marianne]

Almost there?

When I look at your opening post and yesterday's post, I realize that we're back at square one.....

Remote site setup;

Unix server (IBM server)

Tape drives = IBM LTO Ultrium 5

and

Everything looks ok now, i can detect tape drive in ibmdb3 (my media server)

You never mentioned any robot?
IBM or non-IBM robot?

If non-IBM robot, are you familiar with the steps in Device Config Guide to install ovpass driver and mkdev to create device name for the robot? (You should ideally have a documented procedure since you are using Quantum robot at your production site). If not, please download above manual and go through steps in AIX chapter to config the robot.

[Imra_backup]

Hi Marianne,

Thanks again. Sorry it should be IBM robot. But my IBM server at remote side already load with atape driver, version 12.5.9.0. In the device config guide I didn't see steps to config IBM robot. I think the challenge now to add robotic path in Netbackup. Thats why netbackup cannot communicate with IBM tape drive attached to my media server - no robotic path define in robot device column.

Pls advice. Thanks

[Imra_backup]

Hi Marianne,

Sorry I'm confused.

I don't think so my IBM tape drive need to define robot in Netbackup rite..(really sorry if this not correct.)

Or is it Netbackup need to define as a robot in order to initiate backup/restore task to the tape drive.

Please advice..I'm confused now.....

[Marianne]

Please go through Device Config Guide steps to ensure that OS can see robot and that OS has device name for the robot.

*** PS - Apologies if I'm not very helpful today - busy day ahead...

[Marianne]

Please have another look at device config at your main site - you will see that robot is defined in NBU with OS device name and that tape drives are configured with their positions inside the robot. You need this to enable automatic tape mounts and robot inventory.

[Imra_backup]

Hi Marianne,
FYI OS already can see tape drive and OS has define name for it (as below)

root@ibmdb3:>lsdev -Cc tape
root@ibmdb3:>rmt0 Available 04-00-00 IBM 3580 Ultrium Tape Drive (SAS)

Also NBU already with OS device name and that tape drives are configured with their
positions inside the robot. Attached screenshot - tape drive properties

Another on in robot setting (as attached), should it be robot control by a remote host or else ?

Please advice.
many thanks.

[Marianne]

Where is the robot physically connected to?

Please show output of 'scan' as well as 'tpconfig -l' on master and media server (in ...volmgr\bin)

[Imra_backup]

Hi Marianne,

As requested. Do Netbackup support tape drive with SAS connection ?

Master Server (Windows)
C:\Program Files\Veritas\Volmgr\bin>tpconfig -l
Device Robot Drive       Robot                    Drive  Device
Type     Num Index  Type DrNum Status  Comment    Name  Path

C:\Program Files\Veritas\Volmgr\bin>robtest
No robots are configured


Media Server (IBM)
root@ibmdb3:>./tpconfig -l
Device Robot Drive       Robot                    Drive                Device
Type     Num Index  Type DrNum Status  Comment    Name                 Path
robot      0    -    TLD    -       -  -          -                    REMOTE
drive      -    0 hcart2    -      UP  -          IBM.ULT3580-TD5.000  /dev/rmt0.1
 

[Marianne]

You forgot to show output of 'scan' on master and media server.
We still don't know where the robot is physically connected.

Above output shows that robot is NOT configured on the master.

Media server otput shows a robot on the media server controlled by a remote host.
Tape drive on media server is not in a robot ( not indented under the robot and no robot position number).
Output from cmd does not match your screenshot - why would that be?

lsdev output shows a tape drive - no robot.

Have you gone through the AIX chapter in Device Config Guide yet?

Do Netbackup support tape drive with SAS connection ? 

Yes, depending on robot make/model and tape drive make/model.
We still don't know which make/model tape library is at your remote site. (You have only told us about the Quantum scalar i80 at primary site.)
Please find your tape library in NetBackup 7 Hardware Compatibility List (HCL)

[Imra_backup]

Hi Marianne,

Sorry forgot to attached 'scan' output - both master & media server.

But one thing for sure tape drive connect to my Media Server (which is the IBM AIX server)

Let me recap my Master Server is Windows 2008 R2 Server without tape drive/library attached.

Yes indeed robot was not configured in Master (after perform robtest command).And also lsdev in AIX Server shows only tape. Is that mean no need for me to configure robot in netbackup (as AIX only detect IBM tape drive)

Ok no tape library in remote SITE and  model = IBM Ultrium 3580. yes have gone through AIX chapter and all i can see setting & create netbackup device file only for fiber & SCSI connection.

In HCL list netbackup only support up to ultrium 4 for model ibm 3580, means i need to have l

latest drivers etc.Please advice. thank you .

[Marianne]

NBU and OS sees tape drive fine - so should be OK.

Standalone tape drive means you need someone at remote site to manually insert tape in drive when restore jobs asks for tape mount.

Please delete robot from remote site config and restart NBU on master and media server.

Because you cannot do robot inventory to add media-id's to NBU at remote site, you will have to manually add all media id's needed for restore(s). You can use the GUI or vmadd from cmd. Remember that media density must match tape drive's density (hcart2).

Once this is done, you should be able to kick off first restore. When restore prompts for media to be loaded, insert tape in drive.

[Imra_backup]

Hi Marianne,

Ooh ok, now very crystal clear now - different between standalone drives & robot control.

I will do necessary and let you know the outcome.

Just quick one, what IF the tape already insert on the tape drive. Can I start restore now (I will try to remote from home, caused the tape already inside the drive)

Again THANK YOU.