Can't restore NDMP backups from physical tapes

[Hanzo581]

Redacted

PLEASE DELETE THIS THREAD

[Mark_Solutions]

Are you using 3 way NDMP (or remote NDMP) to do this?

I am wondering if the original backup is done by the filer going straight to the VTL - then your SLP allows an alternate read host so that the master server picks up the tape and duplicates it to real tape

So when you come to do the restore it is the Master trying to do the restore rather then the filer itself

If that is the case you will need to configure the master as a valid server for you filer in the same way that you would if you were using remote NDMP - that way it will be allowed to perform a restore back to the filer.

Or change ownership of the copy 2 images so that they are owned by the filer and it can do its won restore (assuming it also has access to the real tape library)

Hope that makes sense!

[SymTerry]

ndmp_data_connect_v3 failed means we are not talking with that filer correctly.

Just to makes sure, is the media server that hosts the physical tapes also the server the hosts the virtual? If its a different server for the physical, make sure that server can talk with and has access to that filer. Does tpautoconf -verify ndmp_hostname work?

[finalcloud]

Greetings, I'm working with Hanzo with this issue.

># /usr/openv/volmgr/bin/tpautoconf -verify finms-nasfil083
Connecting to host "finms-nasfil083" as user "netbackupmaster"...
Waiting for connect notification message...
: NDMP server notified that it is refusing connection: Connection refused
Not connected
: host "finms-nasfil083" failed
NDMP failed to verify host
 

Also you may reference ticket number 06149493 for additional logs.

[SymTerry]

I spoke with your tech and verified possible issues with my backline as well. 

The verify shows your not able to communicate, try to telnet to port 10000 from the media server hosting the physical library to the filer using the DNS name (finms-nasfil083). If that connects, then NetBackup should reach it. If not check your DNS and try with IP address. 

If its reaching the filer then we need to know,  is the filer failing to respond or is it refusing the connection because of credentials. You will want to look at logging on the filer for this. 

[finalcloud]

Looks like it's a connection issue with 83:

Output from finms-nasfil083:

Connecting to host "finms-nasfil083" as user "netbackupmaster"...

Waiting for connect notification message...

: NDMP server notified that it is refusing connection: Connection refused

Not connected

: host "finms-nasfil083" failed

NDMP failed to verify host

: NDMP server notified that it is refusing connection: Connection refused

: host "finms-nasfil083" failed

Internal NDMP error

># telnet finms-nasfil083 10000
Trying 10.36.16.83...
Connected to finms-nasfil083 (10.36.16.83).
Escape character is '^]'.
8SrfConnection refusedConnection closed by foreign host.
PuTTYfinlx-hp079-25:/usr/openv/netbackup/bin># PuTTY
 

[GV89]

As per your above Comment, Your telnet command is not successful on filer; so you are not able to verify connection and credential (># /usr/openv/volmgr/bin/tpautoconf -verify finms-nasfil083) .  Please check at filer end make sure that NDMP services are running on filer. Use these couple of commands to check.

ndmpd on(Starts the NDMP service)
ndmpd off(Stops the NDMP service)
ndmpd status(Displays the status of the NDMP service including any active NDMP sessions)
ndmpd debug n(enable NDMP debug logging - n = 0,10,30,50,70)

[Hanzo581]

We are able to do direct to tape, NDMP backups and even restores from virtual tape currently.  Only issue is when restoring from physical tape.  Could we do all that without the NDMP services being on on the filers?

We don't have direct access to the filers and will have to ask another team so I was wondering this before we bothered them.

[Mark_Solutions]

Did you look at my post earlier?

I think you have pretty much answered one of my questions in your last comment but other parts are outstanding...

So which media server is associated with your NDMP host?

You master server seems to do the duplication in the SLP so that server needs to be the server associated with the filer - restores from "real" tape will then be done from tha Master Server over the network to the filer as per a remote NDMP design

[Hanzo581]

Our master server does not control our physical library, media server finlx-hp079-25 does.  I think this may be the problem.  So as it sits now, our jobs write straight from filer to VTL, then 79 handles the duplication to physical tape.

As far as what media server is associated with our NDMP host, I would say all of them.

[Hanzo581]

Here is our current NDMP host setup, all our filers and our VTL (highlighted) are hosts and all media servers are associated.

[finalcloud]

Mark,

Going over your first post. The policy is set up as a NDMP that uses a SLP. Here is the break down:

1) Backup - Storage Unit (finms-nbmaster-hcart-robot-tld-0-finms-nasfil084) - writes to a virtual pool

2) Duplication - Storage Unit (finms-nbmaster-hcart-robot-tld1-DXi8500-1 - writes to a physical tape and pool. (No alternate read server is set). The media owner is any.

When we do a restore it is from master. If we try to which it to any other media server we get a pipe error.

On each NDMP host we have one username that is used for all media servers.

Would setting an "alternate read server" help us in this effort?

[Mark_Solutions]

Well your  job indicates the Master owns the image:

3/11/2014 12:47:30 PM - Info bptm(pid=10188) Waiting for mount of media id FC4066 (copy 2) on server finms-nbmaster.

That indicates that the Master did the duplication - or maybe it goes back to the original image - or even a restore server override somewhere?

You may need to try having just one media server with access to the NDMP host and use that same media server to do the duplication.

There was a similar post a while ago where getting the actual duplication to work was the issue and if i remember right the only way around that issue was to use remote NDMP for the backup so that the same media server did the duplication too

Worth a try with pinning everything down to just one media server though

[mnolan]

My experience with,

Error ndmpagent(pid=5480) ndmp_data_connect_v3 failed, status = -1 (-1)     ,

is that the filer is reaching back to the media server to pass or retrieve data.

Based on the information above, it does seem as though this has gone from a local backup/restore into a remote restore.

So in this instance, the media server opens its own NDMP server on the server_port_window and request the filer connect to it there.

This range is by default 1025-5000. If you were perfmorning local backups before, you may not have had an issue with this range, specially if the server it is connecting to was not the original media server performing the backup.

Please have this range looked at by the network team to see if there is a firewall that is not currenlty configured to allow this data connection.

[finalcloud]

Is there a way to check, in Netbackup, on what version of ndmp the software is using? 

[GV89]

You can check via tpautoconf -verify command.

# tpautoconf -verify (NDMP HOST)
Connecting to host "NDMP HOST" as user "user1"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4

[finalcloud]

On our master server I'm able to connect and it shows NDMP protocol version 4.  But it fails on the media servers.

EDIT 1:

From our master server

D:\Program Files\Veritas\Volmgr\bin>tpautoconf -verify finms-nasfil084

Connecting to host "finms-nasfil084" as user "%username%"...

Waiting for connect notification message...

Opening session--attempting with NDMP protocol version 4...

Opening session--successful with NDMP protocol version 4

  host supports MD5 authentication

Getting MD5 challenge from host...

Logging in using MD5 method...

Host info is:

  host name "finms-nasfil084"

  os type "NetApp"

  os version "NetApp Release 7.3.3P5"

  host id "0151754847"

Login was successful

Host supports LOCAL backup/restore

Host supports 3-way backup/restore

D:\Program Files\Veritas\Volmgr\bin>

From the media server that controls the Quantum i2000

># ./tpautoconf -verify finms-nasfil084
Connecting to host "finms-nasfil084" as user "%username%"...
Waiting for connect notification message...
: NDMP server notified that it is refusing connection: Connection refused
Not connected
: host "finms-nasfil084" failed
NDMP failed to verify host
>#
 

[Mark_Solutions]

I think this is touching on my comment about it needing to be 3 way NDMP (remote NDMP)

The authentication has been setup on the Master Server so that is the only thing that needs access to the filer and passes it commands to do its backups.

In a 3-way setup it is configured differently which gives the Media Server the access it needs.

There may be one other work around (i dont have a system to test this on myself).... in the SLP you may be able to setup the media owner to be the filer - in that way when you did a restore it would get the filer to do it itself. First you need to create a Server Group (under the devices section) with just the filer name in and then select that in the SLP

You may also be able to move current images to be owned by the filer - if you get the image name of the one you are trying to restore you could test it (this assumes that the copy 1 has expired)

bpimage -newserver filer_name -oldserver current_owner_name -id backupid

As i said i dont have a test system to try this on so it is for you to decide - but from previous discussions what you have and what you are doing will be difficult to make that second copy usable.

[Hanzo581]

Ok, according to our network/storage guys when we do a restore from virtual, the filer only tries to talk to our master server.  It all works perfect.

When we try to do a restore from a physical tape, the filer tries to talk to our VTL and it fails. 

I need somehow for the master server to handle the physical tape restore.  I suppose this is where the 3-way remote NDMP thing come into play?

This is what makes them think the VTL is actually causing the failure (10.36.25.8 is our VTL)

Mar 13 07:41:15 EDT [ndmpd:95]: Remote Mover not reachable on any of following addresses:

Mar 13 07:41:15 EDT [ndmpd:95]: 10.36.25.8:43465

Mar 13 07:41:15 EDT [ndmpd:95]: Error code: NDMP_CONNECT_ERR

Mar 13 07:41:15 EDT [ndmpd:95]: IOException: Connection refused

I will try to find the guide online for remote NDMP in the meantime.

[Hanzo581]

To continue this, after speaking with the network team, does three-way/remote NDMP involve any traffic for restores over ethernet?  They want to avoid that at all costs.

From the NDMP logs it looks like the VTL (DXi8500) is refusing connection for some reason.  (as mentioned above)