Highlighted

NDMP Restore failing

Having an issue with an NDMP restore. It's failing with a 2813 error. Here is what I get in the detailed status:

2/10/2017 4:39:17 PM - Critical bptm(pid=6212) sts_read_image failed: error 2060001 bytes Read = 18339597817245633366one or more invalid arguments
2/10/2017 4:39:17 PM - Error ndmpagent(pid=4960) connection 01231908 ndmp_message_process_one failed, status = 18 (NDMP_XDR_DECODE_ERR)
2/10/2017 4:39:17 PM - Error ndmpagent(pid=4960) NDMP restore failed from path /ifs/LonProdMedia/lonvidediting/PROJECTS
2/10/2017 4:39:22 PM - Error ndmpagent(pid=4960) LONVIDNAS01.ap.org: Failed to read data from daemon: No such file or directoryError reading file data (31009/-1): No such file or directoryWrite error: Invalid argument: ifs/LonProdMedia/lonvidediting/PROJECTS/ZARA WORK IN PROGRESS/zara.prproj
2/10/2017 4:39:22 PM - Error ndmpagent(pid=4960) LONVIDNAS01.ap.org: Could not skip data giving up
2/10/2017 4:39:22 PM - Error ndmpagent(pid=4960) LONVIDNAS01.ap.org: Failed to read archive during DAR for index 0
2/10/2017 4:39:22 PM - Warning ndmpagent(pid=4960) LONVIDNAS01.ap.org: Error extracting entry {ZARA WORK IN PROGRESS/zara.prproj, /ifs/LonProdMedia/lonvidediting/PROJECTS/ZARA WORK IN PROGRESS/zara.prproj} at index 0 of NLIST
2/10/2017 4:39:22 PM - Warning ndmpagent(pid=4960) LONVIDNAS01.ap.org: Operation halted by DMA
2/10/2017 4:39:22 PM - Error ndmpagent(pid=4960) send error status = 12 (NDMP_EOF_ERR)
2/10/2017 4:39:23 PM - Info ndmpagent(pid=4960) done
2/10/2017 4:39:23 PM - Info ndmpagent(pid=4960) done. status: 27: child process killed by signal
2/10/2017 4:39:23 PM - Error bpbrm(pid=4128) client restore EXIT STATUS 27: child process killed by signal
2/10/2017 4:39:23 PM - restored image lonvidnas01.ap.org_1486279880 - (child process killed by signal(27)); restore time 00:00:14
2/10/2017 4:39:23 PM - end Restore; elapsed time: 00:00:15
NDMP policy restore error(2813)

I've looked into the solutions for the status code but can't see an issue. I would appreciate any help you could give.

16 Replies
Highlighted

Re: NDMP Restore failing

The actual error is from ndmpagent, so, no use trying to look for answers in NBU documentation.

I may be wrong, but my interpretation is that the destination path is invalid.

You need to look in the documentation for your make/model NAS appliance. The various types of filers all have different rules for alternate path restore.
Highlighted

Re: NDMP Restore failing

Thanks Marianne,

This is seems weird. I have been backing up this NDMP client for a while with no issues but the restore fails. I'm restoring to the original location. That what I don't get.

Highlighted

Re: NDMP Restore failing

There's a leading '/' missing here:
ifs/LonProdMedia.......

Maybe post screenshots of the last screen?
Highlighted

Re: NDMP Restore failing

I assume you mean this line:

2/10/2017 4:39:22 PM - Error ndmpagent(pid=4960) LONVIDNAS01.ap.org: Failed to read data from daemon: No such file or directoryError reading file data (31009/-1): No such file or directoryWrite error: Invalid argument: ifs/LonProdMedia/lonvidediting/PROJECTS/ZARA WORK IN PROGRESS/zara.prproj

I see that there is no leading /, but where is this truing to read from? 

Not sure what you meant by last screen, though. Here is the screen from the restore status.

Highlighted

Re: NDMP Restore failing

I meant the last screen where you select/specify the restore destination.
Highlighted

Re: NDMP Restore failing

Ok Sorry, feeling a little dense. Here is the screen shot.

Highlighted

Re: NDMP Restore failing

Okay. Nothing strange there.

Next thing to look at is logs.
bprd on master server
ndmpagent, bpbrm and bptm on media server.
bprd and ndmpagent are the first ones I would look at.
bprd to make 100% sure the destination is the ndmp file and not unintendedly another host.
ndmpagent to see what the 'invalid parameters' are.
Highlighted

Re: NDMP Restore failing

I tried it again and watched on the Isilon I'm restoring to. It creates the folder and then starts to write a file but before the file is completed it failes. I looked at the bprd and all I see is an empty text file. The others I pulled out the data pertaining to the issue and have uploaded them. This is not sinking in. Not sure where this error lies. Is this a permmissions issue? I back up this NDMP client with no trouble. 

Highlighted

Re: NDMP Restore failing

Seems it does not like the embedded spaces in "ZARA WORK IN PROGRESS"
Highlighted

Re: NDMP Restore failing

NetBackup must be restarted on the master to enable bprd log.
I will look at uploaded logs on my laptop a bit later. I can only view .txt files on my mobile phone.
Highlighted

Re: NDMP Restore failing

Seems you are trying to restore to the master/media server and not the filer.

This is what I see in bpbrm log:

logparams: -restore -mt 0 -mst 6 -k 0 -c LONVIDNAS01.ap.org -hostname LONVIDNAS01.ap.org -rclnt lonbackup00 -rclnthostname lonbackup00 -browse_clnt LONVIDNAS01.ap.org

Did you choose lonbackup00 as destination client? 

Highlighted

Re: NDMP Restore failing

Okay... now I am confused.

I'm reading your last post again:

"It creates the folder and then starts to write a file..."

So, maybe I am misinterpreting what I see in bpbrm.
I need bprd to know for sure.
Buth I also see only connection requests from IP 10.10.4.134 to same IP address: 10.10.4.134. 

Why would it create a folder if you are restoring back to the same location?
Was your last attempt something different to what you showed in the screenshot on Friday?
Or are you tying to restore to a different filer? 

What confuses me even more are the 'NAS' hostnames I see in bptm and bpbrm:

-c LONVIDNAS01.ap.org -cl LONVIDNAS01EDT

Two different filers?

PS:
ndmpagent is a unified log. You will need vxlogview command to get in plain English.

Highlighted

Re: NDMP Restore failing

So sorry for the confusion. Let me explain. The requester deleted the folder and wanted it restored. I started the restore like any other restore source ans destination was the NDMP client. That failed so I tried again this time only chooing a couple of files in a sub folder of the main folder. That's when I saw the folder creation and data transferuntil it failed again. I tried a few more things.I tried to restore to a different folder on the same destination. My boss kept suggesting I restore it to the master server. I told him it wouldn't work but he was insistant so I tried it to show him. I must have pulled the ndmpagent events from that try. 

I attempted a restore of a file to start fresh and have taken the logs for that. I am attaching them. I did do vxlogview and had it grab the logs pertaining to the job id, but it came back with 244 corrupted log records so I opened the ndmpagent log in wordpad and searched the job id (restore id) and cut and pasted from there to the end.

Highlighted

Re: NDMP Restore failing

BTW, in searching online, I saw a couple of the errors might be caused by DAR so I disabled it and started a restore of one file. Is this going to cause an issue?

 

Highlighted

Re: NDMP Restore failing

I honestly think you will be best off logging a Support call.

With our time differences, it means that I am only viewing VOX posts on my phone.

In light of you being around for so long, I was hoping that you would know how to use vxlogview or at least be prepared to look it up. 
We need vxlogview output for ndmpagent specifically. OID 134.

So, something like this: 
vxlogview -o 134 >C:\temp\ndmp.txt
(You can look at Commands manual for options to extract info for a specific period.)

bprd seems all good with browse_client, destination_client and destination_client_hostname all the same:

browse_client = LONVIDNAS01.ap.org
requesting_client = lonbackup00
destination_client = LONVIDNAS01.ap.org
requesting_client_hostname = lonbackup00
destination_client_hostname = LONVIDNAS01.ap.org


We now need vxlogview output for ndmpagent log. 

IMHO, DAR needs to be enabled for file-level restore.
There is obviously no harm in disabling it before you retry.
Check ndmpagent log with and without DAR. 

 

Highlighted

Re: NDMP Restore failing

You are correct. I should know how to use this and I will do just that. To update you, disabling DAR allowed me to successfully restore the folder. It took a while, but it worked. I will work to be able to check the logs correctly.

Thanks for your help.