the Server returned Error Code 23

[Zahid_Haseeb]

Environment

Netbackup = 7.1

OS = 2003 R2

Client = Linux

Netbackup Client = 7.1

selinux=disabled

linux client able to resolve the netbackup server

/usr/openv/netbackup/bin/bpclntcmd -ip <client_ip_address>    = successful

Problem

I am facing the below error while restoring the backup. I also followed the following TN but nothing positive.

http://www.symantec.com/business/support/index?page=content&id=TECH38975

http://www.symantec.com/business/support/index?page=content&id=TECH52760

[Marianne]

At which point of the restore do you see this error? What kind of backup are you trying to restore? Database? Normal Windows policy? Flashbackup? Can you browse for files/folders? What happens when you start the restore? Does it get to Activity Monitor? Please post bprd and bpdbm logs as attachments. Tell us what time error was seen and which client you are trying to restore.

[Zahid_Haseeb]

When I initiate restore it gives me this error after a while.

Standard

Yes

In Activity Monitor the Restore job shows in queue.

Its a linux redhat client.

[Zahid_Haseeb]

On Master/Media Server

bplist -l /tmp

EXIT STATUS 227:no entity was found

[Marianne]

What is the purpose of bplist command? Without -C client-name, the default is local hostname (master). On Windows master, default type (-t) is 13 - MS-Windows. Without start and end date (-s and -e). The search period is last 24 hours. So, you are trying to list backups done for client 'master' of /tmp? PS /tmp is not even backed up on Unix/Linux client - automatically excluded.

[Zahid_Haseeb]

SOrry for late sharing the required logs. Find the attachment for your reference:

[Zahid_Haseeb]

Kindly see the latest logs for the restore activity which got failed

[Zahid_Haseeb]

Status Code details from below TN:

http://www.symantec.com/business/support/index?page=content&id=HOWTO35098

NetBackup status code: 227

Message: no entity was found

Explanation: The item requested was not in the catalog. For example, the entity can be a file or it can be policy information.

Recommended Action: A common cause for this problem is a query that has no matching images. Specify different parameters or options for the operation and try it again.

===========================================================

 ""or it can be policy information.""

What the above highlighted sentence means

Please Note:  I am restoring the images on a different server(not on a server where I took backup from) ...

[Andy_Welburn]

I agree, it does seem a bit vague to say the least, but if you do not get the policy type correct, for example, in a bplist command then despite there being backups present that conform to all the other options you have supplied then you'll get an EC=227.

[Marianne]

You never told us what the purpose was of bplist command? See my previous explanation of the command that you have typed.

[Marianne]

What exactly are we looking for in the new set of logs?

I can see in bprd log that you started browsing for images for client 192.168.1.102 at 12:15:

12:15:12.593 [1444.2204] <2> process_request: imagelist request = 329199 81 192.168.1.102 tpsbkp.tpsonline.com tpsbkp NONE 0  12 0 1376378112 en en en en en 11 0 0 0 0 0 0 0

Restore was started at 12:23:

12:23:57.606 [5412.4480] <2> process_msgQ_msg: Got new restore request from pid 4496

12:23:57.606 [5412.4480] <2> read_needed_images_from_file: reading C:\Program Files\Veritas\NetBackup\bin\bprd.d\neededimage.4496 for work to do

Media server and media id identified: 

12:23:57.606 [5412.4480] <2> add_to_tape_list: A00023 tpsbkp

Restore started on master/media server:

12:24:33.312 [5412.4480] <2> read_bpbrm_msg: (pid 4172) CURRENT POSITION A00023 2
12:24:33.312 [5412.4480] <2> send_bpbrm_msg: (pid 4172) CONTINUE RESTORE
12:24:34.413 [5412.4480] <2> read_bpbrm_msg: (pid 4172) CKP - 10642.001 1344817359 0 0 6 1 0 Z 22 /home/ali.ahmed/.zshrc
12:24:35.013 [5412.4480] <2> read_bpbrm_msg: (pid 4172) EXIT 192.168.1.102_1341312277 0
12:24:35.013 [5412.4480] <2> handle_image_status: 192.168.1.102_1341312277 restfiles pid 4496 bpbrm pid 4172 status = 0
12:24:35.013 [5412.4480] <2> remove_completed_restore: unlink record with restfiles pid 4496
12:24:35.013 [5412.4480] <2> remove_completed_restore: no more active restores
12:24:35.027 [5412.4480] <2> done_with_bpbrm: bpbrm pid 4172 has no more images to process
12:24:35.027 [5412.4480] <2> send_bpbrm_msg: (pid 4172) TERMINATE
12:24:35.070 [4496.4532] <2> restorefiles: Done with port 0 backup_id 192.168.1.102_1341312277. Status = 0
12:24:35.099 [4496.4532] <2> mail_msg_and_set_exit_status: RESTORE EXIT STATUS = 0

The above restore was successful.

According to bprd log, another restore was submitted at 12:25.

Last entry in bprd log is at 12:39.

No error.

[Zahid_Haseeb]

appologize : typed wrong

(  

On Master/Media Server

bplist -l /tmp

EXIT STATUS 227:no entity was found

   )

[Zahid_Haseeb]

This line is also present under bprd logs.

12:25:12.120 [6036.1152] <2> get_type_of_client_free_browse: db_getCLIENT() failed: no entity was found (227)

[Zahid_Haseeb]

One more thing I would like to add here. In the response of your above explanation. I tried to restore a selected backup ( its actually a very tiny directory ). It got successful (may be showed in the provided logs)

[Marianne]

That status 227 is perfectly normal. You will see that the code is <2> which is notice/debug info.
<8> is warning, <16> is error, <32> is severe error. So, no error.

The status 227 is for  'get_type_of_client_free_browse' . This means that you have not added this client to Master Server's Client Attributes with specific browse/restore permissions for this client.

We still have not any seen any logs for the status 23?

All we can see is logs for successful restore.

Your original screenshot shows error in submitting restore request. Was this a once-off?
What other processes were running on the master at the time?
We often see bprd timeout on 32-bit masters (even on 64-bit masters) when large restore requests are submitted, for example large amount of files/folders from FlashBackup.
 

[Zahid_Haseeb]

when large restore requests are submitted

The above words are making me more confident. Yes its a very large no. of files and folders. I should have increase the timout value more as per the above TN I mentioned.

http://www.symantec.com/business/support/index?page=content&id=TECH38975

Let me do it and will share the result.

[Zahid_Haseeb]

Unfortunately I tried it but could not successful. One more thing I have noticed. I tried to restore the folders one by one and it got successful except three folders which made the restore activity hangged. The three folders I did not restore. See the below for reference:

sudo du -hs /home/xxx_data/ /home/xxxxnix/ /home/admin
32G    /home/xxx_data/
18G    /home/xxxxnix/
7.8G    /home/admin

Second If I try to restore these three folders I see the below screen

 

[Marianne]

I did not have much hope that 'user-directed operation timeout ' in TECH38975 would solve the problem.

I have come to know this as a problem with bprd memory utilization, especially on 32-bit master servers.

The only solution/workaround that I have found is to select smaller sets of data.

Please see this discussion: https://www-secure.symantec.com/connect/forums/status-restore-image-created-mon-may-21-160143-2012-a...

What you are seeing above is that bprd is hanging. 
Only way out is to stop NetBackup Request Manager, then kill all bprd processes (check in Task Manager that all are indeed killed). Restart Request Manager.
Restarting Request Manager will only affect restores and user backups.

If you choose to restart NBU, verify that there are no bprd processes in Task Manager after stopping NBU.
Hung bprd processes may not terminate when NBU is stopped - kill them manually.

[Zahid_Haseeb]

thanks for the workaround.

• Means yet no proper solution ?

• Is not any automated restore which itself take a small chunk of files and did restore and when that chunk finished it again get next chunk of files for restore.

[Marianne]

I remember seeing a fix for this in some 7.5.x documentation. Will see if I can find it. There is no way within NBU to automate restores. You can try to script it.