All restores not working but backup's are

hi,

are you using the Multi-plexing backups?

if yes, please try how restore is moving for non Multiplexing images?

does this restore issue is related to specific media or media server or a Drive?

you would need to isolage this issue in that level.

Are all needed tapes in the library?? Run inventory of library and make sure tape which are needed for restore are there!

you can check that from CLI by issuing such command

vmquery -b -m XXXX014

if there would be NONE in the output the tape is not inside library

here is output for a tape from my env - first which is out of the library, second which is in the library:

C:\>vmquery -b -m QA8658
media   media  robot  robot  robot  side/  optical  # mounts/      last
 ID     type   type     #    slot   face   partner  cleanings    mount time
-------------------------------------------------------------------------------
QA8658  HCART  NONE     -      -     -       -           2     12/28/2012 20:50

C:\>vmquery -b -m QA8651
media   media  robot  robot  robot  side/  optical  # mounts/      last
 ID     type   type     #    slot   face   partner  cleanings    mount time
-------------------------------------------------------------------------------
QA8651  HCART  TLD      0       4     -       -          12     01/23/2013 13:44

Is the restore hanging at 'media id needed'?

If so, it seems that bprd process is hanging. See this discussion:

https://www-secure.symantec.com/connect/forums/master-7503-media-servers-60mp4 

Media mount timeout is a separate isssue.
Post ALL text in the job details as well as media server bptm log as File attachment.

I had already check inventory and it was fine for all robots. No inconsistency.

In the morning I rebooted the master server and now everything is working.

It seems like a netbackup bug since we didn't change anything else.

Let's see how long it will take until ghosts come again ... :)

@ Marianne van den Berg. Your considerations are very relevant.

It explains why after I restarted netbackup master server service restores worked for a few minutes and then stopped working. Most probably there was zombies bprd process still running. With the reboot of today, besides restart all process, all these zombies bprd process got killed.

In which situation(s) bprd process can hanging forever ? Is it a netbackup bug ?

Thanks in advance.

Since I haven't had the problem so far, I marked @Marianne van den Berg post as solution. Seems the most plausible answer. If I had problem again I'll try those recommended actions https://www-secure.symantec.com/connect/forums/master-7503-media-servers-60mp4)

Solved the problem rebooting master server machine.

In which situation(s) bprd process can hanging forever ? Is it a netbackup bug ?

I have only ever seen the problem where FORCE_RESTORE_MEDIA_SERVER entry is in place.

I believe this is a bug. The problem is very few users have level 5 log of bprd available in order to log a support call. It is easy enough to kill hanging bprd process or else restart NBU.

I have once logged a call for NBU 5.1 on AIX master server. This was back in 2005.
At times the various engineers asked for more logs - bpcd on master, bptm and bpcd on media servers, etc...
After long-long time of allowing level 5 bprd logs, monitoring file systems, each day copying previous bprd log away and deleting, we managed to actually get a log that showed the internal comms failure.
Symantec issued new bprd binary.
Problem was never seen again in this environment, even through NBU upgrades.

It seems that Symantec has never acknowledged the problem across all platforms, so this is why the problem is still around.

@Marianne van den Berg.

Need your help again. Having same problem again. Now it's becoming more frequently.

I had the problem during the morning. Only solved restarting machine. I tried to stop Netbackup, killing all bprd process and it didn't work.

Now I having the problem again. I don't know what else I can do ... What kind of logs may I look into ?

In bprd logs I don't see errors. There's a line in bprd logs from one of these restores that hanging...

line handle_image_status: clientname_1359229512 restfiles pid 14080 bpbrm pid 8486 status = 59
 

I didn't find line FORCE_RESTORE_MEDIA_SERVER on my master server.

Which OS is your master server?

What is NBU patch level on your master?

Please show us output of 'bpps' on the master when you see the hanging restore.
Also post bprd log as File attachment.
No guarantees that we will be able to find the issue in bprd log - as per my post above - it took many uploads of level 5 logs before Symantec backline engineer found the issue.

If there are lots of files to be restored , it is always best to select smaller chunks of files/folders at a time. We have seen situations where bprd process runs out of internal memory. This was supposed to have been resolved in some or other NBU patch. will see if I can find details.

OS Version Red Hat Enterprise Linux Server release 6.3 64bits

NBU patch level of master and media server's 7.5.0.4

Output of bpps: file attached

ps -ef | grep bprd

root      3030     1  0 12:41 ?        00:00:02 /usr/openv/netbackup/bin/bprd
root      5104 15944  0 16:14 pts/2    00:00:00 grep bprd
root     14094     1  0 14:05 ?        00:00:00 bprd -dontfork -mpxmain
root     19966  3030  0 14:53 ?        00:00:00 /usr/openv/netbackup/bin/bprd
root     27670  3030  0 15:12 ?        00:00:00 /usr/openv/netbackup/bin/bprd

Found a TECH http://www.symantec.com/business/support/index?page=content&id=TECH33379. Don't know if there's something related to isse. Since in my case all restores are hanging,regardless the medias needed are diferent.

Bprd logs from the moment I started a restore job.