BAR GUI not opening for a particular date's backup , netbackup 7.1 windows

[dixit47]

Hi , 

I have extended the retention of 1 image to January 15th .

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -idonly -U -d 11/10/2014 00:00:00 -e 11/15/2014 00:00:00 -policy XXX
Backed Up         Expires       Files       KB  C  Sched Type   Policy
----------------  ---------- -------- --------  -  ------------ ------------
11/14/2014 17:00  01/30/2015    16616 13194820  N  Full Backup  XXX

When I am trying open the BAR GUI for 14th Backup its getting hung and not opening . For the other dates its working fine .

I have disabled perform default serach option . No luck .

I have tested Duplication job for that image , its failing with below error .

"C:\Program Files\Veritas\NetBackup\bin\admincmd\bpduplicate.exe"  -dstunit abc -dp NB_duplicates -owner *ANY* -rl 5 -fail_on_error 0 -set_primary 0 -primary -number_copies 1 -L "C:\Program Files\Veritas\NetBackup\Logs\user_ops\NA\xyz\logs\jbpDup-20141226211218.log" -Bidfile "C:\Users\C000591\AppData\Local\Symantec\NetBackup\20141226211218.bid" -M XXX

21:12:18 Duplicate started 12/26/2014 9:12:18 PM
21:12:18 Activity monitor job id = 347302
21:12:20 INF - Destination storage unit abc on host XXX
21:12:21 INF - db_IMAGE() failed: unable to process request (228)
21:12:22 INF - Status = no images were successfully processed.

21:12:22 INF - Duplicating policy ActiveDirectory schedule Weekly backup id XXX copy 1 created on 11/14/2014 17:00:01 on source path \\XXX\windows1\su1
21:12:22 INF - Duplicate of backupid XXX failed, unable to process request (228).

21:12:22 INF - Status = no images were successfully processed.

bpdbm logs:-

21:45:53.063 [4484.4508] <2> getIDIRSTRUCT: ?
21:45:53.063 [4484.4508] <2> image_dir_name: ?
21:45:53.063 [4484.4508] <4> db_error_add_to_file: unlink C:\Program Files\Veritas\NetBackup\db\images\XX\1416000000\tmp\xyz_FULL.f.extend has failed: Permission denied (13)
21:45:53.063 [4484.4508] <16> extend_list_add_files: unlink C:\Program Files\Veritas\NetBackup\db\images\XX\1416000000\tmp\xyz_FULL.f.extend has failed: Permission denied (13)
21:45:53.063 [4484.4508] <2> extend_list_add_files: total_selected 0
21:45:53.063 [4484.4508] <2> db_ImgUnlock: db_ImgUnlock(0000000074949F70) unlocked
21:45:53.063 [4484.4508] <2> process_request: request complete: exit status 228 unable to process request; query type: 248
21:45:53.156 [5544.6184] <2> GetParentProcessHandle: OpenProcess(3484) failed: 5. Attempting token adjustment...
21:45:53.156 [5544.6184] <2> GetParentProcessHandle: Got process handle after enabling debug privilege
21:45:53.156 [5544.6184] <2> getIDIRSTRUCT: ?
21:45:53.156 [5544.6184] <2> bpdbm: C:\Program Files\Veritas\NetBackup\Logs\bpdbm
21:45:53.156 [5544.6184] <2> logconnections: BPDBM ACCEPT FROM 

Please suggest how can I open the BAR GUI for 11/14's backup and restore the data .

[Marianne]

21:45:53.063 [4484.4508] <16> extend_list_add_files: unlink C:\Program Files\Veritas\NetBackup\db\images\XX\1416000000\tmp\xyz_FULL.f.extend has failed: Permission denied (13) The above error is what stands out for me. Have you checked OS level ownership and permission of the folder and files in comparison with other images that can be browsed? Is UAC turned off on the master?

[dixit47]

Hi Marianne , The permission is set everyone for folder 1416000000 , I have attached the screenshot .

The files were visible few days back , recently it is getting hung , yes the permissions are same with the files ( .f , .extend file permission ) which can be browsed .

UAC is at level 1 ( out of the 4 ) . I have set that to never notify , but it is asking for a reboot . Not sure whether can go with it , we have to check with the clients for a window .

[INT_RND]

Are you running the BAR GUI with "Run as administrator" ?

[dixit47]

@Marianne: yeah i have checked permission.. its fine

@INT_RND : yeah BAR GUI is run as admin :)

but still BAR GUI is getting hung for 14 Nov. for other dates its working fine...

[Marianne]

This backup is fairly small, so this will not be a result of browse timeout. Curious to know what happens if you try to list from cmd.

Open cmd with 'Run as administrator', then try:

bplist -C <client-name> -s 11/14/2014 17:00 -e 11/14/2014 17:00 -R 5 /

(This should return the top 5 folder/file levels)

About UAC - disable on master server followed by reboot.
Until you can reboot, open GUI and cmd everytime with 'Run as administrator'. 
The 'permission denied' in bpdbm log is probably because of UAC issue.

For further troubleshooting. we will need bprd log as well.
Please post as File attachment.

[dixit47]

Hi INT_RND , Yes i am opening it as admin , still its not opening .
For the other dates and also other servers its working fine , only issue is with 14th's backup ( for which I have extended the retention ).
I am pasting the screen shot .

Hi Marianne 

I have attached the bprd logs from the master . bplist command is not giving any output for that particular date range. Its stuck for 1 hour .

[Marianne]

My advice at this stage is to log a call with Symantec Support. 
They will need the exact command that you used, the time that you tried the listing with level 5 bpdbm and bprd logs.

My gut feel is that .f files for this backup image are probably corrupted.
If backup is on tape, you can try to expire the backup and then import it again, but best to do this under guidance from Symantec Support engineer.

I found a listing that was requested at 04:14:23, but nothing was returned from bpdbm by the time you collected the logs:
04:14:23.148 [4316.1364] <2> fileslist: begin db communication
04:14:23.148 [4316.1364] <2> fileslist: criteria sent to db mgr

PS:

You forgot the '/' in bplist command. Without the / the listing will be requested for your current filepath which was C:\Program Files\Veritas\NetBackup\bin.

[dixit47]

Hi Marianne ,

It is not listing the files with / also , its in hung state for an hour . Sure I will check with my client and log a call with Symantec support . The images are on disk .

[dixit47]

I ran a media verify , its been running for last 200 hours and stuck with the below status 

12/26/2014 9:46:11 PM - Info bpdm(pid=5596) spawning a child process         
12/26/2014 9:46:11 PM - Info bpbrm(pid=5596) child pid: 4644          
12/26/2014 9:46:12 PM - Info bpdm(pid=4644) started            
12/26/2014 9:46:12 PM - started process bpdm (4644)
12/26/2014 9:46:13 PM - begin reading
12/26/2014 9:51:39 PM - end reading; read time: 00:05:26
12/26/2014 9:51:40 PM - begin reading
12/26/2014 9:56:11 PM - Error bpdm(pid=4644) cannot write data to socket, 10054       
12/26/2014 11:14:44 PM - Error bpbrm(pid=6468) could not write EXIT STATUS to OUTSOCK      
12/26/2014 11:14:49 PM - Info tar32(pid=4456) done. status: 24: socket write failed    

Data is showing in disk  . Screenshot attached 

[Marianne]

Seems like something is wrong with the read of fragment 2 and read timeout already occurred after 5 min (09:56). Hopefully you have enabled bpdm and bpbrm logs on the media server at level 5 before verify attempt? Symantec will need those too when you log a call.