Unable to restore hardlink backup with -l -H -y

[TG_syd]

1. create test data directories on a specific file system
[root@tsm-vm6 bin]# ls /usr/
bin etc games i686-pc-mingw32 include kerberos lib lib64 libexec local openv sbin share src tmp
[root@tsm-vm6 bin]# mkdir /usr/testdata1 /usr/testdata2 /usr/testdata3
[root@tsm-vm6 bin]# ls /usr/
bin etc games i686-pc-mingw32 include kerberos lib lib64 libexec local openv sbin share src testdata1 testdata2 testdata3 tmp
2. create test data files
[root@tsm-vm6 bin]# echo "hardlink test" > /usr/testdata1/test_01.file
[root@tsm-vm6 bin]# echo "hardlink test" > /usr/testdata2/test_02.file
[root@tsm-vm6 bin]# echo "hardlink test" > /usr/testdata3/test_03.file

3. create hardlinks to the test data files within the test data directory
[root@tsm-vm6 bin]# cd /usr/testdata2
[root@tsm-vm6 testdata2]# ln /usr/testdata1/test_01.file /usr/testdata2/test_01_link.file
[root@tsm-vm6 testdata2]# ln /usr/testdata2/test_02.file /usr/testdata2/test_02_link.file
[root@tsm-vm6 testdata2]# ln /usr/testdata3/test_03.file /usr/testdata2/test_03_link.file
4. confirm the new created files are hardlinked by checking that they share the same inode number
[root@tsm-vm6 testdata2]# ls -il
total 16
2509622 -rw-r--r-- 2 root root 14 Dec 23 14:41 test_01_link.file
2509623 -rw-r--r-- 2 root root 14 Dec 23 14:42 test_02.file
2509623 -rw-r--r-- 2 root root 14 Dec 23 14:42 test_02_link.file
2509624 -rw-r--r-- 2 root root 14 Dec 23 14:42 test_03_link.file
[root@tsm-vm6 testdata2]# ls -il /usr/testdata1
total 4
2509622 -rw-r--r-- 2 root root 14 Dec 23 14:41 test_01.file
[root@tsm-vm6 testdata2]# ls -il /usr/testdata3
total 4
2509624 -rw-r--r-- 2 root root 14 Dec 23 14:42 test_03.file

5. Create a file system backup of the /usr/testdata1 and /usr/testdata2 directory containing the hardlinks
390 Backup Done 0 Test_HardLink FULL tsm-vm6 PTS00235
6. Delete the test data directory /usr/testdata3 that hosts a hard link target

[root@tsm-vm6 testdata2]# rm -rf /usr/testdata3
[root@tsm-vm6 testdata2]# ls /usr/
bin etc games i686-pc-mingw32 include kerberos lib lib64 libexec local openv sbin share src testdata1 testdata2 tmp
7. Execute bplist against backup object generated
[root@tsm-vm6 bin]# ./bplist -C tsm-vm6 -S PTS00235 -X -s 1387790239 -e 1387790239 -R -t 0 -l "/*/"
drwxr-xr-x root root 0 Dec 23 14:41 /usr/testdata1/
-rw-r--r-- root root 14 Dec 23 14:41 /usr/testdata1/test_01.file
drwxr-xr-x root root 0 Dec 23 14:42 /usr/testdata2/
-rw-r--r-- root root 14 Dec 23 14:42 /usr/testdata2/test_02.file
-rw-r--r-- root root 14 Dec 23 14:42 /usr/testdata2/test_03_link.file
-rw-r--r-- root root 0 Dec 23 14:42 /usr/testdata2/test_02_link.file
-rw-r--r-- root root 0 Dec 23 14:41 /usr/testdata2/test_01_link.file
8. Staging area and Renamer file created successfully.

9. Execute bprestore operation

[root@tsm-vm6 bin]# ./bprestore -C tsm-vm6 -S PTS00235 -H -l -y -R /renamer.rnf -L /bprestore_output.log -X -s 1387790239 -e 1387790239 -t 0 -drs -w "/*/"
EXIT STATUS 2800: Standard policy restore error
[root@tsm-vm6 bin]# echo $?
5

10. Bprestore fails with status Incomplete . Netbackup bprestore log file:
[root@tsm-vm6 bin]# cat /bprestore_output.log
Restore started 12/23/2013 14:53:33

14:53:34 (391.001) Restoring from copy 1 of image created Mon 23 Dec 2013 02:47:19 PM IST
14:53:34 (391.001) TAR STARTED 26356
14:53:35 (391.001) INF - Beginning restore from server PTS00235 to client tsm-vm6.
14:53:35 (391.001) /usr/testdata1/
14:53:35 (391.001) Changed /usr/testdata1/ to /staging_area/usr/testdata1/
14:53:35 (391.001) /usr/testdata1/test_01.file
14:53:35 (391.001) Changed /usr/testdata1/test_01.file to /staging_area/usr/testdata1/test_01.file
14:53:35 (391.001) /usr/testdata2/
14:53:35 (391.001) Changed /usr/testdata2/ to /staging_area/usr/testdata2/
14:53:35 (391.001) /usr/testdata2/test_02.file
14:53:35 (391.001) Changed /usr/testdata2/test_02.file to /staging_area/usr/testdata2/test_02.file
14:53:35 (391.001) /usr/testdata2/test_03_link.file
14:53:35 (391.001) Changed /usr/testdata2/test_03_link.file to /staging_area/usr/testdata2/test_03_link.file
14:53:35 (391.001) /usr/testdata2/test_02_link.file
14:53:35 (391.001) Changed /usr/testdata2/test_02_link.file to /staging_area/usr/testdata2/test_02_link.file
14:53:35 (391.001) Could not link /staging_area/usr/testdata2/test_02_link.file -> /usr/testdata2/test_02.file. Errno = 18: Invalid cross-device link
14:53:35 (391.001) /usr/testdata2/test_01_link.file
14:53:35 (391.001) Changed /usr/testdata2/test_01_link.file to /staging_area/usr/testdata2/test_01_link.file
14:53:35 (391.001) Could not link /staging_area/usr/testdata2/test_01_link.file -> /usr/testdata1/test_01.file. Errno = 18: Invalid cross-device link
14:53:35 (391.001) INF - TAR EXITING WITH STATUS = 0
14:53:35 (391.001) INF - TAR RESTORED 5 OF 7 FILES SUCCESSFULLY
14:53:35 (391.001) INF - TAR KEPT 0 EXISTING FILES
14:53:35 (391.001) INF - TAR PARTIALLY RESTORED 0 FILES
14:53:35 (391.001) Status of restore from copy 1 of image created Mon 23 Dec 2013 02:47:19 PM IST = the restore failed to recover the requested files
14:53:36 (391.xxx) INF - Status = Standard policy restore error.
11. ls –alg of the files restored in staging area path
[root@tsm-vm6 bin]# ls -alg /staging_area/usr/testdata1/
total 12
drwxr-xr-x 2 root 4096 Dec 23 14:41 .
drwxr-xr-x 4 root 4096 Dec 23 14:53 ..
-rw-r--r-- 1 root 14 Dec 23 14:41 test_01.file
[root@tsm-vm6 bin]# ls -alg /staging_area/usr/testdata2/
total 16
drwxr-xr-x 2 root 4096 Dec 23 14:42 .
drwxr-xr-x 4 root 4096 Dec 23 14:53 ..
-rw-r--r-- 1 root 14 Dec 23 14:42 test_02.file
-rw-r--r-- 1 root 14 Dec 23 14:42 test_03_link.file
[root@tsm-vm6 bin]# ls -li /staging_area/usr/testdata2/
total 8
8197 -rw-r--r-- 1 root root 14 Dec 23 14:42 test_02.file
8198 -rw-r--r-- 1 root root 14 Dec 23 14:42 test_03_link.file
[root@tsm-vm6 bin]# ls -li /staging_area/usr/testdata1/
total 4
8195 -rw-r--r-- 1 root root 14 Dec 23 14:41 test_01.file
12. Netbackup bprestore Console Output:
12/23/2013 14:53:24 - begin Restore
12/23/2013 14:53:25 - restoring from image tsm-vm6_1387790239
12/23/2013 14:53:25 - Info bprd (pid=25240) Restoring from copy 1 of image created Mon Dec 23 14:47:19 2013
12/23/2013 14:53:26 - Info bpbrm (pid=25249) tsm-vm6 is the host to restore to
12/23/2013 14:53:26 - Info bpbrm (pid=25249) reading file list from client
12/23/2013 14:53:26 - connecting
12/23/2013 14:53:26 - Info bpbrm (pid=25249) starting bptm
12/23/2013 14:53:26 - Info tar (pid=26356) Restore started
12/23/2013 14:53:26 - connected; connect time: 0:00:00
12/23/2013 14:53:26 - Info bpbrm (pid=25249) bptm pid: 25250
12/23/2013 14:53:26 - Info bptm (pid=25250) start
12/23/2013 14:53:26 - started process bptm (pid=25250)
12/23/2013 14:53:26 - Info bpdm (pid=25250) reading backup image
12/23/2013 14:53:26 - Info bptm (pid=25250) using 30 data buffers
12/23/2013 14:53:26 - Info bptm (pid=25250) spawning a child process
12/23/2013 14:53:26 - Info bptm (pid=25250) child pid: 25252
12/23/2013 14:53:27 - begin reading
12/23/2013 14:53:27 - Info bptm (pid=25250) waited for empty buffer 0 times, delayed 0 times
12/23/2013 14:53:27 - end reading; read time: 0:00:00
12/23/2013 14:53:27 - Info bptm (pid=25250) completed reading backup image
12/23/2013 14:53:27 - Info bptm (pid=25250) EXITING with status 0 <----------
12/23/2013 14:53:27 - Info tar (pid=26356) done. status: 5
12/23/2013 14:53:27 - Info tar (pid=26356) done. status: 5: the restore failed to recover the requested files
12/23/2013 14:53:27 - Error bpbrm (pid=25249) client restore EXIT STATUS 5: the restore failed to recover the requested files
12/23/2013 14:53:27 - restored from image tsm-vm6_1387790239; restore time: 0:00:02
12/23/2013 14:53:27 - Warning bprd (pid=25240) Restore must be resumed prior to first image expiration on Tue 23 Dec 2014 02:47:19 PM IST
12/23/2013 14:53:27 - end Restore; elapsed time 0:00:03
Standard policy restore error (2800)
======================================================================

We further investigated bprestore command and observed excluding -y option for the command will successfully restore the hard links.

The detailed output is as follows:

- Bplist command Output:
[root@tsm-vm6 testdata2]# /usr/openv/netbackup/bin/bplist -C tsm-vm6 -S PTS00235 -X -s 1387876575 -e 1387876575 -R -t 0 -l "/*/"
drwxr-xr-x root root 0 Dec 24 14:41 /usr/testdata1/
-rw-r--r-- root root 14 Dec 24 14:41 /usr/testdata1/test_01.file
drwxr-xr-x root root 0 Dec 24 14:42 /usr/testdata2/
-rw-r--r-- root root 14 Dec 24 14:41 /usr/testdata2/test_02.file
-rw-r--r-- root root 14 Dec 24 14:41 /usr/testdata2/test_03_link.file
-rw-r--r-- root root 0 Dec 24 14:41 /usr/testdata2/test_02_link.file
-rw-r--r-- root root 0 Dec 24 14:41 /usr/testdata2/test_01_link.file
- Cleared Staging Area
[root@tsm-vm6 testdata2]# ls -lia /staging_area/
total 8
2 drwxr-xr-x 2 root root 4096 Dec 24 16:37 .
2 dr-xr-xr-x. 29 root root 4096 Dec 24 16:37 ..
- Bprestore Command:
[root@tsm-vm6 testdata2]# /usr/openv/netbackup/bin/bprestore -C tsm-vm6 -S PTS00235 -H -l -R /renamer.rnf -L /bprestore_test.log -X -s 1387876575 -e 1387876575 -t 0 -drs -w "/*/"
[root@tsm-vm6 testdata2]# echo $?
0
- Bprestore Output:
[root@tsm-vm6 testdata2]# cat /bprestore_test.log
Restore started 12/24/2013 16:49:04
16:49:05 (418.001) Restoring from copy 1 of image created Tue 24 Dec 2013 02:46:15 PM IST
16:49:06 (418.001) TAR STARTED 27707
16:49:06 (418.001) INF - Beginning restore from server PTS00235 to client tsm-vm6.
16:49:06 (418.001) /usr/testdata1/
16:49:06 (418.001) Changed /usr/testdata1/ to /staging_area/usr/testdata1/
16:49:07 (418.001) /usr/testdata1/test_01.file
16:49:07 (418.001) Changed /usr/testdata1/test_01.file to /staging_area/usr/testdata1/test_01.file
16:49:07 (418.001) /usr/testdata2/
16:49:07 (418.001) Changed /usr/testdata2/ to /staging_area/usr/testdata2/
16:49:07 (418.001) /usr/testdata2/test_02.file
16:49:07 (418.001) Changed /usr/testdata2/test_02.file to /staging_area/usr/testdata2/test_02.file
16:49:07 (418.001) /usr/testdata2/test_03_link.file
16:49:07 (418.001) Changed /usr/testdata2/test_03_link.file to /staging_area/usr/testdata2/test_03_link.file
16:49:07 (418.001) /usr/testdata2/test_02_link.file
16:49:07 (418.001) Changed /usr/testdata2/test_02_link.file to /staging_area/usr/testdata2/test_02_link.file
16:49:07 (418.001) Changed link to /usr/testdata2/test_02.file to /staging_area/usr/testdata2/test_02.file
16:49:07 (418.001) /usr/testdata2/test_01_link.file
16:49:07 (418.001) Changed /usr/testdata2/test_01_link.file to /staging_area/usr/testdata2/test_01_link.file
16:49:07 (418.001) Changed link to /usr/testdata1/test_01.file to /staging_area/usr/testdata1/test_01.file
16:49:07 (418.001) INF - TAR EXITING WITH STATUS = 0
16:49:07 (418.001) INF - TAR RESTORED 7 OF 7 FILES SUCCESSFULLY
16:49:07 (418.001) INF - TAR KEPT 0 EXISTING FILES
16:49:07 (418.001) INF - TAR PARTIALLY RESTORED 0 FILES
16:49:07 (418.001) Status of restore from copy 1 of image created Tue 24 Dec 2013 02:46:15 PM IST = the requested operation was successfully completed
16:49:07 (418.xxx) INF - Status = the requested operation was successfully completed.

[root@tsm-vm6 testdata2]#
[root@tsm-vm6 testdata2]# ls -lia /staging_area/usr/testdata2
total 24
114692 drwxr-xr-x 2 root root 4096 Dec 24 14:42 .
114689 drwxr-xr-x 4 root root 4096 Dec 24 16:49 ..
114691 -rw-r--r-- 2 root root 14 Dec 24 14:41 test_01_link.file
114693 -rw-r--r-- 2 root root 14 Dec 24 14:41 test_02.file
114693 -rw-r--r-- 2 root root 14 Dec 24 14:41 test_02_link.file
114694 -rw-r--r-- 1 root root 14 Dec 24 14:41 test_03_link.file
[root@tsm-vm6 testdata2]# ls -lia /staging_area/usr/testdata1
total 12
114690 drwxr-xr-x 2 root root 4096 Dec 24 14:41 .
114689 drwxr-xr-x 4 root root 4096 Dec 24 16:49 ..
114691 -rw-r--r-- 2 root root 14 Dec 24 14:41 test_01.file  

Why the hard link restore fails when the -y option is included ? 

[SymTerry]

From HOWTO43687:

Specify -y to rename UNIX soft links by using the -R rename_file option in the same way as when you rename files. Hard links are unchanged.

Are you also renaming a soft link? As mentioned in the doc, this switch will not effect hard links, and would not attempt to do so if left out.

[TG_syd]

It is needed for a script which can encounter both soft and hard links so.. both -H and -y are used.

[TG_syd]

HOWTO43687 says Hard links are unchanged but still hard links are not restored when the -y switch is used.

[Marianne]

Taran is correct. -y is for soft links only.

Extract from Commands manual:

Specify -y to rename UNIX soft links by using the -R rename_file option in

the same way as when you rename files. Hard links are unchanged.

[CRZ · Accepted solution]

I believe the way the Commands Guide lists the command, you can ONLY specify one of

-l | -H | -y

So, when you have all three, it uses "-y" (and when you remove "-y", it's going with "-H") - but the truth is, you don't want EITHER of them!

"-l" is the only one you would need here.  Think of it as a COMBINATION of BOTH "-H" and "-y" (whereas the other two are independent of each other: -H for hard only and -y for soft only).

Give it a try with JUST "-l" and see if it works.

EDIT: And if it turns out I'm wrong, we'll probably need your version, O/S and to open a case to get escalated to Engineering to look at our code.

[TG_syd]

Yup, That seems to be a solution. [-l | -H | -y] are optional switches i.e. only one should be used.

that is causing the unexpected behavior.