Catalog Restore Partially Successful (Errors in logs)

[PeterDragon]

We are attempting to recover our catalog for DR testing to a server with a fresh install of the same patch level, install path, hostname and OS level. We are using EMC Data Domain with DDBoost. Everything goes well accept after we start the recovery wizard. In the logs we see errors refering to "system.posix_acl_access" errors. The restore completes but with partial success, all of out policies, and other information show up but other items for the pools and sotrage units are missing. And the BAR shows now clients in the list.

Has anyone seen these messages before?

System Information:
HARDWARE: LINUX_RH_X86
VERSION: NetBackup 7.7.1
RELEASEDATE: Thu Sep 10 15:27:12 CDT 2015
BUILDNUMBER: 20150910
OS: Red Hat Enterprise Linux Server release 6.9 (Santiago)

Recovery Log
Restore started 12/21/2017 21:02:33
21:02:41 (3.xxx) Found (126644) files in (1) images for Restore Job ID 3.xxx
21:02:43 (3.xxx) Estimated time to assemble file list: (0) days, (0) hours, (0) min, (2) sec
21:02:44 (3.xxx) Searched (126644) files of (126644) files for Restore Job ID 3.xxx
21:02:44 (3.001) Restoring from copy 1 of image created Thu 21 Dec 2017 06:14:50 AM EST from policy ADM_Catalog
21:02:44 (3.001) TAR STARTED 30263
21:02:46 (3.001) INF - Beginning restore from server rchaxpnb01.mckesson.com to client rchaxpnb01.mckesson.com.
21:02:47 (3.001) /usr/openv/volmgr/vm.conf
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/volmgr/vm.conf. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/netbackup/vault/sessions/
21:02:47 (3.001) Directory /usr/openv/netbackup/vault/sessions already exists.
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/netbackup/vault/sessions. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/lib/ost-plugins/cloudstore.conf
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/lib/ost-plugins/cloudstore.conf. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/lib/ost-plugins/CloudProvider.xml
21:02:47 (3.001) /usr/openv/lib/ost-plugins/CloudInstance.xml
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/lib/ost-plugins/CloudInstance.xml. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/lib/ost-plugins/cacert.pem
21:02:47 (3.001) /usr/openv/tmp/config_parameters
21:02:47 (3.001) /usr/openv/var/global/
21:02:47 (3.001) Directory /usr/openv/var/global already exists.
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/nbservice.conf
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/nbservice.conf. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/external_robotics.txt
21:02:47 (3.001) /usr/openv/var/global/nbcl.conf
21:02:47 (3.001) /usr/openv/var/global/device_mappings.txt
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/device_mappings.txt. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/server.conf
21:02:47 (3.001) /usr/openv/var/global/nbsl.xml
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/nbsl.xml. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/external_mediatypes.txt
21:02:47 (3.001) /usr/openv/var/global/server.conf.forbackup
21:02:47 (3.001) /usr/openv/var/global/logasst.db
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/logasst.db. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/external_densities.txt
21:02:47 (3.001) /usr/openv/var/global/vfm_fim_config_params.xml
21:02:47 (3.001) /usr/openv/var/global/createsrt.conf
21:02:47 (3.001) /usr/openv/var/global/ResourceLimits.xml
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/ResourceLimits.xml. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/databases.conf
21:02:47 (3.001) /usr/openv/var/global/external_types.txt
21:02:47 (3.001) /usr/openv/var/global/external_drivetypes.txt
21:02:47 (3.001) /usr/openv/var/global/i_nbdbinfo.dat
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/i_nbdbinfo.dat. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/nbstserv/

[PeterDragon]

I did some more digging and started looking at the files the restore is trying to update. When I pull a "ls -ltra" I can see that the files appear to have extended attributes. And if I am not mistaken those attributes might be whats stopping them from being updated maybe?

# ls -ltra /usr/openv/volmgr/vm.conf
-rw-rwxr--. 1 root root 41 Aug 18 2015 /usr/openv/volmgr/vm.conf
# ls -ltra /usr/openv/lib/ost-plugins/CloudInstance.xml
-rw-rwxr--. 1 root root 6504 Jan 31 2017 /usr/openv/lib/ost-plugins/CloudInstance.xml
# ls -ltra /usr/openv/var/global/nbservice.conf
-rw-rwxr--. 1 root root 568 Jan 31 2017 /usr/openv/var/global/nbservice.conf

[Mouse]

I am wondering what is the file system on /usr/openv

I saw similar issues on unsupported filesystems like btrfs etc

[PeterDragon]

The filesystem is EXT4. 

# mount | egrep 'tmpfs|openv' (Production Master Server)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/mapper/vg01-lv_openv on /usr/openv type ext4 (rw)

]# mount | egrep 'tmpfs|openv' (DR Master Server)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/rootvg-nbu_lv on /usr/openv type ext4 (rw)

The attribute appears to be SELinux. So I have removed the Extended Attributes using and will rerun the restore to see if the errors still persist. 

# setfattr -x security.selinux /usr/openv/volmgr/vm.conf
# setfattr -x security.selinux /usr/openv/netbackup/vault/sessions
# setfattr -x security.selinux /usr/openv/lib/ost-plugins/cloudstore.conf
# setfattr -x security.selinux /usr/openv/lib/ost-plugins/CloudInstance.xml
# setfattr -x security.selinux /usr/openv/var/global
# setfattr -x security.selinux /usr/openv/var/global/device_mappings.txt
# setfattr -x security.selinux /usr/openv/var/global/nbsl.xml
# setfattr -x security.selinux /usr/openv/var/global/logasst.db
# setfattr -x security.selinux /usr/openv/var/global/ResourceLimits.xml
# setfattr -x security.selinux /usr/openv/var/global/i_nbdbinfo.dat

UPDATE: After removing the ext attribute and then re-running the recovery, the error still persists and the ext attribute was put back?!

Before: root@rchaxpnb01 logs]# ls -ltra /usr/openv/var/global/nbservice.conf
-rw-rwxr-- 1 root root 568 Jan 31 2017 /usr/openv/var/global/nbservice.conf

After: root@rchaxpnb01 logs]# ls -ltra /usr/openv/var/global/nbservice.conf
-rw-rwxr--. 1 root root 568 Jan 31 2017 /usr/openv/var/global/nbservice.conf

Problem still exits. 

[Mouse]

Did you check this TN? http://www.veritas.com/docs/100022474

[PeterDragon]

I did actually, and SELinux is always disabled on our systems. That is what I found odd when I saw the extended file attribute on the files in the error log. But after trying the above commands, we found the extended attribute was trying to be set by the restore. 

# sestatus
SELinux status: disabled

But both systems have SELinux Disabled. 

[Alexis_Jeldrez]

If you have connectivity from the production master to the DR master, try some tests of basic backup ups and restores between each other. I don't believe this is a problem of the catalog restore process but just a filesystem limitation that can be troubleshooted with normal backups.

https://www.veritas.com/support/en_US/article.100027241

[Mike_Gavrilov]

I think that you might have 64-bit version of ext4 on one server and 32-bit on another. Could you run:

On Production Server:

#dumpe2fs /dev/mapper/vg01-lv_openv  | grep featu

On DR-server:

#dumpe2fs /dev/mapper/rootvg-nbu_lv  | grep featu

[PeterDragon]

Production Server
dumpe2fs 1.41.12 (17-May-2010)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Journal features: journal_incompat_revoke
# uname -a
Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Nov 23 16:03:01 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

DR Server
dumpe2fs 1.41.12 (17-May-2010)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Journal features: journal_incompat_revoke
# uname -a
Linux 2.6.32-696.10.3.el6.x86_64 #1 SMP Thu Sep 21 12:12:50 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[Mouse]

Hang on. Can you do 'ls' on the production machine against the files in question and check whether the EA have been set?

If so, NBU restore job will try to get them restored.

[PeterDragon]

Arggg I see the problem now. Yes it is set and the DR servers Logical Volume does not have "user_xattr acl" variable set. Thats got to be the reason why the catalog restore cant set the attribute and why its getting the OS Error. 

ls -ltra /usr/openv
total 23696
drwxr-xr-x+ 2 root bin 4096 Aug 18 2015 kms
-r--rwxr--+ 1 root bin 11938 Aug 18 2015 regid.1992-12.com.symantec_netbackup-7.6.0.1_2.swidtag
-r--rwxr--+ 1 root bin 11962 Aug 18 2015 regid.1992-12.com.symantec_netbackup-7.6.0.3_1.swidtag
-r--rwxr--+ 1 root bin 11957 Aug 25 2015 regid.1992-12.com.symantec_netbackup-7.7.0.0_1.swidtag
drwxr-xr-x+ 7 root bin 4096 Sep 10 2015 wmc
-r--r--r--. 1 root bin 11890 Sep 10 2015 swidtag.xml
-rw-r--r--. 1 root bin 13279941 Sep 10 2015 nbwmc.tar.gz
-r-xr-xr-x. 1 root bin 10739849 Sep 10 2015 pddeserver.tar.gz
-rw-r--r--. 1 eg718t5 domain users 0 Oct 1 2015 test.txt
drwxr-xr-x+ 12 root bin 4096 Oct 22 2015 db
drwxr-xr-x. 3 root bin 4096 Oct 22 2015 man
drwxr-xr-x+ 8 root bin 4096 Oct 22 2015 volmgr
drwxr-xr-x+ 5 root bin 4096 Oct 22 2015 msg
drwxr-xr-x+ 113 root bin 4096 Oct 22 2015 resources
drwxr-xr-x+ 2 root bin 4096 Oct 22 2015 share
drwxr-xr-x. 7 root root 4096 Oct 22 2015 pdde
-r--r--r--. 1 root bin 11959 Oct 22 2015 regid.1992-12.com.symantec_netbackup-7.7.1.0_1.swidtag
drwxr-xr-x+ 9 root bin 20480 Oct 22 2015 lib
drwxr-xr-x. 15 root root 4096 Oct 23 2015 ..
drwxr-xr-x. 30 root root 4096 Nov 4 2016 logs
drwxrwxr-x+ 17 root root 4096 Nov 23 2016 .
drwxr-xr-x. 5 root bin 4096 Dec 9 01:41 java
drwxr-xr-x+ 3 root bin 28672 Dec 22 09:55 tmp
drwxr-xr-x+ 12 root bin 4096 Dec 22 13:27 var
drwxr-xr-x+ 21 root bin 4096 Dec 22 15:20 netbackup

[Amol_Nair]

In the initial post you mentioned that all the policies and stuff are back and stu, pool etc appear to be misisng.. so it kind of seems nbdb isn’t restored..

Apart from the extended attributes part could you check the recovery logs if anything was restored to “/usr/openv/db/staging” or “/usr/openv/db/data” from the staging part..

If files were correctly restored to “/usr/openv/db/staging” simply try to stop nbu services followed by the below steps

1. rename “/usr/openv/db/data” to old
2. create an empty folder “/usr/openv/db/data”
3. copy the contents from staging to this new data folder
4. start the nbdb service
“/usr/openv/db/bin/nbdbms_start_server”
5. try “/usr/openv/db/bin/nbdb_ping”
6. if nbdb is alive then try to start only the emm next - “/usr/openv/netbackup/bin/nbemm”
7. confirm by running the 2 commands below
>> nbemmcmd -listhosts
>> bpstulist -L

if all looks good you can start all the other nbu services as well