Forum Discussion

PeterDragon's avatar
7 years ago

Catalog Restore Partially Successful (Errors in logs)

We are attempting to recover our catalog for DR testing to a server with a fresh install of the same patch level, install path, hostname and OS level. We are using EMC Data Domain with DDBoost. Everything goes well accept after we start the recovery wizard. In the logs we see errors refering to "system.posix_acl_access" errors. The restore completes but with partial success, all of out policies, and other information show up but other items for the pools and sotrage units are missing. And the BAR shows now clients in the list.

Has anyone seen these messages before?

System Information:
HARDWARE: LINUX_RH_X86
VERSION: NetBackup 7.7.1
RELEASEDATE: Thu Sep 10 15:27:12 CDT 2015
BUILDNUMBER: 20150910
OS: Red Hat Enterprise Linux Server release 6.9 (Santiago)

Recovery Log
Restore started 12/21/2017 21:02:33
21:02:41 (3.xxx) Found (126644) files in (1) images for Restore Job ID 3.xxx
21:02:43 (3.xxx) Estimated time to assemble file list: (0) days, (0) hours, (0) min, (2) sec
21:02:44 (3.xxx) Searched (126644) files of (126644) files for Restore Job ID 3.xxx
21:02:44 (3.001) Restoring from copy 1 of image created Thu 21 Dec 2017 06:14:50 AM EST from policy ADM_Catalog
21:02:44 (3.001) TAR STARTED 30263
21:02:46 (3.001) INF - Beginning restore from server rchaxpnb01.mckesson.com to client rchaxpnb01.mckesson.com.
21:02:47 (3.001) /usr/openv/volmgr/vm.conf
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/volmgr/vm.conf. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/netbackup/vault/sessions/
21:02:47 (3.001) Directory /usr/openv/netbackup/vault/sessions already exists.
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/netbackup/vault/sessions. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/lib/ost-plugins/cloudstore.conf
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/lib/ost-plugins/cloudstore.conf. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/lib/ost-plugins/CloudProvider.xml
21:02:47 (3.001) /usr/openv/lib/ost-plugins/CloudInstance.xml
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/lib/ost-plugins/CloudInstance.xml. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/lib/ost-plugins/cacert.pem
21:02:47 (3.001) /usr/openv/tmp/config_parameters
21:02:47 (3.001) /usr/openv/var/global/
21:02:47 (3.001) Directory /usr/openv/var/global already exists.
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/nbservice.conf
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/nbservice.conf. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/external_robotics.txt
21:02:47 (3.001) /usr/openv/var/global/nbcl.conf
21:02:47 (3.001) /usr/openv/var/global/device_mappings.txt
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/device_mappings.txt. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/server.conf
21:02:47 (3.001) /usr/openv/var/global/nbsl.xml
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/nbsl.xml. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/external_mediatypes.txt
21:02:47 (3.001) /usr/openv/var/global/server.conf.forbackup
21:02:47 (3.001) /usr/openv/var/global/logasst.db
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/logasst.db. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/external_densities.txt
21:02:47 (3.001) /usr/openv/var/global/vfm_fim_config_params.xml
21:02:47 (3.001) /usr/openv/var/global/createsrt.conf
21:02:47 (3.001) /usr/openv/var/global/ResourceLimits.xml
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/ResourceLimits.xml. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/databases.conf
21:02:47 (3.001) /usr/openv/var/global/external_types.txt
21:02:47 (3.001) /usr/openv/var/global/external_drivetypes.txt
21:02:47 (3.001) /usr/openv/var/global/i_nbdbinfo.dat
21:02:47 (3.001) Failed to restore attribute 'system.posix_acl_access' (EA) on /usr/openv/var/global/i_nbdbinfo.dat. Errno = 95: Operation not supported
21:02:47 (3.001) /usr/openv/var/global/nbstserv/

 

11 Replies

  • I did some more digging and started looking at the files the restore is trying to update. When I pull a "ls -ltra" I can see that the files appear to have extended attributes. And if I am not mistaken those attributes might be whats stopping them from being updated maybe?

    # ls -ltra /usr/openv/volmgr/vm.conf
    -rw-rwxr--. 1 root root 41 Aug 18 2015 /usr/openv/volmgr/vm.conf
    # ls -ltra /usr/openv/lib/ost-plugins/CloudInstance.xml
    -rw-rwxr--. 1 root root 6504 Jan 31 2017 /usr/openv/lib/ost-plugins/CloudInstance.xml
    # ls -ltra /usr/openv/var/global/nbservice.conf
    -rw-rwxr--. 1 root root 568 Jan 31 2017 /usr/openv/var/global/nbservice.conf

    • Mouse's avatar
      Mouse
      Moderator

      I am wondering what is the file system on /usr/openv

      I saw similar issues on unsupported filesystems like btrfs etc

    • PeterDragon's avatar
      PeterDragon
      Level 3

      The filesystem is EXT4. 

      # mount | egrep 'tmpfs|openv' (Production Master Server)
      tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
      /dev/mapper/vg01-lv_openv on /usr/openv type ext4 (rw)


      ]# mount | egrep 'tmpfs|openv' (DR Master Server)
      tmpfs on /dev/shm type tmpfs (rw)
      /dev/mapper/rootvg-nbu_lv on /usr/openv type ext4 (rw)

      The attribute appears to be SELinux. So I have removed the Extended Attributes using and will rerun the restore to see if the errors still persist. 

      # setfattr -x security.selinux /usr/openv/volmgr/vm.conf
      # setfattr -x security.selinux /usr/openv/netbackup/vault/sessions
      # setfattr -x security.selinux /usr/openv/lib/ost-plugins/cloudstore.conf
      # setfattr -x security.selinux /usr/openv/lib/ost-plugins/CloudInstance.xml
      # setfattr -x security.selinux /usr/openv/var/global
      # setfattr -x security.selinux /usr/openv/var/global/device_mappings.txt
      # setfattr -x security.selinux /usr/openv/var/global/nbsl.xml
      # setfattr -x security.selinux /usr/openv/var/global/logasst.db
      # setfattr -x security.selinux /usr/openv/var/global/ResourceLimits.xml
      # setfattr -x security.selinux /usr/openv/var/global/i_nbdbinfo.dat

      UPDATE: After removing the ext attribute and then re-running the recovery, the error still persists and the ext attribute was put back?!

      Before: root@rchaxpnb01 logs]# ls -ltra /usr/openv/var/global/nbservice.conf
      -rw-rwxr-- 1 root root 568 Jan 31 2017 /usr/openv/var/global/nbservice.conf

      After: root@rchaxpnb01 logs]# ls -ltra /usr/openv/var/global/nbservice.conf
      -rw-rwxr--. 1 root root 568 Jan 31 2017 /usr/openv/var/global/nbservice.conf

      Problem still exits. 

  • I think that you might have 64-bit version of ext4 on one server and 32-bit on another. Could you run:

    On Production Server:

    #dumpe2fs /dev/mapper/vg01-lv_openv  | grep featu

    On DR-server:

    #dumpe2fs /dev/mapper/rootvg-nbu_lv  | grep featu

    • PeterDragon's avatar
      PeterDragon
      Level 3

      Production Server
      dumpe2fs 1.41.12 (17-May-2010)
      Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
      Journal features: journal_incompat_revoke
      # uname -a
      Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Nov 23 16:03:01 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

      DR Server
      dumpe2fs 1.41.12 (17-May-2010)
      Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
      Journal features: journal_incompat_revoke
      # uname -a
      Linux 2.6.32-696.10.3.el6.x86_64 #1 SMP Thu Sep 21 12:12:50 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

      • Mouse's avatar
        Mouse
        Moderator

        Hang on. Can you do 'ls' on the production machine against the files in question and check whether the EA have been set?

        If so, NBU restore job will try to get them restored.

  • In the initial post you mentioned that all the policies and stuff are back and stu, pool etc appear to be misisng.. so it kind of seems nbdb isn’t restored..

    Apart from the extended attributes part could you check the recovery logs if anything was restored to “/usr/openv/db/staging” or “/usr/openv/db/data” from the staging part..

    If files were correctly restored to “/usr/openv/db/staging” simply try to stop nbu services followed by the below steps

    1. rename “/usr/openv/db/data” to old
    2. create an empty folder “/usr/openv/db/data”
    3. copy the contents from staging to this new data folder
    4. start the nbdb service
    “/usr/openv/db/bin/nbdbms_start_server”
    5. try “/usr/openv/db/bin/nbdb_ping”
    6. if nbdb is alive then try to start only the emm next - “/usr/openv/netbackup/bin/nbemm”
    7. confirm by running the 2 commands below
    >> nbemmcmd -listhosts
    >> bpstulist -L

    if all looks good you can start all the other nbu services as well