Forum Discussion

Hamza_H's avatar
Hamza_H
Moderator
5 years ago

Backup of a Redhat server is partial - ERR - Cannot lstat gvfs. Errno = 13: Permission denied

Hi,

 

We have a backup that gets status=1 because of the known error : 

 

ERR - Cannot lstat gvfs. Errno = 13: Permission denied

 

We have been through netbackup technotes & redhat too, but still no solution even with the exclusions :

https://www.veritas.com/support/en_US/article.100024911

** Previously the following command would resolve this error in older versions of Linux:
To allow root user access to the .gvfs filesystem.  :

1. Add user_allow_other to the /etc/fuse.conf file

2. Run the following sequence of commands:

   sudo su - {USER.EN_US}  
   /usr/bin/fusermount -zu $HOME/.gvfs
   /usr/lib64/gvfs/gvfs-fuse-daemon -o allow_root $HOME/.gvfs

   Where "user" is the owner of the gvfs filesystem

** In newer versions of Linux (Example RHEL 7):

RedHat has stated that it is not recommended to backup anything under the .gvfs directory structure for any users.
They recommend to set the following lines to the clients exclude list:
/root/.gvfs
/home/*/.gvfs

file used for exclusion: 

cat exclude_list.client-policy.FULL

/root/.gvfs
/home/*/.gvfs

detailled job:

 

Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) starting backup job (jobid=19709) for client client1, policy policy-client1, schedule INCR
Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=19709, request id:{C8704344-9S04-11JK-B5D4-12ACB0278451})
Feb 14, 2020 9:34:14 AM - requesting resource STU-DD-backup
Feb 14, 2020 9:34:14 AM - requesting resource master.NBU_CLIENT.MAXJOBS.client1
Feb 14, 2020 9:34:15 AM - granted resource master.NBU_CLIENT.MAXJOBS.client1
Feb 14, 2020 9:34:15 AM - granted resource MediaID=@aaaai;DiskVolume=lsu-backup-DD-10g;DiskPool=DP-DD-backup;Path=lsu-backup-DD-10g;StorageServer=DD-10g;MediaServer=mediaserver1
Feb 14, 2020 9:34:15 AM - granted resource STU-DD-backup
Feb 14, 2020 9:34:15 AM - estimated 956512 kbytes needed
Feb 14, 2020 9:34:15 AM - Info nbjm (pid=122652) started backup (backupid=client1_1581669255) job for client client1, policy policy-client1, schedule INCR on storage unit STU-DD-backup
Feb 14, 2020 9:34:16 AM - Info bpbrm (pid=324235) client1 is the host to backup data from
Feb 14, 2020 9:34:16 AM - Info bpbrm (pid=324235) reading file list for client
Feb 14, 2020 9:34:16 AM - started process bpbrm (pid=324235)
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) starting bpbkar on client
Feb 14, 2020 9:34:17 AM - Info bpbkar (pid=17396) Backup started
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) bptm pid: 324238
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/run/user/42/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bptm (pid=324238) start
Feb 14, 2020 9:34:17 AM - connecting
Feb 14, 2020 9:34:17 AM - connected; connect time: 0:00:00
Feb 14, 2020 9:34:18 AM - Info bptm (pid=324238) using 262144 data buffer size
Feb 14, 2020 9:34:18 AM - Info bptm (pid=324238) using 30 data buffers
Feb 14, 2020 9:34:20 AM - Info bptm (pid=324238) start backup
Feb 14, 2020 9:34:20 AM - Info bptm (pid=324238) backup child process is pid 324310
Feb 14, 2020 9:34:20 AM - begin writing
Feb 14, 2020 9:34:21 AM - Info bpbrm (pid=324235) from client client1: TRV - [/var/lib/nfs/rpc_pipefs] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:23 AM - Info bpbrm (pid=324235) from client client1: TRV - [/proc] is on file system type PROC. Skipping
Feb 14, 2020 9:34:23 AM - Info bpbrm (pid=324235) from client client1: TRV - [/sys] is on file system type sysfs. Skipping
Feb 14, 2020 9:34:24 AM - Info bpbrm (pid=324235) from client client1: TRV - [/run] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:27 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/scratch] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bptm (pid=324238) waited for full buffer 39 times, delayed 207 times
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/boot/efi] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/mqueue] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/hugepages] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/pts] is on file system type devpts. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/shm] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bptm (pid=324238) EXITING with status 0 <----------
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) validating image for client client1
Feb 14, 2020 9:34:29 AM - Info bpbkar (pid=17396) done. status: 1: the requested operation was partially successful
Feb 14, 2020 9:34:29 AM - end writing; write time: 0:00:09
The requested operation was partially successful (1)

we tried to exclude a directory/file to confirm that the exclusion works, and it does work using the same exclude file (exclude_list.client-policy.FULL).

here is a snipp from bpbkar :

09:34:27.995 [17396] <2> bpbkar PrintFile: /opt/OpenText/etxcn-11.5/runtime/sessions/52/
09:34:27.995 [17396] <2> bpbkar PrintFile: /opt/OpenText/etxcn-11.5/runtime/sessions/52/.etxproxy-eventq/
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/52 path=spool
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions path=12
09:34:27.995 [17396] <16> bpbkar read_and_sort_dir_entries: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.jcfg
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=xstart_gnome-terminal - Copier2_18_1_gnome-terminal.2020-01-21--16.43.28.log
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.xsession_id
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=xstart_gnome-terminal - Copier2_18_1_gnome-terminal.2020-01-21--16.43.28.jxs
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.pid
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.log

 

should we try to exclude /runtime/sessions/? it is necessary to backup it?

 

Thank you for your help.

8
Linux
  • Hamza_H's avatar
    Hamza_H
    4 years ago

    The problem was resolved after contacting veritas.. (I forgot the update the post with the solution) and it turned out that there was a space in the exclude list.... 

21 Replies

Sort By
  • Mike_Gavrilov's avatar
    Mike_Gavrilov
    Level 6
    5 years ago

    The errors occurs during lstat() system call. It isn't because of ls command itself but because of getting file/link attributes. If NBU can get lstat information. Excluding occurs after collecting information about filesystem objects so you have to ignore all those errors or exclude folders with these files/symlinks on higher OS hierarchy level.  

    • Hamza_H's avatar
      Hamza_H
      Moderator
      5 years ago

      Hi Mike_Gavrilov  & Jivitesh and thank y'all for your assistance.

      Jivitesh for the solution you proposed (NetBackup runs as root), I asked the EC for the output of bpps -x to confirm if the services are started with root I also asked him to run a test with allow multiple data stream enabled.

      Mike_Gavrilov, please how can I do this : "to ignore all those errors or exclude folders with these files/symlinks on higher OS hierarchy level"?

       

      Thanks again.

       

      • Jivitesh's avatar
        Jivitesh
        Level 3
        5 years ago

        Any progress in this, haven't seen any posts for quite some time.

        Is it resolved?

  • Mike_Gavrilov's avatar
    Mike_Gavrilov
    Level 6
    5 years ago

    You have only exluded:

    /root/.gvfs
    /home/*/.gvfs

    but it looks like you have more .gvfs under /opt/OpenText/etxcn-11.5/runtime/sessions*. You need to add them too. Try to find all .gvfs and correct your exclude_list with better pattern. 

    • Hamza_H's avatar
      Hamza_H
      Moderator
      5 years ago

      Hi Mike_Gavrilov 

      Thanks for your reply,

      I just asked the EC to edit the exclude_list by adding:

      I will let you know the result.

      Thanks again for your help.

    • Hamza_H's avatar
      Hamza_H
      Moderator
      5 years ago

      Hi Mike_Gavrilov ,

       

      Still same problem even with exlude_list :

      cat exclude_list

      /opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs

      /opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs

      /opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs

      /run/user/42/gvfs

      /root/.gvfs

      /home/*/.gvfs

       

      any suggestion? should I log a call to veritas support on this?

  • Mike_Gavrilov's avatar
    Mike_Gavrilov
    Level 6
    5 years ago

    You created exclude list for the FULL schedule only:

    cat exclude_list.client-policy.FULL

    /root/.gvfs
    /home/*/.gvfs

    But in the job details the see warnings from the INCR schedule:

    Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) starting backup job (jobid=19709) for client client1, policy policy-client1, schedule INCR

    You need to remove FULL  form the exclude list or add extra exclude list for INCR schedule.

    • Hamza_H's avatar
      Hamza_H
      Moderator
      5 years ago

      Hi Mike_Gavrilov ,

      Thanks for your reply,

      Yes I know, that, we are facing the same problem for the FULL jobs too, I just didn't post the detailled status for the full job.

      schedule name is FULL as in the name of the exclude file.

       

      Thanks again for your reply

  • Mike_Gavrilov's avatar
    Mike_Gavrilov
    Level 6
    5 years ago

    Try to add /opt/OpenText/etxcn-11.5/runtime/sessions/72/gvfs to the exclude list too. 

    • Hamza_H's avatar
      Hamza_H
      Moderator
      5 years ago

      Hi,

      still same problem,

       

      cat exclude_list

      /opt/OpenText/etxcn-11.5/runtime/sessions/72/gvfs

      /opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs

      /opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs

      /opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs

      /run/user/42/gvfs

      /root/.gvfs

      /home/*/.gvfs

      gvfs

       

      Do you think it's worthy to open a case to Redhat?

      • Jivitesh's avatar
        Jivitesh
        Level 3
        5 years ago

        Did you check if ROOT has permissions to the file/folder you are trying to backup?
        If root cannot "ls" the filesystem, then adding in the exclude list might not solve your issue.

        I would like repost below solution which i was able to get from this community(VOX community)

        NetBackup runs as root.

        If ALL_LOCAL_FILES is in the backup selection, the process is as follows (all as root user):

        bpmount is executed on the client to dermine 'local drives'.
        So, if df commands lists it as a local mount point, so will bpmount.
        A separate bpbkar job is created for each local drive.
        Only at this point is exclude_list looked at.
        If root cannot 'ls' the folder, you will see the status 71(Only when Allow Multiple data streams is enabled in policy attributes, if not backups will be partially succesfull)

        There is no solution or workaround other than giving root read access on the top level or else replace ALL_LOCAL_DRIVES with individual mount points.