cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Backup of a Redhat server is partial - ERR - Cannot lstat gvfs. Errno = 13: Permission denied

Hamza_H
Moderator
Moderator
   VIP   

Hi,

 

We have a backup that gets status=1 because of the known error : 

 

ERR - Cannot lstat gvfs. Errno = 13: Permission denied

 

We have been through netbackup technotes & redhat too, but still no solution even with the exclusions :

https://www.veritas.com/support/en_US/article.100024911

** Previously the following command would resolve this error in older versions of Linux:
To allow root user access to the .gvfs filesystem.  :

1. Add user_allow_other to the /etc/fuse.conf file

2. Run the following sequence of commands:

   sudo su - {USER.EN_US}  
   /usr/bin/fusermount -zu $HOME/.gvfs
   /usr/lib64/gvfs/gvfs-fuse-daemon -o allow_root $HOME/.gvfs

   Where "user" is the owner of the gvfs filesystem

** In newer versions of Linux (Example RHEL 7):

RedHat has stated that it is not recommended to backup anything under the .gvfs directory structure for any users.
They recommend to set the following lines to the clients exclude list:
/root/.gvfs
/home/*/.gvfs

file used for exclusion: 

cat exclude_list.client-policy.FULL

/root/.gvfs
/home/*/.gvfs

detailled job:

 

Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) starting backup job (jobid=19709) for client client1, policy policy-client1, schedule INCR
Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=19709, request id:{C8704344-9S04-11JK-B5D4-12ACB0278451})
Feb 14, 2020 9:34:14 AM - requesting resource STU-DD-backup
Feb 14, 2020 9:34:14 AM - requesting resource master.NBU_CLIENT.MAXJOBS.client1
Feb 14, 2020 9:34:15 AM - granted resource master.NBU_CLIENT.MAXJOBS.client1
Feb 14, 2020 9:34:15 AM - granted resource MediaID=@aaaai;DiskVolume=lsu-backup-DD-10g;DiskPool=DP-DD-backup;Path=lsu-backup-DD-10g;StorageServer=DD-10g;MediaServer=mediaserver1
Feb 14, 2020 9:34:15 AM - granted resource STU-DD-backup
Feb 14, 2020 9:34:15 AM - estimated 956512 kbytes needed
Feb 14, 2020 9:34:15 AM - Info nbjm (pid=122652) started backup (backupid=client1_1581669255) job for client client1, policy policy-client1, schedule INCR on storage unit STU-DD-backup
Feb 14, 2020 9:34:16 AM - Info bpbrm (pid=324235) client1 is the host to backup data from
Feb 14, 2020 9:34:16 AM - Info bpbrm (pid=324235) reading file list for client
Feb 14, 2020 9:34:16 AM - started process bpbrm (pid=324235)
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) starting bpbkar on client
Feb 14, 2020 9:34:17 AM - Info bpbkar (pid=17396) Backup started
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) bptm pid: 324238
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/run/user/42/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bptm (pid=324238) start
Feb 14, 2020 9:34:17 AM - connecting
Feb 14, 2020 9:34:17 AM - connected; connect time: 0:00:00
Feb 14, 2020 9:34:18 AM - Info bptm (pid=324238) using 262144 data buffer size
Feb 14, 2020 9:34:18 AM - Info bptm (pid=324238) using 30 data buffers
Feb 14, 2020 9:34:20 AM - Info bptm (pid=324238) start backup
Feb 14, 2020 9:34:20 AM - Info bptm (pid=324238) backup child process is pid 324310
Feb 14, 2020 9:34:20 AM - begin writing
Feb 14, 2020 9:34:21 AM - Info bpbrm (pid=324235) from client client1: TRV - [/var/lib/nfs/rpc_pipefs] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:23 AM - Info bpbrm (pid=324235) from client client1: TRV - [/proc] is on file system type PROC. Skipping
Feb 14, 2020 9:34:23 AM - Info bpbrm (pid=324235) from client client1: TRV - [/sys] is on file system type sysfs. Skipping
Feb 14, 2020 9:34:24 AM - Info bpbrm (pid=324235) from client client1: TRV - [/run] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:27 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/scratch] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bptm (pid=324238) waited for full buffer 39 times, delayed 207 times
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/boot/efi] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/mqueue] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/hugepages] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/pts] is on file system type devpts. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/shm] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bptm (pid=324238) EXITING with status 0 <----------
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) validating image for client client1
Feb 14, 2020 9:34:29 AM - Info bpbkar (pid=17396) done. status: 1: the requested operation was partially successful
Feb 14, 2020 9:34:29 AM - end writing; write time: 0:00:09
The requested operation was partially successful (1)

we tried to exclude a directory/file to confirm that the exclusion works, and it does work using the same exclude file (exclude_list.client-policy.FULL).

here is a snipp from bpbkar :

09:34:27.995 [17396] <2> bpbkar PrintFile: /opt/OpenText/etxcn-11.5/runtime/sessions/52/
09:34:27.995 [17396] <2> bpbkar PrintFile: /opt/OpenText/etxcn-11.5/runtime/sessions/52/.etxproxy-eventq/
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/52 path=spool
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions path=12
09:34:27.995 [17396] <16> bpbkar read_and_sort_dir_entries: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.jcfg
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=xstart_gnome-terminal - Copier2_18_1_gnome-terminal.2020-01-21--16.43.28.log
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.xsession_id
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=xstart_gnome-terminal - Copier2_18_1_gnome-terminal.2020-01-21--16.43.28.jxs
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.pid
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.log

 

should we try to exclude /runtime/sessions/? it is necessary to backup it?

 

Thank you for your help.

1 ACCEPTED SOLUTION

Accepted Solutions

Hamza_H
Moderator
Moderator
   VIP   

The problem was resolved after contacting veritas.. (I forgot the update the post with the solution) and it turned out that there was a space in the exclude list.... 

View solution in original post

21 REPLIES 21

Mike_Gavrilov
Moderator
Moderator
Partner    VIP    Accredited Certified

You created exclude list for the FULL schedule only:

cat exclude_list.client-policy.FULL

/root/.gvfs
/home/*/.gvfs

But in the job details the see warnings from the INCR schedule:

Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) starting backup job (jobid=19709) for client client1, policy policy-client1, schedule INCR

You need to remove FULL  form the exclude list or add extra exclude list for INCR schedule.

Hamza_H
Moderator
Moderator
   VIP   

Hi @Mike_Gavrilov ,

Thanks for your reply,

Yes I know, that, we are facing the same problem for the FULL jobs too, I just didn't post the detailled status for the full job.

schedule name is FULL as in the name of the exclude file.

 

Thanks again for your reply

Hamza_H
Moderator
Moderator
   VIP   

Hi guys,

Any help here, please?

Thanks,

Mike_Gavrilov
Moderator
Moderator
Partner    VIP    Accredited Certified

You have only exluded:

/root/.gvfs
/home/*/.gvfs

but it looks like you have more .gvfs under /opt/OpenText/etxcn-11.5/runtime/sessions*. You need to add them too. Try to find all .gvfs and correct your exclude_list with better pattern. 

Hamza_H
Moderator
Moderator
   VIP   

Hi @Mike_Gavrilov 

Thanks for your reply,

I just asked the EC to edit the exclude_list by adding:

I will let you know the result.

Thanks again for your help.

Hamza_H
Moderator
Moderator
   VIP   

Hi @Mike_Gavrilov ,

 

Still same problem even with exlude_list :

cat exclude_list

/opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs

/opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs

/opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs

/run/user/42/gvfs

/root/.gvfs

/home/*/.gvfs

 

any suggestion? should I log a call to veritas support on this?

quebek
Moderator
Moderator
   VIP    Certified

Hi

I would try to add only 

gvfs

to exclude list - check this link for guide... https://www.veritas.com/content/support/en_US/article.100021742

Mike_Gavrilov
Moderator
Moderator
Partner    VIP    Accredited Certified

Can you share new bpbkar log?

Hamza_H
Moderator
Moderator
   VIP   

Hi @Mike_Gavrilov 

The problem is not resolved yet :(

I tried to upload the logs but its too big, i will try to send you a link in PV so that you can DL them 

again appreciate your help !

 

thanks

Hamza_H
Moderator
Moderator
   VIP   

Hi @Mike_Gavrilov , please find below the rest of the logs

Hamza_H
Moderator
Moderator
   VIP   

Hi @quebek ,

thanks for the reply,

So do you suggest to add only 

gvfs to the exclude_list?

i.e:

# cat exclude_list

gvfs

 

 

I doubt exclude list will not work in this case, If the root user cannot "ls" the filesystem in question, adding it exclude list is of no use.

You can see the same issue when vormetric encryption is enabled to any filesystem.

Try to change backup selection, use individual filesystems instead of "ALL_LOCAL_DRIVES" if you want to eliminated partial backups.

Mike_Gavrilov
Moderator
Moderator
Partner    VIP    Accredited Certified

Try to add /opt/OpenText/etxcn-11.5/runtime/sessions/72/gvfs to the exclude list too. 

Hamza_H
Moderator
Moderator
   VIP   

Hi,

still same problem,

 

cat exclude_list

/opt/OpenText/etxcn-11.5/runtime/sessions/72/gvfs

/opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs

/opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs

/opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs

/run/user/42/gvfs

/root/.gvfs

/home/*/.gvfs

gvfs

 

Do you think it's worthy to open a case to Redhat?

Dollypee
Moderator
Moderator
   VIP    Certified

This error simply mean Netbackup does not have permmission to the file/directory during backup. What's the current permmision set on this file? upload bpbkar log .details should be there.

Did you check if ROOT has permissions to the file/folder you are trying to backup?
If root cannot "ls" the filesystem, then adding in the exclude list might not solve your issue.

I would like repost below solution which i was able to get from this community(VOX community)

NetBackup runs as root.

If ALL_LOCAL_FILES is in the backup selection, the process is as follows (all as root user):

bpmount is executed on the client to dermine 'local drives'.
So, if df commands lists it as a local mount point, so will bpmount.
A separate bpbkar job is created for each local drive.
Only at this point is exclude_list looked at.
If root cannot 'ls' the folder, you will see the status 71(Only when Allow Multiple data streams is enabled in policy attributes, if not backups will be partially succesfull)

There is no solution or workaround other than giving root read access on the top level or else replace ALL_LOCAL_DRIVES with individual mount points.

Mike_Gavrilov
Moderator
Moderator
Partner    VIP    Accredited Certified

The errors occurs during lstat() system call. It isn't because of ls command itself but because of getting file/link attributes. If NBU can get lstat information. Excluding occurs after collecting information about filesystem objects so you have to ignore all those errors or exclude folders with these files/symlinks on higher OS hierarchy level.  

Hamza_H
Moderator
Moderator
   VIP   

Hi @Mike_Gavrilov  & @Jivitesh and thank y'all for your assistance.

@Jivitesh for the solution you proposed (NetBackup runs as root), I asked the EC for the output of bpps -x to confirm if the services are started with root I also asked him to run a test with allow multiple data stream enabled.

@Mike_Gavrilov, please how can I do this : "to ignore all those errors or exclude folders with these files/symlinks on higher OS hierarchy level"?

 

Thanks again.

 

Any progress in this, haven't seen any posts for quite some time.

Is it resolved?