β02-18-2020 08:18 AM
Hi,
We have a backup that gets status=1 because of the known error :
ERR - Cannot lstat gvfs. Errno = 13: Permission denied
We have been through netbackup technotes & redhat too, but still no solution even with the exclusions :
https://www.veritas.com/support/en_US/article.100024911
** Previously the following command would resolve this error in older versions of Linux:
To allow root user access to the .gvfs filesystem. :
1. Add user_allow_other to the /etc/fuse.conf file
2. Run the following sequence of commands:
sudo su - {USER.EN_US}
/usr/bin/fusermount -zu $HOME/.gvfs
/usr/lib64/gvfs/gvfs-fuse-daemon -o allow_root $HOME/.gvfs
Where "user" is the owner of the gvfs filesystem
** In newer versions of Linux (Example RHEL 7):
RedHat has stated that it is not recommended to backup anything under the .gvfs directory structure for any users.
They recommend to set the following lines to the clients exclude list:
/root/.gvfs
/home/*/.gvfs
file used for exclusion:
cat exclude_list.client-policy.FULL
/root/.gvfs
/home/*/.gvfs
detailled job:
Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) starting backup job (jobid=19709) for client client1, policy policy-client1, schedule INCR
Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=19709, request id:{C8704344-9S04-11JK-B5D4-12ACB0278451})
Feb 14, 2020 9:34:14 AM - requesting resource STU-DD-backup
Feb 14, 2020 9:34:14 AM - requesting resource master.NBU_CLIENT.MAXJOBS.client1
Feb 14, 2020 9:34:15 AM - granted resource master.NBU_CLIENT.MAXJOBS.client1
Feb 14, 2020 9:34:15 AM - granted resource MediaID=@aaaai;DiskVolume=lsu-backup-DD-10g;DiskPool=DP-DD-backup;Path=lsu-backup-DD-10g;StorageServer=DD-10g;MediaServer=mediaserver1
Feb 14, 2020 9:34:15 AM - granted resource STU-DD-backup
Feb 14, 2020 9:34:15 AM - estimated 956512 kbytes needed
Feb 14, 2020 9:34:15 AM - Info nbjm (pid=122652) started backup (backupid=client1_1581669255) job for client client1, policy policy-client1, schedule INCR on storage unit STU-DD-backup
Feb 14, 2020 9:34:16 AM - Info bpbrm (pid=324235) client1 is the host to backup data from
Feb 14, 2020 9:34:16 AM - Info bpbrm (pid=324235) reading file list for client
Feb 14, 2020 9:34:16 AM - started process bpbrm (pid=324235)
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) starting bpbkar on client
Feb 14, 2020 9:34:17 AM - Info bpbkar (pid=17396) Backup started
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) bptm pid: 324238
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bpbrm (pid=324235) from client client1: TRV - Cannot process path [/run/user/42/gvfs]: [Permission denied]. Skipping
Feb 14, 2020 9:34:17 AM - Info bptm (pid=324238) start
Feb 14, 2020 9:34:17 AM - connecting
Feb 14, 2020 9:34:17 AM - connected; connect time: 0:00:00
Feb 14, 2020 9:34:18 AM - Info bptm (pid=324238) using 262144 data buffer size
Feb 14, 2020 9:34:18 AM - Info bptm (pid=324238) using 30 data buffers
Feb 14, 2020 9:34:20 AM - Info bptm (pid=324238) start backup
Feb 14, 2020 9:34:20 AM - Info bptm (pid=324238) backup child process is pid 324310
Feb 14, 2020 9:34:20 AM - begin writing
Feb 14, 2020 9:34:21 AM - Info bpbrm (pid=324235) from client client1: TRV - [/var/lib/nfs/rpc_pipefs] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:23 AM - Info bpbrm (pid=324235) from client client1: TRV - [/proc] is on file system type PROC. Skipping
Feb 14, 2020 9:34:23 AM - Info bpbrm (pid=324235) from client client1: TRV - [/sys] is on file system type sysfs. Skipping
Feb 14, 2020 9:34:24 AM - Info bpbrm (pid=324235) from client client1: TRV - [/run] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:27 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Error bpbrm (pid=324235) from client client1: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/scratch] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bptm (pid=324238) waited for full buffer 39 times, delayed 207 times
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/boot/efi] is in a different file system from [/]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/mqueue] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/hugepages] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/pts] is on file system type devpts. Skipping
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) from client client1: TRV - [/dev/shm] is in a different file system from [/dev]. Skipping
Feb 14, 2020 9:34:28 AM - Info bptm (pid=324238) EXITING with status 0 <----------
Feb 14, 2020 9:34:28 AM - Info bpbrm (pid=324235) validating image for client client1
Feb 14, 2020 9:34:29 AM - Info bpbkar (pid=17396) done. status: 1: the requested operation was partially successful
Feb 14, 2020 9:34:29 AM - end writing; write time: 0:00:09
The requested operation was partially successful (1)
we tried to exclude a directory/file to confirm that the exclusion works, and it does work using the same exclude file (exclude_list.client-policy.FULL).
here is a snipp from bpbkar :
09:34:27.995 [17396] <2> bpbkar PrintFile: /opt/OpenText/etxcn-11.5/runtime/sessions/52/
09:34:27.995 [17396] <2> bpbkar PrintFile: /opt/OpenText/etxcn-11.5/runtime/sessions/52/.etxproxy-eventq/
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/52 path=spool
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions path=12
09:34:27.995 [17396] <16> bpbkar read_and_sort_dir_entries: ERR - Cannot lstat gvfs. Errno = 13: Permission denied
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.jcfg
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=xstart_gnome-terminal - Copier2_18_1_gnome-terminal.2020-01-21--16.43.28.log
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.xsession_id
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=xstart_gnome-terminal - Copier2_18_1_gnome-terminal.2020-01-21--16.43.28.jxs
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.pid
09:34:27.995 [17396] <2> bpbkar SelectFile: cwd=/opt/OpenText/etxcn-11.5/runtime/sessions/12 path=etxproxy.log
should we try to exclude /runtime/sessions/? it is necessary to backup it?
Thank you for your help.
Solved! Go to Solution.
β07-15-2021 10:42 AM
The problem was resolved after contacting veritas.. (I forgot the update the post with the solution) and it turned out that there was a space in the exclude list....
β02-18-2020 09:07 AM
You created exclude list for the FULL schedule only:
cat exclude_list.client-policy.FULL
/root/.gvfs
/home/*/.gvfs
But in the job details the see warnings from the INCR schedule:
Feb 14, 2020 9:34:14 AM - Info nbjm (pid=122652) starting backup job (jobid=19709) for client client1, policy policy-client1, schedule INCR
You need to remove FULL form the exclude list or add extra exclude list for INCR schedule.
β02-19-2020 06:13 AM
Hi @Mike_Gavrilov ,
Thanks for your reply,
Yes I know, that, we are facing the same problem for the FULL jobs too, I just didn't post the detailled status for the full job.
schedule name is FULL as in the name of the exclude file.
Thanks again for your reply
β02-20-2020 08:37 AM
Hi guys,
Any help here, please?
Thanks,
β02-20-2020 09:12 AM
You have only exluded:
/root/.gvfs
/home/*/.gvfs
but it looks like you have more .gvfs under /opt/OpenText/etxcn-11.5/runtime/sessions*. You need to add them too. Try to find all .gvfs and correct your exclude_list with better pattern.
β02-24-2020 07:14 AM
Thanks for your reply,
I just asked the EC to edit the exclude_list by adding:
I will let you know the result.
Thanks again for your help.
β02-27-2020 05:37 AM
Hi @Mike_Gavrilov ,
Still same problem even with exlude_list :
cat exclude_list
/opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs
/opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs
/opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs
/run/user/42/gvfs
/root/.gvfs
/home/*/.gvfs
any suggestion? should I log a call to veritas support on this?
β02-27-2020 11:44 AM
Hi
I would try to add only
gvfs
to exclude list - check this link for guide... https://www.veritas.com/content/support/en_US/article.100021742
β02-27-2020 01:46 PM
Can you share new bpbkar log?
β03-02-2020 10:09 AM - edited β03-06-2020 06:19 AM
The problem is not resolved yet :(
I tried to upload the logs but its too big, i will try to send you a link in PV so that you can DL them
again appreciate your help !
thanks
β03-02-2020 10:57 AM - edited β03-06-2020 06:19 AM
Hi @Mike_Gavrilov , please find below the rest of the logs
β03-02-2020 11:05 AM
Hi @quebek ,
thanks for the reply,
So do you suggest to add only
gvfs to the exclude_list?
i.e:
# cat exclude_list
gvfs
β03-02-2020 12:25 PM
I doubt exclude list will not work in this case, If the root user cannot "ls" the filesystem in question, adding it exclude list is of no use.
You can see the same issue when vormetric encryption is enabled to any filesystem.
Try to change backup selection, use individual filesystems instead of "ALL_LOCAL_DRIVES" if you want to eliminated partial backups.
β03-02-2020 02:57 PM
Try to add /opt/OpenText/etxcn-11.5/runtime/sessions/72/gvfs to the exclude list too.
β03-06-2020 06:18 AM
Hi,
still same problem,
cat exclude_list
/opt/OpenText/etxcn-11.5/runtime/sessions/72/gvfs
/opt/OpenText/etxcn-11.5/runtime/sessions/52/gvfs
/opt/OpenText/etxcn-11.5/runtime/sessions/12/gvfs
/opt/OpenText/etxcn-11.5/runtime/sessions/2/gvfs
/run/user/42/gvfs
/root/.gvfs
/home/*/.gvfs
gvfs
Do you think it's worthy to open a case to Redhat?
β03-06-2020 10:55 AM
This error simply mean Netbackup does not have permmission to the file/directory during backup. What's the current permmision set on this file? upload bpbkar log .details should be there.
β03-08-2020 05:30 AM
Did you check if ROOT has permissions to the file/folder you are trying to backup?
If root cannot "ls" the filesystem, then adding in the exclude list might not solve your issue.
I would like repost below solution which i was able to get from this community(VOX community)
NetBackup runs as root.
If ALL_LOCAL_FILES is in the backup selection, the process is as follows (all as root user):
bpmount is executed on the client to dermine 'local drives'.
So, if df commands lists it as a local mount point, so will bpmount.
A separate bpbkar job is created for each local drive.
Only at this point is exclude_list looked at.
If root cannot 'ls' the folder, you will see the status 71(Only when Allow Multiple data streams is enabled in policy attributes, if not backups will be partially succesfull)
There is no solution or workaround other than giving root read access on the top level or else replace ALL_LOCAL_DRIVES with individual mount points.
β03-10-2020 12:23 AM
The errors occurs during lstat() system call. It isn't because of ls command itself but because of getting file/link attributes. If NBU can get lstat information. Excluding occurs after collecting information about filesystem objects so you have to ignore all those errors or exclude folders with these files/symlinks on higher OS hierarchy level.
β03-10-2020 04:34 AM
Hi @Mike_Gavrilov & @Jivitesh and thank y'all for your assistance.
@Jivitesh for the solution you proposed (NetBackup runs as root), I asked the EC for the output of bpps -x to confirm if the services are started with root I also asked him to run a test with allow multiple data stream enabled.
@Mike_Gavrilov, please how can I do this : "to ignore all those errors or exclude folders with these files/symlinks on higher OS hierarchy level"?
Thanks again.
β05-15-2020 10:44 PM
Any progress in this, haven't seen any posts for quite some time.
Is it resolved?