Forum Discussion

2013's avatar
2013
Level 5
12 years ago

Bpbkar Process hung on client

Master Server -7.1 (unix)

Client 6.5.6 (unix)

Bpbkar processes are hung on client and not killing automatically even after server reboot.Only /var backup is failing with 13 error and for that i have enabled the bpbkar logs.

Now need to know what Files are actually causing issue,so that i can check with SA.

Also need to know that the below Mount point is causing an issue:

 

16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv2 on /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logarchive are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>

16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logarchive is /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv3 on /appl/logs
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logs are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logs is /appl/logs
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv4 on /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/lsinfra are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/lsinfra is /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv5 on /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/monitor are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/monitor is /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv6 on /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/problem are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/problem is /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv7 on /usr/local/opt/tools
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /usr/local/opt/tools are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
...skipping...
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd9var on /var
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var is /var
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd3 on /tmp
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /tmp is /tmp
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd1 on /home
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /home is /home
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd10opt on /opt
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /opt is /opt
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hsadm_lv on /home/sadm
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /home/sadm is /home/sadm
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/sysintro_lv on /siopt
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /siopt is /siopt
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (procfs) /proc on /proc
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /proc is /proc
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/local_lv on /usr/local
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/local is /usr/local
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/localcw_lv on /usr/localcw
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw is /usr/localcw
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/patrol_lv on /usr/localcw/opt/patrol
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw/opt/patrol is /usr/localcw/opt/patrol
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/cischk_lv on /var/adm/cischk
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var/adm/cischk is /var/adm/cischk
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/husers_lv on /home/users
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /home/users is /home/users
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/security_lv on /usr/localcw/opt/security
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw/opt/security is /usr/localcw/opt/security
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/cfeng_lv on /usr/localcw/opt/cfengine
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw/opt/cfengine is /usr/localcw/opt/cfengine
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/vsf_lv on /tmp/vsf
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /tmp/vsf is /tmp/vsf
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/livedump_lv on /var/adm/ras/livedump
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var/adm/ras/livedump is /var/adm/ras/livedump
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd11admin on /admin
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /admin is /admin
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/varsa_lv on /var/adm/sa
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var/adm/sa is /var/adm/sa
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/perfstats_lv on /perfstats
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /perfstats is /perfstats
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv2 on /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logarchive are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logarchive is /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv3 on /appl/logs
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logs are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logs is /appl/logs
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv4 on /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/lsinfra are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/lsinfra is /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv5 on /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/monitor are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/monitor is /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv6 on /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/problem are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/problem is /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv7 on /usr/local/opt/tools
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /usr/local/opt/tools are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
log.121512 (0%)
 

Is there ant specific way to find out under /var which files are causing issue bcoz when we check with SA ,as per them VAR is ok.......

Any help will be appreciated.....

  • Hi All,

    As per my findings..i have seen bpbkar logs and i found the NBU is unable to read the below file:

    16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - path = corrals
    16:15:02.019 [9961618] <4> bpbkar PrintFile: /var/adm/corrals/
    16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - cwd = /var/adm
    16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - path = cron
    16:15:02.020 [9961618] <4> bpbkar PrintFile: /var/adm/cron/
    16:15:02.020 [9961618] <2> bpbkar SelectFile: INF - cwd = /var/adm/cron
    16:15:02.021 [9961618] <2> bpbkar SelectFile: INF - path = FIFO
    16:15:02.021 [9961618] <4> bpbkar PrintFile: /var/adm/cron/FIFO
    ---------------------------------------------------------------------------------------------------------------------------------------------------

    Below are the files which are creating issue:

    /var/adm
    /var/eksh.install.out
    /var/ncs
    /var/LicenseUseManagement
    /var/VRTSat
    /var/VRTSweb
    /var/aacct

     I am thinking to put all these files in exclde list....If i am wrong then correct me....

16 Replies

  • Can i exclude the files in ls -R /var which is starting from :The file access permissions do not allow the specified action.

  • hi,

    exclude list is defined based on your organization /appteam/SA requirement..

     

    below is the list that i generally use for /var , you can check them also for test, but before making exclude as perminent you need to check with your Appteam or SA team.to avoide future issues.. 
     
    /var/crash
    /var/adm/crash
    /var/run
    /var/tmp

    ###now for testing 

    1) exclude entire /var from backup and test the backup to make sure /var is causing the failurs.

    2) once you get the confirmation from  saying that the failure is because of /var, then start updating the above exclude list and see how it works..

    and also.. 

    I am seeing below erroes in the output file, did you run the command as root , even with the root its showing access isssue.?

    if you did not run with root, run the command as root and see how much time its taking to compleate the command ls -R.. it should not be more then 1800.

     

    ls: /var/websm/lwi/lib: The file access permissions do not allow the specified action.
    
    /var/websm/lwi/logs:
    ls: /var/websm/lwi/logs: The file access permissions do not allow the specified action.
    
    /var/websm/lwi/native:
    ls: /var/websm/lwi/native: The file access permissions do not allow the specified action.
    
    /var/websm/lwi/rollback_config:
    ls: /var/websm/lwi/rollback_config: The file access permissions do not allow the specified action.
    
    /var/websm/lwi/runtime:
    ls: /var/websm/lwi/runtime: The file access permissions do not allow the specified action.
    
    /var/websm/lwi/security:
    ls: /var/websm/lwi/security: The file access permissions do not allow the specified action.
    
    /var/websm/lwi/update.images:
    ls: /var/websm/lwi/update.images: The file access permissions do not allow the specified action.

    hope this helps.. 

  • Hi Nagalla,

    Thanks for the update.

    1.Excluded /var and Backup completed successfully.

    2.Regarding the exclude file i have checked on others servers its the same file.

    3.I have not ran the command from root.Sending the output of ls -R/var which run from root and yes it completed very fast...

    4.Also i want someone please check bpbkar logs and can tell which files we can exclude.

  • Hi All,

    As per my findings..i have seen bpbkar logs and i found the NBU is unable to read the below file:

    16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - path = corrals
    16:15:02.019 [9961618] <4> bpbkar PrintFile: /var/adm/corrals/
    16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - cwd = /var/adm
    16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - path = cron
    16:15:02.020 [9961618] <4> bpbkar PrintFile: /var/adm/cron/
    16:15:02.020 [9961618] <2> bpbkar SelectFile: INF - cwd = /var/adm/cron
    16:15:02.021 [9961618] <2> bpbkar SelectFile: INF - path = FIFO
    16:15:02.021 [9961618] <4> bpbkar PrintFile: /var/adm/cron/FIFO
    ---------------------------------------------------------------------------------------------------------------------------------------------------

    Below are the files which are creating issue:

    /var/adm
    /var/eksh.install.out
    /var/ncs
    /var/LicenseUseManagement
    /var/VRTSat
    /var/VRTSweb
    /var/aacct

     I am thinking to put all these files in exclde list....If i am wrong then correct me....

  • Issue is now fixed.Cron file was the culprit...