Forum Discussion

arjun7's avatar
arjun7
Level 4
11 years ago

logs

1. If any jobs are failed, where we can see that job logs clearly for troubleshooting ?

  • Adding to Mariannes posts, and the others above.

    There sometimes no clear place to look, you have to look at the type of problem, and then the area where it failed (Activity Monitor details are good for ths, or the job trylogs).

    Then, look in the logs for the processes that were involved at that time.

    Easy example - file system backup.

    Logs of logs involved, but if there was an issue in the backup part, in that the tape did get mounted and the backup started,  you'd be looking to start with

    bptm bpbrm bpbkar

    If the issue was a 'resource' issue, then we are looking earlier in the whole backup process ...

    bptm / nbjm / nbrb / mds / nbrmm

    There is no easy way, there are no documents that specifity what log for what problem (though the troubleshooting guide does touch on this / process flows ). 

  • Depends why it failed. Normally look at bptm and bpbrm (on media server), bpfis is snapshotting is involved and bpbkar and bpcd on the client itself.

    Check the Acitivty monitor to see when it failed and look at the time in these logs.

    It really does depend on WHY it failed - perhaps the master cannot get resources, in time in which case none of those logs will help and you may need to look at nbemm, nbrb and nbjm.

  • First place that you would need to look in is the detail status of the failed job.

    it will have the error and the proces that is using with the process name and PID,. that is the place where you need to look in

    like below,

    you need to look for the error and find the process that is through that error

     
    02/24/2014 03:33:43 - Info bpbkar (pid=34603260) bpbkar waited 0 times for empty buffer, delayed 0 times
    02/24/2014 03:33:46 - Info bptm (pid=39059710) start backup
    02/24/2014 03:33:46 - Info bptm (pid=39059710) waited for full buffer 0 times, delayed 0 times
    02/24/2014 03:33:46 - Info bptm (pid=39059710) EXITING with status 0 <----------
    02/24/2014 03:33:46 - begin writing
    02/24/2014 03:33:47 - Info bpbrm (pid=45351126) validating image for client XXXXXXXXX
    02/24/2014 03:33:48 - Info bpbkar (pid=34603260) done. status: 0: the requested operation was successfully completed
     

    to understand this you need to have the good understanding on the backup process flow.

    please have a look into the backup process flow from Marianne excellent post

    https://www-secure.symantec.com/connect/downloads/netbackup-7x-process-flow

     

     

  •  

    From a previous post I did - a quick guide to NBU and logs.

    Note, that NBU 7.6 has a new 'Logging Assistant' - this allows the type of issue to be inputted and it will suggest and auto setup the logs for that type of issue, on the individual servers concerned.  You can also manually add individual logs for 'fine tuning', or if you don;t want to use its suggestion.  It can also auto collect and upload to the FTP / evidence servers.  In essence, it is nbcplogs command with a graphical interface and some intellegence to know which logs are required.
     
    The purpose of this post is to provide a single location for info to be able to set up logs.
    The details below are just about the shortest version I can make, that still contains detail/ explanation.
     
    Any NetBackup Engineer should be able to set up logs with the minimum of information, all that 'should be required' for example.
     
    "Please set bptm/ bpbrm on windows media xxx at verbose 5/ general 2 and bpdbm on unix master at versbose 5"
    "Please set VX logs at 6 and 6 for OID 111/ 143/ 116/ 117"
     
    The engineer should now how to set up these logs, and check that they are logging.
     
    The details below will allow this to be done.
     
     
     
    Setting up logs in NetBackup
     
     
    For a given issue, it may be necessary to gather multiple logs.  This MUST cover the time the issue happens.
    If an additional log is required, that has to be created, then ALL the logs must be supplied again.
     
    There are two types of logs in NetBackup.  Legacy logs and VX logs.
     
     
    1) Creating Legacy Logs
    2) Setting Verbose Level for Legacy Logs
    3) Collecting Legacy Logs
    4) Creating VX Logs and setting the log level
    5) Collecting VX logs
    6) Volmgr Logs
     
     
    1) Creating Legacy Logs
    ************************
     
    These are created in either 
     
    Unix
    /usr/openv/netbackup/logs/
    /usr/openv/volmgr/debug/
     
    Windows
    <install path>\veritas\netbackup\logs
    <install path>\veritas\volmgr\debug
     
     
    For example to create bptm log, simply create a directory called the <process> name.
     
    mkdir /usr/openv/netbackup/logs/bptm
     
    A newly created log will not log anything /detect a change of verbose level until the process is restarted.  For logs such as bptm, this will be when the next backup runs.  Other logs such as bprm and bpdm may require a restart of the NBU services.  I say 'may', if the process starts a child process, then this would write to a newly created log or pick up a verbose level change.
     
     
    2) Setting Verbose Level for Legacy Logs
    ****************************************
     
    There are two ways this can be done :
     
    (Unix) 
     
    To increase the verbose level of all logs (except vault)
    Add the entry VERBOSE = <level> into /usr/openv/netbackup/bp.conf.  <level> is a value between 0 and 5, with 5 being the highest.
     
     
    (Windows)
     
    On the server you are gathering the logs from, run the BAR GUI
    From the File menu, select Client Properties and in the pop-up window, goto the Troubleshooting tab.
    Set General to 2 and Verbose to 5
     
     
    3) Collecting Legacy Logs
    *************************
     
    The log file is simply found in the <process> name directory.  There is one log per day.
     
    The name of the log file will be log.<date>
     
    If you are sending multiple log files in, they will all have the same name.  Please therefore rename the log files to :
     
    <process>.log.<date>
     
     
    4) Creating VX Logs and setting the log level
    **********************************************
     
    These are more complex, and have to be set with specific commands.  NOTE: Some of these logs, for example, 'mds' do NOT create a log file.  Instead the lines are entered in to other log files.  In the case of mds (143), it logs into EMM (111).
     
     
    The vxlogs cover various processes, for example, nbemm, nbrb, nbjm, nbrb, mds
     
     
    To set these up on either Unix or Windows, use this command :
     
    vxlogcfg -a -p 51216 -o <oid> -s DebugLevel=<1-6> -s DiagnosticLevel=<1-6>
     
    For example, to set the EMM and MDS logs to levels 6 and 6 use
     
    vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6
    vxlogcfg -a -p 51216 -o 143 -s DebugLevel=6 -s DiagnosticLevel=6
     
     
    To confirm the log level has been set, simply look in the nblog.conf file, which is located in the netbackup diorectory.
    (NOTE: DO NOT EDIT THIS FILE MANUALLY)
     
     
    5) Collecting VX Logs
    *********************
     
     
     
    To collect the vx logs, use the nbcplogs command.  This copies the raw logs, which is the preference of Technical Support.
     
    NOTE:  The destination directory MUST be empty.
     
     
    nbcplogs --no-nbsu -d 2hrs --logs nbemm,nbjm /tmp/logs   (Ex.  Coleect the past 2 hrs of logs, RELATIVE, to when the command is run )
      
    nbcplogs --no-nbsu -s 07/11/2012-10:17:58 -e 07/11/2012-12:17:58 --logs nbjm,nbpem /tmp/logs (Collect the logs between two times -s <start> -e <end>  )
     
    In these examples, the nbpem and nbjm logs would be copied to /tmp/logs
     
     
    For details of using vxlogview please see TN:  http://www.symantec.com/docs/TECH75805
     
     
    6) Volmgr Debug Logs
    ********************
     
    These are very similar to legacy logs, the difference being the location and the verbose setting.
    There is no value for verbose level, simply it is turned up by adding the work VERBOSE to a line in the vm.conf file.
     
    To turn on Media Manager logging:
     
    UNIX:
     
    Add VERBOSE to the /usr/openv/volmgr/vm.conf file.  If this file does not exist, just create it.
    If necessary, create the directory /usr/openv/volmgr/debug
     
    mkdir /usr/openv/volmgr/debug/acssi
    mkdir /usr/openv/volmgr/debug/acsd
    mkdir /usr/openv/volmgr/debug/robots
    mkdir /usr/openv/volmgr/debug/daemon
    mkdir /usr/openv/volmgr/debug/ltid
    mkdir /usr/openv/volmgr/debug/oprd
    mkdir /usr/openv/volmgr/debug/reqlib
    mkdir /usr/openv/volmgr/debug/tpcommand
     
    The following empty files increase the details logged further
     
    touch /usr/openv/volmgr/DRIVE_DEBUG
    touch /usr/openv/volmgr/ROBOT_DEBUG 
    touch /usr/openv/volmgr/AVRD_DEBUG 
    touch /usr/openv/volmgr/SSO_DEBUG 
     
     
    Restart ltid :
     
    /usr/openv/volmgr/bin/stopltid
    /usr/openv/volmgr/bin/ltid -v
     
     
    Windows:
     
    Add VERBOSE to the <install path>\veritas\volmgr\vm.conf file.  If this file does not exist, just create it.
    If necessary, create the directory /usr/openv/volmgr/debug
     
    <install path>\veritas\volmgr\debug/acssi
    <install path>\veritas\volmgr\debug/acsd
    <install path>\veritas\volmgr\debug/robots
    <install path>\veritas\volmgr\debug/daemon
    <install path>\veritas\volmgr\debug/ltid
    <install path>\veritas\volmgr\debug/oprd
    <install path>\veritas\volmgr\reqlib
    <install path>\veritas\volmgr\debug\tpcommand
     
    Create the following empty files to increase the details logged further
    Ensure that windows does not craete a 'suffix', for example .txt
     
    <install path>\veritas\volmgr\DRIVE_DEBUG
    <install path>\veritas\volmgr\ROBOT_DEBUG 
    <install path>\veritas\volmgr\AVRD_DEBUG 
    <install path>\veritas\volmgr\SSO_DEBUG 
     
     
    Restart ltid :
     
    <install path>\veritas\volmgr\bin/stopltid
    <install path>\veritas\volmgr\bin/ltid -v
     
    NOTE:
     
    The 'OIDs' for the Unified /VX logs can be determined by looking in the nblog.conf file in the /usr/openv/netbackup directory.  Be careful not to edit the file.
     
    The default logs levels for the Unified /VX log levels can be found in the nblog.conf.  These are set as follows:
     
    NetBackup Server : Default.DiagnosticLevel=6 / Default.DebugLevel=1
    NetBackup Appliance : Default.DiagnosticLevel=6 / Default.DebugLevel=5
  • Adding to Mariannes posts, and the others above.

    There sometimes no clear place to look, you have to look at the type of problem, and then the area where it failed (Activity Monitor details are good for ths, or the job trylogs).

    Then, look in the logs for the processes that were involved at that time.

    Easy example - file system backup.

    Logs of logs involved, but if there was an issue in the backup part, in that the tape did get mounted and the backup started,  you'd be looking to start with

    bptm bpbrm bpbkar

    If the issue was a 'resource' issue, then we are looking earlier in the whole backup process ...

    bptm / nbjm / nbrb / mds / nbrmm

    There is no easy way, there are no documents that specifity what log for what problem (though the troubleshooting guide does touch on this / process flows ).