How to activate logs on a client

[przemol]

Hello,

we have Netbackup standard client installed on two Solaris servers. We check logs from those clients using NBU console.
Is it possible to see the same logs somewhere in a client without running the console ?
There is no 'mklogdir' in /usr/openv/netbackup/logs directory.

[Marianne]

Create logs using mkdir, e.g:
mkdir bpcd bpbkar tar.

The logs cannot be viewed from the NBU console. Use Unix commands on the client such as more, view, etc.

[przemol]

Can I copy 'mklogdir' from other clients (advanced) and run it ?

[Nicolai]

Does not initiate logging. The Netbackup client start writing debug information if the directories exist.

mklogdir is a Windows bat file, it will not run under other operating systems.

[Will_Restore]

there is also a unix shell script by that name

[Nicolai]

So used to create them by hand, that I forgot a UNIX version also exists.

[Andy_Welburn · Accepted solution]

for debugging purposes so as not to fill your filesystem with large log files.

However, with the advent of unified logging from NB6.0 there are only a few processes that still utilise legacy logging:

DOCUMENTATION: A comprehensive list of NetBackup (tm) 6.0 directories and commands relating to Unified Logging.
http://seer.entsupport.symantec.com/docs/278572.htm
"...
A listing of daemons that use Legacy logging include, but are not limited to:

• NetBackup Master Server daemons (bprd, bpdbm, bpjobd, bpdbjobs, admin)
• NetBackup Media Server daemons (bpbrm, bptm, bpdm)
• NetBackup Robotic daemons (vmd, avrd, ltid and robotic daemons)
• NetBackup Client daemons (bpcd, bpbkar, tar)

..."

Also of interest may be:
DOCUMENTATION: Best Practice recommendations for enabling and gathering NetBackup (tm) logging
http://seer.entsupport.symantec.com/docs/282865.htm

& this is how it used to be before unified logging:
DOCUMENTATION: A comprehensive list of VERITAS NetBackup (tm) 3.4, 4.5, 5.x directories, touch files, and commands relating to logging
http://seer.entsupport.symantec.com/docs/243778.htm

As far as mklogdir is concerned this only seems to appear (for us anyway) on our Windows clients (bat file) & Solaris Master/Media (shell script) - it does not appear on any of our Solaris or Linux clients - not sure if there is a valid reason for this "discrimination" or not!

[mph999]

Here's a new one, not published yet, as still 'in progress' ...

NetBackup Unified and VX Logs and Troubleshooting

---

Details:

This Technote is designed to assist in gathering information to allow troubleshooting of an issue with NetBackup.

Initial Information recommended for all issues:

1.   The exact error message (if an error is given)
2.   During what type of operation does the problem happen ?  For example, Backup, Duplication, Vault, Restore etc.
3.   At what point in the job does the issue occur, for example, right at the beginning.
4.   What system(s) are involved.  For example, Master Server, Media Server - Please provide system names.
5.   Please submit the support report or nbsu info from the master and any relevant Media Servers in the environment.
6.   Is this a reoccurring issue ?
7.   What was the system doing at this time ?
8.   Please send in the contents of the details tab in the Activity Monitor for the job.
9.   What logs or screenshots are available?
10. Has this worked previously?  If so, when was the last time it DID work?
11. Have there been any environmental; system; or configuration changes?
12. When did the problem begin?
13. How often does the problem occur?
14. Is a similar configuration working properly?


Support report:

/usr/openv/netbackup/bin/goodies/support/support >/tmp/support.txt

nbsu report:      
/usr/openv/netbackup/bin/support/nbsu,  The output is located in a subdirectory 'output'
For example,  /usr/openv/netbackup/bin/support/output/nbsu


Logs:

Generally, for any reported problem the logs for the daemons/processes involved in that function will be requested. Note, that verbosity level at time of opening a case may not be sufficient and therefore, the issue will need to be captured with the level of logging increased  and perhaps further set up/tuning of logs. In addition, these, initial investigations may lead down a sub-route that may require a repeat collection with additional logs enabled to continue the diagnostic process.  Troubleshooting NetBackup can at times be complex, and it is not always possible to know whether a small set of logs would be required or a larger set.  This then raises the question of gathering perhaps a small set of logs on the understanding that further more detailed logs may be required  (thus saving the time and effort for the customer of gathering a large set that may not be needed).


Time scales:

Dependant on the issue, the logs covering just the time of the issue may be sufficient.  For example, in the case of a Tape Drive going down it is likely that the logs just a few minutes either side of issue will suffice.  In a more complex case, for example a resource or performance problem, then the logs several hours before the issue became apparent may be required.

System Design:

Consideration may be given to creation of separate filesystems for the main log areas, /usr/openv/netbackup/logs and /usr/openv/logs.

Types of logs:

NetBackup (from version 6.0 onwards) uses two types of logging termed :

1)  Legacy Logs :  Used in all version of NetBackup, Legacy Logs are straight text files.
2)  Unified or VX Logs:  Used in NetBackup 6.0 and above.  These logs need processing with the vxlogview command to make them readable.


Legacy logs:

These are located in /usr/openv/netbackup/logs, and /usr/openv/volmgr/debug, for example (bptm process)

/usr/openv/netbackup/logs/<process>/log.<date>
/usr/openv/volmgr/debug/<process>

To enable a legacy log, the directory must be created in /usr/openv/netbackup/logs.  Optionally, the VERBOSE entry can be set in either /usr/openv/netbackup/bp.conf or /usr/openv/volmgr/vm.conf.  This is covered in more detail later.


Unified or VX Logs:

These are located in:

NetBackup 6.0 /6.5x  -  /usr/openv/logs
NetBackup 7.x  -  /usr/openv/logs/<process>

The log files following the naming convention :  

<product id>-<originator id>-<host id>-<date>-<rotention>.log for example, 51216-111-2220372284-091020-0000000017.log

Unlike unified logs, VX logs require processing via the vxlogview command.  The process that the log refers to is represented by the 'originator' id, that is the 2nd, 3 digit number.  In the above example, 111 represents the NBEMM process.

To enable a VX log, the debug and diagnostic levels are set using the vxlogcfg command.

Originator ids:
The originator ids for a given process can be found in /usr/openv/netbackup/nblog.conf  Although this file should not be edited manually, it is a useful reference and also shows the logging levels set, if other than default.

143.L10nResource=emm
143.OIDNames=mds
143.DebugLevel=6
143.DiagnosticLevel=6

The above example shows the originator id for the mds process, a sub process of nbemm.

Some processes do not create separate log files.  For example, the mds process (143) will log into the main nbemm log.


Verbose settings for logs in /usr/openv/netbackup/logs:

By default, the unified verbose level is set to 0, this provides little information in the logs files.  On occasion it may be sufficient, but in most cases there will be a requirement to increase this.

There are two ways this can be done :

To increase the verbose level of all logs (except vault)
Add the entry VERBOSE = <level> into /usr/openv/netbackup/bp.conf.  <level> is a value between 0 and 5, with 5 being the highest.

To increase the verbose level of a specific process:
Add the entry <PROCESS>_VERBOSE = <level> into /usr/openv/netbackup/logs/bp.conf, for example

BPTM_VERBOSE = 5
VAULT_VERBOSE = 5  (note, this must be set to increase the verbose level for /usr/openv/netbackup/logs/vault).

Note:  Not all processes can be individually selected for a custom VERBOSE level in the bp.conf file.


Verbose settings for logs in /usr/openv/volmgr/debug:

For these logs, the only option is to add VERBOSE into /usr/openv/volmgr/vm.conf.   No numerical value is accepted.

Further increased logging can be created by adding creating one or more of the following 'touch' files.

touch /usr/openv/volmgr/DEBUG
touch /usr/openv/volmgr/ROBOT_DEBUG
touch /usr/openv/volmgr/AVRD_DEBUG
touch /usr/openv/volmgr/SSO_DEBUG
touch /usr/openv/volmgr/database/DEBUG
touch /usr/openv/volmgr/VM_MQ_DEBUG
touch /usr/openv/volmgr/DRIVE_DEBUG


A process must restart to pickup changes in either the bp.conf or vm.conf files.  Therefore processes such as bpbrm/ bptm will have an increased logging level as soon as a new backup starts.  However, ltid or robots would not, and so the media manager processes would need to be restarted,

sg debugging (Solaris only)

It is possible to capture a debug log for the sg driver, this can be achieved one of two ways.

Add the following line to the sg configuration /kernel/drv/sg.conf file:
debug-level=1;

Reload the sg driver. This can be achieved either by rebooting the system or unloading and reloading the sg kernel module (see the Solaris man pages for modunload(1M) and modload(1M) for details).

Additionally it is possible to edit the /etc/system file.  The sg driver has a verbosity flag called "sg_err_level" which defaults to 3. The sg driver can be made very verbose by adding the following to the /etc/system file, and then rebooting the machine, or, reloading the sg driver.

set sg:sg_err_level=4

The values between 3 and 9 may be used for different levels of
verbosity. Level 9 is the highest. Engineering advice is not to use
levels beyond level 4.

Java Logs:

On the Master server edit the file /usr/openv/java/Debug.properties

Add or uncomment the following lines :

printcmds=true
printCmdLines=true
debugMask=2

In /usr/openv/netbackup/logs create the following directories

bpjava-msvc
bpjava-susvc

Restart the GUI

When the issue is experienced in the GUI, run the following command :

/usr/openv/java/get_trace


A .log file will be created under /usr/openv/netbackup/logs/user_ops/nbjlogs for each instance of Java GUI you have open.


Further detail on VX logs:

The vx  logs are created in the /usr/openv/logs directory by default, they have the following format, the first and second fields representing the -p and -o options used in the vx... commands.

51216-111-2201603439-080923-0000000000.log

To enable these logs, set the DebugLevel and DiagnosticLevel using the vxlogcfg command:

For example, to increase the 111 verbose level,

vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

The setting change can be confirmed with the following command:

vxlogcfg -l -p 51216 -o 111

The default logging levels for vx logs are as follows: DebugLevel = 1  DiagnosticLevel = 6

vx logs are created in a raw format, and are virtually unreadable until processed through the vxlogview command.  This increases the size of the logs.

For this reason it may be advantageous to send the logs to Symantec in this format.  Additionally, should a case require escalation to Engineering, very often the logs are requested in the raw format.


Collecting VX logs:

The vxlogview command has an option to process the logs for a given jobid.  For this reason, even if vxlogview is run for a specific originator id, all the logs contained in /usr/openv/logs are searched.  Processing individual originator ids can therefore be speeded up by copying the required logs to an alternative location and running the vxlogview command with the -G <location of logs> option.

This method can also be used in general if the raw logs are to be sent in, as it easily collects the required logs into one location.

There are 3 ways to copy the logs :

(1) Copy all the 111 (nbemm) logs for the past day into /tmp/vx  - This is very useful if there are not too many longs and there are not particularly large.
vxlogmgr -o 111 -c -f /tmp/logs -n 1

(2) Copy all the 111 (nbemm) logs between the two times/dates ...   This is perhaps the best and most accurate method.  However, please see the 'Important note' below.
vxlogmgr -o 111 -c -f /tmp/logs -b '10/21/09 12:01:22 PM' -e '10/21/09 12:21:22 PM'

(See the 'Important note regarding format of <date/ time> for -b or -e options with vx commands' section below)

(3) Copy all the 111 (nbemm) logs for the past 1 hour, relative to when the command was run:  This is very simple, however please note that this command is relative  to when it was run, for example if you run the command at 15.00, it will copy the logs between 14.00 - 15.00.   therefore it is vital to ensure that a sufficient time is specified to allow the collection of logs over a time period large enough to capture the issue.
vxlogmgr -o 111 -c -f /tmp/logs -t 01:00:00

Important note regarding format of <date/ time> for -b or -e options with vx commands

The format of the date used in the above commands may vary depending on the locale settings of the system.  Running the vxlogview --help command and looking for the -b or -e options (begin/ end) will display a 'live' example of the date/time format for that machine.



Reading vx logs:

vxlogview -p 51216 -o 111 -d all >/tmp/emm.txt

The command must be run with the -d all option, otherwise vital details will not be included.

Following on from the examples to move the log files, various variations can be run to convert the logs for a specific time period:

View just the past 10 minutes of the nbemm log:
vxlogview -p 51216 -o 111 -d all -t 00:10:00 >/tmp/logs/nbemm.txt

View the nbemm log between two specific times:
vxlogview -p 51216 -o 111 -d all -b '04/30/08 07:00:00 PM' -e '04/30/08 07:30:00 PM' >/tmp/nbemm_log.txt

(See the 'Important note regarding format of <date/ time> for -b or -e options with vx commands' section above)

View the past 20mins of nbemm logs, relative to when the command was run:
vxlogview -p 51216 -o 111 -d all -t 00:20:00 >/tmp/nbemm_log.txt


If the vx logs for a sub-component of a process are also enabled, this detail can be included in the logs b use of the -i option instead of the -o option.

For example, if the mds logging (143) is enabled, this detail will be included in the output of the following command:

vxlogview -p 51216 -i 111 -d all -n 1 >/tmp/nbemm_log.txt


Running vxlogview will cause the logs times to be converted to the timezone of the server the commands are run, this can be inconvenient :

The use of the -z option can show the timezone of the server the logs were created on, eg.

/usr/openv/netbackup/bin/vxlogmgr  -G .   -s -z

Following are the files that were found:

File ./51216-200-2220372284-091105-0000000000.log generated in GMT-8:00 timezone.
File ./51216-231-2220372284-091105-0000000000.log generated in GMT-8:00 timezone.
File ./51216-111-2220372284-091105-0000000000.log generated in GMT-8:00 timezone.


Using the following vxlogview command, we can convert and read the logs with the times adjusted for (example) the timezone -8:00 hours behind GMT.

vxlogview -G . -p 51216 -o 111 -z GMT-8

'Tailing' the vx logs

Using the -S option allows the logs to be display in 'real time', similar to running tail -f /usr/openv/netbackup/<process>/<logfile>

For example, the following command would 'tail' the nbjm log.

vxlogview -p 51216 -i 117 -S