Potentially it is quite good, run the script and pass the logs and times period on the command line. It then copies and FTPs the logs up to the FTP server.
I think a couple of improvements could be made:
1. Allow the option to copy the logs to an area other than /tmp It does ask you to check and confirm there is space but if not, you (as far as I know) cannot change the directory.
2. It FTPs straight from the server, not everyone has internet access from the Netbackup server. Unsure if it leaves the file behind after the FTP in which case you could just move off the machine yourself and FTP up to Symantec from somewhere else.
I'm glad to hear that our customers are exploring this tool. We realized that our log collection process could be quite manual and wanted to ease that process.
rjrumfelt: We thought about a sanitizing option but in environments where this is needed we thought administrators would prefer to do this themselves for security and accountability. What are your thoughts on including a sanitize option and what would that look like?
mph999: Good news! Those options are available, you can use nbcplogs to only bundle logs and place them in any desired directory. Try the following, it will copy the last 60 minutes of logs to the ~/nbcplogsOutput directory.
./nbcplogs -d 1h ~/nbcplogsOutput
As the tool sits today, this tool does zero sanitation of logs. It does however have options that may interest you which could greatly reduce the time it takes for you to sanitize the logs.
Copy subset of logs (the subset with the error in the time frame) and sanitize only those logs.
% nbcplogs --start 01/18 --end 01/19 /tmp/logs-to-sanitize
C:\> nbcplogs --start 01/18 --end 01/19 C:\temp\logs-to-sanitize
How about cutting out 10 minutes of logs (actual 10 minutes only...)?
% nbcplogs --fast --start 01/19-01:00 --end 01/19-01:10 /tmp/logs-to-sanitize
C:\> nbcplogs --fast --start 01/19-01:00 --end 01/19-01:10 C:\temp\logs-to-sanitize
The nbcplogs tool by default copies entire files, but the --fast option (found via --help-long under [algorithms]) actually opens up each log file it encounters and attempts to strip log lines outside of your time frame. This is a very complex process that is not perfected, but will work 99% of the time. (--linebased is the naive way of stripping log lines from a file, --fast is an improved method that won't waste hours to strip those lines)
This way you will be sanitizing FAR less logs than you would normally.
ie: instead of copying an entire 4GB bpdbm log file, it can pull out the 10 minutes you care about and allow you to sanitize a few hundred lines vs millions of lines.
As for sanitation of logs, we could introduce an option for that, but we have no requirements for what needs to be sanitized to comply with customer policy. Hostnames only? If you could enlighten us, we could definitely build it into the tool.
Thanks for the feedback!
As the nbcplogs tool exists today, it does address both of your improvements.
1. Allow alternate temporary location
% nbcplogs --tmpdir=/path/to/temp ...your options/arguments here...
% setenv TMPDIR /path/to/temp
% export TMPDIR=/path/to/temp
C:\> set TMPDIR=C:\path\to\temp
% nbcplogs ...your options/arguments here...
The tool looks to the TMPDIR environment variable, but --tmpdir should override all other environment settings. A few people have asked about this and it's probably just not well documented, so spread the word! The --help-long is a really good resource for power users of this tool!
2. FTP operation failure and proxies...
% nbcplogs -f 123-123-123
... failed to upload ...
** upload failed, but bundle remains at /tmp/logs-hostname-.tar.gz
** please ask tech support to create /pub/support/incoming/123-123-123
** then upload it manually
The tool will leave the log bundle on the host in case of failure. On
successful upload, the bundle will be removed.
A current problem with the tool is its lack of good support for proxies. Hopefully in the future, it will support this better. For now, there are options to manipulate the ftp behavior:
--ftp-host=HOSTNAME hostname to upload logs to
--ftp-user=USERNAME username to use for ftp upload
--ftp-pass="PASSWORD" password to use for ftp upload
--ftp-incoming=PATH base directory for incoming uploaded logs
(from --help-long under [common options])
However, if you'd rather just create a log bundle and place it somewhere:
% nbcplogs --bundle --compress /path/to/bundle
The --ftp (or -f) option auto-enables and names the bundle for you. The above is manually doing the work that ftp does automatically and doesnt upload the bundle. This way, you have finer control over how the bundle is created. This can later be uploaded by whatever means you would like.
Hopefully that helps!
Thanks for the feedback!