03-16-2017 01:34 AM
Hi all,
I'm getting the following alerts from one for the netbackup appliance , this is our media server
I have enabled logging to max, now i started getting alerts that its growing and it is almost 95% full.
From where can cleanup these logs in the cli; kindly help if there is any procedure or article on how to clean up this partition.
+-----------------------------------------------------------------------------+
| Partition Information |
|+---------------------------------------------------------------------------+|
||ID | Partition | Total | Used | Status | State | Acknowledge ||
||---+-------------+---------+------+-----------+-----------+----------------||
||5 |Log |184 GB |95 % |Optimal |Warning |No ||
||---------------------------------------------------------------------------||
|| * The value displayed here may be different from the backup space that is ||
|| available or used on the MSDP partition. The backup space statistics for ||
|| the MSDP partition can be obtained by checking the MSDP disk pool sizes ||
|| from NetBackup Administration Console. ||
|+---------------------------------------------------------------------------+|
+-----------------------------------------------------------------------------+
Solved! Go to Solution.
03-17-2017 05:51 AM
The /log/crash directory is where core dumps from programs that crash end up. Some revs of netbackup have processes that core dump all to regularly. You can pretty safely delete files from /log/core/ assuming you are not using them to debug a problem with netbackup.
The /log/netbackup/ directory has a whole subset of directories. These contain the traditional netbackup logs. You can fairly safely clean up files in the subdirectories under /log/netbackup/. Again you may want them to debug problems, but you can not keep them all forever. Go to /log/netbackup/ and run the du command again and you will see which directories to focus on in that area. You also probably should look at how verbose you have the logging set. You may want to turn it down a bit if it is set at a high level.
Read more here on how to manage netbackup logs: https://www.veritas.com/support/en_US/article.000108090
03-16-2017 01:47 AM
You have not specified the appliance firmware version but in any case, first steps of the technote will be quite useful https://www.veritas.com/support/en_US/article.000108180
03-16-2017 02:19 AM
Hey
You need to clean up some NBU related logs located in /logs/netbackup directory. To do this please elevate to CLISH (https://www.veritas.com/support/en_US/article.000116864) and change to this dir and there run
du -sh *
figure out which one is the biggest and if you dont need these logs remove them
Also can you provide output form this command
cat /usr/openv/netbackup/bp.conf so we can see the debug levels etc...?
03-16-2017 03:45 AM - edited 03-16-2017 03:46 AM
Our Master Server is at 7.6.0.3 and Media servers which are the 5220 Netbackup Appliance at 2.6.0.3
I checked the /log/upload partition in the CLI by logging to the maintenance mode and could see a lot of files generated with the name format like : cmd_out_1489659883457351.log
I did a vi and found this:
s65013:/log/upload # vi cmd_out_1489659883457351.log
Time Ran: Thu Mar 16 2017 11:24:43 CET
CMD: LC_ALL=C /usr/bin/w3m -dump /tmp/9oWiCQD0gC.html
RETVAL: 0
STDOUT:
Compute Node s65013
Time Monitoring Ran: Thu Mar 16 2017 11:22:10 CET
+-----------------------------------------------------------------------------+
| Partition Information |
|+---------------------------------------------------------------------------+|
||ID | Partition | Total | Used | Status | State | Acknowledge ||
||---+-------------+---------+------+-----------+-----------+----------------||
||5 |Log |184 GB |95 % |Optimal |Warning |No ||
||---------------------------------------------------------------------------||
|| * The value displayed here may be different from the backup space that is ||
|| available or used on the MSDP partition. The backup space statistics for ||
|| the MSDP partition can be obtained by checking the MSDP disk pool sizes ||
|| from NetBackup Administration Console. ||
|+---------------------------------------------------------------------------+|
+-----------------------------------------------------------------------------+
STDERR:
Not sure if these logs will be of use for any troubleshooting.
Please let me know if I can just delete them.
These version are out of support now.
Recently i had set the logging to MAX in the master server to get more details when any backup is failed.
I have attached the bp.conf file to here.
Let me know if any information is required.
03-16-2017 07:27 AM
Hi
My take is that majority of the /log is being used by netbackup subdirectory so /log/netbackup - you can go into this location and check this out by running from there
du -sh *
If I am mistaken run this command from /log this will allow you quickly figure out the 'guilty' one folder... and then take actions. Also when backups are failing only certain logs are needed - maybe you do not need all which are configured under /log/netbackup.... ask VRTS which should be enabled...
03-16-2017 09:07 AM - edited 03-16-2017 09:12 AM
And as you said I see the culprit is /netbackup its using up a space of around 95GB out of 184 GB
In /log/netbackup/bptm
the bptm logs are consuming majority of the space
media2:/log # du -sh *
1.7G app_vxul
27G crash
696K diskperflogs
12K initrd_shrink_log_062414122048.txt
32K last_sg_ses_failure.log
0 logrotate.lck
16K lost+found
84M nbappdb
94G netbackup
8.0G openv
4.0K patch
4.0K patch_copy_iso_to_tmpfs_201406161119.log
4.0K patch_log.sh
9.7M patch_log_2.6.0.1_061614101216
9.2M patch_log_2.6.0.1_061614115835
273M patch_log_2.6.0.1_061614121715
296K patch_log_2.6.0.2_062414113225
148K patch_log_2.6.0.3_112514144919
8.0K patch_output_2.6.0.1_061614115835
8.0K patch_output_2.6.0.2_062414113225
8.0K patch_output_2.6.0.3_112514144919
744K puredisk
4.0K s65013_09-23-2015_09h00m38s_logs.tar.gz
33G scsplog
4.0K selftest_report_SYM0910127_06-21-2014.txt
8.0K selftest_report_SYM0910127_06-24-2014.txt
4.0K selftest_report_SYM0910127_11-25-2014.txt
4.0K storage
1.5G upload
272K volmgr
25M webgui
03-17-2017 12:27 AM
Well I would for sure removed the following: 27G crash - I do hope you already cleared out with VRTS possible crashes of this one appliance, than depending on the case you do run I would removed all logs if the job you are debugging did not fail in given time period. That's all - NBU appliance is just a NBU software installed on customized by VRTS linux OS with some add ons - so ...
03-17-2017 01:57 AM
I couldn't check with Veritas as my code is out of support life now. :(
Thanks alot for the guidance.
I'm not sure on what data does the /log/crash holds. I need to check that, is there any document where in I can find all these details, or quickly refer.
03-17-2017 05:51 AM
The /log/crash directory is where core dumps from programs that crash end up. Some revs of netbackup have processes that core dump all to regularly. You can pretty safely delete files from /log/core/ assuming you are not using them to debug a problem with netbackup.
The /log/netbackup/ directory has a whole subset of directories. These contain the traditional netbackup logs. You can fairly safely clean up files in the subdirectories under /log/netbackup/. Again you may want them to debug problems, but you can not keep them all forever. Go to /log/netbackup/ and run the du command again and you will see which directories to focus on in that area. You also probably should look at how verbose you have the logging set. You may want to turn it down a bit if it is set at a high level.
Read more here on how to manage netbackup logs: https://www.veritas.com/support/en_US/article.000108090