In NetBackup 7.5 Master Server, I have a Storage Unit with:
Disk Type: BasicDisk
High Water mark 90%
Low water mark 80%
In Staging Schedule:
storage unit: Tape_Library
volume pool: Full_Pool
The Storage Unit is 9.1 TB in size with 300GB free. All images currently in this unit are duplicated to tapes.
The images have a retension policy of 3 weeks. All of them are created during these 3 weeks, not before!
I now try in this unit to create an image (through manual backup process) of approx 600GB. The job exits with error 129 (Disk Storage Unit full)
Since all images are duplicated to tapes, shouldn't some of them get cleaned up from the disk to free space for the next image to come? Have i misconfigured something?
Thank you for in advance for your answers!
Can you actually see the duplications running on a regular basis?
Please ensure that the following log folders exist under netbackup\logs on the master/media server:
admin log will contain duplication start times and info about each image being duplicated.
bptm will log backup and duplication activitities.
bpdm will log cleanup activities.
Sadly, under C:\Program Files\Veritas\NetBackup\logs, i don't have any of the log folders you note.
But l have full picture from the Activity monitor, after following the dates. Have a look at the attachment.
Inside disk K:, i have an image called xxxx_1475345436_C1_F1.1475345436.img. As you can see, the image was duplicated to tape the very same day successfully. Shouldn't these 300GB be cleaned up to free space for the upcoming backup image?
PS: The rest of the week is assigned with the other 2 Storage Units configured in the same NetBackup Master server.
Image cleanup is supposed to start when HLM is reached.
This TN explains how cleanup works as from NBU 6.5: http://www.veritas.com/docs/000036495
Please create the log folders as suggested to enable logging and troubleshooting.
Also ensure that the K-drive is used for this STU alone. Nothing else.
Create the log folders.
Issue this command on the master:
nbdelete -allvolumes -force
Look at bpdm log to see if cleanup is attempted.
You may also want to lower the LWM so that a bit more will get cleaned up - not only down to 80%.
Ok, after creating C:\Program Files\Veritas\NetBackup\logs\bpdm and the other 2 folders, i went on and run the nbdelete command.
Nothing was returned in stdout and the bpdm folder is empty. Still, i have something in the log under the admin folder:
18:23:09.645 [3896.6112] <16> deletionHandler::cleanDisks: Activity monitor job id = 3902
18:23:09.661 [3896.6112] <2> nbdelete: EXIT status = 0
18:23:09.801 [4660.8932] <8> deletionHandler::cleanDisks: no images need deletion
18:23:09.817 [4660.8932] <2> nbdelete: EXIT status = 0
If it's of any value, here is the image info that could/should be deleted:
C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -backupid xxxx_1475345436 -U
Backed Up Expires Files KB C Sched Type On Hold Index Status Policy
---------------- ---------- -------- -------- - ------------ ------- ------------ ------------
10/01/2016 21:10 10/22/2016 109828 305395532 N Full Backup 0 0 xxxx
Image cleanup on disk will only happen when the HWM is reached and will start with oldest images on disk.
Otherwise the image will expire and be cleaned according to retention / image expiration date.
Have you considered to lower the LWM so that more can be cleaned up? Not only down to 80%?
Is the disk more than 90%?
What else is on K-drive?
The idea with disk storage is to have newest backups on disk for fast restore.
Not to be automatically deleted after duplication...
Thank you for all your info and recomendations.
I am not expecting an image to be deleted after duplication, but i'm expecting it to be deleted when HWM is reached, as you suggest.
K-drive is 9.09TB and now it has 293GB free, meaning that is well above the 90% mark. Nothing else is stored in the K-drive, it is all dedicated to the STU.
Ok, i lowered the LWM down to 70% and re-run the nbdelete command, nothing new happened.
Are the images in some kind of a frozen state maybe? During the last 2 weeks, i face "Bad Tapes" issues and i don't know how this affects the image if the tape went out during the duplication process.
Maybe a good idea to run 'images on disk' report for this particular DSU and compare with what is on disk.
(Other than the valid images that you have listed here.)
There is another current discussion here where half-written images from failed backups are not automatically cleaned up.
So, the problem could be with orphaned images taking up space....