cancel
Showing results for 
Search instead for 
Did you mean: 

Disk storage unit is full(129) after duplication

Petros_Marinos
Level 3

Greetings dears!

In NetBackup 7.5 Master Server, I have a Storage Unit with:

Type: Disk
Disk Type: BasicDisk
High Water mark 90%
Low water mark 80%
In Staging Schedule:
storage unit: Tape_Library
volume pool: Full_Pool

The Storage Unit is 9.1 TB in size with 300GB free. All images currently in this unit are duplicated to tapes.
The images have a retension policy of 3 weeks. All of them are created during these 3 weeks, not before!

I now try in this unit to create an image (through manual backup process) of approx 600GB. The job exits with error 129 (Disk Storage Unit full)

Since all images are duplicated to tapes, shouldn't some of them get cleaned up from the disk to free space for the next image to come? Have i misconfigured something?

Thank you for in advance for your answers!

Best regards!

14 REPLIES 14

Nicolai
Moderator
Moderator
Partner    VIP   

Can you show/tell us the stagning schedule (configured in the storage unit properties) ?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Can you actually see the duplications running on a regular basis?

Please ensure that the following log folders exist under netbackup\logs on the master/media server:

admin
bptm
bpdm 

admin log will contain  duplication start times and info about each image being duplicated.
bptm will log backup and duplication activitities.
bpdm will log cleanup activities.

 

Hi Nicolai!

Ok, it's attached.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

What is in Start Window tab?
Duplications every 6 hours can only run during open windows (similar to backup schedules).

Hello Marianne!

Start Window attached.

Still, i had some Bad Tape incidents lately, so i had to do some manual backups and also trigger a "Manual Relocation", independently of the schedule.

I'll see into the logs you suggest and will let you know of the findings.

Adnan_M_F
Level 4

I think the best way to do this is with SLP, and select the option "Expire after copy" under retention type. 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

So, duplications only to run between Sunday and Wednesday morning?

What about the rest of the week?

@Adnan_M_F BasicDisk cannot be used with SLPs.

 

@Marianne oh, didnt know that. Thanks !!!

Sadly, under C:\Program Files\Veritas\NetBackup\logs, i don't have any of the log folders you note.

But l have full picture from the Activity monitor, after following the dates. Have a look at the attachment.

Inside disk K:, i have an image called xxxx_1475345436_C1_F1.1475345436.img. As you can see, the image was duplicated to tape the very same day successfully. Shouldn't these 300GB be cleaned up to free space for the upcoming backup image?

PS: The rest of the week is assigned with the other 2 Storage Units configured in the same NetBackup Master server.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Image cleanup is supposed to start when HLM is reached.

This TN explains how cleanup works as from NBU 6.5: http://www.veritas.com/docs/000036495 

Please create the log folders as suggested to enable logging and troubleshooting. 
Also ensure that the K-drive is used for this STU alone. Nothing else.

Try this:

Create the log folders.
Issue this command on the master:
nbdelete -allvolumes -force

Look at bpdm log to see if cleanup is attempted.

You may also want to lower the LWM so that a bit more will get cleaned up - not only down to 80%.

 

 

Ok, after creating C:\Program Files\Veritas\NetBackup\logs\bpdm and the other 2 folders, i went on and run the nbdelete command.

Nothing was returned in stdout and the bpdm folder is empty. Still, i have something in the log under the admin folder:

18:23:09.645 [3896.6112] <16> deletionHandler::cleanDisks: Activity monitor job id = 3902
18:23:09.661 [3896.6112] <2> nbdelete: EXIT status = 0

18:23:09.801 [4660.8932] <8> deletionHandler::cleanDisks: no images need deletion
18:23:09.817 [4660.8932] <2> nbdelete: EXIT status = 0

If it's of any value, here is the image info that could/should be deleted:

C:\Program Files\Veritas\NetBackup\bin\admincmd>bpimagelist -backupid xxxx_1475345436 -U


Backed Up           Expires          Files        KB         C Sched Type On Hold Index Status Policy
----------------          ----------       --------   --------          -  ------------      -------     ------------       ------------
10/01/2016 21:10 10/22/2016 109828 305395532 N Full Backup 0            0                   xxxx

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Image cleanup on disk will only happen when the HWM is reached and will start with oldest images on disk. 
Otherwise the image will expire and be cleaned according to retention / image expiration date. 

Have you considered to lower the LWM so that more can be cleaned up? Not only down to 80%?

Is the disk more than 90%? 
What else is on K-drive?

The idea with disk storage is to have newest backups on disk for fast restore. 
Not to be automatically deleted after duplication... 

Thank you for all your info and recomendations.

I am not expecting an image to be deleted after duplication, but i'm expecting it to be deleted when HWM is reached, as you suggest.

K-drive is 9.09TB and now it has 293GB free, meaning that is well above the 90% mark. Nothing else is stored in the K-drive, it is all dedicated to the STU.

Ok, i lowered the LWM down to 70% and re-run the nbdelete command, nothing new happened.

Are the images in some kind of a frozen state maybe? During the last 2 weeks, i face "Bad Tapes" issues and i don't know how this affects the image if the tape went out during the duplication process.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Maybe a good idea to run 'images on disk' report for this particular DSU and compare with what is on disk.
(Other than the valid images that you have listed here.) 

There is another current discussion here where half-written images from failed backups are not automatically cleaned up.

So, the problem could be with orphaned images taking up space....