Dedupe disk is full.

Hi everyone,

I am relatively new to Netbackup and my experience and knowledge are very limited.
We have a dedupe disk connected to a Windows Server 2008 R2 media server. Currently only a single SQL Server with multiple databases is being backed up to this dedupe disk.The total amount of data in all databases is less than 70 GB and retention is 3 days. The dedupe disk capacity is 655 GB, but it keeps getting full (causing  Disk storage unit is full(129) error). Either something is wrong or I am missing something.When I look at the Catalog from Netbackup Admin Console selecting the related PureDisk Pool the backup images are shown as expected, very few and  the oldest one is dated 2 days ago, which as far as I understand implies that some backup image expiring and space reclaiming is taking place. Image Cleaup job appear to be running regularly.


I also ran the commands below(2 days ago) ,but no space has been freed so far:

bpimage -cleanup -allclients

crcontrol --processqueue    (at least 4 times)

crcollect -v -m +1,+2 --noreport


 

Netbackup version is 7.5.0.6
 

Any help will be greatly appreciated.


Regards,

Sezgin


 

 

2 Solutions

Accepted Solutions
Highlighted
Accepted Solution!

Those files are just

Those files are just containers and their size will not actually change - as you are doing de-dupe they could be the first ever backups you ran - after that the same containers get referenced and so they never get removed

As those are quite large it indicates that the first run almost filled your pool up - after that you are just adding fingerprints plus the odd but of data now and again

On that basis you really need to increase the size of your de-dupe pool

Compaction could clear down some of those containers too - all odd concepts when using de-dupe but if you have that many containers of that size it does just put you near the limit

More disk space and even longer retention periods can actually help as you are making the system work pretty hard every day expiring those backups - but if de-dupe is really good all that work actually doesnt clear down any space - and the containers do not shrink anyway without compaction taking place

Hope some of this mankes some sense!

View solution in original post

Highlighted
Accepted Solution!

I am afraid it did not become

I am afraid it did not become clear what fixed the problem. The internal tool crchk was supplied and I ran it , but the output did not give any clues. However, after running crchk,  "crcontrol --processqueue" and "crcontrol --compactstart 100 0 1"  started working properly and space was freed.
I think  running the crchk tool triggered the fix, but I am not sure. Afterwards, I also took the Low Water Mark of the pool to 80 % (it was 94% or 96%). Everything has been fine for the last month.

Thanks again.

View solution in original post

14 Replies
Highlighted

If NBU design pads the

If NBU design pads the deduplication diskpool storage percent used by equl to/more tan 10% which can lead to NetBackup reporting a 'disk storage unit is full' condition when there is still space left on the diskpool.

1)plz verfy below thresholds

http://www.symantec.com/business/support/index?page=content&id=TECH148008

WarningSpaceThreshold=
LowSpaceThreshold=
VeryLowSpaceThreshold=

2)set the high water mark (HWM) on the dedup diskpool to 100% via NetBackup administration console

  1. NetBackup Administration Console, expand Media and Device
  2. Management > Devices > Disk Pools.
  3. Select the disk pool you want to change in the details pane.
  4. On the Edit menu, select Change.

3)Verify that compression is enabled on the MSDP server.

install_path\Veritas\NetBackup\bin\ost-plugins

verify for compression , COMPRESSION=1 means compression is enabled.

Highlighted

if the sql server dumps has

if the sql server dumps has already been compressed, MSDP will not be able to deduplicate them.

Use this tech notes to verify the deduplication ratio:

http://www.symantec.com/docs/HOWTO89020

Else extend the file system 655G is not that much.

 

Highlighted

Thanks for the

Thanks for the replies.
 

High Water Mark is 98%.


Whenever the Job finds enough free space, it runs, but very often fails with error  “Disk storage unit is full(129 )”. Successfull jobs have dedupe rates between 86% and 99%.

  crcontrol  --dsstat 1  output states that 643 GB out of 655 GB is used space. How can I get detailed info about what is taking up that space ?


Regards,

Sezgin                
 

Highlighted

Please verify in reports-disk

Please verify in reports-disk reports.

 

make sure tat pure disk cleaning is happening ..

also plz verify below link:

http://www.symantec.com/business/support/index?page=content&id=TECH124914

if expired backup data doent removed from puredisk server

http://www.symantec.com/business/support/index?page=content&id=TECH147710

Highlighted

A lot of records are shown

A lot of records are shown available for compaction - make sure compaction is turned on

The basic cleanup should run daily but compaction and rebasing will only run once a month

Turn on compaction if it is not on

crcontrol --compactstate will tell you if it is on or not

crcontrol --getmode will also tell you if the compaction (and all other) process is enabled

But it may just be that your disk pool is not big enough

One other thing ... how big is your process queue?

With images getting written and deleted every day it may well not be keeping up - you may need to run it many times to bring the queue down and actually get data cleaned up

crcontrol --queueinfo

 

 

Highlighted

As a percentage, "number of

As a percentage, "number of records available for compaction" and "space needs compaction" appeared small to me, that's why I didn't bother about them.
 

The outputs of the commands are below:

C:\Program Files\Veritas\pdde>crcontrol --compactstate
Data store compaction: ON, DeleteSpaceThreshold: 30%, CompactLBound: 4MB
Compaction busy: No

C:\Program Files\Veritas\pdde>crcontrol --getmode
Mode : GET=Yes PUT=Yes DEREF=Yes SYSTEM=Yes STORAGED=Yes REROUTE=No COMPACTD=Yes
 RECOVERCRDB=No

C:\Program Files\Veritas\pdde>crcontrol --queueinfo
total queue size : 79174598
creation date of oldest tlog : Thu Nov 21 12:20:23 2013

The queue folder as seen from Windows Explorer has exactly the same size as  above, 79174598 , about 75.5. MB

When I checked the dedupe disk from Windows Explorer I saw that the 644 GB of the data is in the folder named "data". When the files in the folder are sorted by Date Modified I see that 578 GB of the data is older than the oldest backup image(dated 20.11.2013) seen in Netbackup Admin Console Catalog. The oldest files in the data folder are dated December, 2012

Regards

 

 

Highlighted
Accepted Solution!

Those files are just

Those files are just containers and their size will not actually change - as you are doing de-dupe they could be the first ever backups you ran - after that the same containers get referenced and so they never get removed

As those are quite large it indicates that the first run almost filled your pool up - after that you are just adding fingerprints plus the odd but of data now and again

On that basis you really need to increase the size of your de-dupe pool

Compaction could clear down some of those containers too - all odd concepts when using de-dupe but if you have that many containers of that size it does just put you near the limit

More disk space and even longer retention periods can actually help as you are making the system work pretty hard every day expiring those backups - but if de-dupe is really good all that work actually doesnt clear down any space - and the containers do not shrink anyway without compaction taking place

Hope some of this mankes some sense!

View solution in original post

Highlighted

Thanks Mark. I forgot to

Thanks Mark.

I forgot to mention that the jobs writing to this PureDisk are using SLP's having Duplication operations.
The problem may be related to incomplete SLP processing.
Anyway, a support case with Symantec Support has been opened.
I will update with developments.


Regards

Highlighted

If the images are not getting

If the images are not getting duplicated (i.e. completing their LifeCycle) then they will have an infinite retention which could cause your disk to fill up

If you do not plan to duplicate them then you could cancel then which will revert them to their original retention period and allow them to expire and clear the disk down

You have some decisions to make by the sounds of it

Let us know if we can assist further

Highlighted

Sorry for the late

Sorry for the late feedback.
The problem was eventually fixed with the help of some internal tools.

Thanks for your help.

Highlighted

A lot of people have given

A lot of people have given you advice on this matter - it would be nice if you could give a bit more detail on what was done to fix your issue so that we know what the actual issue was and this will also help others in the future with similar issue

Thanks

Highlighted
Accepted Solution!

I am afraid it did not become

I am afraid it did not become clear what fixed the problem. The internal tool crchk was supplied and I ran it , but the output did not give any clues. However, after running crchk,  "crcontrol --processqueue" and "crcontrol --compactstart 100 0 1"  started working properly and space was freed.
I think  running the crchk tool triggered the fix, but I am not sure. Afterwards, I also took the Low Water Mark of the pool to 80 % (it was 94% or 96%). Everything has been fine for the last month.

Thanks again.

View solution in original post

Highlighted

Re: If the images are not getting

Hi Mark.

Seems I have exactly this issue.

Months ago I had to disable the SLP deduplications, due space issue on the replication site.

Now according to your message, due this the storage on the source side got full.

How to expire all the not duplicated images and let them expire?

It is really urgent.

thanks

Marian

Highlighted

Re: If the images are not getting

@eccio 

Please carry on with your current post over here - https://vox.veritas.com/t5/NetBackup/Netbackup-appliance-images-not-deleted/m-p/875051

This post has been solved many years ago.