11-19-2013 12:16 PM
Hi everyone,
I am relatively new to Netbackup and my experience and knowledge are very limited.
We have a dedupe disk connected to a Windows Server 2008 R2 media server. Currently only a single SQL Server with multiple databases is being backed up to this dedupe disk.The total amount of data in all databases is less than 70 GB and retention is 3 days. The dedupe disk capacity is 655 GB, but it keeps getting full (causing Disk storage unit is full(129) error). Either something is wrong or I am missing something.When I look at the Catalog from Netbackup Admin Console selecting the related PureDisk Pool the backup images are shown as expected, very few and the oldest one is dated 2 days ago, which as far as I understand implies that some backup image expiring and space reclaiming is taking place. Image Cleaup job appear to be running regularly.
I also ran the commands below(2 days ago) ,but no space has been freed so far:
bpimage -cleanup -allclients
crcontrol --processqueue (at least 4 times)
crcollect -v -m +1,+2 --noreport
Netbackup version is 7.5.0.6
Any help will be greatly appreciated.
Regards,
Sezgin
Solved! Go to Solution.
11-21-2013 04:29 AM
Those files are just containers and their size will not actually change - as you are doing de-dupe they could be the first ever backups you ran - after that the same containers get referenced and so they never get removed
As those are quite large it indicates that the first run almost filled your pool up - after that you are just adding fingerprints plus the odd but of data now and again
On that basis you really need to increase the size of your de-dupe pool
Compaction could clear down some of those containers too - all odd concepts when using de-dupe but if you have that many containers of that size it does just put you near the limit
More disk space and even longer retention periods can actually help as you are making the system work pretty hard every day expiring those backups - but if de-dupe is really good all that work actually doesnt clear down any space - and the containers do not shrink anyway without compaction taking place
Hope some of this mankes some sense!
01-15-2014 04:01 AM
I am afraid it did not become clear what fixed the problem. The internal tool crchk was supplied and I ran it , but the output did not give any clues. However, after running crchk, "crcontrol --processqueue" and "crcontrol --compactstart 100 0 1" started working properly and space was freed.
I think running the crchk tool triggered the fix, but I am not sure. Afterwards, I also took the Low Water Mark of the pool to 80 % (it was 94% or 96%). Everything has been fine for the last month.
Thanks again.
11-19-2013 12:36 PM
If NBU design pads the deduplication diskpool storage percent used by equl to/more tan 10% which can lead to NetBackup reporting a 'disk storage unit is full' condition when there is still space left on the diskpool.
1)plz verfy below thresholds
http://www.symantec.com/business/support/index?page=content&id=TECH148008
WarningSpaceThreshold=
LowSpaceThreshold=
VeryLowSpaceThreshold=
2)set the high water mark (HWM) on the dedup diskpool to 100% via NetBackup administration console
3)Verify that compression is enabled on the MSDP server.
install_path\Veritas\NetBackup\bin\ost-plugins
verify for compression , COMPRESSION=1 means compression is enabled.
11-20-2013 01:33 AM
if the sql server dumps has already been compressed, MSDP will not be able to deduplicate them.
Use this tech notes to verify the deduplication ratio:
http://www.symantec.com/docs/HOWTO89020
Else extend the file system 655G is not that much.
11-20-2013 04:28 AM
Thanks for the replies.
High Water Mark is 98%.
Regards,
Sezgin
11-20-2013 08:23 AM
Please verify in reports-disk reports.
make sure tat pure disk cleaning is happening ..
also plz verify below link:
http://www.symantec.com/business/support/index?page=content&id=TECH124914
if expired backup data doent removed from puredisk server
http://www.symantec.com/business/support/index?page=content&id=TECH147710
11-21-2013 03:36 AM
A lot of records are shown available for compaction - make sure compaction is turned on
The basic cleanup should run daily but compaction and rebasing will only run once a month
Turn on compaction if it is not on
crcontrol --compactstate will tell you if it is on or not
crcontrol --getmode will also tell you if the compaction (and all other) process is enabled
But it may just be that your disk pool is not big enough
One other thing ... how big is your process queue?
With images getting written and deleted every day it may well not be keeping up - you may need to run it many times to bring the queue down and actually get data cleaned up
crcontrol --queueinfo
11-21-2013 04:27 AM
As a percentage, "number of records available for compaction" and "space needs compaction" appeared small to me, that's why I didn't bother about them.
The outputs of the commands are below:
C:\Program Files\Veritas\pdde>crcontrol --compactstate
Data store compaction: ON, DeleteSpaceThreshold: 30%, CompactLBound: 4MB
Compaction busy: No
C:\Program Files\Veritas\pdde>crcontrol --getmode
Mode : GET=Yes PUT=Yes DEREF=Yes SYSTEM=Yes STORAGED=Yes REROUTE=No COMPACTD=Yes
RECOVERCRDB=No
C:\Program Files\Veritas\pdde>crcontrol --queueinfo
total queue size : 79174598
creation date of oldest tlog : Thu Nov 21 12:20:23 2013
The queue folder as seen from Windows Explorer has exactly the same size as above, 79174598 , about 75.5. MB
When I checked the dedupe disk from Windows Explorer I saw that the 644 GB of the data is in the folder named "data". When the files in the folder are sorted by Date Modified I see that 578 GB of the data is older than the oldest backup image(dated 20.11.2013) seen in Netbackup Admin Console Catalog. The oldest files in the data folder are dated December, 2012
Regards
11-21-2013 04:29 AM
Those files are just containers and their size will not actually change - as you are doing de-dupe they could be the first ever backups you ran - after that the same containers get referenced and so they never get removed
As those are quite large it indicates that the first run almost filled your pool up - after that you are just adding fingerprints plus the odd but of data now and again
On that basis you really need to increase the size of your de-dupe pool
Compaction could clear down some of those containers too - all odd concepts when using de-dupe but if you have that many containers of that size it does just put you near the limit
More disk space and even longer retention periods can actually help as you are making the system work pretty hard every day expiring those backups - but if de-dupe is really good all that work actually doesnt clear down any space - and the containers do not shrink anyway without compaction taking place
Hope some of this mankes some sense!
11-21-2013 11:57 AM
Thanks Mark.
I forgot to mention that the jobs writing to this PureDisk are using SLP's having Duplication operations.
The problem may be related to incomplete SLP processing.
Anyway, a support case with Symantec Support has been opened.
I will update with developments.
Regards
11-22-2013 02:26 AM
If the images are not getting duplicated (i.e. completing their LifeCycle) then they will have an infinite retention which could cause your disk to fill up
If you do not plan to duplicate them then you could cancel then which will revert them to their original retention period and allow them to expire and clear the disk down
You have some decisions to make by the sounds of it
Let us know if we can assist further
01-15-2014 02:37 AM
Sorry for the late feedback.
The problem was eventually fixed with the help of some internal tools.
Thanks for your help.
01-15-2014 02:53 AM
A lot of people have given you advice on this matter - it would be nice if you could give a bit more detail on what was done to fix your issue so that we know what the actual issue was and this will also help others in the future with similar issue
Thanks
01-15-2014 04:01 AM
I am afraid it did not become clear what fixed the problem. The internal tool crchk was supplied and I ran it , but the output did not give any clues. However, after running crchk, "crcontrol --processqueue" and "crcontrol --compactstart 100 0 1" started working properly and space was freed.
I think running the crchk tool triggered the fix, but I am not sure. Afterwards, I also took the Low Water Mark of the pool to 80 % (it was 94% or 96%). Everything has been fine for the last month.
Thanks again.
01-13-2020 07:04 AM
Hi Mark.
Seems I have exactly this issue.
Months ago I had to disable the SLP deduplications, due space issue on the replication site.
Now according to your message, due this the storage on the source side got full.
How to expire all the not duplicated images and let them expire?
It is really urgent.
thanks
Marian
01-14-2020 12:45 AM
Please carry on with your current post over here - https://vox.veritas.com/t5/NetBackup/Netbackup-appliance-images-not-deleted/m-p/875051
This post has been solved many years ago.