06-05-2014 04:36 AM
Hi All,
I am troubleshooting some issues where my disk storage is filling up and following this document:
http://www.symantec.com/business/support/index?page=content&id=TECH124914
I want to run the garbage collection but have found that crcollect is missing from the pdde folder. crcontrol.exe is there though. I can find a version of crcollect that appears to have been backed up during an update. I think this may be what is causing the storage to fill up.
Info:
Win 2008 R2 Server
Single Master server with 16TB storage LUN (iSCSI)
~6TB Front end data to back up (licenced on a front end TB basis including deduplication)
2 months retention on disk
I have had the infrastructure up and running for around 6 months now but over the past 2 months the data store has started to fill. It was around 9TB used but now I get warnings that is is full (I do have 600GB reserved). So in short we had the first 2 months where space used grew and this was to be expected then the next 2 months where it remained reasonably constant which I also expected as I had hit the 2 month retention so data was aging out. But then it started growing again and has not stopped...
Monthly writes to tape - infinite retention on the tape using SLP's
Any help would be appreciated.
Regards
Chris
Solved! Go to Solution.
06-05-2014 04:53 AM
Ah - replaced with Data Integrity Check - similar thread here:
https://www-secure.symantec.com/connect/forums/garbage-data-integrity-check
06-05-2014 04:37 AM
Sorry, to add to this. I have expired a week's worth of images as I keep slightly more than the SLA requires (as you do) and then I processed the queue and it cleared up about 1TB of space
06-05-2014 04:49 AM
No crcollect in 7.6.0.1 (and i have checked all of the cab files for it!)
Not in 7.6.0.2 either
crcontol is in data5.cab for both - so i guess it no longer exists - i will check and get back to you
06-05-2014 04:53 AM
Ah - replaced with Data Integrity Check - similar thread here:
https://www-secure.symantec.com/connect/forums/garbage-data-integrity-check
06-05-2014 06:44 AM
Ah OK, I have checked the link but it didn't have an explanation on how to run it manually. I will look into it and let you know if I find anything
Thanks Mark!
06-05-2014 07:08 AM
This is what I have found so far:
http://www.symantec.com/business/support/index?page=content&id=HOWTO70644#v62952974
http://www.symantec.com/business/support/index?page=content&id=HOWTO70642#v62619765
I have run thr following commands from CLI:
C:\Program Files\Veritas\pdde>pddecfg -a getdataintegritycheck
Info: Data integrity check: ON
Info: Data integrity check busy: No
Info: Data integrity check current checked times: 30
Info: Data integrity check total check times : 30
C:\Program Files\Veritas\pdde>pddecfg -a enabledataintegritycheck
Info: Enable data integrity check successfully.
C:\Program Files\Veritas\pdde>Crcontrol.exe --crccheckstate
CRC check : ON
CRC check mode: Normal
CRC check busy: No
Current DCID : 0
Max DCID : 0
Now the amount of free storage appears to be growing. I am not sure if it was my commands as you can see the checks were already enabled. In the past 10 minutes around 10GB of space has come available. I will let everything sit over night as I am almost up to 2TB free so tonight's backup should run through fine.
06-05-2014 07:15 AM
So what time is it where you are?
These things do everything on a regular scheduled basis so it may be doing its regular daily, weekly, monthly run
The old way used to only actually once a month (or 30 days) based on install time or last run if i remember rightly
06-05-2014 07:20 AM
Its 15:16, here in the UK. It is quite possible based on what I read (I see a lot of 'every 12 hours at 20 past the hour' in documentation)
I did want to force a garbage collection specifically though as I have a feeling I have orphaned files. For me to have 6TB of frontend and to be using 14TB to backup 2 months worth in dedupe seems like too much. The rate of change is not high enough to justify it. I guess as you say it may be running something triggered by the multitude of commands I have run today or maybe it's on a cycle but it would be nice to run a command and get some kind of output like 'consistency check complete 1.2TB of orphaned image files removed' or something concrete like that. I guess this is Netbackup and that means a mix between art and science :)
06-05-2014 09:12 AM
The thing with de-dupe is that it is not like normal disk....
If you expire 2TB of images that were de-duped at a rate or 98% then you will only actually free up 40GB of disk space!!!!
The best way is to keep processing the queue after you have done a big expiry (that does run at midday [and midnight]) - it actually reduces space as it runs and then starts to get it back again - and if you keep running it it also triggers the other collectors for you.
06-05-2014 09:16 AM
yes I did run the processqueue jobs and it took a while to complete but I will try again over the next few days. Space is still clearing up. I have been monitoring it every hour and it is freeing up around 180GB/hour over the past 2 hours....may have just needed a kick in the plums!
06-09-2014 01:15 AM
Thanks for the help Mark. It seems that forcing the queue processing task did the trick and I have even more free space after thee weekend's full backups.
The command that did the trick was: crcontrol --processqueue
It was running for some time clearing up space. The 2 commands that I used to monitor the progress were:
crcontrol --processqueueinfo
crcontrol --processqueuestatus