Netbackup 7.6.0.1 - crcollect missing

[ChrisRobertson]

Hi All,

I am troubleshooting some issues where my disk storage is filling up and following this document:

http://www.symantec.com/business/support/index?page=content&id=TECH124914

I want to run the garbage collection but have found that crcollect is missing from the pdde folder. crcontrol.exe is there though. I can find a version of crcollect that appears to have been backed up during an update. I think this may be what is causing the storage to fill up.

Info:

Win 2008 R2 Server

Single Master server with 16TB storage LUN (iSCSI)

~6TB Front end data to back up (licenced on a front end TB basis including deduplication)

2 months retention on disk

I have had the infrastructure up and running for around 6 months now but over the past 2 months the data store has started to fill. It was around 9TB used but now I get warnings that is is full (I do have 600GB reserved). So in short we had the first 2 months where space used grew and this was to be expected then the next 2 months where it remained reasonably constant which I also expected as I had hit the 2 month retention so data was aging out. But then it started growing again and has not stopped...

Monthly writes to tape - infinite retention on the tape using SLP's

Any help would be appreciated.

Regards

Chris

[ChrisRobertson]

Sorry, to add to this. I have expired a week's worth of images as I keep slightly more than the SLA requires (as you do) and then I processed the queue and it cleared up about 1TB of space

[Mark_Solutions]

No crcollect in 7.6.0.1 (and i have checked all of the cab files for it!)

Not in 7.6.0.2 either

crcontol is in data5.cab for both - so i guess it no longer exists - i will check and get back to you

[Mark_Solutions · Accepted solution]

Ah - replaced with Data Integrity Check  - similar thread here:

https://www-secure.symantec.com/connect/forums/garbage-data-integrity-check

[ChrisRobertson]

Ah OK, I have checked the link but it didn't have an explanation on how to run it manually. I will look into it and let you know if I find anything

Thanks Mark!

[ChrisRobertson]

This is what I have found so far:

http://www.symantec.com/business/support/index?page=content&id=HOWTO70644#v62952974

http://www.symantec.com/business/support/index?page=content&id=HOWTO70642#v62619765

I have run thr following commands from CLI:

C:\Program Files\Veritas\pdde>pddecfg -a getdataintegritycheck
Info: Data integrity check: ON
Info: Data integrity check busy: No
Info: Data integrity check current checked times: 30
Info: Data integrity check total check times    : 30

C:\Program Files\Veritas\pdde>pddecfg -a enabledataintegritycheck
Info: Enable data integrity check successfully.

C:\Program Files\Veritas\pdde>Crcontrol.exe --crccheckstate
CRC check     : ON
CRC check mode: Normal
CRC check busy: No
Current DCID  : 0
Max     DCID  : 0

Now the amount of free storage appears to be growing. I am not sure if it was my commands as you can see the checks were already enabled. In the past 10 minutes around 10GB of space has come available. I will let everything sit over night as I am almost up to 2TB free so tonight's backup should run through fine.

[Mark_Solutions]

So what time is it where you are?

These things do everything on a regular scheduled basis so it may be doing its regular daily, weekly, monthly run

The old way used to only actually once a month (or 30 days) based on install time or last run if i remember rightly

[ChrisRobertson]

Its 15:16, here in the UK. It is quite possible based on what I read (I see a lot of 'every 12 hours at 20 past the hour' in documentation)
 

I did want to force a garbage collection specifically though as I have a feeling I have orphaned files. For me to have 6TB of frontend and to be using 14TB to backup 2 months worth in dedupe seems like too much. The rate of change is not high enough to justify it. I guess as you say it may be running something triggered by the multitude of commands I have run today or maybe it's on a cycle but it would be nice to run a command and get some kind of output like 'consistency check complete 1.2TB of orphaned image files removed' or something concrete like that. I guess this is Netbackup and that means a mix between art and science :)

[Mark_Solutions]

The thing with de-dupe is that it is not like normal disk....

If you expire 2TB of images that were de-duped at a rate or 98% then you will only actually free up 40GB of disk space!!!!

The best way is to keep processing the queue after you have done a big expiry (that does run at midday [and midnight]) - it actually reduces space as it runs and then starts to get it back again - and if you keep running it it also triggers the other collectors for you.

[ChrisRobertson]

yes I did run the processqueue jobs and it took a while to complete but I will try again over the next few days. Space is still clearing up. I have been monitoring it every hour and it is freeing up around 180GB/hour over the past 2 hours....may have just needed a kick in the plums!

[ChrisRobertson]

Thanks for the help Mark. It seems that forcing the queue processing task did the trick and I have even more free space after thee weekend's full backups.

The command that did the trick was: crcontrol --processqueue

It was running for some time clearing up space. The 2 commands that I used to monitor the progress were:

crcontrol --processqueueinfo

crcontrol --processqueuestatus