cancel
Showing results for 
Search instead for 
Did you mean: 

MSDP 7.6 data removal

mpiatek39
Level 3

Hello

We have MSDP 7.6.0.3 on RHEL 6.5 and we have a problem with space recalamation.

Current front end data is apparently no more than 3TB,  images were expired in catalog, however, the logical catalog space and space usage on disk is not decreasing.

We could not identify any obvious reason, we have no errors in image cleanup jobs, no errors in storaged.log, queue processing is performed daily, compaction state is on, data integrity check is enabled with 30 days parameter. As it seems some containers get cleaned --> as if the space reclamation worked for images currently backed up but we can not clear the expired ones.

We ran nbdelete -allvolumes -force several times.

No incidents or errors on the MSDP host server either.

Could anyone advise what checks we could still perform? Any ideas of what may be causing this issue? I am awaiting to have the necessary authorizations to open a support case but anyway I would be grateful for this forum`s users opinion.

/usr/openv/pdde/pdcr/bin/crstats --convert-size
Info [139665016317728]: set log options successfully.
Info [139665016317728]: collecting statistics data ...
Info [139665016317728]: sessionStartAgent: Server is Version 8.0003.0014.053, Protocol Version 6.6.1.1
Info [139665016317728]: collecting statistics data was complete
Storage Pool Raw Size=9.6TB
Storage Pool Reserved Space=394.7GB
Storage Pool Size=9.2TB
Storage Pool Used Space=7.1TB
Storage Pool Available Space=2.1TB
Catalog Logical Size=66.0TB
Catalog files Count=8547
Space Allocated For Containers=6.6TB
Deduplication Ratio=10.0
 

/usr/openv/pdde/pdcr/bin/crcontrol --dsstat 1

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   9.6T   9.2T   7.1T   2.1T  77%

Number of containers             : 34619
Average container size           : 209719502 bytes (200.00MB)
Space allocated for containers   : 7260279451996 bytes (6.60TB)
Reserved space                   : 423830069248 bytes (394.72GB)
Reserved space percentage        : 4.0%

1 ACCEPTED SOLUTION

Accepted Solutions

Finally case was resolved,  the root cause of the issue was the orphaned images from  that had PO references in the internal deduplication database to DCID 1, which is not a valid DCID for valid images.  The PoGather script was creating an cleanup script which was not working because DCID 1 was not valid.  So a special cleanup script was created  which cleaned up the orphaned images. Then the remaining piece was to correct the values in the __info__.2 file which resides in /data1/databases/catalog/.  These values are used in calculating the statistics for the crstats command.  After correcting those values, we see the correct information in the storage server properties and the crstats output. Great work of support!

 

View solution in original post

6 REPLIES 6

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

Hello,

 

There is a tool called POgather that you can get from NetBackup. It will gather information from the MSDP and determine if there are orphaned images. Support will give you a script to clean up the orphans once you've uploaded the output of the POgather.

 

You'll need to open a support case.

mpiatek39
Level 3

Hello,

Thanks very much for your reply, looks optimistic, unfortunately I am still waiting for my authorizations (the server is not production so we can not really escalate). I will let you know when I have worked with support.

mpiatek39
Level 3

Support resolved my case, as you wrote they send me POgather script and then a script to clear orphaned images. Space was cleared from disk but there is an inconsistency with logical catalog size:

space usage

                      9.7T   15G  9.2T   1% /data1

 

************ Data Store statistics ************

Data storage      Raw    Size   Used   Avail  Use%

                   9.6T   9.2T 514.3G   8.7T   6%

 

Number of containers             : 63

Average container size           : 131878198 bytes (125.77MB)

Space allocated for containers   : 8308326531 bytes (7.74GB)

Reserved space                   : 423824699392 bytes (394.72GB)

Reserved space percentage        : 4.0%

Storage Pool Raw Size=9.6TB

Storage Pool Reserved Space=394.7GB

Storage Pool Size=9.2TB

Storage Pool Used Space=514.3GB

Storage Pool Available Space=8.7TB

Catalog Logical Size=65.8TB

Catalog files Count=8521

Space Allocated For Containers=7.7GB

Deduplication Ratio=8709.1

We do not find values explaining 65.8 TB anywhere (front end data report, disk images reports, client backup reports). I opened a new case to understand how it works.

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

OK, please share those results as well when resolved.

The case is still open, in fact the pogather script still shows the same orphaned images even if the space on disk was cleared. I will let you know the solution.

Finally case was resolved,  the root cause of the issue was the orphaned images from  that had PO references in the internal deduplication database to DCID 1, which is not a valid DCID for valid images.  The PoGather script was creating an cleanup script which was not working because DCID 1 was not valid.  So a special cleanup script was created  which cleaned up the orphaned images. Then the remaining piece was to correct the values in the __info__.2 file which resides in /data1/databases/catalog/.  These values are used in calculating the statistics for the crstats command.  After correcting those values, we see the correct information in the storage server properties and the crstats output. Great work of support!