Forum Discussion

mpiatek39's avatar
mpiatek39
Level 3
9 years ago

MSDP 7.6 data removal

Hello

We have MSDP 7.6.0.3 on RHEL 6.5 and we have a problem with space recalamation.

Current front end data is apparently no more than 3TB,  images were expired in catalog, however, the logical catalog space and space usage on disk is not decreasing.

We could not identify any obvious reason, we have no errors in image cleanup jobs, no errors in storaged.log, queue processing is performed daily, compaction state is on, data integrity check is enabled with 30 days parameter. As it seems some containers get cleaned --> as if the space reclamation worked for images currently backed up but we can not clear the expired ones.

We ran nbdelete -allvolumes -force several times.

No incidents or errors on the MSDP host server either.

Could anyone advise what checks we could still perform? Any ideas of what may be causing this issue? I am awaiting to have the necessary authorizations to open a support case but anyway I would be grateful for this forum`s users opinion.

/usr/openv/pdde/pdcr/bin/crstats --convert-size
Info [139665016317728]: set log options successfully.
Info [139665016317728]: collecting statistics data ...
Info [139665016317728]: sessionStartAgent: Server is Version 8.0003.0014.053, Protocol Version 6.6.1.1
Info [139665016317728]: collecting statistics data was complete
Storage Pool Raw Size=9.6TB
Storage Pool Reserved Space=394.7GB
Storage Pool Size=9.2TB
Storage Pool Used Space=7.1TB
Storage Pool Available Space=2.1TB
Catalog Logical Size=66.0TB
Catalog files Count=8547
Space Allocated For Containers=6.6TB
Deduplication Ratio=10.0
 

/usr/openv/pdde/pdcr/bin/crcontrol --dsstat 1

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   9.6T   9.2T   7.1T   2.1T  77%

Number of containers             : 34619
Average container size           : 209719502 bytes (200.00MB)
Space allocated for containers   : 7260279451996 bytes (6.60TB)
Reserved space                   : 423830069248 bytes (394.72GB)
Reserved space percentage        : 4.0%

7.6
Backup and Recovery
Linux
NetBackup
  • mpiatek39's avatar
    mpiatek39
    8 years ago

    Finally case was resolved,  the root cause of the issue was the orphaned images from  that had PO references in the internal deduplication database to DCID 1, which is not a valid DCID for valid images.  The PoGather script was creating an cleanup script which was not working because DCID 1 was not valid.  So a special cleanup script was created  which cleaned up the orphaned images. Then the remaining piece was to correct the values in the __info__.2 file which resides in /data1/databases/catalog/.  These values are used in calculating the statistics for the crstats command.  After correcting those values, we see the correct information in the storage server properties and the crstats output. Great work of support!

     

6 Replies

Sort By
Replies have been turned off for this discussion
  • RiaanBadenhorst's avatar
    RiaanBadenhorst
    Level 6
    9 years ago

    Hello,

     

    There is a tool called POgather that you can get from NetBackup. It will gather information from the MSDP and determine if there are orphaned images. Support will give you a script to clean up the orphans once you've uploaded the output of the POgather.

     

    You'll need to open a support case.

  • mpiatek39's avatar
    mpiatek39
    Level 3
    9 years ago

    Hello,

    Thanks very much for your reply, looks optimistic, unfortunately I am still waiting for my authorizations (the server is not production so we can not really escalate). I will let you know when I have worked with support.

  • mpiatek39's avatar
    mpiatek39
    Level 3
    9 years ago

    Support resolved my case, as you wrote they send me POgather script and then a script to clear orphaned images. Space was cleared from disk but there is an inconsistency with logical catalog size:

    space usage

                          9.7T   15G  9.2T   1% /data1

     

    ************ Data Store statistics ************

    Data storage      Raw    Size   Used   Avail  Use%

                       9.6T   9.2T 514.3G   8.7T   6%

     

    Number of containers             : 63

    Average container size           : 131878198 bytes (125.77MB)

    Space allocated for containers   : 8308326531 bytes (7.74GB)

    Reserved space                   : 423824699392 bytes (394.72GB)

    Reserved space percentage        : 4.0%

    Storage Pool Raw Size=9.6TB

    Storage Pool Reserved Space=394.7GB

    Storage Pool Size=9.2TB

    Storage Pool Used Space=514.3GB

    Storage Pool Available Space=8.7TB

    Catalog Logical Size=65.8TB

    Catalog files Count=8521

    Space Allocated For Containers=7.7GB

    Deduplication Ratio=8709.1

    We do not find values explaining 65.8 TB anywhere (front end data report, disk images reports, client backup reports). I opened a new case to understand how it works.

    • mpiatek39's avatar
      mpiatek39
      Level 3
      8 years ago

      The case is still open, in fact the pogather script still shows the same orphaned images even if the space on disk was cleared. I will let you know the solution.

      • mpiatek39's avatar
        mpiatek39
        Level 3
        8 years ago

        Finally case was resolved,  the root cause of the issue was the orphaned images from  that had PO references in the internal deduplication database to DCID 1, which is not a valid DCID for valid images.  The PoGather script was creating an cleanup script which was not working because DCID 1 was not valid.  So a special cleanup script was created  which cleaned up the orphaned images. Then the remaining piece was to correct the values in the __info__.2 file which resides in /data1/databases/catalog/.  These values are used in calculating the statistics for the crstats command.  After correcting those values, we see the correct information in the storage server properties and the crstats output. Great work of support!