Forum Discussion

quebek's avatar
quebek
Moderator
26 days ago

MSDP 2 MSDP duplication

I am participating in a project where I have to free up a MSDP storage server (running on NBA 5350). Backup images which are stored on this MSDP are duplicated (optimized duplication) to other NBA, within the same NBU domain, this time 5240. Both appliances are running 5.3.0.1 SP1 MR2, and have the same patch installed: NBAPP_EEB_ET4145974-5.3.0.1-6.x86_64.

After initially seeding this target appliance 5240 with few days of backups, I cut over the backups directed to 5350 and repointed these to 5240 - played around with the SLPs. 

The problem that was faced is that source MSDP on 5350 was occupied ca. 120 TB. Now after a week after backups are cut off and not having all backup images being duplicated to target 5240 MSDP, that  target pool is occupied in 174 TB. We are running out of disk space on target MSDP, no one knows why we see such an increase in the disk occupancy. IMHO target MSDP should not occupy more than - let's say - 130 TB. 
Do you have any idea as to why we see such discrepancy from our anticipation and real world?

appliance5240
MSDP usage
NETBACKUP 10.3
  • kiyan's avatar
    kiyan
    Level 5
    26 days ago

    not sure if it could be related to your issue,but I faced an issue for a customer, the MSDP was growing unusually near  full after upgraded to 10.3(within two months), after checking the SLP it turned out he has disabled SLP second operations ( tape duplication) but obviously the SLP was keeping all those images in the backlog and was not deleting them.
    And I came to know there a new feature which is enabled in Netbackup 10.3 regarding backup images expiration
    https://www.veritas.com/support/en_US/article.100063195

     

    • quebek's avatar
      quebek
      Moderator
      25 days ago

      Hey.

      Thanks for your input. I am aware of this new future in 10.3, learned during this exercise 😉.

      All our SLPs are being active.

  • X2's avatar
    X2
    Moderator
    25 days ago

    quebek  I am almost sure you have checked this but still thought of bringing it up. Have you tried manually reclaiming the space in the 5240? As the 5240 was being "filled up" initially, it's dedupe ratio would be very low. As data was added, the ratio will increase. That means there might be a need to do GC more frequently than in normal conditions. So, try doing it manually - https://www.veritas.com/support/en_US/article.100000714

  • quebek's avatar
    quebek
    Moderator
    23 days ago

    Of course I did, couple of times. We do have case opened awaiting SME to step in.

    On Friday I did point back the SLP to the 5350 as 5240 ran out of disk space, and during whole weekend the disk occupancy on 5350 was +5 TB while on 5240 from Thu to Fri during night it occupied +22 TB. Weird.

    • chmeyerweck's avatar
      chmeyerweck
      Level 3
      20 days ago

      Hi,

      Since you already have a case open, I'm sure you checked all the basic stuff but can you post the output from "crcontrol --dsstats" from both old and new NBA?

      • quebek's avatar
        quebek
        Moderator
        19 days ago

        Hey

        From 5240:

        /usr/openv/pdde/pdcr/bin/crcontrol --dsstat

        ************ Data Store statistics ************

        Data storage      Raw         Size        Used        Avail       Use%      Free%

                          195.70TiB   185.89TiB   157.00TiB   28.89TiB    85%       15.5%

        Number of containers                : 649972

        Average container size              : 264889745 bytes (252.62MiB)

        Space allocated for containers      : 172170917442951 bytes (156.59TiB)

        Reserved space                      : 9.81TiB (5.0%)

        Reserved space for cloud cache      : 0.00B (0.0%)

        Reserved space for vpfs cloud cache : 0.00B (0.0%)

         

        From 5350:

         /usr/openv/pdde/pdcr/bin/crcontrol --dsstat

        ************ Data Store statistics ************

        Data storage      Raw         Size        Used        Avail       Use%      Free%

                          179.47TiB   170.67TiB   112.43TiB   58.24TiB    66%       34.1%

        Number of containers                : 2092893

        Average container size              : 59116103 bytes (56.38MiB)

        Space allocated for containers      : 123723680230881 bytes (112.53TiB)

        Reserved space                      : 8.80TiB (4.9%)

        Reserved space for cloud cache      : 0.00B (0.0%)

        Reserved space for vpfs cloud cache : 0.00B (0.0%)

  • bhdrkzltn's avatar
    bhdrkzltn
    Level 4
    18 days ago

    Please compare the output on both storage servers generated by the following command.

    catdbutil --count --real_po

    • quebek's avatar
      quebek
      Moderator
      18 days ago

      Hey

      Well these two are not keeping the same data, as the backups are still being duplicated and production was moved to the other one. In other words this is not a mirror...