DSU disk show low utilization percentage and it is near full!

Hi Mark,

 thanks for the link.

I followed suggestion, and the result continue to keep me in odd state, becasue after ran

i have this new state on PureDisk storage:

[root@ph469702]/usr/openv/pdde/pdcr/bin# nbdevquery -U -listdv -stype PureDisk
Disk Pool Name      : MG_PUREDISK
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
Total Capacity (GB) : 10074.11
-->Free Space (GB)     : 3205.44
-->Use%                : 68
 

But very little changes on the disk itself:

/dev/mapper/VolGroup02-dsunew
                      9.9T  8.9T  934G  91% /DSU
 

This is something i can't understand why.

Process i followed was:

crcontrol --dsstat  (to verify)

crcontrol --compactstate   (OK)

crcollect -v -m +1,+2  (20minutes run for 24million records)

crcontrol --processqueueinfo   (answer NO)

crcontrol --processqueue  (OK).

At the end, again

"bpimage -cleanup -allclients" and "nbdelete -allvolumes -force"

The outcome was what you see.

That is pretty the same i was starting from.

But the space to be claimed seems to be more than 934G...

Thanks.

OK - always hard work getting everything just right with these!!

nbdelete should be first followed by bpimage, then the PureDisk commands ...

So first try running the process queue a few times to get the queues cleared right down (they will queue up each time you run it and the queue will get less and less each time but probably need 3 or 4 runs to clear it right down) - keep checking with queueinfo to see how big it is and make sure it is reducing.

Next check the rebase state ( crcollect –rebasestate ) this should be ON

Next run the garbage collection again

Once that is complete do a dsstat -1

Finally wait and see how it looks tomorrow - i have the feeling it does other stuff and not fully sure when NetBackup will pass all of the information to PureDisk / MDSP to do its work

So tomorrow do a dsstat -1 and check the volume usage again.

Let me know the results

Hi Mark, all,

 i did all the steps you suggested me.

I see infact some good result on the puredisk utilization:

[root@ph469702]/usr/openv/pdde/pdcr/bin# nbdevquery -U -listdv -stype PureDisk
Disk Pool Name      : MG_PUREDISK
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
Total Capacity (GB) : 10074.11
->Free Space (GB)     : 3105.03
->Use%                : 69
 

Queue processed and collect done too.

But, after one day, no real space was released to the OS:

/dev/mapper/VolGroup02-dsunew
--->          9.9T  8.8T  1.1T  90% /DSU
 

I've lost around 2T , looking at the two data.

So, it is something i can't figure out, why i can't have this space released, if the space showed on diskpool capacity is right then!

*********** Data Store statistics ************
Data storage       Size   Used  Avail Use%
                   9.8T   5.8T   4.0T  59%
Number of containers             : 38226
Average container size           : 247687453 bytes (236.21MB)
Space allocated for containers   : 9468100583882 bytes (8.61TB)
Space used within containers     : 7047589274259 bytes (6.41TB)
Space available within containers: 2420511309623 bytes (2.20TB)
Space needs compaction           : 744576029770 bytes (693.44GB)
Records marked for compaction    : 784579
Active records                   : 18108612
Total records                    : 18893191
 

Thanks!

Bye

I think this is down to the containers .. you should be fairly cleaned up now but the cleanup just leaves holes in the containers ready for new data - but the containers themselves still occupy the physical disk space on the server - so a df shows not much space available as it it full of containers, but a dsstat shows plenty of space as the PureDisk component knows that there are plenty of partially empty containers to put data in.

The containers are 256MB and stay at that size on the file system even when they are emptied

Should have thought of that in the first place! But you should have a nicely running system now!

Hope this helps and makes sense!

It makes sense for me, but for what i need (shut down alerting) i need to stay some dot under 90% of busy disk (OS realted, df let's say).

Thank you.

Regards,

Not sure if you can do it - you have handed that disk over to de-dupe and it will use it!

You could drop the high watermark on the disk pool to say 80% and then see if that processes things down a bit over a few days but if you are not running capacity managed it would just give you disk full conditions

Maybe a rebase will help - not sure how often that runs

crcontrol --rebasestate

will show if it is turned on and currently running

Have you also compacted? (crcontrol –compactstart) to see if that helps?

So queue processing three time - leave it for a while as garbage collection will then run - then force a compact

See if this helps, but not 100% you will get back what you need

Mark,

 i've started crcontrol --compactstart but it seems to be not helpful.

While, talking of the first command:  crcontrol --rebasestate, this is not the first time you mention it, but it seems to be not recognized by this version of NBU (7.0.1) and PDDE (CR# version6.6.0.35883).

thank you.
 

AH! pure PureDisk - sorry!

So this should be the same as a 5020 type appliance ...

This is the reccomended process for one of those to clean it up ....

1. Make sure no backups start while carrying out the following steps:
2. Log on to the SPA node via webgui as the root user and password, and traverse to the manage Policies section and initiate the following maintenance policies in the same order:
a) PDDO Data removal
b) MB Garbage Collection
c) CR QUEUE PROCESSING & Garbage Collection

Run CR Queue Processing & Garbage collection twice

3. Log on to the SPA using a terminal session and perform manually compact the storage containers:
If compaction is "on" the process is monitored and will automatically occur if the compaction criteria are met.
To start compaction right away run:
/opt/pdcr/bin/crcontrol--compacton
/opt/pdcr/bin/crcontrol--compactstart

See if this helps

Hi Mark,

  the story seems to be very complex!

Yesterday, before performing action on CR process queue and Garbage collecting, i duplicated two big images, aroung 300G to another media, then expired the first copy (i moved as Primary).

So what you suggested me lastly (that is the same present into TECH124914), do not released any space.

Today, magically let's say, i've found 300G free:

/dev/mapper/VolGroup02-dsunew
                      9.9T  8.5T  1.4T  87% /DSU

(it was 1.1T free).

So it seems that NBU perform operations on the background and we are not noticed for them.

Anyway what remain, for me, a big dark hole is the space report and puredisk report that both claims to have around 3T free space, and i can't figure out where they are!

nbdevquery -U -listdv -stype PureDisk
Disk Pool Name      : MG_PUREDISK
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
Total Capacity (GB) : 10074.11
>>Free Space (GB)     : 2883.64<<
Use%                : 71
Status              : UP
 

Thank you.

Regards,