cancel
Showing results for 
Search instead for 
Did you mean: 

DSU disk show low utilization percentage and it is near full!

Michele_Nicosia
Level 5

Hi all,

   thanks in advance for any support on my issue.

I'm using 3 media servers on RHEL5.6 which are running fine.

Each with /DSU disk with 10T.

Here the fact:

[root@ph469702]/usr/openv/netbackup/db/config# nbdevquery -U -listdv -stype PureDisk
Disk Pool Name      : MG_PUREDISK
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
->Total Capacity (GB) : 10074.11   <----
->Free Space (GB)     : 2701.75   <----
->Use%                : 73   <------
Status              : UP
Flag                : ReadOnWrite
Flag                : AdminUp
Flag                : InternalUp
Num Read Mounts     : 0
Num Write Mounts    : 1
Cur Read Streams    : 0
Cur Write Streams   : 0
 

Here the df output:

/dev/mapper/VolGroup02-dsunew
-->                 9.9T  8.9T  891G  92% /DSU  <--
 

Here the dir hierarchy:

-> /DSU/data           9001G  <--

/DSU/databases  5871M

/DSU/etc/            46K

/DSU/history       32G

/DSU/log             252G

/DSU/queue        125M

/DSU/tmp and /DSU/var very low.

So, exist something i can do free up some space on this scenario?

I've executed both these, several times in the few days, but no result:

bpimage -cleanup -allclients

nbdelete -allvolumes -force

 

Thank  you.

 

Regards,

 

Michele

1 ACCEPTED SOLUTION

Accepted Solutions

Mark_Solutions
Level 6
Partner Accredited Certified

After NetBackup has done its expiry and cleanup it passes that information to the puredisk component (MSDP)

This then occasionally does several things - it runs queue processing to keep itself neat and tidy, it runs rebasing which is like a de-dupe defrag and it runs garbage collection to actually delete the data held within the dedupe

If you feel that there is wasted space that nees to be cleaned up you can run the garbage collection manually (as it doesnt run very often)

Here is a link about reclaiming space or your MSDP pool:

http://www.symantec.com/docs/TECH124914

Hope this helps

View solution in original post

11 REPLIES 11

Mark_Solutions
Level 6
Partner Accredited Certified

After NetBackup has done its expiry and cleanup it passes that information to the puredisk component (MSDP)

This then occasionally does several things - it runs queue processing to keep itself neat and tidy, it runs rebasing which is like a de-dupe defrag and it runs garbage collection to actually delete the data held within the dedupe

If you feel that there is wasted space that nees to be cleaned up you can run the garbage collection manually (as it doesnt run very often)

Here is a link about reclaiming space or your MSDP pool:

http://www.symantec.com/docs/TECH124914

Hope this helps

Michele_Nicosia
Level 5

Hi Mark,

 thanks for the link.

I followed suggestion, and the result continue to keep me in odd state, becasue after ran

 

i have this new state on PureDisk storage:

[root@ph469702]/usr/openv/pdde/pdcr/bin# nbdevquery -U -listdv -stype PureDisk
Disk Pool Name      : MG_PUREDISK
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
Total Capacity (GB) : 10074.11
-->Free Space (GB)     : 3205.44
-->Use%                : 68
 

But very little changes on the disk itself:

/dev/mapper/VolGroup02-dsunew
                      9.9T  8.9T  934G  91% /DSU
 

This is something i can't understand why.

 

Process i followed was:

crcontrol --dsstat  (to verify)

crcontrol --compactstate   (OK)

crcollect -v -m +1,+2  (20minutes run for 24million records)

crcontrol --processqueueinfo   (answer NO)

crcontrol --processqueue  (OK).

At the end, again

"bpimage -cleanup -allclients" and "nbdelete -allvolumes -force"

 

The outcome was what you see.

That is pretty the same i was starting from.

But the space to be claimed seems to be more than 934G...

 

Thanks.

Mark_Solutions
Level 6
Partner Accredited Certified

OK - always hard work getting everything just right with these!!

nbdelete should be first followed by bpimage, then the PureDisk commands ...

So first try running the process queue a few times to get the queues cleared right down (they will queue up each time you run it and the queue will get less and less each time but probably need 3 or 4 runs to clear it right down) - keep checking with queueinfo to see how big it is and make sure it is reducing.

Next check the rebase state ( crcollect –rebasestate ) this should be ON

Next run the garbage collection again

Once that is complete do a dsstat -1

Finally wait and see how it looks tomorrow - i have the feeling it does other stuff and not fully sure when NetBackup will pass all of the information to PureDisk / MDSP to do its work

So tomorrow do a dsstat -1 and check the volume usage again.

Let me know the results

Michele_Nicosia
Level 5

Hi Mark, all,

 i did all the steps you suggested me.

I see infact some good result on the puredisk utilization:

 

[root@ph469702]/usr/openv/pdde/pdcr/bin# nbdevquery -U -listdv -stype PureDisk
Disk Pool Name      : MG_PUREDISK
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
Total Capacity (GB) : 10074.11
->Free Space (GB)     : 3105.03
->Use%                : 69
 

Queue processed and collect done too.

But, after one day, no real space was released to the OS:

/dev/mapper/VolGroup02-dsunew
--->          9.9T  8.8T  1.1T  90% /DSU
 

I've lost around 2T , looking at the two data.

So, it is something i can't figure out, why i can't have this space released, if the space showed on diskpool capacity is right then!

 

*********** Data Store statistics ************
Data storage       Size   Used  Avail Use%
                   9.8T   5.8T   4.0T  59%
Number of containers             : 38226
Average container size           : 247687453 bytes (236.21MB)
Space allocated for containers   : 9468100583882 bytes (8.61TB)
Space used within containers     : 7047589274259 bytes (6.41TB)
Space available within containers: 2420511309623 bytes (2.20TB)
Space needs compaction           : 744576029770 bytes (693.44GB)
Records marked for compaction    : 784579
Active records                   : 18108612
Total records                    : 18893191
 

Thanks!

 

Bye

 

Mark_Solutions
Level 6
Partner Accredited Certified

I think this is down to the containers .. you should be fairly cleaned up now but the cleanup just leaves holes in the containers ready for new data - but the containers themselves still occupy the physical disk space on the server - so a df shows not much space available as it it full of containers, but a dsstat shows plenty of space as the PureDisk component knows that there are plenty of partially empty containers to put data in.

The containers are 256MB and stay at that size on the file system even when they are emptied

Should have thought of that in the first place! But you should have a nicely running system now!

Hope this helps and makes sense!

 

 

Michele_Nicosia
Level 5

It makes sense for me, but for what i need (shut down alerting) i need to stay some dot under 90% of busy disk (OS realted, df let's say).

Thank you.

 

Regards,

 

 

Mark_Solutions
Level 6
Partner Accredited Certified

Not sure if you can do it - you have handed that disk over to de-dupe and it will use it!

You could drop the high watermark on the disk pool to say 80% and then see if that processes things down a bit over a few days but if you are not running capacity managed it would just give you disk full conditions

Maybe a rebase will help - not sure how often that runs

crcontrol --rebasestate

will show if it is turned on and currently running

Have you also compacted? (crcontrol –compactstart) to see if that helps?

So queue processing three time - leave it for a while as garbage collection will then run - then force a compact

See if this helps, but not 100% you will get back what you need

Michele_Nicosia
Level 5

Mark,

 i've started crcontrol --compactstart but it seems to be not helpful.

While, talking of the first command:  crcontrol --rebasestate, this is not the first time you mention it, but it seems to be not recognized by this version of NBU (7.0.1) and PDDE (CR# version6.6.0.35883).

thank you.
 

Mark_Solutions
Level 6
Partner Accredited Certified

AH! pure PureDisk - sorry!

So this should be the same as a 5020 type appliance ...

This is the reccomended process for one of those to clean it up ....

1. Make sure no backups start while carrying out the following steps:
2. Log on to the SPA node via webgui as the root user and password, and traverse to the manage Policies section and initiate the following maintenance policies in the same order:
a) PDDO Data removal
b) MB Garbage Collection
c) CR QUEUE PROCESSING & Garbage Collection


Run CR Queue Processing & Garbage collection twice


3. Log on to the SPA using a terminal session and perform manually compact the storage containers:
If compaction is "on" the process is monitored and will automatically occur if the compaction criteria are met.
To start compaction right away run:
/opt/pdcr/bin/crcontrol--compacton
/opt/pdcr/bin/crcontrol--compactstart

 

See if this helps

Michele_Nicosia
Level 5

Hi Mark,

  the story seems to be very complex!

Yesterday, before performing action on CR process queue and Garbage collecting, i duplicated two big images, aroung 300G to another media, then expired the first copy (i moved as Primary).

So what you suggested me lastly (that is the same present into TECH124914), do not released any space.

Today, magically let's say, i've found 300G free:

/dev/mapper/VolGroup02-dsunew
                      9.9T  8.5T  1.4T  87% /DSU

(it was 1.1T free).

So it seems that NBU perform operations on the background and we are not noticed for them.

Anyway what remain, for me, a big dark hole is the space report and puredisk report that both claims to have around 3T free space, and i can't figure out where they are!

nbdevquery -U -listdv -stype PureDisk
Disk Pool Name      : MG_PUREDISK
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
Total Capacity (GB) : 10074.11
>>Free Space (GB)     : 2883.64<<
Use%                : 71
Status              : UP
 

Thank you.

Regards,

Mark_Solutions
Level 6
Partner Accredited Certified

A df will not match with a PureDisk report due to the containers that i mentioned previosly

Think of them if you like as dynamic or thin provisioned disks - the OS sees them as 256MB each even though each of them may not have that much data inside them

So a df says you have little disk space left due to all there 256MB files - but PureDisk knows that many of them have free space so knows that it has 2TB of holes in these containers that it can write data to

Not sure how you would get over this although they can be compacted / cleaned up to a point but will no doubt grow or add more again in the future so not sure how disk space alerts will work for you to e honest - just the way it works!