PureDisk STU Full - Not reclaiming the space.

[MrNetBU]

Hi Everyone,

I need some help to fix my netbackup media server STU not reclaiming space. 

Netbackup Setup:

OS: Microsoft Windows 2008 R2

Netbackup Version : 7.1.0.4

Netbackup Master : 1 

Netbackup Media Server (Tape / Puredisk) : 6 

We are having problem with 1 of our puredisk media server.

Based on Disk Pool Status, Storage Pool is 99.89% Full.

However upon checking the Disk Reports > Images on Disk. Selecting concerned Storage STU to list all images, it shows no result or any images stored in it.

Hope anyone can help me to fix our issue. We need the Storage Unit to be avaiable again for backup.

Thanks.

[Mark_Solutions]

I am guessing that this is MSDP?

In any evernt, on that server run crcontrol --dsstat to see what the real space usage is

Also run crcontrol --queueinfo to see if you have a very large queue and how up to date it is

It may be that you just need to run some queue processing etc. to get some space reclaimed

The figures are always difficult to understand with PureDisk as it uses containers which don't shrink

Show us the outputs of the above 2 commands so that we can help further (and clarify if is is MSDP, Windows Unix, PDDO etc)

[MrNetBU]

Hi Mark,

Here's the output of the two commands.

D:\Program Files\Veritas\pdde>crcontrol --queueinfo
total queue size : 38983121497
creation date of oldest tlog : Mon Aug 05 11:13:55 2013

D:\Program Files\Veritas\pdde>crcontrol --dsstat

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                  14.4T  13.5T  13.5T   0.0M 100%

Number of containers             : 58972
Average container size           : 244524230 bytes (233.20MB)
Space allocated for containers   : 14420082918941 bytes (13.11TB)
Space used within containers     : 14311685546958 bytes (13.02TB)
Space available within containers: 108397371983 bytes (100.95GB)
Space needs compaction           : 0 bytes (0.00MB)
Reserved space                   : 999609622528 bytes (930.96GB)
Reserved space percentage        : 6.3%
Records marked for compaction    : 1630610
Active records                   : 132736063
Total records                    : 134366673

Use "--dsstat 1" to get more accurate statistics

D:\Program Files\Veritas\pdde>

[Mark_Solutions]

Oh dear!

A very old and large queue and a full system!!

You need to stop anything trying to use this disk (though i guess it wont work at the moment anyway!)

Next you need to try and get it cleared down but when this full is very difficult as it actually needs space to create space.

You will need to keep trying a combination of the following until you can get some space back (if you can expand the disk in some way to gain any free space that will make life easier or maybe delete some logs on the drive if you can see some old ones)

crcontrol --processqueue

start with running it twice and then monitor using --processqueueinfo. When none are running then run:

crcollect -v -m +1,+2

Keep on running the --processqueue command after the crcollect command has finished but i think that it will take you a while to get it all cleared down

While this is all happening take a look in NetBackup to see what is actually on this disk and if you have your retentions correct - if NetBackup has not expired anything then it cannot clear it down - but based on your queue size and how old it is that looks to be your issue as garbage collection wont have run.

Keep us updated

[MrNetBU]

Hi  Mark,

Can you advise which logs can be deleted safely? 

I'll be doing what you have suggested on Monday when i got back to work.

Thanks a lot.

[Mark_Solutions]

Before you do anything grab a copy of the storaged.log and spoold.log

In that same location are usually the older copies of those files - worth moving off rather than just deleting so that you have them to hand to help track down the cause of your issues (something is clearly wrong as your queue processing is so far behind - it may be a corrupt t-log that needs deleting before it can continue processing your queues

You will also find directories relating to the servers involved which sometimes have bptm logs etc, in them

Most logs roll over so move off old ones and keep the current ones - but all owrth keeping somewhere as you may need them

Worth logging a call with support anyway to help track down the corruption / error to allow yoru queue processing to start working again.

[rajeshthink]

Check out below two TN just for Puredisk

http://www.symantec.com/business/support/index?page=content&id=TECH180659

http://www.symantec.com/business/support/index?page=content&id=TECH124914

also if its configure as PDDO then first step is to do NBU cleanup

bpimage -cleanup -allclients

and then followed by below steps or policy in puredisk .

1)Data Managment Policy

2)MB Garbage Collector
3)CR Garbegae Collector
4)CR queue Processing