Netbackup 7.1.0.4 - MSDP storaged CRITICAL ERROR

OS?

error says no space. Verify MSDP storage have atleast 10% free space left  if not get it done first.

OS
SLES 11 SP1 (Media Server + MSDP)

Some additional info about space allocation:

DF
nbme3:~ # df -h /nbustage1
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg01-lvol1
                      4.9T  4.6T   76G  99% /nbustage1

CRCONTROL
nbme3:~ # crcontrol --dsstat

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   4.8T   4.6T   4.6T   0.0M 100%

Number of containers             : 19739
Average container size           : 246359744 bytes (234.95MB)
Space allocated for containers   : 4862894990815 bytes (4.42TB)
Space used within containers     : 4846993964232 bytes (4.41TB)
Space available within containers: 15901026583 bytes (14.81GB)
Space needs compaction           : 193103422413 bytes (179.84GB)
Reserved space                   : 299208470528 bytes (278.66GB)
Reserved space percentage        : 5.6%
Records marked for compaction    : 4166145
Active records                   : 86007957
Total records                    : 90174102

Use "--dsstat 1" to get more accurate statistics

For your information yesterday the volume was completley full.

So I copied the database folder to a different mount point and I create a symbolik link in the original location (in my environment is /nbustage1).

thks

mic

nbme3:~ # crcontrol --queueinfo
total queue size : 621840462
creation date of oldest tlog : Tue Jun 26 02:50:59 2012

nbme3:~ # crcontrol --compactstate
Data store compaction: ON, DeleteSpaceThreshold: 30%, CompactLBound: 4MB
Compaction busy: No
nbme3:~ # crcontrol --compacton
Data store compaction turned on
nbme3:~ # crcontrol --compactstart
Data store compaction started
nbme3:~ # crcontrol --processqueueinfo
Busy   : no
Pending: no
nbme3:~ # crcontrol --processqueueinfo
Busy   : no
Pending: no
nbme3:~ #

now check with dsstat 1 . does it show reduction in storage space used by MSDP

Unfortunately no reduction.

nbme3:/nbustage1/log/spoold # crcontrol --dsstat 1

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   4.8T   4.6T   4.6T   0.0M 100%

Number of containers             : 19739
Average container size           : 237690565 bytes (226.68MB)
Space allocated for containers   : 4691774081947 bytes (4.27TB)
Space used within containers     : 4664571389123 bytes (4.24TB)
Space available within containers: 27202692824 bytes (25.33GB)
Space needs compaction           : 164865344893 bytes (153.54GB)
Reserved space                   : 299208474624 bytes (278.66GB)
Reserved space percentage        : 5.6%
Records marked for compaction    : 3637979
Active records                   : 82596678
Total records                    : 86234657

could you perform reboot of MSDP server. if still doesn't work .time to call Support on this. we had  similar cases where reboot did freed space to perform MSDP maintenance but if not then support is  the best choice to get assistance

some space was freed. Is it a good news?

Thanks

crcontrol --dsstat

************ Data Store statistics ************
Data storage      Raw    Size   Used   Avail  Use%
                   4.8T   4.6T   4.4T 139.6G  97%

Number of containers             : 19739
Average container size           : 236416977 bytes (225.46MB)
Space allocated for containers   : 4666634712274 bytes (4.24TB)
Space used within containers     : 4639130276099 bytes (4.22TB)
Space available within containers: 27504436175 bytes (25.62GB)
Space needs compaction           : 139424231869 bytes (129.85GB)
Reserved space                   : 295743315968 bytes (275.43GB)
Reserved space percentage        : 5.6%
Records marked for compaction    : 2996688
Active records                   : 82596678
Total records                    : 85593366

Use "--dsstat 1" to get more accurate statistics

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg01-lvol1
                      4.9T  4.4T  260G  95% /nbustage1
 

as you have got free space. now start processqueue , start compaction and check if you get further reduction

this is time consuming process. so it might not reflect immediately so perform it time interval

At this moment the situation on the MSDP we wrote about yesterday, nbme3, is ok.

The only problem is that also my other MSDP server has the same problems.
I will apply the to-do-list above and I will update you.

0. Got first out of space error.
1. Moved the databases dir to a different file system.
2. Created a symlink to new location.
3. Got a second out-of-space error with scary message in storaged.log.
4. Exectued ervy command you told me:
 4.1 crcontrol --compacton
 4.2 crcontrol --compactstart
 4.3 crcontrol --processqueueinfo
5. Verified erything it's working fine.
6. Resetted the default *SpaceThreshold in contentrouter.cfg. (yes I know I shouldn't have to touch this but unfortunately I did it).
7. Removed the reserved blocks with tune2fs on the msdp file system.

At this moment everything is ok.

Many many thanks.

Michele

LOG

All my MSDP/PDDE logs don't show any error.

DF