Disk storage unit is full, but it's not full (netbackup 10.1.1)

[Bessam]

Hello,

I have failed backups due to disk storage unit is full, but the problem is that there is enough space in diskpool and disk volume.

Diskpool is full at 80%, high water mark is set to 96%, in other FS there is enough space.

Netbackup services has been well restarted (spoold and spad running), master VM has been restarted too.

Error messageStorage Server Error: (Storage server: PureDisk:mastername) pdvfs_io_pwrite: Disk storage unit is full V-454-3 :

Storage Server Error: (Storage server: PureDisk:mastername) write_buf: Failed to write to spoold on storage server mastername(No space left on device). Ensure storage server services are running and operational. V-454-16

sts_close_handle failed: 2060031 out of space

cannot write image to disk, media close failed with status 2060031

Server is at the same time master and media (i'll still available if more informations are needed).

Master: VM

NBU: 10.1.1

Linux Redhat 8.6

[Nicolai]

what does a crcontrol --dsstat say ?

How to reclaim deduplication storage space manually

https://www.veritas.com/support/en_US/article.100000714

[Bessam]

I've already executed crcontrol commands (I forgot to mention it in the topic)

Here is the result of crcontrol --dsstat :

[root@mastername bin]# ./crcontrol --dsstat
Error: -1: sess flags: 2323857407765120008

************ Data Store statistics ************
Data storage      Raw     Size    Used    Avail   Use%    Free%
                  7.0T    6.7T    5.3T    1.4T    80%     20.6%

Number of containers             : 86739
Average container size           : 66752389 bytes (63.66MB)
Space allocated for containers   : 5790035505967 bytes (5.27TB)
Reserved space                   : 308046043136 bytes (286.89GB)
Reserved space percentage        : 4.0%
Reserved space for cloud cache   : 1.1T (15.6%)

Use "--dsstat 1" to get more accurate statistics
Use "--dsstat 2" to get statistics for each partition
Use "--dsstat 3" to get more accurate statistics for each partition

[StefanosM]

Run the same command with 2 as parameter.
crcontrol --dsstat 2
I had in the past problems with a very small partition that was full.

[Axelay91]

Hello,

It is full because you have probably configured a LSU cloud.

From the output, there is 1.1 TB reserved for cache cloud.

Reserved space for cloud cache   : 1.1T (15.6%)

[Bessam]

 @StefanosM : here is result of crcontrol --dsstat 2:

[root@xxxxx bin]# ./crcontrol --dsstat 2
Error: -1: sess flags: 2323857407765120008

************ Data Store statistics ************
Mount point count: 1
=============== Mount point   1 ===============
Path = /msdp/data

Data storage      Raw     Size    Used    Avail   Use%    Free%
                  7.0T    6.7T    1.4T    3.9T    21%     58.2%

Number of containers             : 22494
Average container size           : 66854006 bytes (63.76MB)
Space allocated for containers   : 1503814015939 bytes (1.37TB)
Reserved space                   : 308046041088 bytes (286.89GB)
Reserved space percentage        : 4.0%
Reserved space for cloud cache   : 1.1T (15.6%)

Use "--dsstat 3" to get more accurate statistics for each partition

@Axelay91 what do you suggest as action please ?