Disk storage unit is full, but it's not full (netbackup 10.1.1)

Question

Hello,I have failed backups due to disk storage unit is full, but the problem is that there is enough space in diskpool and disk volume.Diskpool is full at 80%, high water mark is set to 96%, in other FS there is enough space.Netbackup services has been well restarted (spoold and spad running), master VM has been restarted too.Error messageStorage&nbsp;Server&nbsp;Error:&nbsp;(Storage&nbsp;server:&nbsp;PureDisk:mastername)&nbsp;pdvfs_io_pwrite:&nbsp;Disk&nbsp;storage&nbsp;unit&nbsp;is&nbsp;full&nbsp;V-454-3 :Storage&nbsp;Server&nbsp;Error:&nbsp;(Storage&nbsp;server:&nbsp;PureDisk:mastername)&nbsp;write_buf:&nbsp;Failed&nbsp;to&nbsp;write&nbsp;to&nbsp;spoold&nbsp;on&nbsp;storage&nbsp;server&nbsp;mastername(No&nbsp;space&nbsp;left&nbsp;on&nbsp;device).&nbsp;Ensure&nbsp;storage&nbsp;server&nbsp;services&nbsp;are&nbsp;running&nbsp;and&nbsp;operational.&nbsp;V-454-16sts_close_handle&nbsp;failed:&nbsp;2060031&nbsp;out&nbsp;of&nbsp;spacecannot&nbsp;write&nbsp;image&nbsp;to&nbsp;disk,&nbsp;media&nbsp;close&nbsp;failed&nbsp;with&nbsp;status&nbsp;2060031Server is at the same time master and media (i'll still available if more informations are needed).Master: VMNBU: 10.1.1Linux Redhat 8.6&nbsp;

nicolai · Answer

what does a&nbsp;crcontrol --dsstat say ?
How to reclaim deduplication storage space manually
https://www.veritas.com/support/en_US/article.100000714

bessam · Answer

I've already executed crcontrol commands (I forgot to mention it in the topic)Here is the result of crcontrol --dsstat :[root@mastername bin]# ./crcontrol --dsstatError: -1: sess flags: 2323857407765120008************ Data Store statistics ************Data storage&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Raw&nbsp;&nbsp;&nbsp;&nbsp; Size&nbsp;&nbsp;&nbsp; Used&nbsp;&nbsp;&nbsp; Avail&nbsp;&nbsp; Use%&nbsp;&nbsp;&nbsp; Free%&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 7.0T&nbsp;&nbsp;&nbsp; 6.7T&nbsp;&nbsp;&nbsp; 5.3T&nbsp;&nbsp;&nbsp; 1.4T&nbsp;&nbsp;&nbsp; 80%&nbsp;&nbsp;&nbsp;&nbsp; 20.6%Number of containers&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 86739Average container size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 66752389 bytes (63.66MB)Space allocated for containers&nbsp;&nbsp; : 5790035505967 bytes (5.27TB)Reserved space&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 308046043136 bytes (286.89GB)Reserved space percentage&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 4.0%Reserved space for cloud cache&nbsp;&nbsp; : 1.1T (15.6%)Use "--dsstat 1" to get more accurate statisticsUse "--dsstat 2" to get statistics for each partitionUse "--dsstat 3" to get more accurate statistics for each partition&nbsp;

stefanosm · Answer

Run the same command with 2 as parameter.crcontrol --dsstat 2I had in the past problems with a very small partition that was full.

axelay91 · Answer

Hello,&nbsp;It is full because you have probably configured a LSU cloud.From the output, there is 1.1 TB reserved for cache cloud.Reserved space for cloud cache&nbsp;&nbsp; : 1.1T (15.6%)&nbsp;&nbsp;&nbsp;

bessam · Answer

&nbsp;StefanosM&nbsp;: here is result of&nbsp;crcontrol --dsstat 2:[root@xxxxx bin]# ./crcontrol --dsstat 2Error: -1: sess flags: 2323857407765120008************ Data Store statistics ************Mount point count: 1=============== Mount point&nbsp;&nbsp; 1 ===============Path = /msdp/dataData storage&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Raw&nbsp;&nbsp;&nbsp;&nbsp; Size&nbsp;&nbsp;&nbsp; Used&nbsp;&nbsp;&nbsp; Avail&nbsp;&nbsp; Use%&nbsp;&nbsp;&nbsp; Free%&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 7.0T&nbsp;&nbsp;&nbsp; 6.7T&nbsp;&nbsp;&nbsp; 1.4T&nbsp;&nbsp;&nbsp; 3.9T&nbsp;&nbsp;&nbsp; 21%&nbsp;&nbsp;&nbsp;&nbsp; 58.2%Number of containers&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 22494Average container size&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 66854006 bytes (63.76MB)Space allocated for containers&nbsp;&nbsp; : 1503814015939 bytes (1.37TB)Reserved space&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 308046041088 bytes (286.89GB)Reserved space percentage&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : 4.0%Reserved space for cloud cache&nbsp;&nbsp; : 1.1T (15.6%)Use "--dsstat 3" to get more accurate statistics for each partition&nbsp;&nbsp;Axelay91&nbsp;what do you suggest as action please ?&nbsp;

Forum Discussion

Disk storage unit is full, but it's not full (netbackup 10.1.1)

5 Replies

Related Content

Removing host certificate NBU 10.1.1

10.1.1 - Web UI Activity Monitor Filtering

NetBackup Storage Space Full issue

Renaming Netbackup BasicDisk Storage Units?

Netbackup upgrade from 8.3.0.2 to 10.1.1 - Linux RedHat