Forum Discussion

bbot's avatar
bbot
Level 4
9 years ago

NetBackup jobs failing with error 129 (insufficient disk space)

Hello,

We have a couple clients failing to backup to the same storage. There's over 1 TB available on our storage server. The size of all the full client backups are no longer than 300 gb.

Watermark is set to 98% high water mark, 80% low water mark.

I'm fairly new to netbackup,, where should I be looking to find out the root cause?

  • oh my bad.. i read that wrong.. 

     

    Then i would suggest setting the low water mark to 90% and the high leave at 98% see if it lets you store to the unit. the only other thing i can think of without a test environment of my own is that its hitting the low and failing which shouldnt happen but ive seen wierder things. 

     

    *Edit*

    Do you have disk stagin enabled on the basic storage unit? are there images that have yet to duplicate off? 

8 Replies

  • What kind of storage server are you using ?

    Do you have Advandced Disk or Media server deduplication or appliance ?

    What is the size of the diskpool ?

     

    With

      bpimage -cleanup -allclients  and

      nbdelete -allvolumes

    you may cleanup your storage if you have expired images or old fragments on it.

  • Storage server is running Windows Server 2012.

    I believe we use dedupe. Where can I verify these settings? How can I check the diskpool?

     

    Regarding the bpimage and nbdelete commands, shouldn't netbackup be automatically cleaning expired images or old fragments? Or should the command be ran manually when there are issues?

  • Root cause is this setting "Watermark is set to 98% high water mark, 80% low water mark."

     

    NetBackup will only start to cleanup when it reaches the high water mark, in your situation that mean, when its used 980 GB. And as you've said, your backups are 300GB. So by the time it needs to start cleaning, it also needs to have 300gb available to perform the daily backups. 

     

    Clearly this is not possible.

     

    I suggest you lower it to something like High 65% (meaning you can still fit 300 in) and Low 35%

     

  • I understood you have over 1 TB free. This means your diskpool is roundabout at a size of 500TB.

    So what is the size ? In NetBackup GUI: Media and Device Managemnet -> Devices -> Disk Pools

    you'll find a list of your diskpools with all information you need (Kind, Storageserver, size, used used in % etc)

    A high watermark of 98% is default but in most cases very high. On some filesystems it not recommend to fill at this level.

    Normally there is no need to perform the commands nbdelete and bpimage -cleanup. All is done automatically.

    For deduplication (msdp-pool) you should know that a cleanup will take some time to take affect.

  • I'm not seeing the Disk Pool section under Media & Device Management >> Devices.

    Under Storage >> Storage Units, I found the storage unit.

    Storage Unit Type : Disk

    Disk Type : BasicDisk

    Capacity 7388476

    Available 1261722

    % Full 83

  • Based on the above your about full and the water mark will be hit during the backup so NBU will not allow the job and fail out with status 129. According to what you just posted you have 126ish GB of disk space available, so not enough to perform the job.

     

    Basic disk is not deupe and is exactly what it is. the High watermark is the point which NBU will no longer allow backups to occur. the low is the point which NBU will start to error and give warnings. It looks like your watermark is set default, maybe a little higher but not in a bad place. 

     

    You need to add space or expire some images to clear up space to allow the job. 

  • Hi Doug, I have 1.26TB available, not 126gb.

  • oh my bad.. i read that wrong.. 

     

    Then i would suggest setting the low water mark to 90% and the high leave at 98% see if it lets you store to the unit. the only other thing i can think of without a test environment of my own is that its hitting the low and failing which shouldnt happen but ive seen wierder things. 

     

    *Edit*

    Do you have disk stagin enabled on the basic storage unit? are there images that have yet to duplicate off?