cancel
Showing results for 
Search instead for 
Did you mean: 

How to check when the MSDP pool was full?

Mehul_Pal
Level 5

hi,

 

one of the backups which was going to a msdp pool failed with error 129

Aug 28, 2017 10:49:59 PM - Info bpbkar (pid=102636) INF - Backing up vCenter server bmasdc-vc01v, ESX host bmasdc-esx02p-bl.apac.ent.bhpbilliton.net, BIOS UUID 4228b1ef-b3d5-1e42-79da-b30d4f44e561, Instance UUID 5028d801-d14e-9dd9-326e-13b2feaf0686, Display Name BMAGCM-SRV01, Hostname BMAGCM-SRV01.apac.ent.bhpbilliton.net
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) Storage Server Error: (Storage server: PureDisk:bmasdc-bkp03) _pdvfs_fcache_flushfile_norm_cleanup: Disk storage unit is full V-454-1
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) image open failed: error 2060031: out of space
Aug 28, 2017 10:52:29 PM - Warning bptm (pid=102637) disk pool BMASDC-BKP03-MSDP volume PureDiskVolume is full: processing disk full condition
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) image open failed: error 2060031: out of space
Aug 28, 2017 10:52:29 PM - current media @aaaaL complete, requesting next media Any
Aug 28, 2017 10:52:36 PM - Error bptm (pid=102637) NBJM returned an extended error status: Storage unit does not support spanning (2102)
Aug 28, 2017 10:52:37 PM - Info bptm (pid=102637) EXITING with status 129 <----------

my problem is:

currently when i go to my msdp pool its showing up and showing 82 full currently

BMASDC-BKP03-MSDP bmasdc-bkp03 PureDisk 1 82.699394 17 59.08 TB 12.36 TB 71.44 TB 71.44 TB 80 98 99 No -- None

 

I want to know at the time what happened exactly when the backup failed as i am sure it cant be possible in the night it was 100 % full and then it came back to 82 percent

 

 

7 REPLIES 7

Tousif
Level 6

Hello,

 

I think in disk pool property set the high water mark 82%. If it reach the high water mark then next backup get fail with this error. Suggest you to do not increase the value more than 85%. Try to cleanup unwanted images from catalog and disk.

If there is no option to expire the images then add new storage.

Regards,

hi,

 

the high water mark is set to 96% 

William_Jansen_
Level 5
Partner Certified

Hello Just check me on this, but I believe the bperror command on the master server will show when it changed state. 

I think bperror -disk, will be more consise but think there's something like a bperror -msdp or something. just check help on bperror.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@Mehul_Pal

Did you configure email address for notifications on your appliance?

If you did, you would've received an email similar to this:

https://vox.veritas.com/t5/NetBackup-Appliance/MSDP-The-partition-usage-has-exceeded-warning-thresho...

 

yes autogenereated emails comes when the volume pool reaches its capacity however i was unable to find email like that therefore i was amazed?

 

apart from the that then i do crcontrol --processqueueinfo

it shows

busy = no

pending = yes

 

it seems its garbage collection or process is not working?

William_Jansen_
Level 5
Partner Certified

Hi

 

I'd would try running the following:

nbdelete -allvolumes -force

 

This should help with the forced cleanup and then force the garbage collection with:

crcontrol --processqueue

Just check the sintax on those commands, Sintax might vary as I'm typing from memory.

 

EDIT: Sorry the command is: nbdelete -allvolumes -force # it's a safe command, it just checks that everything marked for expiration is expired. You should see an image cleanup job start and once complete run the crcontrol command.

Jim-90
Level 6

Best not to let them get full in the first place by monitoring.

I run this every 20 mins.  It spams you when it exceeds user defined limits.   Can be used for generating reports on MSDP usage.

#!/bin/bash
# If MSDP storage on any pool exceeds $ALERT_LIMIT send an alert message OR
# If MSDP storage on any pool exceeds $CRITICAL_LIMIT send a "CRITICAL" alert message 
# Only sends mail if either limit is exceed
# Title: nbu_list_MSDP.bsh Written for NBU 7.7.3  Jim McDonald
# Assumes sudo access

recipients_1=those_that_like_reports@backupland.com
recipients_2=somebody_that_cares@backupland.com,somebody_that_cares@backupland.com
#set -x
MSG_STATE=""
let ALERT_LIMIT=80  
let CRITICAL_LIMIT=90
let LIMIT_OK=0
#------------------------------------------------------------------------------
function Mail_it(){
MAIL_MSG=$1
RECIPIENTS=$2
/usr/sbin/sendmail -t << THE_END
TO: $RECIPIENTS
FROM: NETBACKUP
REPLY-TO: Please do not attempt to reply to this automatic message
SUBJECT: $MAIL_MSG 

 $(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | awk ' BEGIN {printf"Disk Pool          Media ID    Total(GB)  Free(GB)  %%Used\n"} { printf "%-20s %-10s%9.2f %9.2f  %s%%\n",$2,$5,$6,$7,$8 }')

 NOTE: Check HIGH TIDE levels they may be set to 95% 

 Mail from script $(uname -n): basename $0

THE_END
}
#------------------------------------------------------------------------------
# MAIN 
#------------------------------------------------------------------------------
let LIMIT_OK=$(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | awk  '{ print $8}' | sort -nr | head -1)

if [ "$1" = "-RPT" ]; then # producing a generical report for all locations
   Mail_it "NETBACKUP: MSDP Usage for $(date +'%a %F %r')" $recipients_1
elif [ "$1" = "-MONTH_RPT" ]; then # producing a generical report for all locations
   Mail_it "NETBACKUP: MSDP Monthly Usage for $(date +'%a %F %r')" $recipients_1
fi
    
let LIMIT_OK=$(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | grep "$POOL_STR" | awk  '{ print $8}' | sort -nr | head -1)
if (( $LIMIT_OK >= ALERT_LIMIT  && $LIMIT_OK < CRITICAL_LIMIT )) ; then
          MSG_STATE="ALERT"
          MSG_LIMIT="$ALERT_LIMIT"
elif (( $LIMIT_OK >= CRITICAL_LIMIT )) ; then
          MSG_STATE="CRITICAL"
          MSG_LIMIT="$CRITICAL_LIMIT"
fi
if [ "$MSG_STATE" != "" ] ; then
   Mail_it "NETBACKUP: $MSG_STATE a MSDP storage has exceeded ${MSG_LIMIT}%" $recipients_2 
fi
## END ##