cancel
Showing results for 
Search instead for 
Did you mean: 

How to check when the MSDP pool was full?

hi,

 

one of the backups which was going to a msdp pool failed with error 129

Aug 28, 2017 10:49:59 PM - Info bpbkar (pid=102636) INF - Backing up vCenter server bmasdc-vc01v, ESX host bmasdc-esx02p-bl.apac.ent.bhpbilliton.net, BIOS UUID 4228b1ef-b3d5-1e42-79da-b30d4f44e561, Instance UUID 5028d801-d14e-9dd9-326e-13b2feaf0686, Display Name BMAGCM-SRV01, Hostname BMAGCM-SRV01.apac.ent.bhpbilliton.net
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) Storage Server Error: (Storage server: PureDisk:bmasdc-bkp03) _pdvfs_fcache_flushfile_norm_cleanup: Disk storage unit is full V-454-1
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) image open failed: error 2060031: out of space
Aug 28, 2017 10:52:29 PM - Warning bptm (pid=102637) disk pool BMASDC-BKP03-MSDP volume PureDiskVolume is full: processing disk full condition
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) image open failed: error 2060031: out of space
Aug 28, 2017 10:52:29 PM - current media @aaaaL complete, requesting next media Any
Aug 28, 2017 10:52:36 PM - Error bptm (pid=102637) NBJM returned an extended error status: Storage unit does not support spanning (2102)
Aug 28, 2017 10:52:37 PM - Info bptm (pid=102637) EXITING with status 129 <----------

my problem is:

currently when i go to my msdp pool its showing up and showing 82 full currently

BMASDC-BKP03-MSDP bmasdc-bkp03 PureDisk 1 82.699394 17 59.08 TB 12.36 TB 71.44 TB 71.44 TB 80 98 99 No -- None

 

I want to know at the time what happened exactly when the backup failed as i am sure it cant be possible in the night it was 100 % full and then it came back to 82 percent

 

 

7 Replies

Re: How to check when the MSDP pool was full?

Hello,

 

I think in disk pool property set the high water mark 82%. If it reach the high water mark then next backup get fail with this error. Suggest you to do not increase the value more than 85%. Try to cleanup unwanted images from catalog and disk.

If there is no option to expire the images then add new storage.

Regards,

Re: How to check when the MSDP pool was full?

hi,

 

the high water mark is set to 96% 

Re: How to check when the MSDP pool was full?

Hello Just check me on this, but I believe the bperror command on the master server will show when it changed state. 

I think bperror -disk, will be more consise but think there's something like a bperror -msdp or something. just check help on bperror.

Re: How to check when the MSDP pool was full?

@Mehul_Pal

Did you configure email address for notifications on your appliance?

If you did, you would've received an email similar to this:

https://vox.veritas.com/t5/NetBackup-Appliance/MSDP-The-partition-usage-has-exceeded-warning-thresho...

 

Re: How to check when the MSDP pool was full?

yes autogenereated emails comes when the volume pool reaches its capacity however i was unable to find email like that therefore i was amazed?

 

apart from the that then i do crcontrol --processqueueinfo

it shows

busy = no

pending = yes

 

it seems its garbage collection or process is not working?

Re: How to check when the MSDP pool was full?

Hi

 

I'd would try running the following:

nbdelete -allvolumes -force

 

This should help with the forced cleanup and then force the garbage collection with:

crcontrol --processqueue

Just check the sintax on those commands, Sintax might vary as I'm typing from memory.

 

EDIT: Sorry the command is: nbdelete -allvolumes -force # it's a safe command, it just checks that everything marked for expiration is expired. You should see an image cleanup job start and once complete run the crcontrol command.

Re: How to check when the MSDP pool was full?

Best not to let them get full in the first place by monitoring.

I run this every 20 mins.  It spams you when it exceeds user defined limits.   Can be used for generating reports on MSDP usage.

#!/bin/bash
# If MSDP storage on any pool exceeds $ALERT_LIMIT send an alert message OR
# If MSDP storage on any pool exceeds $CRITICAL_LIMIT send a "CRITICAL" alert message 
# Only sends mail if either limit is exceed
# Title: nbu_list_MSDP.bsh Written for NBU 7.7.3  Jim McDonald
# Assumes sudo access

recipients_1=those_that_like_reports@backupland.com
recipients_2=somebody_that_cares@backupland.com,somebody_that_cares@backupland.com
#set -x
MSG_STATE=""
let ALERT_LIMIT=80  
let CRITICAL_LIMIT=90
let LIMIT_OK=0
#------------------------------------------------------------------------------
function Mail_it(){
MAIL_MSG=$1
RECIPIENTS=$2
/usr/sbin/sendmail -t << THE_END
TO: $RECIPIENTS
FROM: NETBACKUP
REPLY-TO: Please do not attempt to reply to this automatic message
SUBJECT: $MAIL_MSG 

 $(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | awk ' BEGIN {printf"Disk Pool          Media ID    Total(GB)  Free(GB)  %%Used\n"} { printf "%-20s %-10s%9.2f %9.2f  %s%%\n",$2,$5,$6,$7,$8 }')

 NOTE: Check HIGH TIDE levels they may be set to 95% 

 Mail from script $(uname -n): basename $0

THE_END
}
#------------------------------------------------------------------------------
# MAIN 
#------------------------------------------------------------------------------
let LIMIT_OK=$(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | awk  '{ print $8}' | sort -nr | head -1)

if [ "$1" = "-RPT" ]; then # producing a generical report for all locations
   Mail_it "NETBACKUP: MSDP Usage for $(date +'%a %F %r')" $recipients_1
elif [ "$1" = "-MONTH_RPT" ]; then # producing a generical report for all locations
   Mail_it "NETBACKUP: MSDP Monthly Usage for $(date +'%a %F %r')" $recipients_1
fi
    
let LIMIT_OK=$(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | grep "$POOL_STR" | awk  '{ print $8}' | sort -nr | head -1)
if (( $LIMIT_OK >= ALERT_LIMIT  && $LIMIT_OK < CRITICAL_LIMIT )) ; then
          MSG_STATE="ALERT"
          MSG_LIMIT="$ALERT_LIMIT"
elif (( $LIMIT_OK >= CRITICAL_LIMIT )) ; then
          MSG_STATE="CRITICAL"
          MSG_LIMIT="$CRITICAL_LIMIT"
fi
if [ "$MSG_STATE" != "" ] ; then
   Mail_it "NETBACKUP: $MSG_STATE a MSDP storage has exceeded ${MSG_LIMIT}%" $recipients_2 
fi
## END ##