08-28-2017 08:30 PM
hi,
one of the backups which was going to a msdp pool failed with error 129
Aug 28, 2017 10:49:59 PM - Info bpbkar (pid=102636) INF - Backing up vCenter server bmasdc-vc01v, ESX host bmasdc-esx02p-bl.apac.ent.bhpbilliton.net, BIOS UUID 4228b1ef-b3d5-1e42-79da-b30d4f44e561, Instance UUID 5028d801-d14e-9dd9-326e-13b2feaf0686, Display Name BMAGCM-SRV01, Hostname BMAGCM-SRV01.apac.ent.bhpbilliton.net
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) Storage Server Error: (Storage server: PureDisk:bmasdc-bkp03) _pdvfs_fcache_flushfile_norm_cleanup: Disk storage unit is full V-454-1
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) image open failed: error 2060031: out of space
Aug 28, 2017 10:52:29 PM - Warning bptm (pid=102637) disk pool BMASDC-BKP03-MSDP volume PureDiskVolume is full: processing disk full condition
Aug 28, 2017 10:52:29 PM - Critical bptm (pid=102637) image open failed: error 2060031: out of space
Aug 28, 2017 10:52:29 PM - current media @aaaaL complete, requesting next media Any
Aug 28, 2017 10:52:36 PM - Error bptm (pid=102637) NBJM returned an extended error status: Storage unit does not support spanning (2102)
Aug 28, 2017 10:52:37 PM - Info bptm (pid=102637) EXITING with status 129 <----------
my problem is:
currently when i go to my msdp pool its showing up and showing 82 full currently
BMASDC-BKP03-MSDP bmasdc-bkp03 PureDisk 1 82.699394 17 59.08 TB 12.36 TB 71.44 TB 71.44 TB 80 98 99 No -- None
I want to know at the time what happened exactly when the backup failed as i am sure it cant be possible in the night it was 100 % full and then it came back to 82 percent
08-29-2017 01:10 AM
Hello,
I think in disk pool property set the high water mark 82%. If it reach the high water mark then next backup get fail with this error. Suggest you to do not increase the value more than 85%. Try to cleanup unwanted images from catalog and disk.
If there is no option to expire the images then add new storage.
Regards,
09-01-2017 09:58 PM
hi,
the high water mark is set to 96%
09-03-2017 12:25 PM
Hello Just check me on this, but I believe the bperror command on the master server will show when it changed state.
I think bperror -disk, will be more consise but think there's something like a bperror -msdp or something. just check help on bperror.
09-04-2017 01:09 AM
Did you configure email address for notifications on your appliance?
If you did, you would've received an email similar to this:
09-04-2017 05:52 AM
yes autogenereated emails comes when the volume pool reaches its capacity however i was unable to find email like that therefore i was amazed?
apart from the that then i do crcontrol --processqueueinfo
it shows
busy = no
pending = yes
it seems its garbage collection or process is not working?
09-04-2017 06:43 AM - edited 09-04-2017 06:46 AM
Hi
I'd would try running the following:
nbdelete -allvolumes -force
This should help with the forced cleanup and then force the garbage collection with:
crcontrol --processqueue
Just check the sintax on those commands, Sintax might vary as I'm typing from memory.
EDIT: Sorry the command is: nbdelete -allvolumes -force # it's a safe command, it just checks that everything marked for expiration is expired. You should see an image cleanup job start and once complete run the crcontrol command.
09-05-2017 09:55 PM
Best not to let them get full in the first place by monitoring.
I run this every 20 mins. It spams you when it exceeds user defined limits. Can be used for generating reports on MSDP usage.
#!/bin/bash # If MSDP storage on any pool exceeds $ALERT_LIMIT send an alert message OR # If MSDP storage on any pool exceeds $CRITICAL_LIMIT send a "CRITICAL" alert message # Only sends mail if either limit is exceed # Title: nbu_list_MSDP.bsh Written for NBU 7.7.3 Jim McDonald # Assumes sudo access recipients_1=those_that_like_reports@backupland.com recipients_2=somebody_that_cares@backupland.com,somebody_that_cares@backupland.com #set -x MSG_STATE="" let ALERT_LIMIT=80 let CRITICAL_LIMIT=90 let LIMIT_OK=0 #------------------------------------------------------------------------------ function Mail_it(){ MAIL_MSG=$1 RECIPIENTS=$2 /usr/sbin/sendmail -t << THE_END TO: $RECIPIENTS FROM: NETBACKUP REPLY-TO: Please do not attempt to reply to this automatic message SUBJECT: $MAIL_MSG $(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | awk ' BEGIN {printf"Disk Pool Media ID Total(GB) Free(GB) %%Used\n"} { printf "%-20s %-10s%9.2f %9.2f %s%%\n",$2,$5,$6,$7,$8 }') NOTE: Check HIGH TIDE levels they may be set to 95% Mail from script $(uname -n): basename $0 THE_END } #------------------------------------------------------------------------------ # MAIN #------------------------------------------------------------------------------ let LIMIT_OK=$(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | awk '{ print $8}' | sort -nr | head -1) if [ "$1" = "-RPT" ]; then # producing a generical report for all locations Mail_it "NETBACKUP: MSDP Usage for $(date +'%a %F %r')" $recipients_1 elif [ "$1" = "-MONTH_RPT" ]; then # producing a generical report for all locations Mail_it "NETBACKUP: MSDP Monthly Usage for $(date +'%a %F %r')" $recipients_1 fi let LIMIT_OK=$(/usr/bin/sudo /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk | grep "$POOL_STR" | awk '{ print $8}' | sort -nr | head -1) if (( $LIMIT_OK >= ALERT_LIMIT && $LIMIT_OK < CRITICAL_LIMIT )) ; then MSG_STATE="ALERT" MSG_LIMIT="$ALERT_LIMIT" elif (( $LIMIT_OK >= CRITICAL_LIMIT )) ; then MSG_STATE="CRITICAL" MSG_LIMIT="$CRITICAL_LIMIT" fi if [ "$MSG_STATE" != "" ] ; then Mail_it "NETBACKUP: $MSG_STATE a MSDP storage has exceeded ${MSG_LIMIT}%" $recipients_2 fi ## END ##