scripting duplicates

[cmrhine]

Hope someone can help me with this --

My company is running netbackup 6.0MP4 on a Windows Server 2003, but I'm a Unix person and I'm used to having lots of documentation for using line commands, but I'm stuck with this Windows system.  We're currently backing up some jobs to disk and using the duplicate process to move them to tape during the day.  I'd like to script this secondary "duplication" process and set it up so that it will kick off right after the policy script to the disk storage unit has ended.  I know how this can be done if I can just get the command bpduplicate to work on the command line.  Using bpduplicate, I can get a listing of the jobs that I want using:

bpduplicate -policy <policy_name> -M <server name> -PD

This gives me the current two duplicate images it found under the appropriate disk storage unit.  If I take off the -PD to make the command run instead of just giving me a listing, I get:

"Duplicate started Fri Sep 14 2007 15:06:36
Activity monitor job id = 44486
The number of specified storage units must be equal to the number of copies"

And the NB Administration console confirms that the job failed with status code 144.

I basically want the process to find everything under the <policy_name> that hasn't already been backed up to tape and start queuing all of those processes to be backed up.
It'd be great if someone could help and tell me what I'm doing wrong.  The documentation for the command is pretty complete, but does not explain why I might get this error.  It seems like if I get a listing without error, it should be able to start the process without error.


Thanks -- Christy

[AKopel]

Have you played around with the Disk STAGING storage unit? This allows you to schedule the duplications to tape using the built in scheduler (so you don't have to script anything)

Aaron

[cmrhine]

I guess I should have added that we already schedule these and that is not good enough.  We're not manually starting them.  But when you have tight schedules and you want things to run right after one another (we back up a lot of stuff), you really want to know that your process is going to start when another stops.  That's why I want to know how to do this.  Since the ability to script them and start them that way is clearly there, I don't see why I would not want to know how to do this.  Should be simple enough to do and it would give much more control over backups.

Anyone know how to successfully run the bpduplicate command on the command line?

[sdo]

I have a medium sized (400 line) DOS/batch script that I used to use to duplicate media from specific policies in NBU ES v6.0 on Windows.  I know that you are in a Unix shop, but it might help you a bit.  Let me know if you want me to post it.

[jreale_2]

I wouldn't mind seeing your script. 

We have a situation where we use a virtual tape library and use vault jobs to duplicate our virtual tapes to actuals.  When the vault job completes, the virtual copies still reside until the retention of 2weeks is met.  We have looked to see if we could expire the virtual copy right after the vault job, but no option in NBU 6.0MP4 supports expiring virtual tapes.  The disk staging option is quite appealing, however, we have no ability to bring down our backup system to make the conversion from virtual tapes to disks.

I'm hoping scripts have the answer!

- Jim

[sdo]

The duplication script is actually in three parts.

1) The main duplication script written in DOS batch.  It is very specific to a specific situation, so you'll need to work on the "policy names".  If you comment (REM) out the actual duplication, then you can tweak and edit and run as many times as you like as it won't actually do anything then.  Also note that it does run a "bpduplicate -PM" (preview mode) so you can see what the duplication will do, you won't want to comment this line out.

2) A VBScript to calculate the backup "session window", i.e. you don't want to duplicate backups from previous days, or future days (if you a running a duplication of old backups) - so the main script needs to determine the start and end of the backup session window.  Adjust to suit your environment.

3) A sort with no-duplicates routine in VBScript.  This script is used to take a list of media-IDs and simply sort the list down to individual entries.

[sdo]

The main duplication script:

Code:

@echo offREM ******************************************************************************************************************************************************REM ***  File: DUP-BCP-MEDIA.BATREM ***REM ***  Vers  Date  Who DescriptionREM ***  ----  ----  --- -----------REM ***  v0.01  10-APR-2006 DR First version to list images.REM ***  v0.02  11-APR-2006 DR Calculate session window, sort noduplicates, must use "-s" and "-e" with "bpduplicate" otherwise it defaults toREM ***     the last 24 hours.REM ***  v0.03  12-APR-2006 DR Check master server, count tapes found.REM ***  v0.04  20-APR-2006 DR Use a "BID" file instead of requesting duplication, use "-set_primary 0" so that the current primary remainsREM ***     unchanged, copy to volume pool "bcp-test", save BID file, don't abort but offer warning.REM ***  v0.05  21-APR-2006 DR Second test at duplication, split code into streams of "bcp-p1-*" and "bcp-p2-*" policies.REM ***  v0.06  24-APR-2006 DR Add "TIME /T" to the duplication batch files.REM ******************************************************************************************************************************************************set fac=LIST-BCP-MEDIA,set abort=noREM **************************************************************************************************REM *** Check host, as duplications only work on the master server...REM **************************************************************************************************if /i not "%computername%"=="master" (  echo %fac% This script can only be run on the master server.  Duplication requests do not work from other servers.  goto end)set back=0:calculate_sessionREM **************************************************************************************************REM *** Calculate session window...REM **************************************************************************************************echo .echo ####################################time /techo %fac% Session window is...for /f "tokens=1-4" %%a in ('cscript //nologo show-session.vbs %back%') do (  if /i "%%a"=="start" (    set sos-day=%%b    set sos-datetime=%%c %%d  )  if /i "%%a"=="end" (    set eos-day=%%b    set eos-datetime=%%c %%d  ))echo   Start of session:  %sos-datetime%  %sos-day%echo   End of session:    %eos-datetime%  %eos-day%echo .choice /c nyq /m "%fac% Press `n` to go back one day, or press `y` to accept current session window, or `q` to quit —" /t 300 /d qif %errorlevel%==3 goto :endif %errorlevel%==1 (  set /a back=%back%-1  goto calculate_session)REM if not "%1"=="QUIET" pauseREM **************************************************************************************************REM *** List the BCP policies...REM **************************************************************************************************set class-path=D:\Veritas\NetBackup\db\classcall :duplication bcp-p1if /i "%abort%"=="yes" goto :eofcall :duplication bcp-p2if /i "%abort%"=="yes" goto :eof:endREM **************************************************************************************************REM *** End...REM **************************************************************************************************echo .echo ####################################time /techo %fac% Script completed.if not "%1"=="QUIET" pauseexit /bREM **************************************************************************************************REM *** Errors and warnings...REM **************************************************************************************************:abortecho %fac% ERR - Script is aborting...goto endREM **************************************************************************************************REM *** Sub-routines...REM **************************************************************************************************:warningecho %fac% WARNING - Problem encountered...choice /c ny /m "%fac% Do you want to abort this script– Press `n` to continue, or `y` to abort." /t 300 /d yif %errorlevel%==2 set abort=yesgoto :eof:sub_check_tape_is_in_robotset slot=nonefor /f "tokens=1-3" %%b in (%policies%-%1.vmq) do (  if /i "%%b"=="robot" (    if /i "%%c"=="slot:" (      set slot=%%d      echo   Tape: %1   Slot: %%d    )  ))if /i "%slot%"=="none" (  echo %fac% ERR - Tape %1 is missing from the robot, cannot continue, script exiting...  set all-present=no)goto :eof:sub_increment_foundset /a found = %found% + 1goto :eof:duplicationREM **************************************************************************************************REM *** The main duplication sub-routine...REM **************************************************************************************************set policies=%1echo .echo ####################################time /techo %fac% Listing BCP policy names for "%policies%-*"...for /f %%z in ('dir /a:d /b /o:n %class-path%\%policies%-*') do (echo   %%z)REM if not "%1"=="QUIET" pauseREM **************************************************************************************************REM *** Make a "BID" (backup images data) file...REM **************************************************************************************************set bid-file="%cd%\%policies%.bid"set bid-save="%cd%\%policies%.bid-save"if exist %bid-file% del %bid-file%if exist %bid-save% del %bid-save%copy nul %bid-file% > nulecho .echo ####################################time /techo %fac% Creating BID file for "%policies%-*"...for /f %%z in ('dir /a:d /b /o:n %class-path%\%policies%-*') do (  echo   Policy: %%~nz  bpimagelist -policy %%~nz -d %sos-datetime% -e %eos-datetime% -idonly  for /f "tokens=8" %%a in ('bpimagelist -policy %%~nz -d %sos-datetime% -e %eos-datetime% -idonly') do (    (echo %%a) >> %bid-file%  ))REM if not "%1"=="QUIET" pauseecho .echo %fac% The "BID" file %bid-file% contains...type %bid-file%copy %bid-file% %bid-save% > nulREM **************************************************************************************************REM *** Get all the tape numbers for copy 1 "on-site" of the BCP images since start of shift...REM **************************************************************************************************set tapes-dat=%policies%.datset tapes-col=%policies%.colif exist %tapes-dat% del %tapes-dat%if exist %tapes-col% del %tapes-col%copy nul %tapes-dat% > nulcopy nul %tapes-col% > nulecho .echo ####################################time /techo %fac% Retrieving tape numbers from image lists, for policies "%policies%-*"...for /f %%z in ('dir /a:d /b /o:n %class-path%\%policies%-*') do (  echo   Policy: %%~nzREM  bpimagelist -policy %%~nz -d %sos-datetime% -e %eos-datetime%  for /f "tokens=1-10" %%a in ('bpimagelist -policy %%~nz -d %sos-datetime% -e %eos-datetime%') do (    if /i "%%a"=="frag" (      if /i "%%b"=="1" (REM     echo   Copy: %%b  Media: %%i        (echo %%i) >> %tapes-dat%      )    )  ))REM if not "%1"=="QUIET" pauseREM **************************************************************************************************REM *** Sort the list of tapes and remove duplicate tape numbers...REM **************************************************************************************************echo .echo ####################################time /techo %fac% Sorting and collating tape numbers, for policies "%policies%-*"...cscript //nologo sort-no-duplicates.vbs %tapes-dat% %tapes-col%set found=0for /f %%a in (%tapes-col%) do (  echo   Tape: %%a  Call :sub_increment_found)if %found% lss 1 (  echo %fac% WARN - Not enough tapes were found, expected at least 1 tape but found %found%...  call :warning)if /i "%abort%"=="yes" goto :eofREM if not "%1"=="QUIET" pauseREM **************************************************************************************************REM *** Now list the tape contents...REM **************************************************************************************************echo .echo ####################################time /techo %fac% Listing each tape, for policies "%policies%-*"...if exist %policies%-*.bpi del %policies%-*.bpiif exist %policies%-*.bpm del %policies%-*.bpmif exist %policies%-*.vmq del %policies%-*.vmqfor /f %%a in (%tapes-col%) do (  echo   Listing tape "%%a"...  (bpimmedia -mediaid %%a)    > %policies%-%%a.bpi  (bpmedialist -m %%a -L)     > %policies%-%%a.bpm  (vmquery -m %%a -h master1) > %policies%-%%a.vmq)REM if not "%1"=="QUIET" pause:skip-listREM **************************************************************************************************REM *** Check tapes are in slots...REM **************************************************************************************************echo .echo ####################################time /techo %fac% Checking tapes are present in the robot, for policies "%policies%-*"...set all-present=yesfor /f %%a in (%tapes-col%) do (  call :sub_check_tape_is_in_robot %%a)if /i "%all-present%"=="no" (  echo %fac% WARN - Not all tapes are present in the robot...  call :warning)if /i "%abort%"=="yes" goto :eofREM if not "%1"=="QUIET" pauseREM **************************************************************************************************REM *** Show what could be duplicated...REM **************************************************************************************************set dup-bat="%cd%\%policies%.bat"set dup-log="%cd%\%policies%.log"echo .echo ####################################time /techo %fac% Showing possible duplication, for "%policies%-*"...set dup-cmd=bpduplicate -dstunit master-hcart-robot-tld-0 -dp bcp-test -rl 3 -fail_on_error 0 -set_primary 0 -cn 1 -priority 999 -number_copies 1 -Bidfile %bid-file% -M master -mpx -s %sos-datetime% -e %eos-datetime%%dup-cmd% -PM(echo echo on)   >  %dup-bat%(echo time /t)   >> %dup-bat%(echo %dup-cmd% -L %dup-log%) >> %dup-bat%(echo time /t)   >> %dup-bat%(echo pause)   >> %dup-bat%(echo exit)   >> %dup-bat%REM if not "%1"=="QUIET" pauseREM **************************************************************************************************REM *** Sure to copy...REM **************************************************************************************************echo .echo ####################################choice /c ny /m "%fac% Press `n` to skip this duplication, or `y` to start tape duplication ˜" /t 300 /d nif %errorlevel%==1 goto :eofREM **************************************************************************************************REM *** Do the duplication...REM **************************************************************************************************copy %bid-save% %bid-file% > nulecho .echo ####################################time /techo %fac% Attempting duplication, for policies "%policies%-*"...start "%policies%" %dup-bat%REM if not "%1"=="QUIET" pauseREM **************************************************************************************************REM *** End of duplication sub-routine...REM **************************************************************************************************goto :eofREM **************************************************************************************************REM *** A real example from an NBU process...REM **************************************************************************************************"D:\VERITAS\NetBackup\bin\admincmd\bpduplicate.exe"  -dstunit master-hcart-robot-tld-0  -dp bcp-p1-datavolumes  -rl 3  -fail_on_error 0  -set_primary 1  -cn 1  -priority 99999  -number_copies 1  -L "D:\VERITAS\NetBackup\Logs\NBduplicate\Dup-20060420124818.log"  -Bidfile "C:\Documents and Settings\user\Local Settings\Application Data\VERITAS\NetBackup\20060420124818.bid"  -M master  -mpx

 

[sdo]

show-session.vbs

Code:

Option ExplicitCall s_show_session()WScript.QuitSub s_show_session()  Const cs_fac = "%s_show_session, "  Dim ld_now, ll_now_hour, ls_now_date  Dim ld_sos, ls_sos, ls_sos_date, ll_sos_diff, ll_eos_diff  Dim ld_eos, ls_eos  ld_now = Now()  ll_now_hour = Hour( ld_now )  ld_sos = CDate( FormatDateTime( ld_now, vbShortdate ) )    'Assume today, strip off time...  ld_sos = DateAdd( "h", 16, ld_sos )       'Set to 16:00...  If ll_now_hour < 16 Then    ld_sos = DateAdd( "d", -1, ld_sos )       'Go back one day...  End If  If WScript.Arguments.Count >= 1 Then    If IsNumeric( WScript.Arguments.Item(0) ) Then      ld_sos = DateAdd( "d", CLng(WScript.Arguments.Item(0)), ld_sos )  'Go back(-ve)/forward(+ve) even further...    End if  End If  Select Case WeekDay( ld_sos )  Case vbSunday    ll_sos_diff = -2    ll_eos_diff =  3  Case vbSaturday    ll_sos_diff = -1    ll_eos_diff =  3  Case vbFriday    ll_sos_diff =  0    ll_eos_diff =  3  Case Else    ll_sos_diff =  0    ll_eos_diff =  1  End Select  ld_sos = DateAdd( "d", ll_sos_diff, ld_sos )  ls_sos = WeekDayName( WeekDay( ld_sos ) )  ls_sos = ls_sos & " " & fs_zeroes( DatePart( "m",    ld_sos ), 2 )  ls_sos = ls_sos & "/" & fs_zeroes( DatePart( "d",    ld_sos ), 2 )  ls_sos = ls_sos & "/" & fs_zeroes( DatePart( "yyyy", ld_sos ), 4 )  ls_sos = ls_sos & " " & FormatDateTime( ld_sos, vbLongtime )  ld_eos = DateAdd( "d", ll_eos_diff, ld_sos )  ls_eos = WeekDayName( WeekDay( ld_eos ) )  ls_eos = ls_eos & " " & fs_zeroes( DatePart( "m",    ld_eos ), 2 )  ls_eos = ls_eos & "/" & fs_zeroes( DatePart( "d",    ld_eos ), 2 )  ls_eos = ls_eos & "/" & fs_zeroes( DatePart( "yyyy", ld_eos ), 4 )  ls_eos = ls_eos & " " & FormatDateTime( ld_eos, vbLongtime )  WScript.Echo "start " & ls_sos  WScript.Echo "end   " & ls_eosEnd SubFunction fs_zeroes( pl_number, pl_length )  Const cs_fac = "%fs_zeroes, "  Dim ls_result  ls_result = String( pl_length, "0" ) & CStr( pl_number )  ls_result = Right( ls_result, pl_length )  fs_zeroes = ls_resultEnd Function

 

[sdo]

sort-no-duplicate.vbs

Code:

Option ExplicitCall s_sort_no_duplicates()WScript.Quit(0)Sub s_sort_no_duplicates()  Const cs_fac = "%s_sort_no_duplicates, "  Const ci_for_reading  = 1  Const ci_for_writing  = 2  Const ci_for_appending = 8    Dim lo_fso, ls_inp_spec, ls_out_spec, lo_inp_chan, lo_out_chan  Dim ld_data, ls_data, ls_datum, lb_swapped, ll_i, ll_j  Set lo_fso = CreateObject( "Scripting.FileSystemObject" )  If WScript.Arguments.Count <> 2 Then    Call s_abort( cs_fac & "Script requires two parameters, an input file name and an output file name..." )  End If  ls_inp_spec = WScript.Arguments.Item(0)  ls_out_spec = WScript.Arguments.Item(1)  If lo_fso.GetFile( ls_inp_spec ).Size = 0 Then    WScript.Echo cs_fac & "The input file is empty, nothing to sort, exiting..."    WScript.Quit(0)  End If  On Error Resume Next  Set lo_inp_chan = lo_fso.OpenTextFile( ls_inp_spec, ci_for_reading )  If Err.Number <> 0 Then Call s_error( cs_fac & "Failed to open `" & ls_inp_spec & "`..." )  On Error Goto 0  Set ld_data = CreateObject( "Scripting.Dictionary" )  Do    ls_datum = Trim( lo_inp_chan.ReadLine )    If ls_datum <> "" Then      If Not ld_data.Exists( ls_datum ) Then        ld_data.Add ls_datum, ""      End If    End If  Loop Until lo_inp_chan.AtEndOfStream  lo_inp_chan.Close    If ld_data.Count = 0 Then    WScript.Echo cs_fac & "No data found."    WScript.Quit(1)  End If    Redim ls_data( ld_data.Count )  ll_i = 0  For Each ls_datum In ld_data.Keys    ll_i = ll_i + 1    ls_data( ll_i ) = ls_datum  Next  For ll_i = UBound( ls_data ) - 1 To 1 Step -1    lb_swapped = False    For ll_j = 1 To ll_i      If ls_data( ll_j ) > ls_data( ll_j + 1 ) Then        ls_datum   = ls_data( ll_j )        ls_data( ll_j )  = ls_data( ll_j + 1 )        ls_data( ll_j + 1 ) = ls_datum        lb_swapped = True      End If    Next    If Not lb_swapped Then Exit For  Next    Set lo_out_chan = lo_fso.OpenTextFile( ls_out_spec, ci_for_writing, True )  For ll_i = 1 To UBound( ls_data )    lo_out_chan.WriteLine ls_data( ll_i )  Next  lo_out_chan.Close    WScript.Echo cs_fac & "Written `" & UBound( ls_data ) & "` records to `" & ls_out_spec & "`..."  End SubSub s_error( ps_message )  WScript.Echo ps_message & vbCrlf & "Error `" & Err.Number & "`, description `" & Err.Description & "`, source `" & Err.Source & "`..."  WScript.Quit( Err.Number )End SubSub s_abort( ps_message )  WScript.Echo ps_message  WScript.Quit(1)End Sub