10-28-2016 01:23 PM
I've been asked to complete a restore of multiple user drives, across multiple servers, over multiple years....I know...sounds like fun, right? Here are my challenges...
1. Generate a list of files to be restored. I have the user list, but they could be on any of 20 different client servers. I was thinking of using bplist to find them, but can't for the life of me get past the dreaded 227 error when I run it from the Master server. Oh, and some of those clients don't exist anymore....just the backups.
- I've tried variations for bplist such as: bplist -S master -C client -t 13 -R c:\pathname on client
2. Use powershell, to take the list from above and generate the restores with the files being directed to another location using bprestore is my guess.
I'm running NBU 7.6.0.1 on W2K8 and my install path is not the default.
Any help would be really appreciated as this is probably going to take weeks as it is.
Solved! Go to Solution.
11-02-2016 02:45 AM - edited 11-02-2016 03:11 AM
cd to your working folder where your script resides...
you should have two files already:
list-of-source-clients.txt (contains 7 to 20 old server - i.e. backup client names)
list-of-usernames.txt (contains 50 to 51 old usernames, partial strings, i.e. not full paths)
...then use this to generate a list of all MS-Windows type backup images in a two year date range:
for each record in list-of-source-clients.txt
extract clientname
bpimagelist -idonly -d twoyearsagodatetime -e todaydatetime -pt MS-Windows -client $clientname > images.txt
for each record in images.txt
extract imagename
extract imagedatetime
for each record in list-of-usernames.txt
extract username
bplist -C $clientname -s $imagedatetime -e $imagedatetime -t 13 -I -PI -R 999 -nt_files *$username* >>files-$imagename.txt
next
next
next
...then you would end up with a bunch of files named "file-*.txt" which contain a list of files to be restored from each image.
The above was the easy part.
Here comes the tricky part:
dir /b files-*.txt > list-of-files.txt
set targetserver = blahblah (i.e. the name of the server that restores will be written to)
delete restore-commands.txt
for each record in list-of-files.txt
extract backupfilename
extract imagename (from $backupfilename)
extract clientname (from $imagename)
extract imagectime (from $imagename)
convert imagectime to imagedatetime
set renamefile = restore-$imagename.txt
for each record in $backupfilename
extract sourcefilename
set sourcelen = len( sourcefilename )
set targetfilename = "F:\$imagename\" + Mid( $sourcefilename, 4, 999 )
set targetlen = len( targetfilename )
write $renamefile "rename $sourcelen $sourcefilename $targetlen $targetfilename"
next
write restore-commands.txt "bprestore -B -R $renamefile -C $clientname -s $imagedatetime -e $imagedatetime -f $sourcefilename -D $targetserver"
next
FYI - it is important that there is a one-to-one relationship between each file name entry in:
-f sourcenames
-R renamenames
...and so the above should achieve:
- separate restore folder for each file in backups
- restored folder structure maintainted underneath F:\imagenames\
- and then you can run a tool to detect duplicate files and delete, so that only unique files are left.
- and a list of restore commands in "restore-commands.txt"
...and so could pick the first one to test an actual restore.
It may be possible to also capture and calculate the total size of what is to be restored. But I'll let you work out how to do that :)
HTH.
10-28-2016 03:48 PM
10-28-2016 03:49 PM
Do you have Opscneter in place...if yes use the restore option in Opscenter.. it will search for you if you give the directory or file name for restore
you might need to give the client names and dirctory or file names to search.. you can trigger the restore from Opscneter as well..
even if you get issues triggering the restores from Opscneter.. at lest you will get to know what are the clietns havign the data and date details... to trigger them from Master server...
10-28-2016 04:07 PM
10-31-2016 01:14 AM
Have you confirm that the Client name used in bplist corresponds with client name used at time of backup?
Can you browse ...netbackup\db\images folder and locate correct client name in subfolders?
Bear in mind that NBU is case sensitive and you need to determine if clients were baked up using shortname or FQDN.
Client1 != client1 != client1.fqdn
Once you have located the client folder under images, can you browse the 10-digit-number-folders in the client folder and see valid catalog entries?
Once you have correct client folder name and 10-digit folder containing valid images, you can convert the timestamp for one of the oldest images with bpdbm command:
bpdbm -ctime <10-digit-number>
e.g.
10-31-2016 06:19 AM - edited 10-31-2016 06:23 AM
Yes, bplist can be a troublesomely command to use, but you can get it to behave :D
I have written a small blog about bplist
http://www.mass.dk/netbackup-quick-hints/bplist-a-handy-tool-for-the-netbackup-admin/
Try the bplist command again, but add the -R qualifier.
11-01-2016 07:47 AM
Hey SDO,
Thanks, for the most part they should be known users with known folder paths. I have some PoSh knowledge and can get help as necessary. PowerwShell was the direction I was thinking of going, so any help will be appreciated.
Dan
11-01-2016 07:49 AM
Hey Marianne,
After a small amount of fighting, I have been able to rid myself of the nasty 227 error, thanks.So now I can get an output list of all the files across all the backups that i'll need. So now, it comes to how do I restore all the files, across all the versions of backups, for the 51 users that are required?
11-01-2016 07:53 AM
Hey Nicolai, thanks for the link. It definitely did help. I'm still trying to figure out how to browse a list of clients to figure out which one the User's drive resided on. 50 users out of 10,000 isn't that easy for sorting.
11-01-2016 12:22 PM - edited 11-01-2016 12:34 PM
hey, arent you the lucky one !!!!!
i have underwent similar case, first things first
are you restoring from tapes? or disk (may be DataDomain or something), if its tape then you have more trouble in identifying the tapes needed for restore, if its DISK then its lots simpler
1) is your source files under diffrent servers same ? (with user name diffrents)
2) do you have the dates from which your planning to restore
3) chose same destination path (e:\restore\) for all restoer you want to perform.
4) use bprestore command to restore the file
bprestore -s <start date> -e <end date> -t 13 -C <source client> -D <dest. client> -L c:/temp/restore/logs/Restore1.log -R <path to file name where you have defined restore destiantion> -f <path of teh file name where you defined list of files>
restore command example : from which i did (it was unix though)
/usr/openv/netbackup/bin/bprestore -s 3/26/2015 -e 3/26/2015 -t 13 -C Source_Client -D Dest_client -L /tmp/restore/logs/Restore_`date +%h%d_%R`.log -R /tmp/restore/alt_location -f /tmp/restore/filelist
11-01-2016 01:48 PM
Thanks for the info....but still looking for more.
I need to figure out which client (out of 7) the user existed on during the time frame. My date range is everything that was created or modified over a two year period, and the destination will be a different server entirely. The date range can be extrapolated via bplist, so really it will just be finding their original client. Powershell will end up being my friend by the end of this.
Tapes are tapes....we'll be loading lots of them for this restore.
11-01-2016 07:06 PM - edited 11-01-2016 07:08 PM
for each record in list-of-7-clients
read clientname
bpimagelist -idonly -s twoyearsago -e today -pt MS-Windows > images-$clientname.txt
for each record in images-$clientname.txt
extract imagedatetime
for each record in list-of-usernames.txt
extract username
bplist -C $clientname -t 13 -s $imagedatetime -e $imagedatetime -nt_files -I -PI -R 999 *$username* >> files-$clientname.txt
next
next
next
for each filename in files-*.txt
extract clientname from filename
create a restore-$clientname.txt
call bprestore
next
11-02-2016 12:11 AM
If you need to restore everything that was backed up over a period of 2 years, how will you handle different versions of the same file?
Seems bprestore has only 2 options - overwrite existing file or to keep existing file (no option to restore with temp filename).
Extract from Commands manual:
-K
Causes bprestore to keep existing files rather than overwrite them when it
restores files with the same name. The default condition is to overwrite existing
files.
11-02-2016 01:38 AM
Good point Marianne - I'd missed that re needing to restore all instances. I'll re-do the psuedo code.
11-02-2016 02:45 AM - edited 11-02-2016 03:11 AM
cd to your working folder where your script resides...
you should have two files already:
list-of-source-clients.txt (contains 7 to 20 old server - i.e. backup client names)
list-of-usernames.txt (contains 50 to 51 old usernames, partial strings, i.e. not full paths)
...then use this to generate a list of all MS-Windows type backup images in a two year date range:
for each record in list-of-source-clients.txt
extract clientname
bpimagelist -idonly -d twoyearsagodatetime -e todaydatetime -pt MS-Windows -client $clientname > images.txt
for each record in images.txt
extract imagename
extract imagedatetime
for each record in list-of-usernames.txt
extract username
bplist -C $clientname -s $imagedatetime -e $imagedatetime -t 13 -I -PI -R 999 -nt_files *$username* >>files-$imagename.txt
next
next
next
...then you would end up with a bunch of files named "file-*.txt" which contain a list of files to be restored from each image.
The above was the easy part.
Here comes the tricky part:
dir /b files-*.txt > list-of-files.txt
set targetserver = blahblah (i.e. the name of the server that restores will be written to)
delete restore-commands.txt
for each record in list-of-files.txt
extract backupfilename
extract imagename (from $backupfilename)
extract clientname (from $imagename)
extract imagectime (from $imagename)
convert imagectime to imagedatetime
set renamefile = restore-$imagename.txt
for each record in $backupfilename
extract sourcefilename
set sourcelen = len( sourcefilename )
set targetfilename = "F:\$imagename\" + Mid( $sourcefilename, 4, 999 )
set targetlen = len( targetfilename )
write $renamefile "rename $sourcelen $sourcefilename $targetlen $targetfilename"
next
write restore-commands.txt "bprestore -B -R $renamefile -C $clientname -s $imagedatetime -e $imagedatetime -f $sourcefilename -D $targetserver"
next
FYI - it is important that there is a one-to-one relationship between each file name entry in:
-f sourcenames
-R renamenames
...and so the above should achieve:
- separate restore folder for each file in backups
- restored folder structure maintainted underneath F:\imagenames\
- and then you can run a tool to detect duplicate files and delete, so that only unique files are left.
- and a list of restore commands in "restore-commands.txt"
...and so could pick the first one to test an actual restore.
It may be possible to also capture and calculate the total size of what is to be restored. But I'll let you work out how to do that :)
HTH.
11-02-2016 02:57 AM - edited 11-02-2016 02:58 AM
IMO, and it is only an opinion, I myself personally wouldn't bother attempting to do this in PowerShell, which has terrible string manipulation. It should be far easier and quicker to develop in VBScript, and easier to debug and work out what is going on.
Let me know how you get on. If you make a decent attempt but find that you struggle, then I might find some time to help you do this in VBScript.
11-02-2016 10:55 AM
Thanks for your help everyone.
SDO, i may look at VB, but my skills there are pretty limited. Will probably just tackle vi PoSh. Thanks for the help with the script. Looks to be very useful.
Danbert
11-03-2016 03:45 AM
Thanks for the solution mark.
And good luck with this task.
11-03-2016 05:31 AM
To get bplist to list only files and not folders...
...then you could:
bplist -switches -switches -etc | findstr /v "\\$"
...so lines/records/paths ending in "\" (i.e. directories) do not make it in to your lists, i.e. so that only actual file names make it in to your lists.
11-03-2016 06:22 AM
That will certainly help for determining the size, etc...but this will end up being for a forensic restore, so i'll need to keep it intact as much as possible. But a great suggestion that i'll keep in mind for future restores.