cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Folders not 'visible' for granular restore from FlashBackup-Windows policy

Logicalis_PAF
Level 3
Partner

Hello Everyone!

This is the current configuration for this backup client.

Backup client:   Windows 2008 R2 x64 w/NetBackup 7.1.0.2
NetBackup:        NetBackup 7.6.0.1
Policy:                FlashBackup-Windows

The backup client has a rather large volume (D:) that is approximately 5.6TB and contains close to 40 million files.  We are backing this volume up differently than the OS volume due to it's size and content.  We are currently running a full FlashBackup-Windows job monthly, running weekly cumulatives, and daily differentials.  In attempting to restore specific files from this backup image, it was noticed that while the size of the job indicates that indeed, the entire volume backed up, selecting certain files for a granular restore is not possible.  ***There is a single folder in the root of that volume that contains the vast majority of the 5.6TB of data.  Its named "vol" and is not listed as a folder in the FULL FlashBackup-Windows backup image from either the BAR gui on Windows machines, or the Java gui on either Windows or Linux machines.  Interestingly enough, it IS listed in the cumulative and differential INCREMENTAL backups however...***

I am attempting to set up a new, temporary client that I can restore the entire raw partition image to just to confirm that the folders are being backed up and that in a DR scenario, we would be able to recover the entire volume, however even if that is the case, granular restores will be rather cumbersome unless we can get this ironed out.  Symantec currently IS working on this as I opened a support case when I originally thought that the backup was just somehow missing that folder, but I have updated Symantec Support that it appears the "file mapping" aspect of the FlashBackup-Windows policy is failing in this particular case.  I've shared this issue with the community because we're all very good at solving complex issues, and if anyone else runs into this, it would be nice to be able to find the solution without having to jump through the Symantec Support hoops.  Once resolved, if resolved, I will certainly update this post.

To try and head off the little stuff, I will confirm that we checked and double-checked that there were no exclusion issues.  We re-created the policy from scratch to confirm that there were no policy "corruption" issues that we were running into, and the problem persists.  My guess is that it may possible be due to the folder-depth or sheer size that is causing the process to fail, but until I can do the restore test, its all conjecture.  I also know that this will be much less painful if the customer were to segment their data somewhat, and that is one of my recommendations but is something to be addressed after this 'restorability' issue is resolved.  I also have upgraded the NetBackup client to 7.6.0.1 but as it takes about 18hrs to run this particular backup, I don't know if that has resolved it yet...

I've attached screen-scrapes of the drive size and its folder contents from the Windows client, the full restore as well as the incremental restore views from BAR to help show that the 'vol' folder is not visible or selectable to restore from the full FlashBackup-Windows backup but is in the smaller incrementals.

Hopefully, someone has run across this before and can provide some insight.
 

Thanks all!
 

Pat

1 ACCEPTED SOLUTION

Accepted Solutions

Logicalis_PAF
Level 3
Partner

Final update:

So, updating the client from 7.1.0.2 to 7.6.0.1 resolved the issue.  All new full backups with the existing policies and the upgraded client properly show all folders in the root of the drive and allow browsing and restoration granularity at the file level.  According to Symantec, there is an issue with dynamic Windows partitions and the 7.1.x clients, although this information doesnt appear readily available anywhere else.

 

Thank you everyone for your input, insight, and assistance.  As always, a phenomenol group of people out there supporting NetBackup in their own capacities.

 

Pat

View solution in original post

21 REPLIES 21

Veerendra
Level 3
Partner Accredited Certified

Can you verify  your Windows client backups , their image size and file count (compare full with incrementals).

 

bpimagelist -client clientname -d 10/01/2013 00:00:00 -e 2/22/2014 00:00:00 -U

Logicalis_PAF
Level 3
Partner
Absolutely, and the interesting thing is that by looking through the restore GUI,
there are less that 50 files visible in the few folders that can be browsed, however
the full backups we are choosing to restore from where the 'vol' folder is not
visible shows file counts well over 38 million:


root@mymediaserver ~]# bpimagelist -client mybackupclient -d 02/01/2014 00:00:00 -e 2/22/2014 00:00:00 -U
Backed Up         Expires       Files       KB  C  Sched Type      On Hold Index Status Policy
----------------  ---------- -------- --------  -  --------------- ------- ------------ ------------
02/21/2014 00:04  03/07/2014    65465 13752291  N  Differential In 0       0            CUSTOMER-flash-D
02/21/2014 00:00  03/07/2014    16592 11530458  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/20/2014 20:04  02/20/2015 38888632 6123567084N  Full Backup     0       0            CUSTOMER-flash-D-new
02/20/2014 00:03  03/06/2014    39288 56014264  N  Differential In 0       0            CUSTOMER-flash-D
02/20/2014 00:00  03/06/2014    16459 11217487  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/19/2014 00:02  03/05/2014   398556 33409628  N  Differential In 0       0            CUSTOMER-flash-D
02/19/2014 00:00  03/05/2014    16378 11170540  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/18/2014 00:04  03/04/2014    31740 30350471  N  Differential In 0       0            CUSTOMER-flash-D
02/18/2014 00:00  03/04/2014    16274 11032614  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/17/2014 00:03  03/03/2014    10513  2668954  N  Differential In 0       0            CUSTOMER-flash-D
02/17/2014 00:00  03/03/2014    16201 10952068  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/16/2014 00:02  03/02/2014     9443  2594315  N  Differential In 0       0            CUSTOMER-flash-D
02/16/2014 00:00  03/02/2014    16185 10919216  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/15/2014 00:02  03/01/2014   416795 69235363  N  Cumulative Incr 0       0            CUSTOMER-flash-D
02/15/2014 00:00  03/01/2014   117871 21331753  N  Full Backup     0       0            CUSTOMER-ald-win
02/14/2014 00:03  02/28/2014    33793  6890997  N  Differential In 0       0            CUSTOMER-flash-D
02/14/2014 00:00  02/28/2014    16460 11222217  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/13/2014 00:03  02/27/2014    39442  7592444  N  Differential In 0       0            CUSTOMER-flash-D
02/13/2014 00:00  02/27/2014    16415 11179253  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/12/2014 00:02  02/26/2014    38441  7169505  N  Differential In 0       0            CUSTOMER-flash-D
02/12/2014 00:00  02/26/2014    16364 11144262  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/11/2014 00:04  02/25/2014    40364  7787699  N  Differential In 0       0            CUSTOMER-flash-D
02/11/2014 00:01  02/25/2014    16338 11095744  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/10/2014 00:03  02/24/2014     9437  2581152  N  Differential In 0       0            CUSTOMER-flash-D
02/10/2014 00:01  02/24/2014    16264 11035458  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/09/2014 00:03  02/23/2014    10970  2615159  N  Differential In 0       0            CUSTOMER-flash-D
02/09/2014 00:00  02/23/2014    16182 10953086  N  Cumulative Incr 0       0            CUSTOMER-ald-win
02/08/2014 00:24  04/12/2014   231279 38863176  N  Cumulative Incr 0       0            CUSTOMER-flash-D
02/08/2014 00:01  04/12/2014   117736 21280777  N  Full Backup     0       0            CUSTOMER-ald-win
02/01/2014 00:02  02/01/2015 38102180 5807075212N  Full Backup     0       0            CUSTOMER-flash-D
02/01/2014 00:00  02/01/2015   117622 21185215  N  Full Backup     0       0            CUSTOMER-ald-win

Logicalis_PAF
Level 3
Partner

Just as an update, I have initated a restore of the raw partition file to a Windows server with a 6TB volume to see if the folders are present within the backup, which I'm almost 100% certain they are given the size and filecount of the backup.

Pat

Veerendra
Level 3
Partner Accredited Certified

Its look like you have mismatched the policy name ( but i am not sure). Verify the policy name on BAR Console. Those screenshot itself indicating both policies are different.

 

Backed Up          Expires        Files        KB  Sched Type      On Hold Index Status Policy
2/21/2014 0:04   03/07/2014     65465 13752291  Differential In 0       0            CUSTOMER-flash-D
2/20/2014 0:03   03/06/2014     39288 56014264  Differential In 0       0            CUSTOMER-flash-D
2/19/2014 0:02   03/05/2014    398556 33409628  Differential In 0       0            CUSTOMER-flash-D
2/18/2014 0:04   03/04/2014     31740 30350471  Differential In 0       0            CUSTOMER-flash-D
2/17/2014 0:03   03/03/2014     10513  2668954  Differential In 0       0            CUSTOMER-flash-D
2/16/2014 0:02   03/02/2014      9443  2594315  Differential In 0       0            CUSTOMER-flash-D
2/15/2014 0:02   03/01/2014    416795 69235363  Cumulative Incr 0       0            CUSTOMER-flash-D
2/14/2014 0:03   02/28/2014     33793  6890997  Differential In 0       0            CUSTOMER-flash-D
2/13/2014 0:03   02/27/2014     39442  7592444  Differential In 0       0            CUSTOMER-flash-D
2/12/2014 0:02   02/26/2014     38441  7169505  Differential In 0       0            CUSTOMER-flash-D
2/11/2014 0:04   02/25/2014     40364  7787699  Differential In 0       0            CUSTOMER-flash-D
2/10/2014 0:03   02/24/2014      9437  2581152  Differential In 0       0            CUSTOMER-flash-D
2/9/2014 0:03   02/23/2014     10970  2615159  Differential In 0       0            CUSTOMER-flash-D
2/8/2014 0:24   04/12/2014    231279 38863176  Cumulative Incr 0       0            CUSTOMER-flash-D
2/1/2014 0:02   02/01/2015 38102180 5807075212 Full Backup     0       0            CUSTOMER-flash-D
                 
Backed Up          Expires        Files        KB  Sched Type      On Hold Index Status Policy
2/21/2014 0:00   03/07/2014     16592 11530458 Cumulative Incr 0       0            CUSTOMER-ald-win
2/20/2014 0:00   03/06/2014     16459 11217487  Cumulative Incr 0       0            CUSTOMER-ald-win
2/19/2014 0:00   03/05/2014     16378 11170540  Cumulative Incr 0       0            CUSTOMER-ald-win
2/18/2014 0:00   03/04/2014     16274 11032614  Cumulative Incr 0       0            CUSTOMER-ald-win
2/17/2014 0:00   03/03/2014     16201 10952068  Cumulative Incr 0       0            CUSTOMER-ald-win
2/16/2014 0:00   03/02/2014     16185 10919216  Cumulative Incr 0       0            CUSTOMER-ald-win
2/15/2014 0:00   03/01/2014    117871 21331753  Full Backup     0       0            CUSTOMER-ald-win
2/14/2014 0:00   02/28/2014     16460 11222217  Cumulative Incr 0       0            CUSTOMER-ald-win
2/13/2014 0:00   02/27/2014     16415 11179253  Cumulative Incr 0       0            CUSTOMER-ald-win
2/12/2014 0:00   02/26/2014     16364 11144262  Cumulative Incr 0       0            CUSTOMER-ald-win
2/11/2014 0:01   02/25/2014     16338 11095744  Cumulative Incr 0       0            CUSTOMER-ald-win
2/10/2014 0:01   02/24/2014     16264 11035458  Cumulative Incr 0       0            CUSTOMER-ald-win
2/9/2014 0:00   02/23/2014     16182 10953086  Cumulative Incr 0       0            CUSTOMER-ald-win
2/8/2014 0:01   04/12/2014    117736 21280777  Full Backup     0       0            CUSTOMER-ald-win
2/1/2014 0:00   02/01/2015    117622 21185215  Full Backup     0       0            CUSTOMER-ald-win

 

Backed Up          Expires        Files        KB  Sched Type      On Hold Index Status Policy
2/20/2014 20:04   02/20/2015 38888632 6123567084 Full Backup     0       0            CUSTOMER-flash-D-new

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Seems you have Full and Incr schedules in different policies?

NBU needs them to be in the SAME policy for Incrementals to work properly. 
FlashBackup also needs Incr Backup config to be set to Modification date/time (the default is Archive bit).

Please also check if 'vol' is a separate physical volume that is mounted under D.
If so, you need to specify it separately:
\\.\D:\vol\

Logicalis_PAF
Level 3
Partner

Yeah, I apologize for that.  I left the OS backup policy in that list, which are NON-FlashBackup-Windows
policies.  For THAT volume only, we use FlashBackup-Windows policies.  The following list is just backups
for that volume.  Sorry for the confusion...


root@mymediaserver ~]# bpimagelist -client mybackupclient -d 02/01/2014 00:00:00 -e 2/22/2014 00:00:00 -U
Backed Up         Expires       Files       KB  C  Sched Type      On Hold Index Status Policy
----------------  ---------- -------- --------  -  --------------- ------- ------------ ------------
02/21/2014 00:04  03/07/2014    65465 13752291  N  Differential In 0       0            CUSTOMER-flash-D
02/20/2014 00:03  03/06/2014    39288 56014264  N  Differential In 0       0            CUSTOMER-flash-D
02/19/2014 00:02  03/05/2014   398556 33409628  N  Differential In 0       0            CUSTOMER-flash-D
02/18/2014 00:04  03/04/2014    31740 30350471  N  Differential In 0       0            CUSTOMER-flash-D
02/17/2014 00:03  03/03/2014    10513  2668954  N  Differential In 0       0            CUSTOMER-flash-D
02/16/2014 00:02  03/02/2014     9443  2594315  N  Differential In 0       0            CUSTOMER-flash-D
02/15/2014 00:02  03/01/2014   416795 69235363  N  Cumulative Incr 0       0            CUSTOMER-flash-D
02/14/2014 00:03  02/28/2014    33793  6890997  N  Differential In 0       0            CUSTOMER-flash-D
02/13/2014 00:03  02/27/2014    39442  7592444  N  Differential In 0       0            CUSTOMER-flash-D
02/12/2014 00:02  02/26/2014    38441  7169505  N  Differential In 0       0            CUSTOMER-flash-D
02/11/2014 00:04  02/25/2014    40364  7787699  N  Differential In 0       0            CUSTOMER-flash-D
02/10/2014 00:03  02/24/2014     9437  2581152  N  Differential In 0       0            CUSTOMER-flash-D
02/09/2014 00:03  02/23/2014    10970  2615159  N  Differential In 0       0            CUSTOMER-flash-D
02/08/2014 00:24  04/12/2014   231279 38863176  N  Cumulative Incr 0       0            CUSTOMER-flash-D
02/01/2014 00:02  02/01/2015 38102180 5807075212N  Full Backup     0       0            CUSTOMER-flash-D

Logicalis_PAF
Level 3
Partner

Further background information.  The volume we are backing up in this policy is a spanned Windows volume totalling around 5.8TB.  All of the folders that should be in the backup are standard NTFS folders on that volume, hence the backup specification of \\.\D:\.

The OS disk is being backed up with a MS-Windows policy that contains weekly full backups and daily incrementals.  When I generated the list I sent, I failed to filter out those non-FlashBackup-Windows policies, so I apologize for the confusion. 

 

Backed Up Expires Files KB C Sched Type On Hold Index Status Policy
2/21/2014 0:04 3/7/2014 65465 13752291 N Differential In 0 0 CUSTOMER-flash-D
2/20/2014 0:03 3/6/2014 39288 56014264 N Differential In 0 0 CUSTOMER-flash-D
2/19/2014 0:02 3/5/2014 398556 33409628 N Differential In 0 0 CUSTOMER-flash-D
2/18/2014 0:04 3/4/2014 31740 30350471 N Differential In 0 0 CUSTOMER-flash-D
2/17/2014 0:03 3/3/2014 10513 2668954 N Differential In 0 0 CUSTOMER-flash-D
2/16/2014 0:02 3/2/2014 9443 2594315 N Differential In 0 0 CUSTOMER-flash-D
2/15/2014 0:02 3/1/2014 416795 69235363 N Cumulative Incr 0 0 CUSTOMER-flash-D
2/14/2014 0:03 2/28/2014 33793 6890997 N Differential In 0 0 CUSTOMER-flash-D
2/13/2014 0:03 2/27/2014 39442 7592444 N Differential In 0 0 CUSTOMER-flash-D
2/12/2014 0:02 2/26/2014 38441 7169505 N Differential In 0 0 CUSTOMER-flash-D
2/11/2014 0:04 2/25/2014 40364 7787699 N Differential In 0 0 CUSTOMER-flash-D
2/10/2014 0:03 2/24/2014 9437 2581152 N Differential In 0 0 CUSTOMER-flash-D
2/9/2014 0:03 2/23/2014 10970 2615159 N Differential In 0 0 CUSTOMER-flash-D
2/8/2014 0:24 4/12/2014 231279 38863176 N Cumulative Incr 0 0 CUSTOMER-flash-D
2/1/2014 0:02 2/1/2015 38102180 5807075212 N Full Backup 0 0 CUSTOMER-flash-D

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I am curious to see bpdbm and bprd logs on the master to see the difference in browsing for Incremental and Full backups.

It honestly seems as if a completely different policy is used for the 2 types (yes, I have seen the latest output that shows same policy name).

I am also curious to see what the difference is between the 3 policies for this client: 
CUSTOMER-ald-win
CUSTOMER-flash-D
CUSTOMER-flash-D-new

Which backup date did you select when browsing for backups?
bpimagelist output shows that all of these policies ran on the same day (20 Feb).

 

Logicalis_PAF
Level 3
Partner

   So, just to update everyone, we did a WebEx with Symantec Support last night where they verified everything was configured properly with the policies, client, and any excludes.  They recommended that we run a full backup of this client with the upgraded 7.6.0.1 client installed.  They mentioned that they have seen similar issues with the 7.1 client and large spanned dynamic disk volumes on Windows clients.  We are currently running that backup and its approximately 50% completed.  I expect to have an answer to whether that solved the problem or not, so I will be sure to post an update at that time.

  Unfortunately, the full partition restore we were doing ended when the job manager on the master server crashed while forcfully ending a rogue backup process, so I've begun duplicating the backup image off those tapes to disk-based storage in an effort to facilitate the restore process and avoid the cumbersome nature of multi-tape restores.  Hope to have a result of the restore process I will begin once that completes by tomorrow as well.

Thanks to everyone for their input - as always, its much appreciated.

Pat

Logicalis_PAF
Level 3
Partner

The three policies are as follows:

CUSTOMER-ald-win
       This is a MS-Windows policy that specifies ALL_LOCAL_DRIVES as the selection list.  On the client, there is a policy specific exclusion of drive D: for this.

CUSTOMER-flash-D
      This is the original flashbackup policy with the \\.\D: selection list that generated all of the backups in question.

CUSTOMER-flash-D-new
     This is the new flashbackup policy I had the engineer create and run on 2/20 to confirm that it was not an issue with that specific policy.

For the backup date when browsing, ANY full backup we select is 'missing' several folders, vol being the main one in question.  If there are changes in any of these directories, they do show up in the incrementals and cumulatives, just not in ANY full on ANY date that the fulls ran.

I'm not sure where you're seeing the same date for all of those images in the bpimagelist as I see from 02/01/2014 to 02/21/2014 in the "Backed Up" column, or are you referring to something else?

RoundTower_JGL
Level 3
Partner Accredited Certified

Pat,

Have you tried using the bplist or bpflist commands to see if your vol directory shows up in the images? I think for Flashbackup-Windows policy types you'll want to use the -t 29 option.

Joe

Mark_Solutions
Level 6
Partner Accredited Certified

Have you tried viewing the restore via a Windows BAR GUI?

There have been known differences between the Java and Windows GUi and maybe that would help.

It is also worth putting the client read timeouts on the Media Server and Master server (and Admin Console PC) right up to prevent a timeout during the image browse making some file not become available to view (though  I would have expected a timeout / database error to be honest)

hope this helps

Logicalis_PAF
Level 3
Partner

Final update:

So, updating the client from 7.1.0.2 to 7.6.0.1 resolved the issue.  All new full backups with the existing policies and the upgraded client properly show all folders in the root of the drive and allow browsing and restoration granularity at the file level.  According to Symantec, there is an issue with dynamic Windows partitions and the 7.1.x clients, although this information doesnt appear readily available anywhere else.

 

Thank you everyone for your input, insight, and assistance.  As always, a phenomenol group of people out there supporting NetBackup in their own capacities.

 

Pat

Logicalis_PAF
Level 3
Partner

Hi Joe,

      Its interesting you brought that up, but that is literally the first thing I tried, followed by a command-line bprestore to see if I could request what wasn't visible.  Ultimately, no dice.  Some non-error producing glitch with the client version (7.1.0.2) and a massive spanned volume of multiple dynamic disks in Windows 2008 R2.

Thanks for the input.

Pat

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Glad to know issue was solved.

Don't we just love these 'known issues'? Just not to the general public??

Please as your Support engineer to publish a TechNote.

CRZ
Level 6
Employee Accredited Certified

DISCLAIMER: It's always easier to find an answer after you've resolved your issue.  wink

I believe you were hitting the defect covered in this TechNote (although the symptoms were different):

BUG REPORT: After applying NetBackup 7.1.0.2 browsing in the BAR GUI hangs for some images and the restore fails with status 233 "premature eof encountered"
 http://symantec.com/docs/TECH179269

Going to 7.1.0.3 or 7.1.0.4 should also have fixed you up.

This is another example of Support's oft-repeated advice, "please upgrade to the latest version," being the best advice.  If you had been at 7.1.0.4, you wouldn't have seen this problem.

Heck, even 7.1.0.3 - which came out in November 2011! - would have resolved this one.

Mark_Solutions
Level 6
Partner Accredited Certified

But doesnt that relate more to the bpdbm process on the Master and as it also involves the "/" then relates to the Java GUI being used?

An yet upgrading the client resolves it?

Maybe i am missing something here but unless the Java GUI was being run on the client why would that tech note apply?

Out of interest Pat did you try the Windows BAR GUI as i suggested?

Logicalis_PAF
Level 3
Partner

Thank you Chris, but this isnt actually the issue.  The issue was that the file mapping was incomplete on the existing images, not that the browsing of properly structured and mapped images was faulty.  The backup image was broken, and in order to fix it, the client had to be upgraded past 7.1.0.2.

Pat

Logicalis_PAF
Level 3
Partner

I apologize as I thought I had put that information into the original post, but yes, I tried the Windows BAR GUI, as well as the Java Admin Console on both Linux and Windows.  Essentially, NetBackup did not know about the folders that were missing from the backup images, although the file count and the size were recorded properly in the catalog.  Restoring the entire raw-partition image yields the expected results and the folders and files are present on the partition post-restore.

I will most definitely request that the support tech post a tech-note to make this image gathering process less painful for those who encounter them after me.