Missing history in backup archive and restore including CLI

[unkn0wnn]

Hello,

After changing slp policies to expire after copy our backup history is only available for the last month or so.

I think this it could be due to an issue as explained in ===> https://www.veritas.com/support/en_US/article.000094999
Unless some kind of cleanup routines removed / archived history.

Problem
If assignment of a custom retention level causes an image's expiration time to be set beyond the UNIX epoch time of January 19, 2038 03:14:08 UTC, images managed by a Storage Lifecycle Policy (SLP) will expire after duplication is complete.

I am trying to recover this valuable information without importing each tape as we have over 600 of them..

So I far I have found full catalog backup in 1 months old tape (month end backup), i have 2 images but the DR one in the usual location does not contain NBDB data but just cat files.

I have done bprecovery -wizard and it's restored over 30,000 files, these went to ./db/images/*/ ,

bprecovery failed to recover nbdb data due to the dr backup not having these files in image, wizard tried to expire month end infinity image as per its routine but it's failed as well..

So is there an easy way to restore backup history if images have expired ? Personally I do not think these have expired but could have been archived etc...

We are running master and disaster recovery appliances (both n5220, 7.5.0.5).

I do not think that bprecovery is an option as it contains 30 days old dat meaning that current catalog database would have been lost right ?

Catalog Merge sounds ok to me however could we not just read these files somehow ?  I remember that in older versions of netbackup all you do is restore db/images/ directories and history is available..

Any help from your side is much appreciated,

[Marianne]

Please show us a screenshot of your SLP (similar to the one in the TN).

Any reason why you are still using an EOSL version of NBU?
We see that the issue was solved in a later version/patch of 7.5.x.

[unkn0wnn]

Hello, Please find attached.

Beacause I have a very limited time to perform an upgrade, I think we have patched in early 2013.

Could you please help to restore backup history so I can do restores ? I think I we need to merge catalogs somehow...

[Marianne]

You don't have the issue that is explained in the TN.

Your issue is with with short retentions - your longest retention is 27 days - less than a month.
That is why you don't have any history older than a month.

After 2 weeks, your backups expire. After 27 days, your duplications expire. 

At this point all catalog entries are deleted and backups on disk deleted as well.

There is nothing you can do to recover the backups. Catalog recovery or merge won't help. The backups are gone.

PS: Schedule upgrade as a matter of urgency. 
You have NO support with your current version.

[mph999]

Following on from Mariannes excelleny response - once the backup expire, indeed, all catalog information for that image(s) is removed from NBU.

If the images are on tape, then providing the tape is not overwritten, there is a possiblity to import the tape, and rebuild the catalog information for the images it holds.

The problem with this, is that it takes 'on average', several hours to import a single tape.

[unkn0wnn]

Hello, thank you for your help

I am trying to see history for initifty backups, is there no way to merge older catalg to recover as  these images should never expire ?, see slp please:

[Marianne]

Last time you showed us SLP with 27 days retention.
When did you add this SLP?
When did you update policies with this SLP? Can you tell us when backups and duplication ran against this SLP?

If you are sure that backups and duplications ran against this SLP, then the history WILL be in NBU. Unless....
Did someone perhaps expire media/images?

Recovering catalog is not the way to get back expired (missing) images. You will need to import tapes as per @mph999 's post above. Provided the tapes have not been overwritten.

[unkn0wnn]

Hello,

We have more than 1 SLP in place, we have changed 2 slps to expire copy on disk once copy to tape is created and due to that setting images have expired prematurely.

We have done this on 2 different systems and have experienced exactly the same problem.

Image that I've attached contains current setting.

So in the event of disaster failure we would have to re-image all the month end tapes as well ?

Is there no way to recalculate infinity backups somehow ? Would netbackup not create some kind of archive information once it expired history in DB then ?

Sorry to be a pain, I still believe this is doable as image history is kept in catalog.

[Marianne]

Let me re-phrase my question - were any NEW backups done with the new settings?

Any backups that were submitted against the old settings will still have the 2-week on disk and 27 days on tape retention.

Only new backups done with the new SLP settings will have the Infinity retention on tape.

There is no way to 'recalculate' expired images.
Expired images remove all records in the catalog for these images.

If tapes have not been overwritten, your only way to get the images back will be to import the tapes.

[unkn0wnn]

Let me re-phrase my question - were any NEW backups done with the new settings?

Yes, plenty of daily backups.

Only new backups done with the new SLP settings will have the Infinity retention on tape.

Why would that be? When it comes to month end slp then previously we have only amended retention period on disk to expire after copy... then netbackup expired all images somehow.. retention period for tapes was left as infinity.

Month end tapes are write protected so no problem however we only have 2 drives, these are always occupied by netbackup.

Is there no way to even read these files somehow using some linux editor etc ?

Old Backup history for infinity backups is gone due to netbackup setting "expire after copy"...I think we should never amend existing policy but create new one instead to avoid that behaviour right ?

[Marianne]

Please show us output of some of the daily (not yet expired) images that were done with this SLP.
e.g.
bpimagelist -L -policy policy_name -sl schedule_name -d date -e date 
(e.g. to list backups for Monday and Tuesday where policy name is Policy1 and schedule name is Daily:
bpimagelist -L -policy Policy1 -sl Daily -d 11/13/2017 -e 11/14/2017  )
and policy config :
bppllist policy-name -U 

Only new backups done with the new SLP settings will have the Infinity retention on tape.
This is how SLPs works.
Any outstanding SLP duplications at the time when the change was made, where backups were done with old 2 week backup retention and 27 day retention for duplications, would still have those retentions. 

"Old Backup history for infinity backups is gone due to netbackup setting "expire after copy".."

Absolutely impossible.
ONLY new backups done with the new SLP will have the backup and duplication retentions.
Also, when the backup copy expires, you will still have the tape copy with the configured retention.

I am sure @mph999 will confirm this.

[mph999]

Modified SLPs, will only affect new backups taken from that point onwards, unles you goto the trouble of modifying old versions, which is not possible to do by accident (command line only, and the syntax required for the command is a bit awkard).

Fom Mariannes post:

Only new backups done with the new SLP settings will have the Infinity retention on tape.
This is how SLPs works.
Any outstanding SLP duplications at the time when the change was made, where backups were done with old 2 week backup retention and 27 day retention for duplications, would still have those retentions. 

"Old Backup history for infinity backups is gone due to netbackup setting "expire after copy".."

Absolutely impossible.
ONLY new backups done with the new SLP will have the backup and duplication retentions.
Also, when the backup copy expires, you will still have the tape copy with the configured retention.

I would agree with all of this.

If you change the promary (backup copy ) of SLP to 'Expire after duplication)' - then this will only affect the 1st copy, and secondary (duplication) copies made will have the retention period 'infnity', or whatever value you used.

If however, the first copy was made, and the image was canceled from SLP control before the copies were made, then the 'Expire after Duplication' setting will cause the 1st copy to be epired immediately - a canceled images reverts back to the 'try-to-keep time, or retention period set, which in this case is 'Expire').  I'm not saying this happened, but it is a way that you could get into this situation.

If the 2nd copy was made, and is no loner there, then something happened to it outside of SLP.

M

[unkn0wnn]

Thank you very much so far guys, Marianne please find requested below:

Also spotted this:

Files File Name:   dev-server_Filesystem_1510603203_INCR.f

I have plenty of f files availabe to be restored from catalg backup, so far I know there is also db required and we cannot merge it however if I could at least read media id from it then that would be very useful, looks like this kind of remedy is our last hope

bpimagelist -L -policy dev-server_Filesystem -sl Incremental -d 11/13/2017 -e 11/14/2017

Client:            dev-server
Backup ID:         dev-server_1510690204
Policy:            dev-server_Filesystem
Policy Type:       MS-Windows (13)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       Incremental
Schedule Type:     INCR (1)
Retention Level:   1 week (0)
Backup Time:       Tue Nov 14 20:10:04 2017 (1510690204)
Elapsed Time:      499 second(s)
Expiration Time:   Mon Dec 11 20:10:04 2017 (1513023004)
Compressed:        no
Client Encrypted:  no
Kilobytes:         5394263
Number of Files:   1385
Number of Copies:  1
Number of Fragments:   1
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   dev-server_Filesystem_1510690204_INCR.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               0
TIR Info:          0
TIR Expiration:    Thu Jan  1 01:00:00 1970 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  no
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:  Thu Jan  1 01:00:00 1970 (0)
BI Full Time:      Thu Jan  1 01:00:00 1970 (0)
Request Pid:       0
Backup Status:     0
Stream Number:     0
Backup Copy:       Standard (0)
Files File size:     165703
PFI type:     0
IMAGE_ATTRIBUTE:     0
Primary Copy:      2
Image Type:        0  (Regular)
Job ID:            241753
Num Resumes:       1
Resume Expiration: Thu Jan  1 01:00:00 1970 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    UK-AdvDisk-Tape-27dt-7od
Storage Lifecycle Policy Version:    0
STL_Completed:      3
Remote Expiration Time: Thu Jan  1 01:00:00 1970 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:      Thu Jan  1 01:00:00 1970 (0)
IR Enabled:      no
Client Character Set:     1
Image On Hold:     0
Indexing Status:   0
Copy number:       2
 Fragment:         1
 Kilobytes:        5394263
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         52
 ID:               0335L5
 Host:             n5220uk
 Block Size:       65536
 Offset:           38623700
 Media Date:       Thu Nov 16 16:30:21 2017 (1510849821)
 Dev Written On:   3
 Flags:            0x0
 Media Descriptor:        ?
 Expiration Time:  Mon Dec 11 20:10:04 2017 (1513023004)
 MPX:              0
 retention_lvl:    27 days (3)
 Try to Keep Time:  Mon Dec 11 20:10:04 2017 (1513023004)
 Copy Creation Time:  Fri Nov 17 04:32:50 2017 (1510893170)
 Data Format:      Tar
 checkpoint:       0
 resume num:       0
 Key tag:          *NULL*
 STL tag:          *NULL*
 Copy on hold:     0

Client:            dev-server
Backup ID:         dev-server_1510603203
Policy:            dev-server_Filesystem
Policy Type:       MS-Windows (13)
Proxy Client:      (none specified)
Creator:           root
Name1:             (none specified)
Sched Label:       Incremental
Schedule Type:     INCR (1)
Retention Level:   1 week (0)
Backup Time:       Mon Nov 13 20:00:03 2017 (1510603203)
Elapsed Time:      621 second(s)
Expiration Time:   Sun Dec 10 20:00:03 2017 (1512936003)
Compressed:        no
Client Encrypted:  no
Kilobytes:         20757187
Number of Files:   1442
Number of Copies:  1
Number of Fragments:   1
Histogram:         0 0 0 0 0 0 0 0 0 0
DB Compressed:     no
Files File Name:   dev-server_Filesystem_1510603203_INCR.f
Previous Backup Files File Name:   (none specified)
Parent Backup Image File Name:   (none specified)
SW Version:        (none specified)
Options:           0x0
MPX:               0
TIR Info:          0
TIR Expiration:    Thu Jan  1 01:00:00 1970 (0)
Keyword:           (none specified)
Ext Security Info: no
File Restore Raw:  no
Image Dump Level:  0
File System Only:  no
Object Descriptor: (none specified)
Previous BI Time:  Thu Jan  1 01:00:00 1970 (0)
BI Full Time:      Thu Jan  1 01:00:00 1970 (0)
Request Pid:       0
Backup Status:     0
Stream Number:     0
Backup Copy:       Standard (0)
Files File size:     174207
PFI type:     0
IMAGE_ATTRIBUTE:     0
Primary Copy:      2
Image Type:        0  (Regular)
Job ID:            241321
Num Resumes:       1
Resume Expiration: Thu Jan  1 01:00:00 1970 (0)
Data Classification:    (none specified)
Data_Classification_ID: (none specified)
Storage Lifecycle Policy:    UK-AdvDisk-Tape-27dt-7od
Storage Lifecycle Policy Version:    0
STL_Completed:      3
Remote Expiration Time: Thu Jan  1 01:00:00 1970 (0)
Origin Master Server:  (none specified)
Origin Master GUID:    (none specified)
Snap Time:      Thu Jan  1 01:00:00 1970 (0)
IR Enabled:      no
Client Character Set:     1
Image On Hold:     0
Indexing Status:   0
Copy number:       2
 Fragment:         1
 Kilobytes:        20757187
 Remainder:        0
 Media Type:       Media Manager (2)
 Density:          hcart (6)
 File Num:         43
 ID:               0479L5
 Host:             n5220uk
 Block Size:       65536
 Offset:           19151408
 Media Date:       Thu Nov 16 13:53:43 2017 (1510840423)
 Dev Written On:   1
 Flags:            0x0
 Media Descriptor:        ?
 Expiration Time:  Sun Dec 10 20:00:03 2017 (1512936003)
 MPX:              0
 retention_lvl:    27 days (3)
 Try to Keep Time:  Sun Dec 10 20:00:03 2017 (1512936003)
 Copy Creation Time:  Thu Nov 16 21:39:41 2017 (1510868381)
 Data Format:      Tar
 checkpoint:       0
 resume num:       0
 Key tag:          *NULL*
 STL tag:          *NULL*
 Copy on hold:     0

bppllist dev-server_Filesystem -U
------------------------------------------------------------

Policy Name:       dev-server_Filesystem

  Policy Type:         MS-Windows
  Active:              yes
  Effective date:      04/12/2016 09:28:39
  Backup network drvs: no
  Collect TIR info:    no
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           UK-AdvDisk-Tape-27dt-7od
  Volume Pool:         NetBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    yes
  Application Discovery:      no
  Discovery Lifetime:      0 seconds
ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  Windows-x86   Windows2003   dev-server

  Include:  ALL_LOCAL_DRIVES
            C:\Dev-server.bch

  Schedule:              Full
    Type:                Full Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     0 (1 week)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Calendar sched: Enabled
      Saturday, Week 1
      Saturday, Week 2
      Saturday, Week 3
      Saturday, Week 4
      Saturday, Week 5
      EXCLUDE DATE 0 - 04/30/2016
      EXCLUDE DATE 1 - 05/28/2016
      EXCLUDE DATE 2 - 07/02/2016
      EXCLUDE DATE 3 - 07/30/2016
      EXCLUDE DATE 4 - 08/27/2016
      EXCLUDE DATE 5 - 10/01/2016
      EXCLUDE DATE 6 - 10/29/2016
      EXCLUDE DATE 7 - 11/26/2016
      EXCLUDE DATE 8 - 12/31/2016
      EXCLUDE DATE 9 - 01/28/2017
      EXCLUDE DATE 10 - 02/25/2017
      EXCLUDE DATE 11 - 04/01/2017
      EXCLUDE DATE 12 - 04/29/2017
      EXCLUDE DATE 13 - 05/27/2017
      EXCLUDE DATE 14 - 07/01/2017
      EXCLUDE DATE 15 - 07/29/2017
      EXCLUDE DATE 16 - 08/26/2017
      EXCLUDE DATE 17 - 09/30/2017
      EXCLUDE DATE 18 - 12/30/2017
      EXCLUDE DATE 19 - 02/03/2018
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Saturday   16:00:00  -->  Sunday     08:00:00

  Schedule:              Period_End
    Type:                Full Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     0 (1 week)
    Number Copies:       1
    Fail on Error:       0
    Residence:           UK-AdvDisk-Tape-Infinity
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Calendar sched: Enabled
      SPECIFIC DATE 0 - 04/30/2016
      SPECIFIC DATE 1 - 05/28/2016
      SPECIFIC DATE 2 - 07/02/2016
      SPECIFIC DATE 3 - 07/30/2016
      SPECIFIC DATE 4 - 08/27/2016
      SPECIFIC DATE 5 - 10/01/2016
      SPECIFIC DATE 6 - 10/29/2016
      SPECIFIC DATE 7 - 11/26/2016
      SPECIFIC DATE 8 - 12/31/2016
      SPECIFIC DATE 9 - 01/28/2017
      SPECIFIC DATE 10 - 02/25/2017
      SPECIFIC DATE 11 - 04/01/2017
      SPECIFIC DATE 12 - 04/29/2017
      SPECIFIC DATE 13 - 05/27/2017
      SPECIFIC DATE 14 - 07/01/2017
      SPECIFIC DATE 15 - 07/29/2017
      SPECIFIC DATE 16 - 08/26/2017
      SPECIFIC DATE 17 - 09/30/2017
      SPECIFIC DATE 18 - 10/28/2017
      SPECIFIC DATE 19 - 11/25/2017
      SPECIFIC DATE 20 - 12/30/2017
      SPECIFIC DATE 21 - 02/03/2018
    Residence is Storage Lifecycle Policy:         1
    Schedule indexing:     0
    Daily Windows:
          Saturday   14:00:00  -->  Sunday     08:00:00

  Schedule:              Incremental
    Type:                Differential Incremental Backup
    Maximum MPX:         1
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     0 (1 week)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Calendar sched: Enabled
      Monday, Week 1
      Tuesday, Week 1
      Wednesday, Week 1
      Thursday, Week 1
      Friday, Week 1
      Monday, Week 2
      Tuesday, Week 2
      Wednesday, Week 2
      Thursday, Week 2
      Friday, Week 2
      Monday, Week 3
      Tuesday, Week 3
      Wednesday, Week 3
      Thursday, Week 3
      Friday, Week 3
      Monday, Week 4
      Tuesday, Week 4
      Wednesday, Week 4
      Thursday, Week 4
      Friday, Week 4
      Monday, Week 5
      Tuesday, Week 5
      Wednesday, Week 5
      Thursday, Week 5
      Friday, Week 5
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:
          Monday     20:00:00  -->  Tuesday    05:00:00
          Tuesday    20:00:00  -->  Wednesday  05:00:00
          Wednesday  20:00:00  -->  Thursday   05:00:00
          Thursday   20:00:00  -->  Friday     05:00:00
          Friday     20:00:00  -->  Saturday   05:00:00

[Marianne]

Seems you are still using the 27-day SLP:

Residence: UK-AdvDisk-Tape-27dt-7od

The expiration date for this image is Sun Dec 10 20:00:03 2017.

[unkn0wnn]

Expire after copy option is not in use any more within all SLPs. More interested in getting old image client backup history and media id using preferably some experimental method so if needed then can import one two tapes instead of 20...

[mph999]

If there is an entry in the Image Table of NBDB for a backup, you can find this by searching for the ctime in nbdb_unload output.  (you cannot sarch for the backupid as it doesn't exist in the image table, it is split, the 8th filed is the client name and the 11th the ctime).

From this, the first field is the ImageKey, a unique number.

The ImageFragment table contains the mediaID for each fragement of the backup, and is related to the Image table by the ImageKey, which in the ImageFragment table is the 2nd field - you can then fine the mediaid(s) for a given backup.

The reload.sql shows which .dat file (from nbdb_unload) is which table (tables differ between NBU versions).

If the catalog information is completely missing, as would seem to be the case, there is no way to work out which backup is on which media.

[unkn0wnn]

Hello and thank you for your support. This sounds promising.

Is it ok to run nbdb_unload on database files that I have restored ?

e.g. i have all the db files in directory /tmp/databases_check/

Will usage of this command actually cause any problems ? By default nbdb is unloaded hmm