0 byte 0 file incremental backups show up in images on disk, but not in catalog

[John_Snodgrass]

I have a customer that gets a lot of special reports.  One of them is to check and make sure their backup images have been replicated to the DR side.  I have some issues i need to work out with the reporting group, but I've come across a problem I've never seen before.  Onne of the images that says was not replicated is correct, but I think I know why, but I don't fully understand why it works like it appears to,

From the Disk Reports - Images on disk.

One thing I figured out, the number of files and kilibytes are both zero, which for an incremental backup makes sense and I understand that.

From the catalog that image is not there.

That image is not seen in the catalog.  I can understand why, since there is no actual data associated with it, but where does that image that is in the images on disk report come from.  I think I need to tell my reporting team to pull the image/backup data a different way, as the way they are doing it now makes it seem like something is not working, and best I can tell it's working perfectly.

[mph999]

It's going to be in the catalog, well NBDB.

If there are no fragments, or 0 size it won't show as being in the 'catalog' as such (your 2nd screenshot above), but given it's reported in the Disk Report it must be referenced somewhere.

I suspect that there is a zero size fragment entry though (in the fragment tabe) as this is the only place in NBDB where an image is associated with a backupid (I think).

Personally, I'd run nbdb_unload <some dir> get a readable dump of NBDB.

cd <some dir> and grep for the ctime of the backupid only, it should be unique, so the entries retured will reference this image - it will appear in at least one .dat file.

Look in reload.sql and confirm that the x.dat is the emm image table - this should be one of the files the grep returned.

The first fileld of the line from this table is the ImageKey.

From sql.dat, find the .dat file that is the ImageCopy table, call it y.dat.  The first filed is the ImageCopy key , the second field is the Image Key, so you can find the copy like from this table that referenced your imange (you got the Image key from x.dat) by looking for the number in the 2nds field of the ImageCopy table - depending on how many copies there are, there could be more than one entry (the IMageTable contains 1 line for each copy of an image).

Finally, find the dat file that corresnds to the EMM ImageFragment table - z.dat

From memory, I think the ImageCopy key is the 2nd filed, but it might be the third, from y.dat (Image table) you have the ImageCopy key, this can be used to find the fragment lines for this image in z.dat

The .dat files appear a few times in reload.sql, so it takes a few mins, but somewhere you have sections like this 

<database table name>

.dat file 

field1,field2,field3 etc

Where the field1,field2,field3 are the names of the fields in order for that table  - so although I am not sure from memory which table is which for the fragment table, it's easy enough to work out - just takes a few mins.

M

[RiaanBadenhorst]

Hi,

See below example from 7.7.1 reload.sql environment to illustrate Martin's explanation (please note the dat file numbers are almost always different in each maintenance release)

INPUT INTO "DBM_MAIN"."DBM_ImageCopy" 
    FROM 'C:/temp/riaan/824.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("ImageCopyKey","ImageKey","CopyNumber","SLPIndex","CopyType","ExpireTime","MpxState","RetentionLevel","CopyDate","DataFormat","KMSKeyTag","TryToKeepTime","Residence","CopyState","JobID","ExpirationType","RetryCount","LastRetryTime","LifecycleDestinationTag","SLPSourceIndex","DeferredDuplicationTime","ExpireLifecycleTime","IsReplica","MirrorParent","Hold","CreatedDateTime","LastModifiedDateTime","Resumable","CopySize","WGkey","SourceCopyExpirationTime","ValidationCount","ExpirationCount")

INPUT INTO "DBM_MAIN"."DBM_ImageFragment" 
    FROM 'C:/temp/riaan/826.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("ImageFragmentKey","ImageKey","ImageCopyKey","CopyNumber","FragmentNumber","ResumeCount","MediaID","MediaServerKey","StorageUnitType","StuSubType","FragmentState","FragmentSize","FragmentID","Density","FileNum","BlockSize","Offset","MediaDate","DeviceWrittenOn","FFlags","MediaDescription","FragmentCheckpoint","MediaSequenceNum","MediaExtents","SnapshotClientMountHost","CreatedDateTime","LastModifiedDateTime")

[RiaanBadenhorst]

I forgot to mention, take the headers mentioned in the reload.sql and add them to the top of the respective DAT file. Then open the DAT file in excel as a delimited file so its easy to review.

[John_Snodgrass]

I'm not sure if it's worth doing all that.  I did some checking today and I found one of those images is found if I use bpimedia or bpimagelist, but it's still not seen in the catalog section of the GUI.  Which I thought was just a graphical representation of bpimagelist.

[johnsnod@aliwpl01-ius-02-001-nbma ~]$ sudo bpimmedia -disk -U -client 333oem03 | grep 1241
Backup-ID:            333oem03_1452301241

Here is the full output from bpimedia for the two related images.

Backup-ID:            333oem03_1452301241
 Policy:              Amalgamated-AIR-TKLM_aliwpl01-ius-02-001-nbma_MAN_2k8_VFIL_0_0
 Schedule Type:       INCR
 Retention Level      3
 Number of Files:     0
 Compression:         N
 Encryption:          N
 Image Type:          Regular
 Primary Copy:        1
 Expires:             20:00 02/08/2016
 Image on Hold:       0
 Indexing Status:     0

  Copy Number:        1
  Fragment Number:    1
  Fragment Size (KB): 0
  Media Type:         Disk
  Density:            -
  File Number:        -
  Offset:             -
  Host:               aliwpl01-ius-02-001-nbma
  Device Written On:  -
  MPX:                N
  Expires:            20:00 02/08/2016
  Retention Level:    3
  MediaID:            @aaaab
  Copy on Hold:       0
--------------------------------------------------------------------------------
Backup-ID:            333oem03_1452301240
 Policy:              Amalgamated-AIR-TKLM_aliwpl01-ius-02-001-nbma_MAN_2k8_VFIL_0_0
 Schedule Type:       INCR
 Retention Level      3
 Number of Files:     5804
 Compression:         N
 Encryption:          N
 Image Type:          Regular
 Primary Copy:        1
 Expires:             20:00 02/08/2016
 Image on Hold:       0
 Indexing Status:     0

  Copy Number:        1
  Fragment Number:    IDX
  Fragment Size (KB): 1
  Media Type:         Disk
  Density:            -
  File Number:        -
  Offset:             -
  Host:               aliwpl01-ius-02-001-nbma
  Device Written On:  -
  MPX:                -
  Expires:            -
  Retention Level:    -
  MediaID:            @aaaab
  Copy on Hold:       0

  Copy Number:        1
  Fragment Number:    IDX
  Fragment Size (KB): 710
  Media Type:         Disk
  Density:            -
  File Number:        -
  Offset:             -
  Host:               aliwpl01-ius-02-001-nbma
  Device Written On:  -
  MPX:                -
  Expires:            -
  Retention Level:    -
  MediaID:            @aaaab
  Copy on Hold:       0

  Copy Number:        1
  Fragment Number:    1
  Fragment Size (KB): 10107006
  Media Type:         Disk
  Density:            -
  File Number:        -
  Offset:             -
  Host:               aliwpl01-ius-02-001-nbma
  Device Written On:  -
  MPX:                N
  Expires:            20:00 02/08/2016
  Retention Level:    3
  MediaID:            @aaaab
  Copy on Hold:       0

[johnsnod@aliwpl01-ius-02-001-nbma ~]$ sudo bpimagelist -backupid 333oem03_1452301241 -U
Backed Up         Expires       Files       KB  C  Sched Type   On Hold Index Status Policy
----------------  ---------- -------- --------  -  ------------ ------- ------------ ------------
01/08/2016 20:00  02/08/2016        0        0  N  Differential 0       0            Amalgamated-AIR-TKLM_aliwpl01-ius-02-001-nbma_MAN_2k8_VFIL_0_0

[johnsnod@aliwpl01-ius-02-001-nbma ~]$ sudo bpimagelist -backupid 333oem03_1452301240 -U
Backed Up         Expires       Files       KB  C  Sched Type   On Hold Index Status Policy
----------------  ---------- -------- --------  -  ------------ ------- ------------ ------------
01/08/2016 20:00  02/08/2016     5804 10107716  N  Differential 0       0            Amalgamated-AIR-TKLM_aliwpl01-ius-02-001-nbma_MAN_2k8_VFIL_0_0

The 333oem03_1452301240 image does show up in the catalog GUI.  So why if both of those images are found using bpimagelist, why would they not both be in the catalog section of the GUI??  I think this is my real question.  I did not check the bpimagelist on the source master the other day.  Only the 1240 gets replicated to the DR master, based on the null data in 1241, makes sense to me.

My reporting person is on vacation this week, so I'll have to wait until they get back to get that part figured out.  I think it's safe to just have them filter out any "null" images from the replication report so our customer is not seeing any phantom errors.

[RiaanBadenhorst]

Maybe the catalog (and AIR) has some inteligence to not show you something that's not really relevant. If nothing changed since date xx/xx/xx, then it is showing you the last "important" image, i.e. the one with actual data. Same with AIR, maybe it just ignores it so as not to create unnecessary work.

Support case might be required to get an official answer :)

[Amol_Nair]

This happens when clients are running on NetBackup version lower than 7.6.1 and seen especially with SLP.

In such cases if there are no changes when an incremental backup is triggered the image created on the storage is 0kb in size. After 7.6.1 version of NetBackup (on the client) for some reason NetBackup creates 2kb image file on the storage even if there are no changes. So this issue is not seen for clients running NetBackup 7.6.1 version and above.

In the previous versions, these images only appear in the output of bpimagelist and bpimmedia. The image for some-reason does not show up in the NetBackup catalog. The problem with such images is that the SLP is directly marked as "complete" (since the image is 0kb in size no secndary operations are triggered for them)

Currently I am working with another customer on a similar issue and the case has been reported to engineering already. A EEB has been proposed specifically for the customer I am working with at the moment. A slight change in the issue where I am working is that the original backup copy in the SLP is set to Expire after copy. And since the secondary operation is never triggered these 0kb files are never removed from the storage.

But there is no update from the engineering team yet and not sure when the EEB would be created.

In-your case these 0kb images would be present on the storage till the expiration time is reached. So you can safely ignore these reports as long as these backupids are for incremental backups which have not change in data. If you cannot wait for the images to reach their respective expiration time, I would recommend upgrading the NetBackup version to 7.6.1 (this would include the NetBackup version on the client as well). This will ensure that 0kb images are not created. I would say this is the easiest way out. And for the previous 0kb images, you could probably try a for loop with the bpimagelist command to list all such backupids, followed by running the bpexpdate command to remove them. I think I had created a for loop while testing this out to get the list of such 0kb images in my test kit.

[Amol_Nair · Accepted solution]

Although the post seems dead, just thought of updating it in-case if anyone else refers to it in future.

This issue has been fixed in NetBackup 7.7.2. Below are the changes made in the SLP processing in NetBackup 7.7.2

If expire after copy is selected and a backup image is 0 bytes in length, the expiration date of the 0 byte image will be set to that of the shortest copy 2 or 3. Inshort if copy 2 has an expiration date of 2 months, copy 1 of zero bytes will not expire until 2 months after the backup. It will not get expired immediately.