understanding bpduplicate

Forgot to attach log.

Attaching log.

Hi Nicolai,

Thanks for such a good example.

I saw the article that you have provided above.

Its a good article. But I want to ask one thing. 

How netbakup handles this -

BOT|server1part1|server2part1|server1part2|server2part2|EOT

Now suppose the image server2part2 is expired (the last one expired) i.e. blankspace is created, other images are yet to be expired.

Will the next backup use the tape and wite it there where the last image expired since it writes linearly and there is blank space created after expiration of last image.

Please suggest.

No!

NetBackup will not write to any "space" on a tape UNTIL the very last image on tape has expired and the tape has been deassigned.

NetBackup does not remove the image from tape, it just removes the image from the catalog.

Hi revarooo,

when this is the case it writes to the blank space -

BOT|server1part1|server2part1|server1part2|Blank space|EOT

When this is the case it nt writes on the tape according to you)-

BOT|server1part1|server2part1|server1part2|expired image|EOT

That means netbackup is not treating the last expired space as blank space, right?

I think Netbackup will start writing at the last valid (valid is important here) image on tape - that is the end of "server1part2".

But pratical this will never happen, because Netbackup doesn't mix retention. The oldest backup will always be in front and the newest always in the end.

I can think of two scenarios where Netbackup would re-use space at end of tape:

1. if a new non-multiplxed backup fails, next backup will start at end of server1part2 and reuse space at "Non MPX failed Backup" 
2. you manual expire the last backup on tape & the backup is non-mulitplexed.

BOT|server1part1|server2part1|server1part2|Non MPX failed Backup|EOT

Hi Nicolai,

That means when this is the case -

BOT|server1part1|server2part1|server1part2|expired image|Blank space|EOT

it will write the new backup images at "expired image"  space (if it is in the end of the tape or before blank space) and then proceed for Blank Space if any.

Did I understand right?

yes - that is my understanding (please note: I could be wrong)  - but the expired image must be non-mpx else the next image will start at blank space.

Seems Nicolai is correct ...

At the end of a backup, either mpx or non-mpx NBU writes an empty header :

root@womble 160106 $ scsi_command -f /dev/rmt/1cbn
Inquiry data: removable dev type 1h HP      Ultrium 1-SCSI  E38W
root@womble 160106 $ scsi_command -map -f /dev/rmt/1cbn
00000000: file 1: record 1: size 1024: NBU MEDIA header (A00001)
00000001: file 1: eof after 1 records: 1024 bytes
00000002: file 2: record 1: size 1024: NBU BACKUP header
          backup_id womble_1451909069: frag 1: file 1: copy 1
          expiration 1452513869: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000003: file 2: record 2: size 262144
00000004: file 2: record 3: size 98304
00000005: file 2: eof after 3 records: 361472 bytes
00000006: file 3: record 1: size 1024: NBU BACKUP header
          backup_id womble_1451909106: frag 1: file 2: copy 1
          expiration 1452513906: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000007: file 3: record 2: size 262144
00000008: file 3: record 3: size 98304
00000009: file 3: eof after 3 records: 361472 bytes
00000010: file 4: record 1: size 1024: NBU EMPTY header (file 3)    <<<<<<<<  HERE IS THE EMPTY HEADER
00000011: file 4: eof after 1 records: 1024 bytes
eot

When a new backup is written to the tape, NBU searches for the Empty header, well, in fact it knows where the Empty header 'should' be , it then reads it to confirm it really is in the correct place.  It then rewinds, overwrites the Empty header with a Filemark, and then writes the new backup header and continues writing the data.  This can be clearly seen in the bptm log

11:35:08.567 [9715] <2> io_position_for_write: position media id A00001, copy 1, current number images = 2
11:35:08.567 [9715] <2> io_position_for_write: locating to absolute block number 10, copy 1
11:35:08.620 [9715] <2> io_position_for_write: locate block is done

11:35:08.567 [9715] <2> io_position_for_write: position media id A00001, copy 1, current number images = 2
11:35:08.567 [9715] <2> io_position_for_write: locating to absolute block number 10, copy 1
11:35:08.620 [9715] <2> io_position_for_write: locate block is done
11:35:08.649 [9715] <2> io_position_for_write: processing empty header, filenum = 3, bid = (empty_file), copy 1   <<< Found the empty header
11:35:08.649 [9715] <2> io_position_for_write: empty header found on A00001, OK, copy 1      <<<<  Checks it really is the empty header
11:35:08.649 [9715] <2> io_ioctl: command (2)MTBSF 1 0x0 from (bptm.c.22814) on drive index 0   <<<< Rewinds to before the empty header
11:35:08.786 [9715] <2> io_ioctl: command (0)MTWEOF 1 0x0 from (bptm.c.22866) on drive index 0   <<<<<  Overwrites the empty header with a file mark

I expired the last backup on the tape (womble_1452080100), and wrote a new one ( womble_1452080389 ) ...

So we have some 'free space' now on the tape ... before the empty header ...

Lookinf at scsi_command -map output ....

00000006: file 3: record 1: size 1024: NBU BACKUP header    <<<<<<<<  BACKUP HEADER OF IMAGE BEFORE womble_1452080389
          backup_id womble_1451909106: frag 1: file 2: copy 1
          expiration 1452513906: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000007: file 3: record 2: size 262144
00000008: file 3: record 3: size 98304
00000009: file 3: eof after 3 records: 361472 bytes
00000010: file 4: record 1: size 1024: NBU BACKUP header   <<<<<< BACKUP HEADER FOR womble_1452080389
          backup_id womble_1452080389: frag 1: file 3: copy 1
          expiration 1452685189: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000011: file 4: record 2: size 262144
00000012: file 4: record 3: size 98304
00000013: file 4: eof after 3 records: 361472 bytes
00000014: file 5: record 1: size 1024: NBU EMPTY header (file 4)
00000015: file 5: eof after 1 records: 1024 bytes
eot

We see teh previous image on the tape is womble_1452080389, so indeed, it has overwritten the womble_1452080100 image which I expired.

This isn't possible for mpx images, as we don;t actually know where the data is for the image (it's all mixed in with other images)