cancel
Showing results for 
Search instead for 
Did you mean: 

understanding bpduplicate

smith_jones
Level 6

Hi All,

I ran duplicates using bpduplicate command using BID file.

I was looking at activity monitor logs. Please tell me what does this mean -

Capture.JPG

What is happening at "positioning to file"?

It does not come when we run duplicates manually one by one from Netbackup Administration console --->catalog.

 

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Nicolai
Moderator
Moderator
Partner    VIP   

See this blog 

https://www-secure.symantec.com/connect/blogs/understanding-how-netbackup-writes-tape

Each "file" is a netbackup fragment with the fragment (file) size configured in the storage unit definition.

If you think of a old cassette music tape, with your favorite as song 3. You then has to fast forward 2 songs to hear it or fast fordward to the end to record a new performance.  

Netbackup work in the same way. It can fast forward before listing (reading) or fast forward to the end, to record a new song (backup).

If you want to copy a lot of song (backups), to a new tape, you may have to fast forward to get song 2, 6, 12 and 24. The diffrence here is that NBU actual tells you where it is on tape and what its doing right now.

The messages are informational only, but can be used to find performance issue during write operations.

 

View solution in original post

11 REPLIES 11

smith_jones
Level 6

Forgot to attach log.

Attaching log.

Nicolai
Moderator
Moderator
Partner    VIP   

See this blog 

https://www-secure.symantec.com/connect/blogs/understanding-how-netbackup-writes-tape

Each "file" is a netbackup fragment with the fragment (file) size configured in the storage unit definition.

If you think of a old cassette music tape, with your favorite as song 3. You then has to fast forward 2 songs to hear it or fast fordward to the end to record a new performance.  

Netbackup work in the same way. It can fast forward before listing (reading) or fast forward to the end, to record a new song (backup).

If you want to copy a lot of song (backups), to a new tape, you may have to fast forward to get song 2, 6, 12 and 24. The diffrence here is that NBU actual tells you where it is on tape and what its doing right now.

The messages are informational only, but can be used to find performance issue during write operations.

 

smith_jones
Level 6

Hi Nicolai,

Thanks for such a good example.

I saw the article that you have provided above.

Its a good article. But I want to ask one thing. 

How netbakup handles this -

BOT|server1part1|server2part1|server1part2|server2part2|EOT

Now suppose the image server2part2 is expired (the last one expired) i.e. blankspace is created, other images are yet to be expired.

Will the next backup use the tape and wite it there where the last image expired since it writes linearly and there is blank space created after expiration of last image.

Please suggest.

revarooo
Level 6
Employee

No!

NetBackup will not write to any "space" on a tape UNTIL the very last image on tape has expired and the tape has been deassigned.

NetBackup does not remove the image from tape, it just removes the image from the catalog.

 

smith_jones
Level 6

Hi revarooo,

when this is the case it writes to the blank space -

BOT|server1part1|server2part1|server1part2|Blank space|EOT

When this is the case it nt writes on the tape according to you)-

BOT|server1part1|server2part1|server1part2|expired image|EOT

That means netbackup is not treating the last expired space as blank space, right?

Nicolai
Moderator
Moderator
Partner    VIP   

I think Netbackup will start writing at the last valid (valid is important here) image on tape - that is the end of "server1part2".

But pratical this will never happen, because Netbackup doesn't mix retention. The oldest backup will always be in front and the newest always in the end.

I can think of two scenarios where Netbackup would re-use space at end of tape:

  1. if a new non-multiplxed backup fails, next backup will start at end of server1part2 and reuse space at "Non MPX failed Backup" 
  2. you manual expire the last backup on tape & the backup is non-mulitplexed.

BOT|server1part1|server2part1|server1part2|Non MPX failed Backup|EOT

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
You will agree that Nicolai has answered the query in your opening post? Please mark his first reply as Solution. Thanks!

smith_jones
Level 6

Hi Nicolai,

That means when this is the case -

BOT|server1part1|server2part1|server1part2|expired image|Blank space|EOT

it will write the new backup images at "expired image"  space (if it is in the end of the tape or before blank space) and then proceed for Blank Space if any.

Did I understand right?

Nicolai
Moderator
Moderator
Partner    VIP   

yes - that is my understanding (please note: I could be wrong)  - but the expired image must be non-mpx else the next image will start at blank space.

mph999
Level 6
Employee Accredited

Seems Nicolai is correct ...

At the end of a backup, either mpx or non-mpx NBU writes an empty header :

root@womble 160106 $ scsi_command -f /dev/rmt/1cbn
Inquiry data: removable dev type 1h HP      Ultrium 1-SCSI  E38W
root@womble 160106 $ scsi_command -map -f /dev/rmt/1cbn
00000000: file 1: record 1: size 1024: NBU MEDIA header (A00001)
00000001: file 1: eof after 1 records: 1024 bytes
00000002: file 2: record 1: size 1024: NBU BACKUP header
          backup_id womble_1451909069: frag 1: file 1: copy 1
          expiration 1452513869: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000003: file 2: record 2: size 262144
00000004: file 2: record 3: size 98304
00000005: file 2: eof after 3 records: 361472 bytes
00000006: file 3: record 1: size 1024: NBU BACKUP header
          backup_id womble_1451909106: frag 1: file 2: copy 1
          expiration 1452513906: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000007: file 3: record 2: size 262144
00000008: file 3: record 3: size 98304
00000009: file 3: eof after 3 records: 361472 bytes
00000010: file 4: record 1: size 1024: NBU EMPTY header (file 3)    <<<<<<<<  HERE IS THE EMPTY HEADER
00000011: file 4: eof after 1 records: 1024 bytes
eot

When a new backup is written to the tape, NBU searches for the Empty header, well, in fact it knows where the Empty header 'should' be , it then reads it to confirm it really is in the correct place.  It then rewinds, overwrites the Empty header with a Filemark, and then writes the new backup header and continues writing the data.  This can be clearly seen in the bptm log

11:35:08.567 [9715] <2> io_position_for_write: position media id A00001, copy 1, current number images = 2
11:35:08.567 [9715] <2> io_position_for_write: locating to absolute block number 10, copy 1
11:35:08.620 [9715] <2> io_position_for_write: locate block is done


11:35:08.567 [9715] <2> io_position_for_write: position media id A00001, copy 1, current number images = 2
11:35:08.567 [9715] <2> io_position_for_write: locating to absolute block number 10, copy 1
11:35:08.620 [9715] <2> io_position_for_write: locate block is done
11:35:08.649 [9715] <2> io_position_for_write: processing empty header, filenum = 3, bid = (empty_file), copy 1   <<< Found the empty header
11:35:08.649 [9715] <2> io_position_for_write: empty header found on A00001, OK, copy 1      <<<<  Checks it really is the empty header
11:35:08.649 [9715] <2> io_ioctl: command (2)MTBSF 1 0x0 from (bptm.c.22814) on drive index 0   <<<< Rewinds to before the empty header
11:35:08.786 [9715] <2> io_ioctl: command (0)MTWEOF 1 0x0 from (bptm.c.22866) on drive index 0   <<<<<  Overwrites the empty header with a file mark

 

 

I expired the last backup on the tape (womble_1452080100), and wrote a new one ( womble_1452080389 ) ...

So we have some 'free space' now on the tape ... before the empty header ...

Lookinf at scsi_command -map output ....

 

00000006: file 3: record 1: size 1024: NBU BACKUP header    <<<<<<<<  BACKUP HEADER OF IMAGE BEFORE womble_1452080389
          backup_id womble_1451909106: frag 1: file 2: copy 1
          expiration 1452513906: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000007: file 3: record 2: size 262144
00000008: file 3: record 3: size 98304
00000009: file 3: eof after 3 records: 361472 bytes
00000010: file 4: record 1: size 1024: NBU BACKUP header   <<<<<< BACKUP HEADER FOR womble_1452080389
          backup_id womble_1452080389: frag 1: file 3: copy 1
          expiration 1452685189: retention 0: block_size 262144
          flags 0x0: mpx_headers 0: resume_count 0: media A00001
00000011: file 4: record 2: size 262144
00000012: file 4: record 3: size 98304
00000013: file 4: eof after 3 records: 361472 bytes
00000014: file 5: record 1: size 1024: NBU EMPTY header (file 4)
00000015: file 5: eof after 1 records: 1024 bytes
eot

 

We see teh previous image on the tape is womble_1452080389, so indeed, it has overwritten the womble_1452080100 image which I expired.

 

This isn't possible for mpx images, as we don;t actually know where the data is for the image (it's all mixed in with other images)

 

smith_jones
Level 6

Great. Thanks mph999 and Nicolai and others.