block read is not a NetBackup or Backup Exec media header, len = 65536, media id xxxx

[Sagar_Kolhe]

Hi All,

We are facing below issue in our setup during restoration. Please help us to understand the issue.

11/18/2014 20:55:31 - Waiting for scan drive stop Driveyyy, Media server: xxxxxxxx

11/18/2014 20:55:34 - Info bptm (pid=31261322) Waiting for mount of media id yyyy (copy 1) on server xxxxxxxx

11/18/2014 20:55:34 - started process bptm (pid=31261322)
11/18/2014 20:55:34 - mounting yyyy

11/18/2014 20:55:34 - granted resource  yyyyy

11/18/2014 20:55:34 - granted resource  Driveyyy11/18/2014 20:55:35 - Info bptm (pid=31261322) INF - Waiting for mount of media id yyyyy on server xxxxxxxx for reading.
11/18/2014 20:56:02 - Error bptm (pid=31261322) block read is not a NetBackup or Backup Exec media header, len = 65536, media id yyyyy, drive index 4, data is unknown
11/18/2014 20:56:50 - Info bpbrm (pid=59244832) got ERROR 172 from media manager
11/18/2014 20:56:50 - restored from image xxxxxxxx_1413695720; restore time: 0:04:06
11/18/2014 20:56:50 - Info bpbrm (pid=59244832) terminating bpbrm child 64750326 jobid=635081
11/18/2014 20:56:51 - Warning bprd (pid=23130) Restore must be resumed prior to first image expiration on Sat Nov 29 23:59:59 2014
11/18/2014 20:56:51 - end Restore; elapsed time 0:04:12
Failed to get status code information (2850)

also we have tried to duplicate the image but failed with 191 status with same information.

11/19/2014 00:27:54 - Error bptm (pid=54985434) block read is not a NetBackup or Backup Exec media header, len = 65536, media id yyyyy, drive index 4, data is unknown
11/19/2014 00:28:40 - Info bptm (pid=54985434) EXITING with status 172 <----------
11/19/2014 00:28:41 - Error bptm (pid=64291518) media manager terminated by parent process
11/19/2014 00:29:38 - Error bpduplicate (pid=29424) host xxxxx  backup id xxxx_1413695720 read failed, cannot read media header, may not be NetBackup media or is corrupted (172).
11/19/2014 00:29:38 - Error bpduplicate (pid=29424) host xxxxxx backupid xxxxxx_1413695720 write failed, media manager killed by signal (82).
11/19/2014 00:29:38 - Error bpduplicate (pid=29424) Duplicate of backupid xxxxxx_1413695720 failed, media manager killed by signal (82).
11/19/2014 00:29:38 - Error bpduplicate (pid=29424) Status = no images were successfully processed.
11/19/2014 00:29:38 - end Duplicate; elapsed time 1:27:16
no images were successfully processed  (191)

AS THIS IS ORIGINAL MEDIA , WE CANNOT AFFORD TO LOOSE DATA.

[mph999]

I can tell you what has happened, but only give a guess as to how.

At the beginning of the tape should be a media header that is 1024 bytes in size.  What you have is a block that is 65536 bytes in size, and there is only one thing on a tape that is that size, and that is data.

So, at the beginning of your tape you have a data block, not a header - NBU can;t understaand this, and there is no way to fix it - impossible.  The tape might be readable via tar command, or data recovery specialists.

What happened, well, as a guess the tape originally had a header but midway through the backup, there was a scsi reset on the san which caused the tape to rewind, which is invisable to NB and the OS - the backup however continued which overwrites the header.  There is no way to detect this, so you only find out on restore, or by looking in the NBU logs where a mis-match in the real tape position and where NBU things the tape will be will have been shown, the backup would also have failed.  We can't detect it when it happpens (in fact the OS writes the data, not NBU anyway) but can on the position check.

The most common cause for this is when NDMP devices (filer) share (SSO) tape drives with NBU media servers, and the scsi reservation type set on the media server is different to the reservation type set on the filer, eg. SPC2 type on the filer and Persistent reservation in NBU.  Other possibilities are firmware/ diver errors or faulty hardware.

The other possibility is that the tape was overwriteen outside of NBU by a 3rd part applliaction of manual OS command.  Unfortunately, there is no way to fix this tape in this instance.

[mph999]

The reason this tape can't be fixed ...

1.  If a new header was written it would overwrite the first 1k of the data on the tape

2.  After a header is writtem (eg label a tape) a logical end-of-data mark is written, the tape drive would be unable to read past this point to get at the data beyond the 'new' header.  (Usually when a new backup is writte the eod mark is deleted and re-written at the end of the new backup, for restores, the drive will refuse to read past this point.  The only drives that can, are those with special modified firmware as used by data recovery specialists - however I mention this only out of interest we aren't in that situation.

[Sagar_Kolhe]

Hi mph,

you only find out on restore, or by looking in the NBU logs where a mis-match in the real tape position and where NBU things the tape will be will have been shown

Would you please provide more clarification. Which logs need to see n is there any 0.1% chance to get the data.

Regards,

Sagar

[mph999]

I am sorry, but this data is not going to be recovered by NBU, even Symantec Engineering won't be able to help in this instance.

I don't know the OS used, but if Unix/ Linux you might find tar can read the tape :

tar tvf bs=65536 <tape drive path>

This will only display the contects (if it works) use cvf to restore, acually use

/usr/openv/netbackup/bin/tar ...

The NBU version of tar is pretty much the same as the 'Linux /Unix' version but very importantly it drops the leading / so the data will not overwrite anything on the machine, it will just end up in a sub directory of where you are when the command is run, so I would mkdir /new_dir, cd into new_dir and run it fom there.  You may have to run the command multiple times without rewinding the tape inbetween.

If this fails, the only other option would be data recover firms - I know a good one here in the UK if needed.  This may/ may not work (no guarantees and is usually very expensive).

The logs won't help you get anything back in this case, and you would need to have them from when the backup is written but you look for 'External event has caused scsi rewind' in bptm, from the media server that wrote the image.  Maybe bperror report might show something, not sure.

[mph999 · Accepted solution]

Hmm, apologies, the command should be 

tar bs=65536 tvf <path to tape drive>

You need to first put the tape into the drive and then rewind it mt -f <path> rewind

Of course, if I am correct, then you only have 'half and image' on the tape anyway, as the beginning was overwritten.

Unfortunately, I have seen many of these sorts of issues and I don't recall any real success stories in getting the data back.  It's a bit odd, we detect this after it has happened and fail the backup which deletes the image from the catalog - so probably the most likely cause is something other than NBU wrote to the tape.

I've just had a thought, are you using MSEO or some other type of encryption ? - that could be another cause, maybe.

[mph999]

Have to goto bed now, will check this post tomorrow

[Sagar_Kolhe]

Yes we are using KMS at our site.

Regards,

Sagar

[sksujeet]

Are you still using KMS and is the enryption working fine? If the header has been encrypted by KMS and if something is changted or if you are trying to do this without KMS it will not work.

Are you doing this on the same environment where you have the KMS setup or to remote enviornment.

http://www.symantec.com/business/support/index?page=content&id=TECH203420

[Nicolai]

Do you also have tape based encryption that is managed by the library ?

[mph999]

KMS could cause an issue, though you shouldn't get a 65536 block at the beginning of the tape, KMS or not. When you try and read the tape coud be worth checking the bptm log to confirm it gets the correct key.

[Marianne]

Ensure firmware and drivers on backup and restore drives are the same or newer on restore drives.

Also check firmware and drivers on hba.

We have seen in the past that backups worked fine at one particular client, but restores were impossible.
If memory serves me right, it was a similar error.
No encryption was used.

After lengthy troubleshooting and support calls with all vendors, the hba was replaced. 
Restores worked fine after that....

[Sagar_Kolhe]

Yes KMS is working fine except for that media. We have used different media for duplication(not the same data) n restored the data for client.

Thank you for all your support.

Regards,

Sagar