I am testing restore from one site to another. i backed up a server in one location in tape and loaded it is another location. However when i tried to import it in another location first phase it only shows below logs .
19:04:26 INF - Created bpimport process, pid: 292
Import phase 1 started 05/02/2018 19:04:26
19:04:26 INF - Create DB information for media id XXXXX.
19:04:26 INF - Initiation of bptm process to phase 1 import media id XXXXX was successful.
I also observed that kilobytes in tape is 0. I am not sure what has happened. When i verify in primary site catalogs it shows that backup image exists in this tape .
Has anyone faced this scenario?
import has two phases. First one imports just image header information, and the second one imports image content information.
In Catalog section in Admin Console, first is under "Initiate Import", and second is under "Import".
Image content is browsable and restorable after both phases were done.
First phase completed. The second phase fails with 191 error. Data is written to the encryption pool and keys have been imported in destination master server.
Job logs :
May 6, 2018 9:40:20 AM - Info bptm (pid=268171) start
May 6, 2018 9:40:20 AM - started process bptm (pid=268171)
May 6, 2018 9:40:20 AM - Info bptm (pid=268171) reading backup image
May 6, 2018 9:40:21 AM - Info bptm (pid=268171) Waiting for mount of media id XXXXXX (copy 1) on server <media_server>
May 6, 2018 9:40:21 AM - started process bptm (pid=268171)
May 6, 2018 9:40:21 AM - mounting XXXXXX
May 6, 2018 9:40:22 AM - Info bptm (pid=268171) INF - Waiting for mount of media id XXXXXX on server <media_server> for reading.
May 6, 2018 9:41:01 AM - mounted XXXXXX; mount time: 0:00:40
May 6, 2018 9:41:01 AM - Info bptm (pid=268171) XXXXXXL
May 6, 2018 9:41:02 AM - Info bptm (pid=268171) INF - Waiting for positioning of media id XXXXXX on server <media_server> for reading.
May 6, 2018 9:41:02 AM - positioning XXXXXX to file 2
May 6, 2018 9:41:02 AM - positioned XXXXXX; position time: 0:00:00
May 6, 2018 9:41:02 AM - begin reading
May 6, 2018 9:56:03 AM - Error bptm (pid=268171) cannot read image from media id XXXXXX, drive index 0, Input/output error
May 6, 2018 9:56:03 AM - Warning bptm (pid=268171) TapeAlert Code: 0x15, Type: Warning, Flag: CLEAN PERIODIC, from drive IBM.ULTRIUM-TD6.002 (index 0), Media Id XXXXXX
May 6, 2018 9:56:04 AM - Info bptm (pid=268171) EXITING with status 85 <----------
May 6, 2018 10:06:39 AM - begin Import
May 6, 2018 10:06:41 AM - requesting resource XXXXXX
May 6, 2018 10:06:41 AM - granted resource XXXXXX
May 6, 2018 10:06:41 AM - granted resource IBM.ULTRIUM-TD6.002
May 6, 2018 10:22:23 AM - Error bpimport (pid=9452) Import of policy <policy_name>, schedule Weekly_Full (hostname_1524589090) failed, media read error.
May 6, 2018 10:22:24 AM - Error bpimport (pid=9452) Status = no images were successfully processed.
May 6, 2018 10:22:24 AM - end Import; elapsed time 0:15:45
no images were successfully processed (191)
This seems to be the reason for the failure:
TapeAlert Code: 0x15, Type: Warning, Flag: CLEAN PERIODIC, from drive IBM.ULTRIUM-TD6.002 (index 0),
Do you have cleaning tapes in the robot?
Maybe do a manual clean and try again?
Please ensure that bptm log folder exists and logging level for bptm is set to 3.
We contacted vendor and he suggested the same thing . There is no cleaning tape now so we will fetch that and run the manual cleaning. Is there any other area we need to lok into apart from that ?
Sorry for delay in reply but cleaning also did not help. We have raised case with vendor but they say that it is hardware issue and hardware vendor keeps denying that they do not find hardware errors. I am attaching bptm logs with maximum verbosity. Can anyone help us track down the actual issue.
We have uploaded the same logs to veritas already but they said its a hardware issue. We are not really convinced by that as level 1 import would have failed in such a case.Correct me if i am wrong.
We are just stuck without any resolution here.
I had a couple of minutes to look at the log.
I am sure the Veritas engineer pointed out the errors that can be seen in the logs. These errors are coming from the tape drive firmware.
The tape is mounted and positioned over here:
11:51:49.996  <2> io_position_for_read: positioning 00270L to file number 2
11:51:49.996  <2> io_position_for_read: locating to absolute block number 5
11:51:50.003  <2> io_position_for_read: locate block is done
It then goes through the decryption keys, and then attempts to read data:
11:51:50.570  <4> read_backup: begin reading backup id czstmv-vmpsc002.cognizantdc.com_1524589090 (import), copy 1, fragment 1, from media id 00270L on drive IBM.ULTRIUM-TD6.000 (index 3)
The drive firmware now reports error 5:
11:51:50.592  <2> read_data: attempting read error recovery, err = 5
11:51:50.592  <2> tape_error_rec: error recovery to block 6 requested
11:51:50.592  <2> tape_error_rec: attempting error recovery, delay 3 minutes before next attempt, tries left = 5
NBU now retries every 3 minutes - 5 more times, but each time receives the error 5.
Last attempt was 12:06:
12:06:51.199  <2> read_data: attempting read error recovery, err = 5
12:06:51.199  <2> tape_error_rec: error recovery to block 6 requested
12:06:51.214  <2> set_job_details: Tfile (585479): LOG 1528200411 16 bptm 114297 cannot read image from media id 00270L, drive index 3, Input/output error
12:06:51.353  <16> read_data: cannot read image from media id 00270L, drive index 3, Input/output error
2:06:51.514  <2> check_error_history: just tpunmount: called from bptm line 12394, EXIT_Status = 85
IBM says the following about error 5:
Read failure :
This flag is set for any unrecoverable read error where the isolation is uncertain and the failure could be either faulty media or faulty drive hardware. It is cleared when the cartridge is removed from the drive.
You will understand that NetBackup is merely reporting the error.
There is honestly nothing that can be done from NBU point of view to make this work.
There are several things you can do to try and pinpoint the problem (media or drive issue):
Compare firmware versions of tape drive that wrote the backup with firmware on this drive.
Try another tape drive.
See if you can restore from this tape in the source environment.
See if you can import other tapes in this environment.