01-25-2021 09:04 PM
Dear All,
From past few days i am facing a problem with netbackup 8.1.2
=========
Jan 26, 2021 12:31:33 PM - begin writing
Jan 26, 2021 12:37:06 PM - Error bptm (pid=1136) cannot open file C:\Program Files\Veritas\NetBackup\db\media\tpreq\drive_IBM.ULT3580-HH6.000, The system cannot find the file specified. (2)
Jan 26, 2021 12:40:06 PM - Error bptm (pid=1136) cannot open file C:\Program Files\Veritas\NetBackup\db\media\tpreq\drive_IBM.ULT3580-HH6.000, The system cannot find the file specified. (2)
Jan 26, 2021 12:43:06 PM - Error bptm (pid=1136) cannot open file C:\Program Files\Veritas\NetBackup\db\media\tpreq\drive_IBM.ULT3580-HH6.000, The system cannot find the file specified. (2)
Jan 26, 2021 12:46:06 PM - Error bptm (pid=1136) cannot open file C:\Program Files\Veritas\NetBackup\db\media\tpreq\drive_IBM.ULT3580-HH6.000, The system cannot find the file specified. (2)
Jan 26, 2021 12:49:06 PM - Error bptm (pid=1136) cannot open file C:\Program Files\Veritas\NetBackup\db\media\tpreq\drive_IBM.ULT3580-HH6.000, The system cannot find the file specified. (2)
Jan 26, 2021 12:49:06 PM - Error bptm (pid=1136) cannot write image to media id LTO008, drive index 0, The request could not be performed because of an I/O device error.
Jan 26, 2021 12:49:06 PM - Info bptm (pid=1136) EXITING with status 84 <----------
==============
Any idea how to fix this issue?
Solved! Go to Solution.
01-28-2021 10:29 PM
You have different errors now.
The OS has lost connectivity to tape drive and library and all drives have been DOWN'ed.
TLD(0) unavailable: initialization failed: Unable to open robotic path
TLD(0) [9064] Could not find SCSI coordinates {3,0,0,1} in the registry
09:54:49.518 [11384.284] <16> openTpreqFile: CreateFile on \\.\Tape0 failed, The system cannot find the file specified. (2)
There should be evidence of this in Event Viewer System log as well.
I have seen this happening some years ago with an aging tape library. The backups regularly failed with errors like these.
A power cycle of the tape library brought it back for a day or two.
The customer eventually replaced the tape library.
I am not saying that this is the same issue here.
From NBU point of view, it is difficult to troubleshoot.
All NBU knows is that it has lost connectivity to the devices via the OS paths.
NBU is merely reporting the error.
This means that you need to troubleshoot at OS level and the physical hardware.
Bear in mind that there is more involved than just the tape library and drive(s).
You need to troubleshoot the entire hardware path, starting at the server's hba.
01-26-2021 12:18 AM
This info is not sufficient to troubleshoot.
Do you have bptm log folder on this media server?
We need to see all info for PID 1136
bptm logging level should preferrably on level 3 (level 5 for a Support call with Veritas).
Please copy the log to bptm.txt and upload here as attachment.
If you do not have btpm folder on the media server, please create it in C:\Program Files\Veritas\NetBackup\logs directory and increase logging level for bptm in Host Properties.
Please also run Media Logs report for a period that covers about 12 hours before this backup until current time.
Save the report as .txt file and upload here.
Check C:\Program Files\Veritas\NetBackup\db\media\errors for entries for this tape drive.
Please share what you see.
Hopefully @mph999 will be along soon with ideas on what to check.
01-26-2021 08:18 AM
01-26-2021 10:50 PM
tpreq is the command that NBU uses to mount a tape in a drive. Part of the process is to create a tpreq file that is a link to the drive path. This file is created when the tape mount is successful.
See explanation of tpreq: https://www.veritas.com/support/en_US/doc/123533878-127136857-0/v123556035-127136857
We need to see bptm log to see what happened to the mount request.
01-27-2021 01:31 AM
Hi Marianne,
I've re-ran the backup job and enclosed please find the bptm log file.
Below are the Media Logs report:
01/27/21 15:35:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 15:38:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 15:41:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 15:44:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 15:47:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 15:47:41 LTO009 0 WRITE_ERROR IBM.ULT3580-HH6.000
01/27/21 16:10:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 16:13:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 16:16:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 16:19:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 16:22:41 LTO009 0 OPEN_ERROR IBM.ULT3580-HH6.000
01/27/21 16:22:41 LTO009 0 WRITE_ERROR IBM.ULT3580-HH6.000
01-27-2021 01:32 AM
Here is the bptm log
01-27-2021 03:51 AM
Please share the new Job Details so that we know what to look for in bptm log.
It seem that you shared the contents of the errors file, not the Media Logs report?
01-27-2021 04:23 AM
The bptm log is small, so I had a look.
I see the following in the log:
16:04:07.098 [14584.11896] <2> manage_drive_attributes: Media is protected with APPEND_ONLY
Is this a WORM tape?
From the errors file it seems that NBU is only trying to use one tape. Do you have other tapes that can be added?
Best to enable device-level logging as well.
Please add VERBOSE entry to ..\volmgr\vm.conf and restart NBU Device Management Services.
Media Manager actions and errors will be logged in Event Viewer System and Application logs.
01-28-2021 07:00 PM
Hi Marianne,
I re-ran the jobs and failed again. Below are the errors log in Event Viewer:
TLD(0) unavailable: initialization failed: Unable to open robotic path
TLD(0) [9064] Could not find SCSI coordinates {3,0,0,1} in the registry
=============
Below are the job detailed status history
Jan 29, 2021 10:07:53 AM - Info nbjm (pid=6604) starting backup job (jobid=1950) for client hkhkgpbku161, policy Daily_Backup, schedule Daily_Full
Jan 29, 2021 10:07:53 AM - Info nbjm (pid=6604) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1950, request id:{087E0271-14B0-4311-A3C1-1A1AE87B1483})
Jan 29, 2021 10:07:53 AM - requesting resource __ANY__
Jan 29, 2021 10:07:53 AM - requesting resource hkhkgpbku161.NBU_CLIENT.MAXJOBS.hkhkgpbku161
Jan 29, 2021 10:07:53 AM - Error nbjm (pid=6604) NBU status: 2009, EMM status: All compatible drive paths are down, but media is available
Jan 29, 2021 10:07:56 AM - Info bpbrm (pid=5464) Starting delete snapshot processing
Jan 29, 2021 10:07:58 AM - Info bpfis (pid=11844) Backup started
Jan 29, 2021 10:07:58 AM - Warning bpbrm (pid=5464) from client hkhkgpbku161: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.hkhkgpbku161_1611884752.1.0
Jan 29, 2021 10:07:58 AM - Info bpfis (pid=11844) done. status: 4207
Jan 29, 2021 10:07:58 AM - Info bpfis (pid=11844) done. status: 4207: Could not fetch snapshot metadata or state files
All compatible drive paths are down but media is available (2009)
=======================
However, I tried it succeeded to backup if I just select few small files(under 10 GB) to backup.
01-28-2021 07:01 PM
Here is the bptm log
01-28-2021 10:29 PM
You have different errors now.
The OS has lost connectivity to tape drive and library and all drives have been DOWN'ed.
TLD(0) unavailable: initialization failed: Unable to open robotic path
TLD(0) [9064] Could not find SCSI coordinates {3,0,0,1} in the registry
09:54:49.518 [11384.284] <16> openTpreqFile: CreateFile on \\.\Tape0 failed, The system cannot find the file specified. (2)
There should be evidence of this in Event Viewer System log as well.
I have seen this happening some years ago with an aging tape library. The backups regularly failed with errors like these.
A power cycle of the tape library brought it back for a day or two.
The customer eventually replaced the tape library.
I am not saying that this is the same issue here.
From NBU point of view, it is difficult to troubleshoot.
All NBU knows is that it has lost connectivity to the devices via the OS paths.
NBU is merely reporting the error.
This means that you need to troubleshoot at OS level and the physical hardware.
Bear in mind that there is more involved than just the tape library and drive(s).
You need to troubleshoot the entire hardware path, starting at the server's hba.
02-07-2021 04:22 PM
thank you Marianne,
After IBM replaced the tape drive, the backup issue has been fixed.