11-24-2014 07:08 AM
Unable to open NDMP device.
Logs from BPTM:
07:54:34.569 [11928.14988] <16> ndmp_setup_for_write: EEB: ndmp_media_first_time = 0, NdmpMediaSession_is_mover_open = 0
07:54:34.569 [11928.14988] <2> ndmp_setup_for_write: EEB: calling first time in ndmp_setup_for_write (FALSE)
07:54:34.569 [11928.14988] <16> ndmp_setup_for_write: EEB: ndmp_media_first_time = 0, NdmpMediaSession_is_mover_open = 0
07:58:16.006 [13012.13008] <16> open_ndmp_device: cannot open ndmp device /dev/alias/nst/XXXXXXXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
-------------------
Detail Job logs
Nov 24, 2014 7:36:11 AM - started process bptm (pid=13012)
Nov 24, 2014 7:36:11 AM - mounting AS0588
Nov 24, 2014 7:36:11 AM - granted resource AS0588
Nov 24, 2014 7:36:11 AM - granted resource stu-hart2-005-02
Nov 24, 2014 7:36:11 AM - granted resource media_server-hcart2-robot-tld-5-ndmp_server
Nov 24, 2014 7:36:56 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:39:37 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:40:22 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:40:22 AM - Warning bptm (pid=13012) media id AS0588 load operation reported an error
Nov 24, 2014 7:40:22 AM - current media AS0588 complete, requesting next media Any
Nov 24, 2014 7:45:31 AM - started process bptm (pid=13012)
Nov 24, 2014 7:45:31 AM - mounting AS0588
Nov 24, 2014 7:45:31 AM - granted resource AS0588
Nov 24, 2014 7:45:31 AM - granted resource stu-hart2-005-02
Nov 24, 2014 7:45:31 AM - granted resource media_server-hcart2-robot-tld-5-ndmp_server
Nov 24, 2014 7:46:16 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:49:26 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:50:11 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:50:11 AM - Warning bptm (pid=13012) media id AS0588 load operation reported an error
Nov 24, 2014 7:50:11 AM - current media AS0588 complete, requesting next media Any
Nov 24, 2014 7:54:04 AM - started process bptm (pid=13012)
Nov 24, 2014 7:54:04 AM - mounting AS0588
Nov 24, 2014 7:54:03 AM - granted resource AS0588
Nov 24, 2014 7:54:03 AM - granted resource stu-hart2-005-02
Nov 24, 2014 7:54:03 AM - granted resource media_server-hcart2-robot-tld-5-ndmp_server
Nov 24, 2014 7:54:49 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:57:30 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:58:15 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:58:16 AM - Warning bptm (pid=13012) media id AS0588 load operation reported an error
Nov 24, 2014 7:58:16 AM - current media AS0588 complete, requesting next media Any
Nov 24, 2014 8:02:02 AM - current media -- complete, awaiting next media Any. Waiting for resources.
Reason: Drives are in use, Media server: media_server.test.com,
Robot Type(Number): TLD(5), Media ID: N/A, Drive Name: N/A,
Volume Pool: stu_Offsite, Storage Unit: media_server-hcart2-robot-tld-5-ndmp_server, Drive Scan Host: N/A,
Disk Pool: N/A, Disk Volume: N/A
-------------------------------------------------
Followed this Tech Note and created NO_SIXTEEN touch file as well
http://www.symantec.com/business/support/index?page=content&id=TECH48631
----------------------------------------------------
Tpautoconf output From media server and Robotic hosts
C:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -verify ndmp_server
Connecting to host "ndmp_servero" as user "backup"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
host supports TEXT authentication
host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
Host info is:
host name "ndmp_server"
os type "Quantum DXiXXXX"
os version "2.1.3.47427 Build2"
host id "XXXXX"
Login was successful
Host supports LOCAL backup/restore
Host supports 3-way backup/restore
Opening SCSI device "/dev/alias/mc/ADICXXXXXXXXX_LL1"...
Inquiry result is "ADIC Scalar i2000 XXXXXQ.GMXXXXX "
C:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -probe ndmp_servero
.com
Host "ndmp_servero" SCSI device model "ADIC Scalar":
Device "/dev/alias/mc/ADICXXXXXXXXX_LL1" attributes=0x0
SERIAL_NUMBER=ADICXXXXXXXXX_LL1
------------------------------------
Please assist what could be done.
11-24-2014 07:21 AM
Hello
Do you see any other Deives other than "SCSI device model "ADIC Scalar":" when you do the tpautoconf -probe command.?
11-24-2014 07:24 AM
A bit of background please....
This is Quantum DXi seen as ADIC Scalar robot?
Are you trying 'NDMP Direct' for VTL -> Robot direct copy?
11-24-2014 07:26 AM
Look like /dev/alias/nst/F0015XXXXX is being used by another server (NDMP DEVICE BUSY). Either becuase tape drive is writing data for another host or it has been reserved by a host but never released again. Given the age of the tech note I don't think it apply.
Do you use the SSO option in Netbackup ?
Are FC zoning correct ?
11-24-2014 07:42 AM
Ram,
Do you see any other Deives other than "SCSI device model "ADIC Scalar":" when you do the tpautoconf -probe command.?
Not sure, but the probe command output I shared above.
Marianne,
This is Quantum DXi seen as ADIC Scalar robot?
Are you trying 'NDMP Direct' for VTL -> Robot direct copy?
Please assist how to check this.
Niloai,
Look like /dev/alias/nst/0015XXXXX is being used by another server (NDMP DEVICE BUSY). Either becuase tape drive is writing data for another host or it has been reserved by a host but never released again. Given the age of the tech note I don't think it apply.
(Drive are not shared with other host, these are dedicated to NDMP)
Do you use the SSO option in Netbackup ? (SSO is there, but these are dedicated drives for NDMP.)
Are FC zoning correct ? How to check this, plz assist.
11-24-2014 08:26 AM
if the above is the all the output that you have with probe command... its not listing the tape Drives...
it cnfirms that the drive is not ready to take bakups and that is the reason its giving the error (NDMP_DEVICE_BUSY_ERR)
see if you can reset the tape alisase from the NDMP deivce end
11-24-2014 09:08 AM
You need to speak to the SAN admin and ask him/her if multiple initiators are in the same zone.
You could also power cycle the tape drive assigned to the NDMP filer, this will clear any scsi reservation
Some background info using Linux/UNIX
http://www.symantec.com/docs/HOWTO32844
11-24-2014 09:46 AM
Are you trying 'NDMP Direct' for VTL -> Robot direct copy?
Please assist how to check this.
Surely this is not a new installation and SOMEONE in your company should know?
Who installed and configured it?
Where is your NBU config documentation?
You should know what make/model devices are in your environment and what they are used for, right?
At least someone in your company should know...
11-24-2014 11:52 AM
From NDMP host, how can the drives be reset ? Can I run the configuration from NBU console ?
One drive have been replaced on Friday, after that drive configuration wizard has been ran. Would that cause an issue ?
Nicolai: SAM admin, you mean someone from Storage team ?
Marianne,
Offcourse, its not new....but unfortunately I am new to this environment, and all other old buddies left before I jumped in.
So many issue related to understand the environment.
And struggling with such issue daily. :(
Sorry to bother you guys....but just looking for a Hand who can assist me.
11-24-2014 12:18 PM
Troubleshooting depends on whether this is 'normal' NDMP or 'NDMP DirectCopy'.
You may want to have a look at diagrams in the manual and see which one resembles your environment.
See NBU 7 version of NDMP Admin Guide http://www.symantec.com/docs/TECH127075)
p. 27 covers this topic: NDMP direct copy with VTL
"NetBackup copies backup images from the VTL disk to a physical tape in an NDMP storage unit. It copies without using media server I/O or network bandwidth. NetBackup can also copy NDMP images directly between NDMP tape drives attached to an NDMP host. In both cases, this feature is called NDMP direct copy. This feature also enables NetBackup to restore data directly from either the image in the VTL or from the physical NDMP tape.
p.28 contains a diagram and this explanation:
The NDMP direct copy feature uses a VTL that has an embedded NDMP tape server using the NDMP protocol. The embedded NDMP tape server moves the image from the VTL disk directly to a physical tape. The image does not pass through the NetBackup media server or travel over the network.
p.29 contains a diagram with explanation of data flow
Also see this post: https://www-secure.symantec.com/connect/forums/quantum-dxi-path-tape
11-24-2014 03:04 PM
11-24-2014 07:31 PM
You need to help us to see what you see.
You did a copy & paste selecttivelywithout telling us anything.
In job details we see:
- started process bptm (pid=13012)
What kind of process? Is this a backup or duplication?
My thoughts were duplication because we see 2 storage units assigned:
Nov 24, 2014 7:36:11 AM - granted resource stu-hart2-005-02
Nov 24, 2014 7:36:11 AM - granted resource media_server-hcart2-robot-tld-5-ndmp_server
But we see only one media id.
Weird thing is, media is mounting before the resource is granted.
Nov 24, 2014 7:36:11 AM - mounting AS0588
Nov 24, 2014 7:36:11 AM - granted resource AS0588
So, it seems we are missing information in Job Details prior to what is posted.
At the end of Job details we see:
Waiting for resources.
Reason: Drives are in use, Media server: media_server.test.com,
Have you checked device usage at this point on media server?
vmoprcmd -d
Maybe all drives are really busy?
What other Active jobs do you see in Activity Monitor?
What does 'nbrburil -dump' show in MDS Allocation section at the bottom?
What other
12-08-2014 12:12 PM
Sorry for late reply,
Yes, Marianne this is Duplication job.
Devices are not shared, and that are only being used for Duplication.
No NDMP policy even configured to use NDMP devices.
==============================================================
The backups are going to disk, and from disk they are duplicating to Tape via SLP.