cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted

<16> open_ndmp_device: cannot open ndmp device, error code 2 (NDMP_DEVICE_BUSY_ERR)

Unable to open NDMP device.

Logs from BPTM:

07:54:34.569 [11928.14988] <16> ndmp_setup_for_write: EEB: ndmp_media_first_time = 0, NdmpMediaSession_is_mover_open = 0
07:54:34.569 [11928.14988] <2> ndmp_setup_for_write: EEB: calling first time in ndmp_setup_for_write (FALSE)
07:54:34.569 [11928.14988] <16> ndmp_setup_for_write: EEB: ndmp_media_first_time = 0, NdmpMediaSession_is_mover_open = 0

07:58:16.006 [13012.13008] <16> open_ndmp_device: cannot open ndmp device /dev/alias/nst/XXXXXXXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)

-------------------

Detail Job logs

Nov 24, 2014 7:36:11 AM - started process bptm (pid=13012)
Nov 24, 2014 7:36:11 AM - mounting AS0588
Nov 24, 2014 7:36:11 AM - granted resource  AS0588
Nov 24, 2014 7:36:11 AM - granted resource  stu-hart2-005-02
Nov 24, 2014 7:36:11 AM - granted resource  media_server-hcart2-robot-tld-5-ndmp_server
Nov 24, 2014 7:36:56 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:39:37 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:40:22 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:40:22 AM - Warning bptm (pid=13012) media id AS0588 load operation reported an error
Nov 24, 2014 7:40:22 AM - current media AS0588 complete, requesting next media Any
Nov 24, 2014 7:45:31 AM - started process bptm (pid=13012)
Nov 24, 2014 7:45:31 AM - mounting AS0588
Nov 24, 2014 7:45:31 AM - granted resource  AS0588
Nov 24, 2014 7:45:31 AM - granted resource  stu-hart2-005-02
Nov 24, 2014 7:45:31 AM - granted resource  media_server-hcart2-robot-tld-5-ndmp_server
Nov 24, 2014 7:46:16 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX,  error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:49:26 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX,  error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:50:11 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX,  error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:50:11 AM - Warning bptm (pid=13012) media id AS0588 load operation reported an error
Nov 24, 2014 7:50:11 AM - current media AS0588 complete, requesting next media Any
Nov 24, 2014 7:54:04 AM - started process bptm (pid=13012)
Nov 24, 2014 7:54:04 AM - mounting AS0588
Nov 24, 2014 7:54:03 AM - granted resource  AS0588
Nov 24, 2014 7:54:03 AM - granted resource  stu-hart2-005-02
Nov 24, 2014 7:54:03 AM - granted resource  media_server-hcart2-robot-tld-5-ndmp_server
Nov 24, 2014 7:54:49 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:57:30 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:58:15 AM - Error bptm (pid=13012) cannot open ndmp device /dev/alias/nst/F0015XXXXX, error code 2 (NDMP_DEVICE_BUSY_ERR)
Nov 24, 2014 7:58:16 AM - Warning bptm (pid=13012) media id AS0588 load operation reported an error
Nov 24, 2014 7:58:16 AM - current media AS0588 complete, requesting next media Any
Nov 24, 2014 8:02:02 AM - current media -- complete, awaiting next media Any. Waiting for resources. 
          Reason: Drives are in use, Media server: media_server.test.com, 
          Robot Type(Number): TLD(5), Media ID: N/A, Drive Name: N/A, 
          Volume Pool: stu_Offsite, Storage Unit: media_server-hcart2-robot-tld-5-ndmp_server, Drive Scan Host: N/A, 
          Disk Pool: N/A, Disk Volume: N/A 

-------------------------------------------------

Followed this Tech Note and created NO_SIXTEEN touch file as well

http://www.symantec.com/business/support/index?page=content&id=TECH48631

----------------------------------------------------

Tpautoconf output From media server and Robotic hosts

C:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -verify ndmp_server
Connecting to host "ndmp_servero" as user "backup"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
  host supports TEXT authentication
  host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
Host info is:
  host name "ndmp_server"
  os type "Quantum DXiXXXX"
  os version "2.1.3.47427 Build2"
  host id "XXXXX"
Login was successful
Host supports LOCAL backup/restore
Host supports 3-way backup/restore
Opening SCSI device "/dev/alias/mc/ADICXXXXXXXXX_LL1"...
Inquiry result is "ADIC    Scalar i2000    XXXXXQ.GMXXXXX       "

C:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -probe ndmp_servero
.com
Host "ndmp_servero" SCSI device model "ADIC Scalar":
  Device "/dev/alias/mc/ADICXXXXXXXXX_LL1" attributes=0x0
    SERIAL_NUMBER=ADICXXXXXXXXX_LL1

------------------------------------

Please assist what could be done.

 

12 Replies
Highlighted

Hello Do you see any other

Hello

Do you see any other Deives other than "SCSI device model "ADIC Scalar":" when you do the tpautoconf -probe command.?

Highlighted

A bit of background

A bit of background please....

This is Quantum DXi seen as ADIC Scalar robot?

Are you trying 'NDMP Direct' for VTL -> Robot direct copy?

Highlighted

Look

Look like  /dev/alias/nst/F0015XXXXX is being used by another server (NDMP DEVICE BUSY). Either becuase tape drive is  writing data for another host or it has been reserved by a host but never released again. Given the age of the tech note I don't think it apply.

Do you use the SSO option in Netbackup ?

Are FC zoning correct ?

Highlighted

Ram, Do you see any other

Ram,

Do you see any other Deives other than "SCSI device model "ADIC Scalar":" when you do the tpautoconf -probe command.?

    Not sure, but the probe command output I shared above.

Marianne,

This is Quantum DXi seen as ADIC Scalar robot?

Are you trying 'NDMP Direct' for VTL -> Robot direct copy? 

Please assist how to check this.

Niloai,

Look like  /dev/alias/nst/0015XXXXX is being used by another server (NDMP DEVICE BUSY). Either becuase tape drive is  writing data for another host or it has been reserved by a host but never released again. Given the age of the tech note I don't think it apply.
(Drive are not shared with other host, these are dedicated to NDMP)

Do you use the SSO option in Netbackup ? (SSO is there, but these are dedicated drives for NDMP.)

Are FC zoning correct ? How to check this, plz assist.

Highlighted

if the above is the all the

if the above is the all the output that you have with probe command... its not listing the tape Drives... 

it cnfirms that the drive is not ready to take bakups and that is the reason its giving the error (NDMP_DEVICE_BUSY_ERR)

see if you can reset the tape alisase from the NDMP deivce end 

 

Highlighted

You need to speak to the SAN

You need to speak to the SAN admin and ask him/her if multiple initiators are in the same zone.

You could also power cycle the tape drive assigned to the NDMP filer, this will clear any scsi reservation

Some background info using Linux/UNIX

http://www.symantec.com/docs/HOWTO32844

Highlighted

Are you trying 'NDMP Direct'

Are you trying 'NDMP Direct' for VTL -> Robot direct copy? 

Please assist how to check this.

 

Surely this is not a new installation and SOMEONE in your company should know?

Who installed and configured it?
Where is your NBU config documentation?

You should know what make/model devices are in your environment and what they are used for, right?
At least someone in your company should know... 

Highlighted

From NDMP host, how can the

From NDMP host, how can the drives be reset ? Can I run the configuration from NBU console ?

One drive have been replaced on Friday, after that drive configuration wizard has been ran. Would that cause an issue ?

Nicolai: SAM admin, you mean someone from Storage team ?

Marianne,

Offcourse, its not new....but unfortunately I am new to this environment, and all other old buddies left before I jumped in.

So many issue related to understand the environment.

And struggling with such issue daily. :-(

Sorry to bother you guys....but just looking for a Hand who can assist me.

 

Highlighted

Troubleshooting depends on

Troubleshooting depends on whether this is 'normal' NDMP or 'NDMP DirectCopy'.

You may want to have a look at diagrams in the manual and see which one resembles your environment.

See NBU 7 version of NDMP Admin Guide http://www.symantec.com/docs/TECH127075)

 p. 27 covers this topic: NDMP direct copy with VTL

"NetBackup copies backup images from the VTL disk to a physical tape in an NDMP storage unit. It copies without using media server I/O or network bandwidth. NetBackup can also copy NDMP images directly between NDMP tape drives attached to an NDMP host. In both cases, this feature is called NDMP direct copy. This feature also enables NetBackup to restore data directly from either the image in the VTL or from the physical NDMP tape.

p.28 contains a diagram and this explanation:

The NDMP direct copy feature uses a VTL that has an embedded NDMP tape server using the NDMP protocol. The embedded NDMP tape server moves the image from the VTL disk directly to a physical tape. The image does not pass through the NetBackup media server or travel over the network.

p.29 contains a diagram with explanation of data flow

 

Also see this post: https://www-secure.symantec.com/connect/forums/quantum-dxi-path-tape  

 

 

Highlighted

Do people not have system

Do people not have system documentation these days that describes the environment and any changes made to it ...
Highlighted

You need to help us to see

You need to help us to see what you see.

You did a copy & paste selecttivelywithout telling us anything.

In job details we see: 

 - started process bptm (pid=13012)

What kind of process? Is this a backup or duplication?

My thoughts were duplication because we see 2 storage units assigned:

Nov 24, 2014 7:36:11 AM - granted resource  stu-hart2-005-02
Nov 24, 2014 7:36:11 AM - granted resource  media_server-hcart2-robot-tld-5-ndmp_server

But we see only one media id. 
Weird thing is, media is mounting before the resource is granted

Nov 24, 2014 7:36:11 AM - mounting AS0588
Nov 24, 2014 7:36:11 AM - granted resource  AS0588

So, it seems we are missing information in Job Details prior to what is posted.

At the end of Job details we see:

Waiting for resources. 
          Reason: Drives are in use, Media server: media_server.test.com, 

Have you checked device usage at this point on media server? 
vmoprcmd -d

Maybe all drives are really busy?
What other Active jobs do you see in Activity Monitor?

What does 'nbrburil -dump' show in MDS Allocation section at the bottom?

What other 

Highlighted

Sorry for late reply, Yes,

Sorry for late reply,

Yes, Marianne this is Duplication job.

Devices are not shared, and that are only being used for Duplication.

No NDMP policy even configured to use NDMP devices.

==============================================================

The backups are going to disk, and from disk they are duplicating to Tape via SLP.