Trouble with Tape drive library

[ipmanyak]

I set Netbackup 8.1.1 recently and some time everything worked normally.

My tape library is  HP MSL-2024   with LTO7 catridges.

.  Now my job have status queued and say me:

25.05.2018 15:34:31 - Info nbjm (pid=5072) starting backup job (jobid=647) for client TMN-FS01.utair.dom, policy TEST-POLICY, schedule Full
25.05.2018 15:34:31 - Info nbjm (pid=5072) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=647, request id:{F8CC856E-49A4-4DCD-B338-0E9E5799FB59})
25.05.2018 15:34:31 - requesting resource  tmn-nbu-hcart-robot-tld-0
25.05.2018 15:34:31 - requesting resource  tmn-nbu.utair.dom.NBU_CLIENT.MAXJOBS.TMN-FS01.utair.dom
25.05.2018 15:34:31 - requesting resource  tmn-nbu.utair.dom.NBU_POLICY.MAXJOBS.TEST-POLICY
25.05.2018 15:34:31 - awaiting resource tmn-nbu-hcart-robot-tld-0. No drives are available.

----------------------------------------------------------------------------------------------------------------------

My library is visible and Inventory run succefully. I see 24 tapes. But drive diagnostics failed.

At first it try mount tape eg. ATA178L7 then wrote than cannot read barcode label.

On second try it wrote that "Drive Information    Failed    Unable to allocate resources for diagnostics"

Service "NetBackup Device Manager"  ( ltid) have status  - running

Some information from utilities

tpconfig -l

vmoprcmd -d

robotest amd other

--------------------------------------

vmoprcmd -d -h tmn-nbu

C:\Program Files\Veritas\Volmgr\bin>vmoprcmd -d -h tmn-nbu

                                PENDING REQUESTS
                                     <NONE>
                                  DRIVE STATUS
Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart    TLD               No                     Yes     Yes        0
                             ADDITIONAL DRIVE STATUS
Drv DriveName            Shared    Assigned        Comment
  0 HP.ULTRIUM7-SCSI.000  No       -

---------------------------------------------------

C:\Program Files\Veritas\Volmgr\bin>tpconfig -l
Device Robot Drive       Robot                    Drive                 Device
Type     Num Index  Type DrNum Status  Comment    Name                  Path
robot      0    -    TLD    -       -  -          -                     {4,0,3,1}
  drive    -    0  hcart    1      UP  -          HP.ULTRIUM7-SCSI.000  {4,0,3,0}

---------------------------------------------------------------

C:\Program Files\Veritas\Volmgr\bin>robtest.exe
Configured robots with local control supporting test utilities:
  TLD(0)     robotic path = {4,0,3,1}

Robot Selection
---------------
  1)  TLD 0
  2)  none/quit
Enter choice: 1
1
Robot selected: TLD(0)   robotic path = {4,0,3,1}
Invoking robotic test utility:
C:\Program Files\Veritas\Volmgr\bin\tldtest.exe -rn 0 -r {4,0,3,1}
Opening {4,0,3,1}
MODE_SENSE complete
Enter tld commands (? returns help information)
s s 5
slot 5 (addr 1005) contains Cartridge = yes
Source address = 1005
Barcode = ATA164L7

----------------------------------------------------------------

C:\Program Files\Veritas\Volmgr\bin>vmoprcmd -h tmn-nbu -timeout 3600 -autoconfig  -a
TPAC60 - - - -1~-1~-1~-1 2 - - - 0 - - - - - - - 0 0 - - tmn-nbu.utair.dom 4 - 0 - - - -
 TPAC60 HP.ULTRIUM7-SCSI.000 HP~~~~~~Ultrium~7-SCSI~~G9Q1 - 4~0~3~0 1 0 0 1 8 3 - - - 9C1730E870 - 6 0 0 - - tmn-nbu.uta
ir.dom 0 - 0 - - - -
 TPAC60 0 HP~~~~~~MSL~G3~Series~~~7.00 - 4~0~3~1 0 0 0 - 8 - 24 1 0 DEC73408KB - 5 0 - tmn-nbu.utair.dom tmn-nbu.utair.d
om tmn-nbu.utair.dom 0 - 0 - - - -

--------------------------------------------------------------------------------------------------------------------------

C:\Program Files\Veritas\NetBackup\bin\admincmd> bpstulist -label tmn-nbu-hcart-robot-tld-0 -U

Label:                tmn-nbu-hcart-robot-tld-0
Storage Unit Type:    Media Manager
Host Connection:      tmn-nbu.utair.dom
Number of Drives:     24
On Demand Only:       no
Max MPX/drive:        1
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    TLD / 0
Max Fragment Size:    1048576 MB

----------------------------------------------------------------------

I again start and configure  thru master wizard storage  successfully but library dont work.

Yesterday drive test run succesfully/ But test job not run.

[ipmanyak]

Thanks mph999!

[mph999]

You are most welcome.

M

[ipmanyak]

My trouble appeared again. I dont understand why it occurs.  But test policy works normally when I freezed problems tape label eg. A170L7.  Also earlier I saw in Device two pathes for robot library (connected on fiber channel), now I do not see any. Any ideas ? I think I create a case in support veritas

---------------
29.05.2018 15:36:06 - Error bptm (pid=8204) ioctl (MTFSF) failed on media id A170L7, drive index 0, The request was not executed because of an input-output error on the device.. (1117) (../bptm.c.7046)
29.05.2018 15:36:06 - Info bptm (pid=8204) EXITING with status 86 <----------
29.05.2018 15:36:11 - Error bpduplicate (pid=6928) host tmn-nbu.utair.dom backup id TMN-FS01.utair.dom_1527586379 read failed, termination requested by administrator (150).
29.05.2018 15:36:11 - Error bpduplicate (pid=6928) host tmn-nbu.utair.dom backupid TMN-FS01.utair.dom_1527586379 write failed, media position error (86).
29.05.2018 15:36:11 - Error bpduplicate (pid=6928) Duplicate of backupid TMN-FS01.utair.dom_1527586379 failed, media position error (86).
29.05.2018 15:36:11 - Error bpduplicate (pid=6928) Status = no images were successfully processed.
29.05.2018 15:36:11 - end Duplicate; elapsed time 0:02:01
no images were successfully processed (191)
----------------------------------------------------------
29.05.2018 15:39:10 - Error nbjm (pid=3752) NBU status: 830, EMM status: All compatible drive paths are down, but media is available
29.05.2018 15:39:10 - begin Duplicate
29.05.2018 15:39:11 - end Duplicate; elapsed time 0:00:01
All compatible drive paths are down but media is available (2009)

---------------------

[mph999]

It looks like there is a another problem media.

How old are these tapes - they do not last forever, depending on how often they are used and how they are stored (if not in a tape library).  Also, if a drive has gone bad, this can physically damage media and hence you end up if a loop of everything going down.

[Marianne]

---

@ipmanyak wrote:

.......  Also earlier I saw in Device two pathes for robot library (connected on fiber channel), now I do not see any. Any ideas ? I think I create a case in support veritas

---------------------

---

This is NOT a Veritas problem. If devices 'disappear' from Windows Device Manager, you need to troubleshoot at hardware and OS-level.
NetBackup relies on the OS for device access.

Look in Windows System log for driver/hardware errors. Check FC switch logs for errors. 

[mph999]

Ohhh, thanks Marianne, I forgot to comment on that part ...

[ipmanyak]

This robot library is very new. We have bougth this device 4 month ago

[ipmanyak]

Marianne, this strimer is present in  Windows Device Manager. I wanted to say that I dont see drive paths in Device manager of NBU. Test backup policy has fulfilled normally when I have frozen the  problem tape.

[mph999]

If the paths have diasappeared from NBU, then at some point they disappeared from the OS, even if they are there now.

If the paths change, this will cause the issue, they disappear from NBU but are sen in the OS but have changed.  You need to set persistent bining to prevent them changing.  This is done at HBA level via the HBA utility (Sansurfer of HBAnywhere, depending on HBA make).

[mph999]

Forgot to say - as with disks, you sometimes get faulty batches of tapes.  You may wish to talk with your hardware vendor.

[Marianne]

---

@ipmanyak wrote:

This robot library is very new. We have bougth this device 4 month ago

---

Another 2c from my side...

Please bear in mind that the tape library is not the only hardware in the backup path. 

There is firstly the HBA in the server. This should not be shared with disk.
Disk and tape need different settings on the HBA - consult your HBA documentation.
Other hardware such as fiber gbic, the cable, the switch, etc.

Then there is also firmware and drivers for the hba and tape drive. 
Any one of these could cause the ongoing postioning errors.

Have you checked that you have the latest vendor firmware and drivers for the HBA and the tape drive? 

Have you checked Windows System log for hardware/driver errors?

[ipmanyak]

Firmware and driver is the lastest. In Windows System log for hardware/driver no errors except for

"Operator/EMM server has DOWN'ed drive HP.ULTRIUM7-SCSI.000 (device 0)" , when there is a positioning error

[mph999]

Did you open a case with Veritas, if so, what is the number.

[ipmanyak]

case # 180529-000839

[Alexis_Jeldrez]

You mention the tape library uses fibre channel. Can you confirm the server HBA(s) used for the connection against the library's tape drives is ONLY used for tape and NOT also used for connection to external LUN/disks?

[ipmanyak]

NBU is on virtual machine together other VM and use  Fiber channel  for other external LUN/disks.

[Marianne]

Are you aware that tape is NOT supported on virtual machines? Precisely because of issues that you are experiencing.
This is not a NetBackup limitation, but with VMware and Hyper-V.

[mph999]

Ahhh, VM ...  that changes things.

I'll stick my neck out a bit here - as Marianne says it is unsupported, which is correct.  Sometimes, although things may be unsupported, this often means it's not been tested, so it 'might' work.  For example, I have an 'unsupported' tape drive attached to one of my test servers, it works perfecty ...

However, in the case of drives attached to VM machines, you are going to have major issues due to the issues between hardware and VMWare (or Hyper-V).

Your only solution is to move the tapes drives/ library to a physical machine.

[ipmanyak]

We haven't a physical machine and haven't a single fiber channel controller, so it is impossible for us.We have Fibre Channel Fabrics .

[mph999]

I am sorry, but there is no other solution, well, not that is supported.

The problem is that VMWare / Hyper-V do not work with fibre attached tape devices/ libraries.  Unless the devices work correctly on the operating system level, NetBackup has no chance.

As you can see - it is not working which is why you are geting all these issues issues - tape drives going down etc ...

As Marianne explained, it is nothing to do with NetBackup, if you installed somewthing else like EMC Networker you will get the same issue.

So, as I mentioned, in this case there are no other options than to move the tape devices to a physical server.