Backup exex is unable to communicate with LTO autoloader.

[BackupMonkey50]

Since my last post was marked as spam and any queries to get it unmarked as such are ignored I'll repost it.

Hi all.

Server 2019 OS (Updated)
HP P212 HBA (None Raid) Single MSL2024 connected via external connector, nothing via the internal.
HP LTO5 SAS Drive

I've hit a small issue in that Backup Exec can't communicate with my HP Autoloader. Fresh install of Windows and BE up to latest version shows this in the adamm.log

**************************** LOG FILE*************************

[05764] 01/12/20 16:47:39.276 [StorageManager::RemoveUnusedLocalDisksFromDB] Results:
0 non-present unconfigured local disks were deleted
0 local disk parent objects with no local disks were deleted
[07068] 01/12/20 16:47:46.228 DeviceIo: Discover: serialize mode_sense FAILED, lib=(0,5,0), serial="MXA050Z3WP"
[07068] 01/12/20 16:47:46.260 DeviceIo Discovery - end
[07068] 01/12/20 16:47:46.260 DeviceIo Devices:
Scsi Address
Prt :Bus :Tar :Lun Attributes
------------------- -------------------------------------------------------------------------------
0003:0000:0004:0000 DeviceIo[01]: "HP Ultrium 5-SCSI HUJ7439GCF", Cert=Y, "\\.\Tape2147483646"
0003:0000:0005:0000 DeviceIo[02]: "HP MSL G3 Series MXA050Z3WP", Cert=N, "\\.\Changer0"
------------------- -------------------------------------------------------------------------------
[07068] 01/12/20 16:47:46.260 PnP Device Paths:
Scsi Address
Prt :Bus :Tar :Lun Device Path
----------------------------------- ---------------------------------------------------------------
00000003:00000000:00000004:00000000 \\?\scsi#sequential&ven_hp&prod_ultrium_5-scsi#7&74514f4&0&000400#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}
----------------------------------- ---------------------------------------------------------------
[07068] 01/12/20 16:47:46.260 Read Device Inquiries - start
[07068] 01/12/20 16:47:46.276 Backup Exec was unable to initialize and communicate with the device [HP MSL G3 Series 6.10] (The system cannot find the path specified.). Click the link below for more information on how to diagnose the problem.

**************************** LOG FILE END*************************

Windows can see the autoloader and tape drive just fine.
If I leave BE discovering services for long enough, I get this in the adamm.log and then it shows up as "online" but it doesn't work as an endpoint.

**************************** LOG FILE *************************

0003:0000:0004:0000 Device Name "\\.\Tape2147483646"
Secondary Name "\\?\scsi#sequential&ven_hp&prod_ultrium_5-scsi#7&74514f4&0&000400#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
Primary Inquiry "HP Ultrium 5-SCSI Z6MW"
Serial Number "HP Ultrium 5-SCSI HUJ7439GCF"
Device Flags UMD, SCSI, SN(TYPE 0)
Device State 3, Online
Device IDs 1050, {011D99CC-5C32-417E-9A50-8E951DD0CA8F}
Device Name "Tape drive 0001"
Device Type 134283265, "LTO 64K (64K,10,0,E,HU)"
Device Features 0x005BFA7F: PB,SFB,SRB,WFM,SFF,SRM,PEOD,EL,LU,E,SBS,DC,SDC,TA,HU,RR,W,EN
Device Element 0, 0
Device Block Limits 512 min, 65536 max
Device Hard Errors 0 write, 0 read
Device Soft Errors 0 write, 0 read
0003:0000:0005:0000 Device Name "\\.\Changer0"
Primary Inquiry "HP MSL G3 Series 6.10"
Serial Number "HP MSL G3 Series MXA050Z3WP"
Device Flags UMD, SN(TYPE 0)
Device State 3, Online
Device IDs 1052, {F0F6BDE3-5E47-4517-9E9D-57967B73A29E}
Device Name "Robotic library 0001"
Device Type 2131755008, "CHANGER FS=1"
Device Features 0x00026000: RMP,RRD,SMCE
1st Slot Number 1
Number Of Slots 0
Portal Slots 0
Import/Export Manual
Drives 0
0003:0000:0005:0000 Device Name "\\.\Changer0"
Primary Inquiry "HP MSL G3 Series 6.10"
Serial Number "HP MSL G3 Series MXA050Z3WP"
Device Flags UMD, SN(TYPE 0)
Device State 3, Online
Device IDs 1052, {F0F6BDE3-5E47-4517-9E9D-57967B73A29E}
Device Name "Robotic library 0001"
Device Type 2131755008, "CHANGER FS=1"
Device Features 0x00026000: RMP,RRD,SMCE
1st Slot Number 1
Number Of Slots 0
Portal Slots 0
Import/Export Manual
Drives 0

**************************** LOG FILE END*************************

Any advice?

[Larry_Fine]

"Number Of Slots 0" and "Drives 0" definitly doesn't look right, so either the library is confused or theire is bad communication issues.

Please make sure you are NOT using HP drivers for the robot OR the tape drive.

Please make sure you are NOT using an unsupported RAID HBA.  Edit - now I see you are using the P212 which is supported https://www.veritas.com/content/support/en_US/article.100021088  But you might need to check with HPE to see if it is configured correctly for use with tape.

Can you post your adamm.log file?

[BackupMonkey50]

Hi Larry.

The MSL is listed as a Unknown Medium Changer, driver provided by Microsoft.
The LTO is listed as HP LTO Ultrium-5 Drive, driver provided by HP.

I've been unable to locate a driver for this that isn't provided by HP.

Adamm.log is attached.

I'll keep looking for a driver. 
Thanks for the help. 

[BackupMonkey50]

New file after using the just the basic LTO driver.

[Larry_Fine]

It looks like something is wrong with your hardware or configuration.  Normally, BE detects tape libraries properly with no user intervention.

I was going to say that I had never seen this error before

DeviceIo: Discover: serialize mode_sense FAILED

but apparently somebody else asked about it here in 2008, but never followed up :(

https://vox.veritas.com/t5/Backup-Exec/Problem-cannot-detect-media-in-Storageloader-LTO4-slots/td-p/...

I would suggest follwoing the troubleshooting document closely and let us know what you find.

https://www.veritas.com/support/en_US/article.100016050

[BackupMonkey50]

Hi Larry.

Every check came back fine.
Discover.txt (Attached) is fine.
Tracer comes back as fine (attached)

Quick assist has a few issues such as..

Local user will have an issue backing up or restoring exchange - Exchange isn't installed so no issue.
BE Dedup service isn't running - Dedupe option isn't installed. :)

Nothing massive.
The hunt progresses.

[Larry_Fine]

Your discover log is incorrect and incomplete, so there are obvious issues with the server or the robotic library, or communications between them.

03:00:05:00 08 -- \\.\Scsi3:
INQ 00 "HP MSL G3 Series 6.10"
INQ cc "HP MSL 2024 6.10"
INQ dc ""
INQ 83 "HP MSL G3 Series MXA050Z3WP"
INQ 80 "MXA050Z3WP"
TUR 00 2048e
RES 00 00 00000
REL 00 00 00000
MTE 0000 0
STE 0000 0
IEE 0000 0
DTE 0000 0
BUS RAID(8)
MAX controller bytes 0x10000 pages 0x11

MTE should be 1

STE shoud be your number of slots

IEE should be your number of portals

DTE should be your number of tape drives and there should be an additional line below it with the tape drive serial number.

https://www.veritas.com/support/en_US/article.100023117

[BackupMonkey50]

Missed that one. 

I'll start looking into that, thanks!

[BackupMonkey50]

Running the HP Tape Library and Tools gives a blue screen on hardware discovery. :( Never looked at it given that Windows can see it.

New HBA will be here Wed\Thurs so will do the swap and see what we get, I'll update the thread for historical and anyone else that comes across it. 

Issue isn't fixed but the root cause isn't BE.

Thanks Larry.

[Larry_Fine]

Wow, you are the second person recently to mention getting a blue screen with L&TT.

https://vox.veritas.com/t5/Backup-Exec/Backup-Exec-and-HPE-L-amp-TT/m-p/874134#M358878

unfortunately, that thread never got answered.

[BackupMonkey50]

Interesting, thanks for asking him for an update.

Tomorrow I'm going to build up an old Gen7 HP Microserver with the same HP P212 and see what I can get out of it.

I'll post results.

[Larry_Fine]

Since it appears that both blue screens occured on Server 2019, you may want to try a different, older, verion of Windows?

[BackupMonkey50]

Hi Larry.
Good news, and I'm not saying I understand it :)

The HP P212 doesn't have a UEFI boot ROM, just a legacy\BIOS one. (To boot the card, not the OS from the card)
My machine was set to only allow PCI boot rooms that were UEFI.

So from my understanding on this part of hardware, the P212 could initilize but not much else.

This also fixes the HPE Tape application blue screening the machine :)

Working ADAMM log attached. 

[Larry_Fine]

Cool, glad to see progress and thanks for the follow through.

I still see a few weird things in your adamm.log, but if it works, then you may live with it.

0003:0000:0004:0000 Device Name "\\.\Tape2147483646"  *** that is an unusually long number
Secondary Name "\\?\scsi#sequential&ven_hp&prod_ultrium_5-scsi#7&74514f4&1&000400#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
Primary Inquiry "HP Ultrium 5-SCSI Z6MW"
Serial Number "HP Ultrium 5-SCSI HUJ7439GCF"
Device Flags UMD, SCSI, SN(TYPE 0)
Device State 3, Online

Device IDs 1092, {0390A5A1-375F-4EFD-9629-76665F45ABF3}
Device Name "Tape drive 0001"
Device Type 134283265, "LTO 64K (64K,10,0,E,HU)"
Device Features 0x005BFA7F: PB,SFB,SRB,WFM,SFF,SRM,PEOD,EL,LU,E,SBS,DC,SDC,TA,HU,RR,W,EN
Device Element 0, 0
Device Block Limits 512 min, 65536 max
Device Hard Errors 0 write, 0 read
Device Soft Errors 0 write, 0 read

0003:0000:0005:0000 Device Name "\\.\Changer0"
Secondary Name "\\.\Changer0"
Primary Inquiry "HP MSL G3 Series 6.10"
Serial Number "HP MSL G3 Series MXA050Z3WP"
Device Flags UMD, SCSI, SN(TYPE 0)
Device State 3, Online

Device IDs 1093, {04E9B335-A205-4B57-B683-744A9F8BA637}
Device Name "Robotic library 0001"
Device Type 2131755008, "CHANGER FS=1"
Device Features 0x00006000: RMP,RRD
1st Slot Number 1
Number Of Slots 24
Portal Slots 0
Import/Export Manual
Drives 2

Drive Element 0 0, ""  These quotes should be filled in with the tape drive serial numbers
Drive Element 1 0, ""  These quotes should be filled in with the tape drive serial numbers

How many tape drives do you have?  I only see one tape drive being discovered by Windows and BE, but the library thinks that it has two tape drives.  Is one tape drive not connected to the server via SAS cable?

[BackupMonkey50]

I'm fighting that as we speak.
I moved the one drive into the other slot to see if it was that.

I've moved the drive back but BE keeps seeing 'both' drives now even throu the MSL shows one.

I've just removed everything, rebooted everything and reinstalling drivers and such.

& we've gone back to blue screening the OS. :(

[Larry_Fine]

If you only have one tape drive, but at some point the library had two tape drives (or maybe just the one drive moved to the second drive bay), then the libray itself may be confused about how many drives it should report.  I would consider doing a factory default on the library itself.

[BackupMonkey50 · Accepted solution]

Factoy reset everything now which didn't work. 

However, I moved the drive from bay0 to bay1 on the silo, cleared everything out and now everything works.
I can only think is that the connection is 'dirty' or damaged in that bay on the silo. 

Erase tape job:

Job started     : 15 January 2020 16:26:37
Job ended       : 15 January 2020 16:30:04
Completed status: Completed with exceptions

System Status

View Legend
Updated: Wednesday,1/15/2020 16:32:44
Status	Ready
Drive 1 Status	Ready
Slots (Free/Total)	0/24
Mailslot	Disabled
Library Time	01/15/20 16:32

Can you check over the attached adamm.log?

[Larry_Fine]

I don't see anything obviously concerning in the adamm.log.  I see the tape library now agrees that it only has one tape drive. :)

Is it all working OK now?

[BackupMonkey50]

Yup.
Backups are working just as before.
1 drive, 1 MSL, 1 happy owner.

Thanks for the help Larry.