Backup Exec 2014 loses tape robot after reboot

[today]

I have an HP 1x8 G2 autoloder with LTO-6 drive.  I installed Backup Exec 2014 on my Server 2012 machine that is connected to this tape robot.  Without rebooting (because nothing I installed asked me to reboot), I was able to set up the tape robot within Backup Exec and run one backup to a tape.

I then rebooted the server.  When I brought up the Backup Exec gui, the robot came up in an offline state.  I tried to bring it back online, but that did nothing.  I still got errors when sending it an initailize or scan command.  I then deleted the robot and the drive from the Backup Exec interface, shut down all Backup Exec services, turned the tape drive off/on, waited for tape drive display to say it was ready, verified in Device Manager that both tape robot and tape drive are seen, then restarted all Backup Exec services.  Now, only the tape drive comes up.  When I try the Configure Storage button, and select Tape Storage, it leads me through the hotswap process.  Again, only the tape drive is seen, not the tape robot.

I have also tried changing the drivers for the robot from the HP version to the Microsoft Unknown Media Changer version (it was previously working with the HP drivers).  Backup Exec still can't see the robot.

I've looked through all the services for any other software I might have installed that has openen a connection to the robot, but I can't see anything suspicious.

I have the HP L&TT tools installed, so I ran them and verified that the robot is seen.  I'm even able to send it commands and do a scan.

Any ideas on what to do next?  This is truly baffling.  The tape robot shows up in device manager, so Backup Exec should be able to see it.  Could this be some tape library licensing issue?  I'm running the Trial version with the Agents for Linux, Mac, VMware/Hyper-V, and Windows installed, plus the Remote Media Agent for Linux.  But no other options have been installed.

At this point, I cannot even use Backup Exec at all, because I can't move media to the drive.

[CraigV]

Hi,

1. Make sure you're using the Symantec drivers for the drive. Use tapeinst.exe to install them.

2. What Windows sees, BE sees...so make sure that the library shows up correctly in Device Manager. The robotics should show up as: Unknow Medium Changer. If not, change the driver to reflect this.

Thanks!

[today]

Thanks for the quick reply.  However, tapeinst.exe is no longer used by Backup Exec 2014

I have also tried changing the drivers for the robot from the HP version to the Microsoft Unknown Media Changer version (it was previously working with the HP drivers).  Backup Exec still can't see the robot.

Just to emphasize this, the library shows up find in Device Manager.  The HP driver worked with the tape library before the reboot.

And after I changed the Microsoft drivers, BE 2014 still didn't see it.

To emphasize even more, I have gone through the "hot swap" process in Configure Storage and have gone as far as to stop all BE services and restart them again.  Plus, the HP tools I have see the tape library just fine.

[VJware]

Any errors/warnings under system logs in the event viewer ?

Check adamm.log as well if there any errors. This log is located at \Program Files\Symantec\Backup Exec\Logs

[CraigV]

...fantastic...something useful being removed from BE!

Have you run a device initialisation test with HP LTT?

Thanks!

[VJware]

Tapeinst.exe is no longer installed if BE 2014 is installed on a Win 2012 or above OS. However, you can install Symantec drivers. Couple of methods (verbatim from admin guide)

To get the latest device drivers installed to the Backup Exec installation folder
1 Click the Backup Exec button, select Technical Support, and then click Latest Backup Exec Device Drivers.
2 Run the Symantec Device Driver Installation Wizard, in the following procedure.

To install Symantec tape device drivers by using the Configure Storage wizard
1 On the Storage tab, in the Configure group, click Configure Storage.
2 On the wizard panel, select Tape Storage, and then click Next.
3 Select Install tape device drivers, and then click Next.
4 On the Symantec Device Driver Installation Wizard, follow the on-screen prompts.

[CraigV]

...what was the reason for removing it?

[VJware]

Tapeinst was primarily used for installing symantec kernel mode drivers. With media server running Win 2012 and above, user mode drivers are preferred as they are stable and can be installed without a reboot.

[CraigV]

Thanks very much!

[MSI_iqbal]

After you reboot your Tape Library and restart BE services it would be fine, sometime it just very slow to detect Tape Library. 

[today]

I tried this - didn't work for me.  The "Latest Backup Exec Device Drivers" button opens my browser to the URL http://go.symantec.com/support/BEWS-downloads-drivers/ which for me is getting redirected to the Enterprise Support query page with "Backup Exec" filled into one of the boxes.  There are a lot of articles listed under "Technical Solution", but if you select 2014 for version, they all disappear.  Furthermore, if you instead click on "Drivers (62)", there are no drivers listed for 2014.

I don't think the manual is right about installing tape drivers in BE 2014, at least if run on a Windows 2012 Server.  I am not offered the choice of "Install tape device drivers" on the second page of that wizard.

[today]

Thanks for leading me to that log file.  That has given me a lot of clues.  My problem is as described on this page.

http://www.symantec.com/business/support/index?page=content&id=TECH60088

Basically, it appears to be claming that it cannot find the serial number for this changer.  But that is a lie for a number of reasons, as I will show.

First, the tape library firmware is fully up-to-date.  It is an HP 1/8 G2 Tape Autoloader, and is running firmware 4.20.  According to the change notes...

Implemented between 4.00 and 4.10:  Added support for tape drive reporting autoloader serial number over SCSI.

Second, I ran Discover.exe, and got this for output.  Note that INQ 80 is the serial number.

ByGuid: Tape

  00:00:01:00 01 2147483646 \\?\scsi#sequential&ven_hp&prod_ultrium_6-scsi#5&1e4d4873&0&000100#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}
              INQ 00 "HP      Ultrium 6-SCSI  33HW"
              INQ cc "HP      ULTRIUM8000 DRV 33HW"
              INQ dc ""
              INQ 83 "HP      Ultrium 6-SCSI  HU14101AW6"
              INQ 80 "HU14101AW6"
              TUR 00 23a00
              RES 00 00 00000
              REL 00 00 00000
              MAX drive bytes 0xffffff
              MIN drive bytes 0x1
              BUS SAS(10)
              MAX controller bytes 0xffffff pages 0x101

ByGuid: Changer

  00:00:01:01 08 2147483646 \\?\scsi#changer&ven_hp&prod_1x8_g2_autoldr#5&1e4d4873&0&000101#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}
              INQ 00 "HP      1x8 G2 AUTOLDR  4.20"
              INQ cc "HP      1x8 G2 AUTOLDR  4.20"
              INQ dc ""
              INQ 83 "HP      1x8 G2 AUTOLDR  MXA416Z315"
              INQ 80 "MXA416Z315"
              TUR 00 00000
              RES 00 00 00000
              REL 00 00 00000
              MTE 0000   1
              STE 03e9   8
              IEE 0065   0
              DTE 0001   1
                0 "HP      Ultrium 6-SCSI  HU14101AW6"
              BUS SAS(10)
              MAX controller bytes 0xffffff pages 0x101

Here's what happens in the adamm.log

When I first installed the drivers for the tape robot and tape drive, it never asked me to reboot the system.  So I didn't.  I then installed Backup Exec 2014, which also never asked me to reboot the system.  Even after installing Windows Updates that got dragged in by the BE install, I still didn't have to reboot the system.  When BE first ran, it found the tape library and tape drive at Changer0 and Tape0 (I'm cutting out some of the logs for space here).

0000:0000:0001:0000  Device Name             "\\.\Tape0"
                     Secondary Name          "\\?\scsi#sequential&ven_hp&prod_ultrium_6-scsi#5&1e4d4873&0&000100#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
                     Primary Inquiry         "HP      Ultrium 6-SCSI  33HW"
                     Serial Number           "HP      Ultrium 6-SCSI  HU14101AW6"
                     Device Flags            UMD, SCSI, SN(TYPE 0)
                     Device State            3, Online

0000:0000:0001:0001  Device Name             "\\.\Changer0"
                     Secondary Name          "\\.\Changer0"
                     Primary Inquiry         "HP      1x8 G2 AUTOLDR  4.20"
                     Serial Number           "HP      1x8 G2 AUTOLDR  MXA416Z315"
                     Device Flags            UMD, SCSI, SN(TYPE 0), SN(ELEMENT)
                     Device State            3, Online

After it got rebooted, Tape0 became Tape2147483646 (that's what happens when you take -2 and convert it to an unsigned number).  This is due to a registry entry Microsoft added 10 years ago called Persistence.  It is explained more here.  http://support.microsoft.com/kb/873337

Anyway, I have no idea why this key wasn't set to 1 right away so the first time the unit came up, it would be at the right name.  But it didn't.  However, Changer0 simply disappeared entirely (like in the TECH60088 link above).  Note that the serial number is missing as well.

0000:0000:0001:0000  Device Name             "\\.\Tape2147483646"
                     Secondary Name          "\\?\scsi#sequential&ven_hp&prod_ultrium_6-scsi#5&1e4d4873&0&000100#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
                     Primary Inquiry         "HP      Ultrium 6-SCSI  33HW"
                     Serial Number           "HP      Ultrium 6-SCSI  HU14101AW6"
                     Device Flags            KMD, SCSI, SN(TYPE 1)
                     Device State            3, Online

0000:0000:0001:0001  Device Name             ""
                     Secondary Name          "\\.\Changer0"
                     Primary Inquiry         "HP      1x8 G2 AUTOLDR  4.20"
                     Serial Number           ""
                     Device Flags            KMD, SCSI
                     Device State            1, Offline, ERROR = 0x0000007B (ERROR_INVALID_NAME)

Now it turns out there is a similar registry key for Persistence for the tape changer as well (more here http://www.data-protector.org/wordpress/2013/05/persistent-binding-hp-lto-tape-drives-windows/ ), and that was unset.  So I thought that could be causing this problem.  So I set it and rebooted.

Nope.  Changer0 changed to Changer2147483646 as expected, but still, it can't find the Serial Number nor the Device Name.

0000:0000:0001:0000  Device Name             "\\.\Tape2147483646"
                     Secondary Name          "\\?\scsi#sequential&ven_hp&prod_ultrium_6-scsi#5&1e4d4873&0&000100#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
                     Primary Inquiry         "HP      Ultrium 6-SCSI  33HW"
                     Serial Number           "HP      Ultrium 6-SCSI  HU14101AW6"
                     Device Flags            KMD, SCSI, SN(TYPE 1)
                     Device State            3, Online

0000:0000:0001:0001  Device Name             ""
                     Secondary Name          "\\.\Changer2147483646"
                     Primary Inquiry         "HP      1x8 G2 AUTOLDR  4.20"
                     Serial Number           ""
                     Device Flags            KMD, SCSI
                     Device State            1, Offline, ERROR = 0x0000007B (ERROR_INVALID_NAME)

Looking at the following registry key...

HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\Scsi\Scsi Port 0\Scsi Bus 0\Target Id 1\Logical Unit Id 1

...you can see that the SerialNumber has been found for the tape changer.

So despite my firmware being up to date and Discover.exe being able to find the Serial Number for my tape changer and the OS having found the number and stored it in the Registry, somehow the main BE 2014 software cannot find this number.  But it was working fine before I rebooted the machine after initial device install.  Very odd.

[VJware]

Would you pls post adamm.log.

And I would recommend to log a formal support case so that your issue can be resolved asap.

[today]

PS - there were only these entries in the application event log

Backup Exec Alert: Storage Error
(Server: "USSBAHYP02") Backup Exec was unable to initialize and communicate with the device  [HP      1x8 G2 AUTOLDR  4.20] (The system cannot find the path specified.).  Click the link below for more information on how to diagnose the problem.

ADAMM configuration warning.
The device driver may not be installed for device :
 Identifier : HP      1x8 G2 AUTOLDR  4.20.
 Type : MediumChangerPeripheral.

[Larry_Fine]

I think that you changed persistence in the wrong direction and that BE doesn't like the huge device numbers, like Tape2147483646.  I would suggest deleting the keys HKey_Local_Machine\System\CurrentControlSet\Control\Tape\Persistence and HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\MChgr\Persistence (or set them to 0), then reboot the server.  BE should be happy with Tape0 and Changer0.

It also sounds like you have a vendor driver loader for your changer.  BE recommends using the MS driver.  So, changing your changer driver may fix it also.

http://www.symantec.com/docs/TECH33002

[Larry_Fine]

I think the admin guide is mostly correct.  Page 361 and 362 talk about the difference between Windows Server 2012 and previous versions of Windows.  But the part that VJware posted above should be made more obvious that it does NOT apply to Windows Server 2012 so it isn't taken out of context.

[today]

Thanks for your reply.

The tape drive and tape robot are considered separate entities.  I am able to connect to the tape drive from BE 2014 when it is named Tape0 and when it is named Tape2147483646.  The problem is that the BE 2014 software for some reason claims it can no longer find the Device Name or Serial Number of the Tape library whether it is named Changer0 or Changer2147483646.  However, the C:\Program Files\Symantec\Backup Exec\Discover.exe tool has no problem finding both the tape drive and the changer and the serial number of both of them.

I have tried both the HP driver and the MS driver for the changer - both produce the same results.

I suppose it is possible that the Persistance key on the Tape drive is somehow killing the serial number stuff on the Changer.  After all, it was named Tape0 when the tape library was working.  But Backup Exec ought to be able to deal with these super long numbers in the name (and it seems like it can).  They have been around for 10 years.  Anyway, I will try killing both Persistence keys and rebooting and then also using the MS drivers tonight and see what happens.

Below are the full results of Discover.exe.

System Page Size: 4096

ByName: Tape

ByName: MediumChanger

ByName: Changer

ByPort: Scsi

  01:01:64:00 1f -- \\.\Scsi1:
              INQ 00 "  RAID   DummyDevice    0001"
  01:01:64:00 IOCTL_SCSI_PASS_THROUGH_DIRECT on \\.\Scsi1: failed, 1117 - The request could not be performed because of an I/O device error.

  01:01:64:00 IOCTL_SCSI_PASS_THROUGH_DIRECT on \\.\Scsi1: failed, 1117 - The request could not be performed because of an I/O device error.

  01:01:64:00 IOCTL_SCSI_PASS_THROUGH_DIRECT on \\.\Scsi1: failed, 1117 - The request could not be performed because of an I/O device error.

  01:01:64:00 IOCTL_SCSI_PASS_THROUGH_DIRECT on \\.\Scsi1: failed, 1117 - The request could not be performed because of an I/O device error.

ByGuid: Tape

  00:00:01:00 01 2147483646 \\?\scsi#sequential&ven_hp&prod_ultrium_6-scsi#5&1e4d4873&0&000100#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}
              INQ 00 "HP      Ultrium 6-SCSI  33HW"
              INQ cc "HP      ULTRIUM8000 DRV 33HW"
              INQ dc ""
              INQ 83 "HP      Ultrium 6-SCSI  HU14101AW6"
              INQ 80 "HU14101AW6"
              TUR 00 23a00
              RES 00 00 00000
              REL 00 00 00000
              MAX drive bytes 0xffffff
              MIN drive bytes 0x1
              BUS SAS(10)
              MAX controller bytes 0xffffff pages 0x101

ByGuid: Changer

  00:00:01:01 08 2147483646 \\?\scsi#changer&ven_hp&prod_1x8_g2_autoldr#5&1e4d4873&0&000101#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}
              INQ 00 "HP      1x8 G2 AUTOLDR  4.20"
              INQ cc "HP      1x8 G2 AUTOLDR  4.20"
              INQ dc ""
              INQ 83 "HP      1x8 G2 AUTOLDR  MXA416Z315"
              INQ 80 "MXA416Z315"
              TUR 00 00000
              RES 00 00 00000
              REL 00 00 00000
              MTE 0000   1
              STE 03e9   8
              IEE 0065   0
              DTE 0001   1
                0 "HP      Ultrium 6-SCSI  HU14101AW6"
              BUS SAS(10)
              MAX controller bytes 0xffffff pages 0x101

ByGuid: ChangerIomega

ByGuid: CDRom

[VJware]

See if this helps ~ http://www.symantec.com/business/support/index?page=content&id=TECH36281

Secondly, try setting the BE services to manual start. Perform a power-cycle and let the device drivers load before the starting of Backup Exec. Check if the autoloader is detected then or not.

[pkh]

Check that the HBA that you are using does not support RAID.  BE has very limited support for HBA with RAID support.  See this document

http://www.symantec.com/business/support/index?page=content&id=TECH70907

[today · Accepted solution]

I did what I told Larry I was going to do - set all the Persistence keys to 0, go back to the Microsoft driver, and reboot.  Well, the tape changer came back.  Unfortunately, it came back with the name "Scsi0:".  Yes, including that colon.  BE 2014 had no problem interacting with it, however, it scared me that it might disappear again in the future.  So I changed the tape changer driver back to the official HP one, rebooted, and ended up with the tape library being named Changer0, just like it was in the first place.  Yippee!

Just to make this official and easy for others to digest should they stumble across this looking for a fix...

Problem: BE 2014 installed on Server 2012.  One HP tape library system.  BE 2014 doesn't show HP tape library connected, even though it appears in Device Manager under Medium Changer Devices.  BE's adamm.log says tape changer is offline with "ERROR_INVALID_NAME".  Log looks like the one on this KB page.

Solution: In the registry, track down the Persistence values for the changer and the tape drivers.  If you find that the Persistence value is set to 1, set it to 0.  If you see a subkey like Tape2147483646 or Changer2147483646, delete it.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\MChgr

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Tape

Next track down the PersistencePrivateCopy value for the driver's service.  If it is set to 1, set it to 0.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MChgr

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tape

Finally, make sure your tape library changer device is using the latest HP drivers, NOT THE MICROSOFT DEFAULT DRIVER.

Now reboot your machine.  Run the BE 2014 administration tool - your tape robot and tape deck should both appear.  You aren't done - check the very end of your adamm.log file again.  Make sure that the tape drive shows up as "\\.\Tape0" and your tape changer shows up as "\\.\Changer0".  Just to make sure this setup will survive future reboots, I would completely powerdown your server and tape drive system, then bring them up again, and test that all of the above still holds true.