Quantum i40 and BE2010 drive offline error

[Paul_L]

Hi,

We have had our Quantum Scalar i40 library setup for around two weeks now but have encountered problems getting scheduled jobs to run and getting the library to perform tasks through Backup Exec 2010 and 2010 R2.

We have the library connected to a Dell PowerEdge 1950 server running server 2003, via SAS cable, and have a single half height HP SAS LTO-4 drive in the Quantum.

The library shows in Backup Exec as Quantum001 and the HP tape drive HP001 plus Slots are beneath it.

When a tape is imported via the library's interface, and we run a scan then inventory and start a job off it runs without issue, but once the job has run it seems that the device becomes pretty much unusable until it is rebooted.

As a typical example of what happens.....

• Job runs, kicked off manually. Job completes.
• Attempt to export tape, it fails and drive goes offline.
• Set drive back to online, and attempt inventory, fails drive goes offline.
• Set drive to online and attempt initialize, fails and drive goes offline.
• Attempt to export tape via library web interface, error that do not have permission due to library being locked by an application.
• Attempt to unlock library via Backup Exec, fails drive goes offline.

The error messages which we get when the drive goes offline are either....

# The drive hardware is offline, Please confirm that the drive hardware is powered on and properly cabled. ID 58053

# [job name] - The job failed with the following error: Physical Volume Library Drive not available. ID 34113

# Robotic library hardware error - The robotic library has reported a general hardware error condition... ID 58053

We were able to install the Symantec drivers for the HP tape drive, but the Quantum device was not detected when running tapeinst.exe, i've done much searching on these and other forums and found that setting the library to Unknown Medium Changer should work. So currently it's set to that.

The SCSI information for each is HP Drive Port: 3 Bus: 0 Target ID: 45 LUN: 0 and library Port: 3 Bus: 0 Target ID: 45 LUN: 1.

Which ties in with what i've read about having the drive on a lower LUN than the library.

When we first configured the library it was using BE 2010, but we have since upgraded to 2010 R2 and the results are exactly the same.

Does anyone have experience of a similar problem or any advice on things we could try. 

If we reboot the library with the BE services stopped then the library works and we can import a tape, and perform a backup job with no problems at all, but once a job has run Backup Exec seems unable to perfrom any tasks, be it inventory, initialize, scan or unlock.

Any help will be greatly appreciated, thanks.

Paul.

[AmolB]

Hi Paul 

 Open BEutility.exe and check the status of the server if its "Paused" or "Running"

 BEutility is located at X:\Program Files\Symantec\Backup Exec.

 Also check Windows system log for any error message.

 Make sure the library and the tape device is reflecting in the Windows device manager.

[Paul_L]

..for the quick response,

BEUtil shows the server is running, and Windows event logs only show the three listed errors either with either 34113 or 58053 IDs all in the application log.

In device manger the devices all show fine..

Unknown Medium Changer (have also tried using Quantum supplied drivers)

DELL PERC 6/E Adapter RAID Controller

HP Ultrium 4-SCSI SCSI Sequential Device

thanks again, Paul.

[AmolB]

Paul, check the web console of the library I guess tape is stucked in the drive.

EDIT: Please post the description of event id 58053.

[Paul_L]

Yes, that's right the web interface (and operation panel) both refuse to allow the tape to be removed from the drive.

Once that happens the only solution is to stop the services in Backup Exec and reboot the library. After we do that everything works OK for a while, until a job completes and then same problem again.

These are the two errors the first happens after a job fails when i try to attempt an inventory etc. The second error appears when the scheduled job fails.

------------------------

Event Type: Error

Event Source: Backup Exec

Event Category: None

Event ID: 58053

Date: 22/12/2010

Time: 10:32:14

User: N/A

Computer: MARATHON

Description:

Backup Exec Alert: Device Error

(Server: "MARATHON") The drive hardware is offline.

Please confirm that the drive hardware is powered on and properly cabled. 

-------------------------

Event Type: Error

Event Source: Backup Exec

Event Category: None

Event ID: 58053

Date: 21/12/2010

Time: 09:10:49

User: N/A

Computer: MARATHON

Description:

Backup Exec Alert: Device Error

(Server: "MARATHON") (Job: "Inventory Device 00025") Robotic library hardware error.

The robotic library has reported a general hardware error condition. Manual intervention, calibration, etc., may be required. This may be a device related hardware problem, or it may be some other hardware problem (i.e. SCSI card/bus etc.). The library and drive states have been set to offline. Please attend to this condition. 

Cheers, Paul.

[AmolB]

Paul the issue seems to be with the medium changer, I guess you will have to contact 

hardware vendor to fix it.

[Paul_L]

Ok, thanks for the advice Amol.

If anyone else has any other suggestions then please feel free to contribute, but i'll get in touch with the vendor for some advice.

Thanks again.

[Paul_L]

Just to bump this thread.... i pulled the following from my adamm.log file earlier, currently waiting for Quantum to get back to me on the issue, but does anyone know what this error suggests?

[4100] 12/23/10 13:46:39.682 PvlChanger::MapErrorCode() - offline device

       Job = {299CF0D8-5B57-468D-BEF4-43549D243475} "Inventory Device 00026"

       Changer = 1013 "QUANTUM 0001"

       ERROR = 0xE000820A (E_CHG_HARDWARE)

[4100] 12/23/10 13:46:39.698 PvlChanger::UpdateOnlineState()

       Changer = 1013 "QUANTUM 0001"

       ERROR = The device is offline!

[4100] 12/23/10 13:46:39.698 Begin dump of device's SCSI history

[4100] 12/23/10 13:46:39.838 End dump of device SCSI history

[4100] 12/23/10 13:46:39.870 PvlDrive::UpdateOnlineState()

       Drive = 1012 "HP 0001"

       ERROR = The device is offline!

[4100] 12/23/10 13:46:39.870 Begin dump of device's SCSI history

[4100] 12/23/10 13:46:39.870 End dump of device SCSI history

[4100] 12/23/10 13:46:39.870 PvlChanger::MapErrorCode() - offline device

       Drive = 1012 "HP 0001"

       ERROR = 0xE000820A (E_CHG_HARDWARE)

[Gurvinder]

Follow this Hardware Troubleshooting TechNote step by step :-

www.symantec.com/docs/TECH24414

Also check "Support Policy for Host-Bus-Adapter that feature RAID"

www.symantec.com/docs/TECH70907

Somethings that you can check :-

- Does the front panel for RL report any errors

- Check for any Tape jams

- Make sure Tape Library is set to automatic or random

- Make sure RL is supported as per SCL

- Medium Changer and Drive have latest drivers

Test with OEM and Symantec Drivers

-- Check events in event viewer (system logs)

-- You can also run tracer.exe while running the Job. Export the log as text

and look for scsi errors

[Paul_L]

Hi Gurvinder, Thanks for the reply. I have read through and followed the steps in the hardware troubleshooting, all of the steps were performed and the only thing which it picked out were the above adamm log entries. and this error pulled from the tracer run. Event: 8 Start: 14:02:02.754 Stop: 14:02:02.754 Duration: 0.000000 SCSI Address: 03:00:45:00 Function SRB_FUNCTION_EXECUTE_SCSI SCSI Status SCSISTAT_CHECK_CONDITION Sense Length 20 Data Length 36 Driver Result STATUS_SUCCESS Raw CDB 12 01 CC 00 24 00 ....$. CDB Operation INQUIRY LUN 0 EVPD True Page 204 Alloc Length 36 Control 0x00 Data 01 80 05 02 5B 10 10 02 48 50 20 20 20 20 20 20 ....[...HP 55 6C 74 72 69 75 6D 20 34 2D 53 43 53 49 20 20 Ultrium 4-SCSI Sense Data 70 00 05 00 00 00 00 10 00 00 00 00 24 00 00 CF p...........$... 00 02 00 00 .... Filemark False EOM False ILI False Sense key ILLEGAL_REQUEST ASC INVALID_FIELD_IN_CDB It's the first time i've read through the information on RAID controllers being unsupported, we do have a PowerVault MD1000 connected to the same PERC 6/E HBA so perhaps that could be the issue. Regarding your other points.... - Does the front panel for RL report any errors Only errors are that it is not possible to remove a tape when Backup Exec reports drive offline, have to cycle BE services to unload tape. No actuall error message to report. - Check for any Tape jams Tape does become stuck, until BE services are restarted. Only after a job runs, prior to a job running import export etc seems fine. - Make sure Tape Library is set to automatic or random Not sure if this setting is present on the Scalar i40 but i will look into it when i speak to Quantum. - Make sure RL is supported as per SCL Library is on the HCL for BE 2010 and 2010 R2 - Medium Changer and Drive have latest drivers Have updated drivers for both drive and library, and tested with Symantec and OEM and tried library as Unknown media changer too. All same result, single job works fine but afterwards nothing is possible. - Test with OEM and Symantec Drivers as above. -- Check events in event viewer (system logs) Only thing found was posted in previous post above. -- You can also run tracer.exe while running the Job. Export the log as text and look for scsi errors as above --------------------------- Thanks for the hint about the RAID HBA not being supported i'll see if i can try another HBA card in the next couple of days.

[Gurvinder]

keeps us posted..thanks

[Paul_L]

Just to update anyone reading this thread i found a possible solution to my issue.

After installing Dell OpenManage server adminisatrator on the server i had a look to see if there were any settings to adjust on my HBA, and it highlighted the fact that the Storport Driver Version for Windows 2003 was below the supported version of the OpenManage software.

I updated this with a hotfix from Microsoft to 5.2.3790.4173  and things do seem to be working better, the drive has only gone offline once in the last two days and i have been using the drive \ library a lot in that time.

I'm still waiting for delivery of a Dell non-RAID SAS 5/E card to install alongside the PERC 6/E for the library to connect but for now it does seem at least that this works.

[Paul_L · Accepted solution]

I installed a new HBA yesterday alongside our existing one.

Rather than using the  DELL PERC 6/E RAID enabled card we are now using the DELL SAS 5/E HBA with no RAID support and no other devices. Ran a manual job yesterday and that was fine, scheduled job ran fine last night too.

Looks like this can be closed, thanks for your help Gurvinder and Amol ;)

[AmolB]

Cheers!!!