cancel
Showing results for 
Search instead for 
Did you mean: 

tape drive dropping offline in the middle of a job

Julie_Barnes
Level 4


The issue we’re seeing is as follows: Backup Exec 2010 R3 SP2: We have a Tandberg Storage Library T24 with two HP LTO5 drives in it. It’s attached by fiber to the Q-Logic 8Gb FC Dual-port HBA. We have 2 backup jobs that run nightly. One goes directly to tape and the other goes to disk and then to tape. Tandberg has swapped out both tape drives and the tape library. We’ve tried different fiber cables and now we’ve swapped out the Q-Logic card. One of the tape drives drops offline in the middle of the disk-to-tape portion of the job. I am unable to find WHY it drops anywhere. I’ve verified the SCSI configuration and I do not see any problems. I’m not seeing the tell-tale Event ID 9 or 11 that would suggest a problem.

If I restart BE services, I can rerun the duplicate to tape job fine, but it fails again at it's next scheduled run.

I'm about to pull my hair out! Any help would be greatly appreciated.

26 REPLIES 26

Jaydeep_S
Level 6
Employee Accredited Certified

Look at this time stamp and search the Windows Event viewer. Also are you using a RAID controller to connect a Tape Drive. Also verify the cables for any damage and confirm that they are secure.


[5108] 10/18/12 04:59:21.273 DeviceIo: 05:00:00:00 - Device error 55 on "\\.\Tape2", SCSI cmd 34, 1 total errors
[3560] 10/18/12 04:59:27.756 PvlDrive::DisableAccess() - ReserveDevice failed, offline device
       Drive = 1026 "HP 0001"
       ERROR = 0x0000001F (ERROR_GEN_FAILURE)

[3560] 10/18/12 04:59:27.783 PvlDrive::UpdateOnlineState()
       Drive = 1026 "HP 0001"
       ERROR = The device is offline!

[3560] 10/18/12 04:59:27.783 Begin dump of device's SCSI history

Julie_Barnes
Level 4

The drives are attached via a Q-Logic HBA. The only errors are coming from Backup Exec.

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:33 AM
Event ID:      58053
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
Backup Exec Alert: Device Error
(Server: "SVRTAPE") The drive hardware is offline.

Please confirm that the drive hardware is powered on and properly cabled.

 For more information, click the following link:
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">58053</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:33.000000000Z" />
    <EventRecordID>40777</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Backup Exec Alert: Device Error
(Server: "SVRTAPE") The drive hardware is offline.

Please confirm that the drive hardware is powered on and properly cabled.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:28 AM
Event ID:      34113
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
Backup Exec Alert: Job Failed
(Server: "SVRTAPE") (Job: "Utility Full Backup Policy") Utility Full Backup Policy -- The job failed with the following error: The device timed out.
 

 For more information, click the following link:
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">34113</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:28.000000000Z" />
    <EventRecordID>40776</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Backup Exec Alert: Job Failed
(Server: "SVRTAPE") (Job: "Utility Full Backup Policy") Utility Full Backup Policy -- The job failed with the following error: The device timed out.
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:27 AM
Event ID:      58053
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
Backup Exec Alert: Device Error
(Server: "SVRTAPE") The drive hardware is offline.

Please confirm that the drive hardware is powered on and properly cabled.

 For more information, click the following link:
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">58053</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:27.000000000Z" />
    <EventRecordID>40775</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Backup Exec Alert: Device Error
(Server: "SVRTAPE") The drive hardware is offline.

Please confirm that the drive hardware is powered on and properly cabled.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:21 AM
Event ID:      57665
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
Storage device "HP 0001" reported an error on a request to rewind the media.

Error reported:
A device attached to the system is not functioning.
.

 For more information, click the following link:
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">57665</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:21.000000000Z" />
    <EventRecordID>40774</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>HP 0001</Data>
    <Data>rewind the media</Data>
    <Data>A device attached to the system is not functioning.
</Data>
    <Binary>1F000000F48400E000000000000000009B030000</Binary>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:21 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[0868] 10/18/12 04:59:21 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 1a, new handle 7a0, error 0

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:21.000000000Z" />
    <EventRecordID>40773</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[0868] 10/18/12 04:59:21 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 1a, new handle 7a0, error 0
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:21 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[0868] 10/18/12 04:59:21 Adamm Mover Error: DeviceIo: 05:00:00:00 - Device error 55 on "\\.\Tape2", SCSI cmd 01, 6 total errors

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:21.000000000Z" />
    <EventRecordID>40772</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[0868] 10/18/12 04:59:21 Adamm Mover Error: DeviceIo: 05:00:00:00 - Device error 55 on "\\.\Tape2", SCSI cmd 01, 6 total errors
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:21 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[0868] 10/18/12 04:59:21 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 01, new handle 7a0, error 87

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:21.000000000Z" />
    <EventRecordID>40771</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[0868] 10/18/12 04:59:21 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 01, new handle 7a0, error 87
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      57665
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
Storage device "HP 0001" reported an error on a request to write data to media.

Error reported:
This operation returned because the timeout period expired.
.

 For more information, click the following link:
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">57665</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40770</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>HP 0001</Data>
    <Data>write data to media</Data>
    <Data>This operation returned because the timeout period expired.
</Data>
    <Binary>B4050000F08400E0000001000000000093030000</Binary>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 1a, new handle 950, error 0

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40769</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 1a, new handle 950, error 0
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 00:00:00:00 - Retry Logic: Failed to enable/disable compression on device: HP       Ultrium 5-SCSI  

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40768</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 00:00:00:00 - Retry Logic: Failed to enable/disable compression on device: HP       Ultrium 5-SCSI  
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 1a, new handle 950, error 0

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40767</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 1a, new handle 950, error 0
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 00:00:00:00 - Retry Logic: 2 TURs Failed on device: HP       Ultrium 5-SCSI  

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40766</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 00:00:00:00 - Retry Logic: 2 TURs Failed on device: HP       Ultrium 5-SCSI  
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 00, new handle 950, error 55

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40765</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 00, new handle 950, error 55
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 00, new handle 950, error 0

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40764</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Refresh handle on "\\.\Tape2", SCSI cmd 00, new handle 950, error 0
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:59:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 00:00:00:00 - Retry Logic: Retry logic was engaged on device: HP       Ultrium 5-SCSI  

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:59:05.000000000Z" />
    <EventRecordID>40763</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:59:05 Adamm Mover Error: DeviceIo: 00:00:00:00 - Retry Logic: Retry logic was engaged on device: HP       Ultrium 5-SCSI  
</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Backup Exec
Date:          10/18/2012 4:54:05 AM
Event ID:      33152
Task Category: (1)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SvrTape.mcc.local
Description:
[3176] 10/18/12 04:54:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Device error 1167 on "\\.\Tape2", SCSI cmd 0a, 1 total errors

%2
%3
%4
%5
%6
%7
%8
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Backup Exec" />
    <EventID Qualifiers="8192">33152</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-10-18T09:54:05.000000000Z" />
    <EventRecordID>40762</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SvrTape.mcc.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>[3176] 10/18/12 04:54:05 Adamm Mover Error: DeviceIo: 05:00:00:00 - Device error 1167 on "\\.\Tape2", SCSI cmd 0a, 1 total errors
</Data>
  </EventData>
</Event>

 

 

Larry_Fine
Moderator
Moderator
   VIP   

Still looks like a hardware issue to me.  I would focus on FC errors.  Do you have the QLogic management software installed?  If so, does that show errors?  Might yuou be able to contact QLogic for assistance? 

One thing that is pretty unusual about your environment is the lack of an FC switch.  So possibly your FC HBA is seeing events or glitches from the library end that are normally masked by an FC swtich?

I might also try forcing the HBA to the speed of your tape drives (ie 4GB or 8GB?, rather than the "auto" setting) and try forcing the HBA into "loop" mode.

Julie_Barnes
Level 4

I'd love to have some FC errors to focus on. No related errors in the event log or from the SANSurfer Manager software. I will try to force the speed and mode as you suggest. Thanks!

Julie_Barnes
Level 4

Failed again last night. Errors were a bit different. Sending them to Symantec as well. Attaching here in case anyone sees something and has any thoughts.

ddonley_lvs
Not applicable

We have this nearly identical issue, and we have not found any resolution.  only difference is we are using a single drive HP MSL2024 and a 4 GB HP1142SR HBA.

 

Was this ever resolved for you?

Having the same problem with same HP library but with two LT05 drives. I am planning to replace the SCSI card as soon as I can have the server down for a period to install it. 

HP L&T tools provide no "smoking gun" for this problem. Replaced the drives with no change. Waiting for renewal of our HP contract to call ot HP