BE 2012 - The request could not be performed because of an I/O device error

Hi,

Have you tried to isolate the SAS HBA to see if there is an issue there or not? You could consider upgrading the firmware on the card and see if this fixes it.

I also see that the tape drive itself doesn't show up in Device Manager. You need to fix this otherwise BE itself won't see the drive.

If it does show up, run tapeinst.exe and install the Symantec drivers for the drive and see if this fixes it.

Thanks!

Hi CraigV,

The tape drive does show up in Device manager. I just uploaded the wrong screenshot. Sorry bout that. I have edited it now.

I will follow your suggestions and come back with the results. However, since this is an on and off problem, the success/failure may not show immediately even if there are 3 daily running jobs on this BE server.

...does that SAS card support tape, or is it a RAID controller?

Thanks!

I believe the DELL 6gbps SAS HBA is for tape. The DELL SAS 6/Ir Integrated Controller is the one used for RAID.

...the library you have is listed as compatible. I'd upgrade the firmware on both to see if this fixes things.

Why did you use the IBM tape diagnostic tool when it is a Dell library? Is it OEM'd?

Thanks!

Hi Craigv,

The device is a Dell-branded IBM device (this got me very confused early on). Hence, I was advised from DELL support forums to use the IBM tape diagnostic tool.

Please let me know if I understood your suggestion correctly:

1) update the driver on the SAS - test the backup

2) if number 1 fails, update the firmware on the SAS - test the backup

3) if number 2 fails, update the driver on the Library - test the backup

4) if number 3 fails, update the firmware on the library, test the backup

#3 - I don't think that there is a library driver that you can update.  The library driver shown in device manager is just the Microsoft Unknown Medium Changer driver, which BE uses in a pass-through mode anyway.

You can update things step by step if you have the time and the patience.  Generally speaking, both BE and Dell tech support probably want you fully patched and updated on everything in your system.

I wouldn't worry about the BE Remote Agents as all indications are that you have a tape hardware or communication issue.  The "refersh handle" error and the other errors in your event log support this.

I assume that you only have 1 tape drive in this library.  If by chance you have 2, then are you seeing issues with both tape drives or only one?

Are you seeing a high soft or hard error count (in the BE tape drive Statistics tab)?

Can you post your adamm.log file?

I suspect that you may need a new tape drive, but I am curious that the IBM diagnostics didn't report any issues.

Hi Larry,

Thanks for your inputs. The problem happened again over the weekend.

Yup, there is only one tape drive. It seems, I was not running the diags tool properly and was actually testing the library, not the drive. The new test I ran with diags does show an initialization failed on the tape drive.

These are the stats for write errors on it:

The ADAMM.log file content for the past few days:

[3792] 03/25/15 08:37:17.664 PnP Device Paths:

Scsi Address
Prt     :Bus     :Tar     :Lun       Device Path
-----------------------------------  ---------------------------------------------------------------

00000003:00000000:00000003:00000000  \\?\scsi#sequential&ven_ibm&prod_ult3580-hh4#5&3433385&1&000300#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}

-----------------------------------  ---------------------------------------------------------------

[3792] 03/25/15 08:37:17.664 Read Device Inquiries - start

[3792] 03/25/15 08:37:17.664 Device: \\.\Tape0
[3792] 03/25/15 08:37:18.101 Device: \\.\MediumChanger1
[3792] 03/25/15 08:37:18.397 Device: \\.\Bkup2Dsk

[3792] 03/25/15 08:37:18.678 Read Device Inquiries - end

[3792] 03/25/15 08:37:18.678 Serialize Devices:

[3792] 03/25/15 08:37:18.974 Device Discovery:

[3792] 03/25/15 08:37:18.974 Device Driver Status:

Scsi Address
Prt :Bus :Tar :Lun   Status
-------------------  -----------------------------------------------

0003:0000:0003:0000  IBM     ULT3580-HH4     BBH5 [IBM     ULT3580-HH4     1K10121873] (White List User Mode Driver)
0003:0000:0003:0001  IBM     3573-TL         B.20 [IBM     3573-TL         00X2U78R5291_LL0] (White List User Mode Driver)
9999:0112:0004:0001  FIXEDB2DDevice          1.00 [FIXEDB2DDevice           {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}]
-------------------  -----------------------------------------------

Scsi Address
Prt :Bus :Tar :Lun   Attributes
-------------------  -------------------------------------------------------------------------------

0003:0000:0003:0000  Device Name             "\\.\Tape0"
                     Secondary Name          "\\?\scsi#sequential&ven_ibm&prod_ult3580-hh4#5&3433385&1&000300#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
                     Primary Inquiry         "IBM     ULT3580-HH4     BBH5"
                     Serial Number           "IBM     ULT3580-HH4     1K10121873"
                     Device Flags            UMD, SCSI, SN(TYPE 0)
                     Device State            3, Online

                     Device IDs              1021, {D762016E-D7B9-4E91-909C-0EB867EC5F7E}
                     Device Name             "Tape drive 0001"
                     Device Type             134283265, "LTO 64K (64K,10,0,E,HU)"
                     Device Features         0x005BFA7F: PB,SFB,SRB,WFM,SFF,SRM,PEOD,EL,LU,E,SBS,DC,SDC,TA,HU,RR,W,EN
                     Device Element          1024, 0
                     Device Block Limits     512 min, 1048576 max
                     Device Hard Errors      2 write, 0 read
                     Device Soft Errors      790 write, 23 read

0003:0000:0003:0001  Device Name             "\\.\MediumChanger1"
                     Primary Inquiry         "IBM     3573-TL         B.20"
                     Serial Number           "IBM     3573-TL         00X2U78R5291_LL0"
                     Device Flags            UMD, SCSI, SN(TYPE 0), SN(ELEMENT)
                     Device State            3, Online

                     Device IDs              1024, {E909BB03-4253-4EFA-938E-F6BA2FD267B9}
                     Device Name             "Robotic library 0001"
                     Device Type             2131755008, "CHANGER FS=1"
                     Device Features         0x00006200: PP,RMP,RRD
                     1st Slot Number         1
                     Number Of Slots         23
                     Portal Slots            1
                     Import/Export           Robotic

                     Drive Element 0         1021, "IBM     ULT3580-HH4     1K10121873"

-------------------  -------------------------------------------------------------------------------

9999:0112:0004:0001  Device Name             ""
                     Secondary Name          "\\?\Volume{ce7b9da0-7453-11e1-8b98-d4ae526e58cc}"
                     Primary Inquiry         "FIXEDB2DDevice          1.00"
                     Serial Number           "FIXEDB2DDevice           {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}"
                     Device Flags            SPO_B2D, LOCAL, SN(TYPE 0)
                     Device State            3, Online

                     Device IDs              1009, {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}
                     Device Name             "Disk storage 0001"
                     Folder Path             "E:\BEData\"

-------------------  -------------------------------------------------------------------------------

[3792] 03/25/15 08:37:19.988 Start Rsm Support:

[3792] 03/25/15 08:37:20.004 Device Discovery End:

[3792] 03/25/15 08:37:20.113 Disk mapping for Win32:

Win32 name               NT internal name
------------------------ -------------------------------------------------------------------------------

C:                      \Device\HarddiskVolume2
D:                      \Device\HarddiskVolume3
E:                      \Device\HarddiskVolume4
F:                      \Device\CdRom0
PhysicalDrive0          \Device\Harddisk0\DR0
Harddisk0Partition1     \Device\HarddiskVolume1
Harddisk0Partition2     \Device\HarddiskVolume2
Harddisk0Partition3     \Device\HarddiskVolume3
Harddisk0Partition4     \Device\HarddiskVolume4

------------------------ -------------------------------------------------------------------------------

[StorageManager::RemoveUnusedLocalDisksFromDB] Results:
1 non-present unconfigured local disks were deleted
1 local disk parent objects with no local disks were deleted

[6360] 03/25/15 08:38:38.804 PvlDrive::UpdateOnlineState()
       Drive = 1025 "Unconfigured disk cartridge 0001"
       ERROR = The device is offline!

[7000] 03/25/15 08:38:53.044 PvlDrive::UpdateOnlineState()
       Drive = 1025 "Unconfigured disk cartridge 0001"
       ERROR = The device is offline!

[3336] 03/26/15 17:11:45.863 The following event was reported on 3/26/2015 5:10:04 PM:

The driver detected a controller error on \Device\RaidPort1.

--------------------

This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:

- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.

If these errors continue to occur, contact the hardware manufacturer.
[3336] 03/26/15 20:57:17.138 The following event was reported on 3/26/2015 8:49:05 PM:

The driver detected a controller error on \Device\RaidPort1.

--------------------

This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:

- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.

If these errors continue to occur, contact the hardware manufacturer.
[3336] 03/28/15 01:31:16.113 The following event was reported on 3/28/2015 1:31:11 AM:

The driver detected a controller error on \Device\RaidPort1.

--------------------

This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:

- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.

If these errors continue to occur, contact the hardware manufacturer.
[4008] 03/29/15 23:23:59.038 Adamm Log Stopped!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~