cancel
Showing results for 
Search instead for 
Did you mean: 

BE 2012 - The request could not be performed because of an I/O device error

pxtian
Level 4

Hi,

I have been experiencing this problem for some months now and it has been happening on and off, but more frequently now. I thought it was just a nuisance that tended to wear out and expire my cleaning tapes until I noticed that it has caused 3 cleaning tapes to be expired within the last 3 months. THis is using a DELL Powervault TL2000. The error happens when BE is backing up the server where it is installed and 2 other remote servers with BE agents.

IOError.JPG

This is one of the persistent alert information errors that I see in BE.

 

BEAlertERror.JPG

 

The errors I see in Event Viewer.

BEEventViewer.JPG

EventViewer1.JPG

EventViewer2.JPG

Eventviewer3.JPG

Eventviewer4.JPG

 

I have read a couple of the articles posted here and what I have done so far are:

1. Cleaned the drive. Used different blank tapes.

2. Powercycled the library

3. Updated the drivers via device manager - rolled them back to original since no improvement happened

4. Updated BE 2012 and checked that the agents in the remote servers are connecting to the BE server

5. Uninstalled the drivers in Device Manager, deleted the devices in BE Storage and rediscovered them using the Configure storage option

6. Ran the IBM Tape diagnostic tool - found no errors.

 

As far as I know, the connection of the drive and library to the BE server is through SAS. Here are the current drivers.

LibraryDriver.JPG

 

TapeDriver.JPG

 

Please, any help will be very much appreciated. This device is already out of warranty from the vendor so I am trying to exhaust all means first before throwing in the towel and resorting to a new device purchase. Thanks in advance.

 

 

9 REPLIES 9

CraigV
Moderator
Moderator
Partner    VIP    Accredited

Hi,

 

Have you tried to isolate the SAS HBA to see if there is an issue there or not? You could consider upgrading the firmware on the card and see if this fixes it.

I also see that the tape drive itself doesn't show up in Device Manager. You need to fix this otherwise BE itself won't see the drive.

If it does show up, run tapeinst.exe and install the Symantec drivers for the drive and see if this fixes it.

Thanks!

pxtian
Level 4

Hi CraigV,

The tape drive does show up in Device manager. I just uploaded the wrong screenshot. Sorry bout that. I have edited it now.

I will follow your suggestions and come back with the results. However, since this is an on and off problem, the success/failure may not show immediately even if there are 3 daily running jobs on this BE server.

CraigV
Moderator
Moderator
Partner    VIP    Accredited

...does that SAS card support tape, or is it a RAID controller?

Thanks!

pxtian
Level 4

I believe the DELL 6gbps SAS HBA is for tape. The DELL SAS 6/Ir Integrated Controller is the one used for RAID.

CraigV
Moderator
Moderator
Partner    VIP    Accredited

...the library you have is listed as compatible. I'd upgrade the firmware on both to see if this fixes things.

Why did you use the IBM tape diagnostic tool when it is a Dell library? Is it OEM'd?

Thanks!

pxtian
Level 4

Hi Craigv,

The device is a Dell-branded IBM device (this got me very confused early on). Hence, I was advised from DELL support forums to use the IBM tape diagnostic tool.

Please let me know if I understood your suggestion correctly:

1) update the driver on the SAS - test the backup

2) if number 1 fails, update the firmware on the SAS - test the backup

3) if number 2 fails, update the driver on the Library - test the backup

4) if number 3 fails, update the firmware on the library, test the backup

 

 

Larry_Fine
Level 6
   VIP   

#3 - I don't think that there is a library driver that you can update.  The library driver shown in device manager is just the Microsoft Unknown Medium Changer driver, which BE uses in a pass-through mode anyway.

You can update things step by step if you have the time and the patience.  Generally speaking, both BE and Dell tech support probably want you fully patched and updated on everything in your system.

I wouldn't worry about the BE Remote Agents as all indications are that you have a tape hardware or communication issue.  The "refersh handle" error and the other errors in your event log support this.

I assume that you only have 1 tape drive in this library.  If by chance you have 2, then are you seeing issues with both tape drives or only one?

Are you seeing a high soft or hard error count (in the BE tape drive Statistics tab)?

Can you post your adamm.log file?

I suspect that you may need a new tape drive, but I am curious that the IBM diagnostics didn't report any issues.

pxtian
Level 4

Hi Larry,

Thanks for your inputs. The problem happened again over the weekend.

Yup, there is only one tape drive. It seems, I was not running the diags tool properly and was actually testing the library, not the drive. The new test I ran with diags does show an initialization failed on the tape drive.

BE Tape Drive Logs.JPG

These are the stats for write errors on it:

BE hard write Errors.JPG

 

The ADAMM.log file content for the past few days:

 

[3792] 03/25/15 08:37:17.664 PnP Device Paths:

Scsi Address
Prt     :Bus     :Tar     :Lun       Device Path
-----------------------------------  ---------------------------------------------------------------

00000003:00000000:00000003:00000000  \\?\scsi#sequential&ven_ibm&prod_ult3580-hh4#5&3433385&1&000300#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}

-----------------------------------  ---------------------------------------------------------------

[3792] 03/25/15 08:37:17.664 Read Device Inquiries - start

[3792] 03/25/15 08:37:17.664 Device: \\.\Tape0
[3792] 03/25/15 08:37:18.101 Device: \\.\MediumChanger1
[3792] 03/25/15 08:37:18.397 Device: \\.\Bkup2Dsk

[3792] 03/25/15 08:37:18.678 Read Device Inquiries - end

[3792] 03/25/15 08:37:18.678 Serialize Devices:

[3792] 03/25/15 08:37:18.974 Device Discovery:

[3792] 03/25/15 08:37:18.974 Device Driver Status:

Scsi Address
Prt :Bus :Tar :Lun   Status
-------------------  -----------------------------------------------

0003:0000:0003:0000  IBM     ULT3580-HH4     BBH5 [IBM     ULT3580-HH4     1K10121873] (White List User Mode Driver)
0003:0000:0003:0001  IBM     3573-TL         B.20 [IBM     3573-TL         00X2U78R5291_LL0] (White List User Mode Driver)
9999:0112:0004:0001  FIXEDB2DDevice          1.00 [FIXEDB2DDevice           {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}]
-------------------  -----------------------------------------------

Scsi Address
Prt :Bus :Tar :Lun   Attributes
-------------------  -------------------------------------------------------------------------------

0003:0000:0003:0000  Device Name             "\\.\Tape0"
                     Secondary Name          "\\?\scsi#sequential&ven_ibm&prod_ult3580-hh4#5&3433385&1&000300#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
                     Primary Inquiry         "IBM     ULT3580-HH4     BBH5"
                     Serial Number           "IBM     ULT3580-HH4     1K10121873"
                     Device Flags            UMD, SCSI, SN(TYPE 0)
                     Device State            3, Online

                     Device IDs              1021, {D762016E-D7B9-4E91-909C-0EB867EC5F7E}
                     Device Name             "Tape drive 0001"
                     Device Type             134283265, "LTO 64K (64K,10,0,E,HU)"
                     Device Features         0x005BFA7F: PB,SFB,SRB,WFM,SFF,SRM,PEOD,EL,LU,E,SBS,DC,SDC,TA,HU,RR,W,EN
                     Device Element          1024, 0
                     Device Block Limits     512 min, 1048576 max
                     Device Hard Errors      2 write, 0 read
                     Device Soft Errors      790 write, 23 read

0003:0000:0003:0001  Device Name             "\\.\MediumChanger1"
                     Primary Inquiry         "IBM     3573-TL         B.20"
                     Serial Number           "IBM     3573-TL         00X2U78R5291_LL0"
                     Device Flags            UMD, SCSI, SN(TYPE 0), SN(ELEMENT)
                     Device State            3, Online

                     Device IDs              1024, {E909BB03-4253-4EFA-938E-F6BA2FD267B9}
                     Device Name             "Robotic library 0001"
                     Device Type             2131755008, "CHANGER FS=1"
                     Device Features         0x00006200: PP,RMP,RRD
                     1st Slot Number         1
                     Number Of Slots         23
                     Portal Slots            1
                     Import/Export           Robotic

                     Drive Element 0         1021, "IBM     ULT3580-HH4     1K10121873"

-------------------  -------------------------------------------------------------------------------

9999:0112:0004:0001  Device Name             ""
                     Secondary Name          "\\?\Volume{ce7b9da0-7453-11e1-8b98-d4ae526e58cc}"
                     Primary Inquiry         "FIXEDB2DDevice          1.00"
                     Serial Number           "FIXEDB2DDevice           {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}"
                     Device Flags            SPO_B2D, LOCAL, SN(TYPE 0)
                     Device State            3, Online

                     Device IDs              1009, {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}
                     Device Name             "Disk storage 0001"
                     Folder Path             "E:\BEData\"

-------------------  -------------------------------------------------------------------------------

[3792] 03/25/15 08:37:19.988 Start Rsm Support:

[3792] 03/25/15 08:37:20.004 Device Discovery End:

[3792] 03/25/15 08:37:20.113 Disk mapping for Win32:

Win32 name               NT internal name
------------------------ -------------------------------------------------------------------------------

C:                      \Device\HarddiskVolume2
D:                      \Device\HarddiskVolume3
E:                      \Device\HarddiskVolume4
F:                      \Device\CdRom0
PhysicalDrive0          \Device\Harddisk0\DR0
Harddisk0Partition1     \Device\HarddiskVolume1
Harddisk0Partition2     \Device\HarddiskVolume2
Harddisk0Partition3     \Device\HarddiskVolume3
Harddisk0Partition4     \Device\HarddiskVolume4

------------------------ -------------------------------------------------------------------------------

[StorageManager::RemoveUnusedLocalDisksFromDB] Results:
1 non-present unconfigured local disks were deleted
1 local disk parent objects with no local disks were deleted

[6360] 03/25/15 08:38:38.804 PvlDrive::UpdateOnlineState()
       Drive = 1025 "Unconfigured disk cartridge 0001"
       ERROR = The device is offline!

[7000] 03/25/15 08:38:53.044 PvlDrive::UpdateOnlineState()
       Drive = 1025 "Unconfigured disk cartridge 0001"
       ERROR = The device is offline!

[3336] 03/26/15 17:11:45.863 The following event was reported on 3/26/2015 5:10:04 PM:

 

The driver detected a controller error on \Device\RaidPort1.

 

 

--------------------

 

This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:

- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.

If these errors continue to occur, contact the hardware manufacturer.
[3336] 03/26/15 20:57:17.138 The following event was reported on 3/26/2015 8:49:05 PM:

 

The driver detected a controller error on \Device\RaidPort1.

 

 

--------------------

 

This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:

- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.

If these errors continue to occur, contact the hardware manufacturer.
[3336] 03/28/15 01:31:16.113 The following event was reported on 3/28/2015 1:31:11 AM:

 

The driver detected a controller error on \Device\RaidPort1.

 

 

--------------------

 

This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:

- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.

If these errors continue to occur, contact the hardware manufacturer.
[4008] 03/29/15 23:23:59.038 Adamm Log Stopped!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

 

pkh
Moderator
Moderator
   VIP    Certified
The error seems to indicate that your tape library HBA support RAID. Check that your tape library HBA is one of those listed in the document below. http://www.symantec.com/docs/TECH70907 If not, check with the manufacturer to confirm that it is not supporting RAID.