03-25-2015 01:36 AM
Hi,
I have been experiencing this problem for some months now and it has been happening on and off, but more frequently now. I thought it was just a nuisance that tended to wear out and expire my cleaning tapes until I noticed that it has caused 3 cleaning tapes to be expired within the last 3 months. THis is using a DELL Powervault TL2000. The error happens when BE is backing up the server where it is installed and 2 other remote servers with BE agents.
This is one of the persistent alert information errors that I see in BE.
The errors I see in Event Viewer.
I have read a couple of the articles posted here and what I have done so far are:
1. Cleaned the drive. Used different blank tapes.
2. Powercycled the library
3. Updated the drivers via device manager - rolled them back to original since no improvement happened
4. Updated BE 2012 and checked that the agents in the remote servers are connecting to the BE server
5. Uninstalled the drivers in Device Manager, deleted the devices in BE Storage and rediscovered them using the Configure storage option
6. Ran the IBM Tape diagnostic tool - found no errors.
As far as I know, the connection of the drive and library to the BE server is through SAS. Here are the current drivers.
Please, any help will be very much appreciated. This device is already out of warranty from the vendor so I am trying to exhaust all means first before throwing in the towel and resorting to a new device purchase. Thanks in advance.
03-25-2015 02:08 AM
Hi,
Have you tried to isolate the SAS HBA to see if there is an issue there or not? You could consider upgrading the firmware on the card and see if this fixes it.
I also see that the tape drive itself doesn't show up in Device Manager. You need to fix this otherwise BE itself won't see the drive.
If it does show up, run tapeinst.exe and install the Symantec drivers for the drive and see if this fixes it.
Thanks!
03-25-2015 02:17 AM
Hi CraigV,
The tape drive does show up in Device manager. I just uploaded the wrong screenshot. Sorry bout that. I have edited it now.
I will follow your suggestions and come back with the results. However, since this is an on and off problem, the success/failure may not show immediately even if there are 3 daily running jobs on this BE server.
03-25-2015 02:32 AM
...does that SAS card support tape, or is it a RAID controller?
Thanks!
03-25-2015 04:02 AM
I believe the DELL 6gbps SAS HBA is for tape. The DELL SAS 6/Ir Integrated Controller is the one used for RAID.
03-25-2015 04:08 AM
...the library you have is listed as compatible. I'd upgrade the firmware on both to see if this fixes things.
Why did you use the IBM tape diagnostic tool when it is a Dell library? Is it OEM'd?
Thanks!
03-25-2015 07:51 AM
Hi Craigv,
The device is a Dell-branded IBM device (this got me very confused early on). Hence, I was advised from DELL support forums to use the IBM tape diagnostic tool.
Please let me know if I understood your suggestion correctly:
1) update the driver on the SAS - test the backup
2) if number 1 fails, update the firmware on the SAS - test the backup
3) if number 2 fails, update the driver on the Library - test the backup
4) if number 3 fails, update the firmware on the library, test the backup
03-25-2015 11:01 AM
#3 - I don't think that there is a library driver that you can update. The library driver shown in device manager is just the Microsoft Unknown Medium Changer driver, which BE uses in a pass-through mode anyway.
You can update things step by step if you have the time and the patience. Generally speaking, both BE and Dell tech support probably want you fully patched and updated on everything in your system.
I wouldn't worry about the BE Remote Agents as all indications are that you have a tape hardware or communication issue. The "refersh handle" error and the other errors in your event log support this.
I assume that you only have 1 tape drive in this library. If by chance you have 2, then are you seeing issues with both tape drives or only one?
Are you seeing a high soft or hard error count (in the BE tape drive Statistics tab)?
Can you post your adamm.log file?
I suspect that you may need a new tape drive, but I am curious that the IBM diagnostics didn't report any issues.
03-29-2015 09:49 PM
Hi Larry,
Thanks for your inputs. The problem happened again over the weekend.
Yup, there is only one tape drive. It seems, I was not running the diags tool properly and was actually testing the library, not the drive. The new test I ran with diags does show an initialization failed on the tape drive.
These are the stats for write errors on it:
The ADAMM.log file content for the past few days:
[3792] 03/25/15 08:37:17.664 PnP Device Paths:
Scsi Address
Prt :Bus :Tar :Lun Device Path
----------------------------------- ---------------------------------------------------------------
00000003:00000000:00000003:00000000 \\?\scsi#sequential&ven_ibm&prod_ult3580-hh4#5&3433385&1&000300#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}
----------------------------------- ---------------------------------------------------------------
[3792] 03/25/15 08:37:17.664 Read Device Inquiries - start
[3792] 03/25/15 08:37:17.664 Device: \\.\Tape0
[3792] 03/25/15 08:37:18.101 Device: \\.\MediumChanger1
[3792] 03/25/15 08:37:18.397 Device: \\.\Bkup2Dsk
[3792] 03/25/15 08:37:18.678 Read Device Inquiries - end
[3792] 03/25/15 08:37:18.678 Serialize Devices:
[3792] 03/25/15 08:37:18.974 Device Discovery:
[3792] 03/25/15 08:37:18.974 Device Driver Status:
Scsi Address
Prt :Bus :Tar :Lun Status
------------------- -----------------------------------------------
0003:0000:0003:0000 IBM ULT3580-HH4 BBH5 [IBM ULT3580-HH4 1K10121873] (White List User Mode Driver)
0003:0000:0003:0001 IBM 3573-TL B.20 [IBM 3573-TL 00X2U78R5291_LL0] (White List User Mode Driver)
9999:0112:0004:0001 FIXEDB2DDevice 1.00 [FIXEDB2DDevice {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}]
------------------- -----------------------------------------------
Scsi Address
Prt :Bus :Tar :Lun Attributes
------------------- -------------------------------------------------------------------------------
0003:0000:0003:0000 Device Name "\\.\Tape0"
Secondary Name "\\?\scsi#sequential&ven_ibm&prod_ult3580-hh4#5&3433385&1&000300#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
Primary Inquiry "IBM ULT3580-HH4 BBH5"
Serial Number "IBM ULT3580-HH4 1K10121873"
Device Flags UMD, SCSI, SN(TYPE 0)
Device State 3, Online
Device IDs 1021, {D762016E-D7B9-4E91-909C-0EB867EC5F7E}
Device Name "Tape drive 0001"
Device Type 134283265, "LTO 64K (64K,10,0,E,HU)"
Device Features 0x005BFA7F: PB,SFB,SRB,WFM,SFF,SRM,PEOD,EL,LU,E,SBS,DC,SDC,TA,HU,RR,W,EN
Device Element 1024, 0
Device Block Limits 512 min, 1048576 max
Device Hard Errors 2 write, 0 read
Device Soft Errors 790 write, 23 read
0003:0000:0003:0001 Device Name "\\.\MediumChanger1"
Primary Inquiry "IBM 3573-TL B.20"
Serial Number "IBM 3573-TL 00X2U78R5291_LL0"
Device Flags UMD, SCSI, SN(TYPE 0), SN(ELEMENT)
Device State 3, Online
Device IDs 1024, {E909BB03-4253-4EFA-938E-F6BA2FD267B9}
Device Name "Robotic library 0001"
Device Type 2131755008, "CHANGER FS=1"
Device Features 0x00006200: PP,RMP,RRD
1st Slot Number 1
Number Of Slots 23
Portal Slots 1
Import/Export Robotic
Drive Element 0 1021, "IBM ULT3580-HH4 1K10121873"
------------------- -------------------------------------------------------------------------------
9999:0112:0004:0001 Device Name ""
Secondary Name "\\?\Volume{ce7b9da0-7453-11e1-8b98-d4ae526e58cc}"
Primary Inquiry "FIXEDB2DDevice 1.00"
Serial Number "FIXEDB2DDevice {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}"
Device Flags SPO_B2D, LOCAL, SN(TYPE 0)
Device State 3, Online
Device IDs 1009, {8AC8D2F1-E121-44D0-89D8-CE670AFEAE66}
Device Name "Disk storage 0001"
Folder Path "E:\BEData\"
------------------- -------------------------------------------------------------------------------
[3792] 03/25/15 08:37:19.988 Start Rsm Support:
[3792] 03/25/15 08:37:20.004 Device Discovery End:
[3792] 03/25/15 08:37:20.113 Disk mapping for Win32:
Win32 name NT internal name
------------------------ -------------------------------------------------------------------------------
C: \Device\HarddiskVolume2
D: \Device\HarddiskVolume3
E: \Device\HarddiskVolume4
F: \Device\CdRom0
PhysicalDrive0 \Device\Harddisk0\DR0
Harddisk0Partition1 \Device\HarddiskVolume1
Harddisk0Partition2 \Device\HarddiskVolume2
Harddisk0Partition3 \Device\HarddiskVolume3
Harddisk0Partition4 \Device\HarddiskVolume4
------------------------ -------------------------------------------------------------------------------
[StorageManager::RemoveUnusedLocalDisksFromDB] Results:
1 non-present unconfigured local disks were deleted
1 local disk parent objects with no local disks were deleted
[6360] 03/25/15 08:38:38.804 PvlDrive::UpdateOnlineState()
Drive = 1025 "Unconfigured disk cartridge 0001"
ERROR = The device is offline!
[7000] 03/25/15 08:38:53.044 PvlDrive::UpdateOnlineState()
Drive = 1025 "Unconfigured disk cartridge 0001"
ERROR = The device is offline!
[3336] 03/26/15 17:11:45.863 The following event was reported on 3/26/2015 5:10:04 PM:
The driver detected a controller error on \Device\RaidPort1.
--------------------
This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:
- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.
If these errors continue to occur, contact the hardware manufacturer.
[3336] 03/26/15 20:57:17.138 The following event was reported on 3/26/2015 8:49:05 PM:
The driver detected a controller error on \Device\RaidPort1.
--------------------
This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:
- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.
If these errors continue to occur, contact the hardware manufacturer.
[3336] 03/28/15 01:31:16.113 The following event was reported on 3/28/2015 1:31:11 AM:
The driver detected a controller error on \Device\RaidPort1.
--------------------
This error signifies controller errors. These errors are generally caused by hardware that is not functioning correctly. To resolve these errors, try the following actions:
- Slow the SCSI bus.
- Verify that the latest SCSI drivers and firmware are loaded.
If these errors continue to occur, contact the hardware manufacturer.
[4008] 03/29/15 23:23:59.038 Adamm Log Stopped!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
03-29-2015 10:59 PM