Tape Drive Going OffiIne

[revenge_of_the_]

Hi Everybody

I have a HP Proliant ML350 G6 with 2003 R2 (64) and a HP DAT160 USB tape drive with HP drivers and Backup Exec 2010R3 with all the updates applied.

After restarting the Backup Exec services I can run one backup but that night the next back up will fail with the tape going offline.  I have checked the event logs and there is no entry for events 5/7/9/11/15.  I have also disabeled the Removable Storage service.

I have also run the HP tape drive check utility (HP L&TT) which updated the firm ware but does not indicate any issues with the hardware.

Any ideas please?

Thanks

[CraigV]

Hi,

Do the following:

1. Delete the tape drive from Backup Exec Devices tab, and from Windows Device Manager.

2. Disconnect the drive, and restart the server. Make sure the tape drive is not listed in WDM.

3. Shut down the server, and reconnect the drive before turning it on. Start up the server.

4. Run tapeinst.exe to install the latest tape drives.

Also make sure that the RSM service is stopped and disabled, along with the HP Insight agents. The ProLiant ML350 G6 would have them installed with the SmartStart CD.

Thanks!

EDIT: You might also want to consider disabling tape drive polling as per the HP TN below:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?spf_p.tpst=kbDocDispl...

[revenge_of_the_]

Hi

Do you know how to stop the HP Insite Agents?

The  HP article you sent is for LTO not DAT.

Thanks

[CraigV]

Yes, and it is also for fibre channel. However, I've had to do something similar with a SCSI tape autoloader as part of troubleshooting.

Go into Services.msc, and anything starting with HP (like HP Storage Agents) can be stopped.

Thanks!

[revenge_of_the_]

Thanks, i will give your suggestions a go.

Weird thing is though, i seem to be able to manually run job after job no problem.  Its just the scheduled ones at nighty that send it off line.

[CraigV]

OK, that's a bit more information to work with. This COULD be corruption of sorts with either the job, or in the BEDB itself. Try 1, or both, of these:

1. Recreate the job and selection list and schedule the job to run. See what happens and report back...

2. Open up BEutility.exe and repair the BEDB. Try the normal scheduled job again and report back.

Thanks!

[Larry_Fine]

check the tail end of the adamm.log for more info about why the drive went offline.  Post the adamm.log file here and we can help you interpret the file.

[revenge_of_the_]

Hi

Adamm.log attached (server names changed to protect the innocent).

Thanks

[Siddhant_Saini]

I did review the adamm.log file and found the following: 

[13532] 11/01/13 01:00:02.599 DeviceIo: 99:00:00:00 - Device error 55 on "\\?\usbstor#sequential&", SCSI cmd 16, 1 total errors

[13532] 11/01/13 01:00:07.599 PvlDrive::DisableAccess() - ReserveDevice failed, offline device

       Drive = 1005 "HP 0001"

       ERROR = 0x000005B4 (ERROR_TIMEOUT)

[13532] 11/01/13 01:00:07.646 PvlDrive::UpdateOnlineState()

       Drive = 1005 "HP 0001"

       ERROR = The device is offline!

The ERROR_TIMEOUT part indicates that this operation returned because the time-out period expired(a strange scenario for a scheduled job..considering that after as service restart is done, manual jobs complete).

Please refer to the following article and look for the Event ID's mentioned in it(not only 5,7,9,11 and 15) http://www.symantec.com/business/support/index?page=content&id=TECH128041 

 

[revenge_of_the_]

Hi

I don't have any of those Event IDs in either System or Application logs.

Thanks

[CraigV]

...I don't think that HP Library and Tape Tools will work with a USB drive, but you can try it and see if it gives you errors. You need to stop the BE services first.

Otherwise, try running a backup using NTbackup. Stop the BE services, and see if you get a similar error/s.

Thanks!

[revenge_of_the_]

HP LT&T worked as it updated the firware.

I can try NTBackup but i think it will work fine as Backup Exec works fine if i restart the services.

Will let you know after I have tested.

Thanks

[revenge_of_the_]

OK, three manual jobs were fine on NTBackup. 

I have set up three more jobs to run at 30 mins intervals this afternoon.

[revenge_of_the_]

NTBackup is fine.

[revenge_of_the_]

Hi Everyone

I have been in touch with HP who have run full diagnostics and think  the drive is fine with  the latest drivers amd firmware. 

I have shown them the below log and want to know what an error 55 is and why the SCSI references when it is USB?

Thanks

[15416] 11/06/13 02:00:00.467 DeviceIo: 99:00:00:00 - Device error 55 on "\\?\usbstor#sequential&", SCSI cmd 16, 1 total errors

[15416] 11/06/13 02:00:05.467 PvlDrive::DisableAccess() - ReserveDevice failed, offline device

       Drive = 1005 "HP 0001"

       ERROR = 0x000005B4 (ERROR_TIMEOUT)

[15416] 11/06/13 02:00:05.499 PvlDrive::UpdateOnlineState()

       Drive = 1005 "HP 0001"

       ERROR = The device is offline!

[15416] 11/06/13 02:00:05.499 Begin dump of device's SCSI history

[15416] 11/06/13 02:00:05.936 End dump of device SCSI history

[revenge_of_the_]

Hi

I have gone through lots of work with HP who say the drive is fine and updated the firmaware and have confirmed i have the latest drivers.

What is a device error55?

And why is the error indicating SCSI when it is USB?

Any more ideas?

Thanks

[Larry_Fine]

What is a device error55?

Error Code 55

System error code 55 means "The specified network resource or device is no longer available." This error code may also display as "ERROR_DEV_NOT_EXIST" or as the value 0x37.

And why is the error indicating SCSI when it is USB?

SCSI cmd 16

16h

RESERVE UNIT

Reserves the unit.

that is a very common command, so for that command to fail generally indicates a general communication failure.

http://www.symantec.com/docs/TECH49432

I suspect that you have some sort of a hardware issue or some sort of issue with the HP software installed on that server that is interferring with device communication.  There have been multiple weird issues seen over the years and many were solved by removing the HP software.  I have no idea why it isn't consistent.  YMMV.

[revenge_of_the_]

Hi

I have tried a couple of test jobs with tracer.exe running and they have all gone through fine.

It always seems to be when it fires off in the nigth it fails.

Would it be possible that the drive is going into some sort of "sleep" or power save?

I have also stopped all the HP services.

HP have also come back saying the ML350G6, windows 2003 (64) and BackupExec 2010 are NOT on their compatibility matrix and I should upgrade to 2012.  I find this somewhat suprising.

Thanks

[CraigV]

...weird, because Symantec don't have a server compatibility list...and I have used Backup Exec 2010 on an HP ProLiant DL385 G2, ProLiant DL165 G5, ProLiant DL385 G5, ProLiant 585 G7 before with no hassles.

HP Have the Enterprise Backup Solutions matrix, but I haven't seen servers on this. Take that information with a pinch-of-salt.

Symantec is concerned with a supported Operating System, tape library/drive/disk target, and sometimes an HBA...not the physical server it runs on.

Thanks!

[revenge_of_the_]

Hi

Attached is what HP sent me.

Any ideas where i go next?

I am thinking of leaving tracer.exe running overnight to try to capture the failure.   I know the log file will get big but will it just be a pain to go through?  it wont eat up masses of space and make my server fall over will it?

Thanks