Random Job Failing - device error timed out

[TimPoulter]

Hi,

Im trying to troubleshoot a random issue that is happening. The difficulty is that the issue is not consistent. 

We have 3 jobs setup to backup 3 server, and it will fail with the following error.  Now this can happen randomly with either Job1,Job2,Job3

Or Job1 will succed, Job2 will fail, Job3 will succed, or another night it will be Job1 Fail, Job2 Success,Job3 Success

Basiscally the jobs fail with the following error. 

Backup-"JOB NAME"An unknown error occurred on device "TAPE DEVICE".
V-79-57344-34029 - A hardware error occurred.

An unknown error occurred on device "TAPE DEVICE".
V-79-57344-34030 - An invalid command was sent to the storage device.

The storage device may no longer be accessible to the operating system. Review the event logs for more information.

An unknown error occurred on device "TAPE DEVICE".
V-79-57344-34030 - An invalid command was sent to the storage device.

The storage device may no longer be accessible to the operating system. Review the event logs for more information.

Now this has been happending few days.  Some of the things I have tried so far are

1) Ran the HP LLT Tools - passed

2) Tried a new job to test, will fail randomly if either started manually or scheduled

3) The jobs that we already have defined, will randomly fail, either started manually or scheduled

4) Happens to different physical Tapes

5) When I try to go to the links for the V- errors the pages come up cannot find page

6) Windows Event Logs do not show anything in the time period of the jobs

Specs are as follow

Backup Exec 2014 Small Business Edition with Service Pack 2

Windows Server 2008 R2 with all updates

HP Proliant ML350 Gen 9 with HP H240 HBA card on a HP Ultrium 1760 LTO-4 Tape Drive with Vsphere ESXI 5.5 with update 2 and latest patches.

Now this was working prior to the hardware upgrade above this tape drive was running similar setup of

Backup Exec 2014 Small Business Edition with Service Pack 2

Windows Server 2008 R2 with all updates

HP ProLiant DL380 G6 with HP P212 HBA card on a HP Ultrium 1760 LTO-4 Tape Drive with Vsphere ESXI 5.5 with update 2 and latest patches.

[CraigV]

...you have a virtualised media server...strictly this is not supported by Symantec, and if I remember correctly, VMware doesn't support passthrough on ESXi 5.5 anymore.

Best practice is to run BE on a physical server in order to get around any hardware issues you will have.

Thanks!

EDIT: Please see below:

http://www.symantec.com/business/support/index?page=content&id=TECH130609

[TimPoulter]

I know that its "strictly this is not supported by Symantec", which I have thoughts about that,which is  not really for this topic....

But im just wondering how to try to isloate this becasue the errors that it provides are not helpful.

[CraigV]

Tim:

Have you researched whether or not ESXi 5.5 supports passthrough for SAS? What I've seen shows that this might not be the case...

http://h30499.www3.hp.com/t5/Tape-Libraries-and-Drives/Standalone-LTO5-SAS-Tape-Drive-in-VMWare-ESX-...

https://communities.vmware.com/message/2337023

Thanks!

[TimPoulter]

Hi Sorry for the delay,

Yes I looked into the passthrough option and its seems that for 5.5 its does not work at all for hba.

Just as a further update, it now seems more stable. Where as before it would randomly fail any of the 3 jobs it now only seems to fail on a specific job.

And it seems to do it consitantly after abut 4-6 minutes one the job has started.

But the odd thing is when I manually run the job. It completes successfully

I have re-created the job. Re-installed the Agent on that server and it still does the same thing

[CraigV]

...the fastest way to get to a supported configuration Tim is to invest in a physical backup server. From this perspective you're also going to get support from Symantec.

Your configuration is considered an alternative configuration and they don't have to support it at all.

Thanks!

[TimPoulter · Accepted solution]

Well it started to work fine out of the blue now. The past week I had been manually running the job and it was fine, and the last two days it worked fine without me having to run the job manually.