Troubleshooting Error (E00084C8)

[Andrew_Cameron]

Hi Everyone,

  Hoping someone may have run into this problem and knows of a solution.  Whenever I do a full backup to tape of my Hypver-V host (parent partition) and all the guests the backup will fail with the error E00084C8 stating the information below:

Replaced server names with generic name, see full configuration listing below error.

Error Information

Verify- VRTSRV::\\Host\Hyper-V?Virtual?Machine\Guest_1
Storage device "QUANTUM 0001" reported an error on a request to read data from media.

Error reported: The request could not be performed because of an I/O device error. V-79-57344-33992 - The backup storage device has failed.

Verify
A selection on device VRTSRV::\\Host\Hyper-V?Virtual?Machine\Guest_2 was skipped because of previous errors with the job.

A selection on device C: was skipped because of previous errors with the job.

A selection on device VRTSRV::\\Host\Hyper-V?Virtual?Machine\Guest_3 was skipped because of previous errors with the job.

A selection on device Host\BKUPEXEC was skipped because of previous errors with the job.

A selection on device VRTSRV::\\Host\Hyper-V?Store\Initial Store was skipped because of previous errors with the job.

A selection on device Shadow?Copy?Components was skipped because of previous errors with the job.

A selection on device System?State was skipped because of previous errors with the job.

Configuration

Host (Parent partition) Software

Windows Server 2008 R2 SP1

Symantec Backup Exec 2010 R3 (Version 13.0 Rev 52.04 (64-bit))

Hypver-V

Host (Parent partition) Hardware

Dell PowerEdge 2950

2x Xeon E5420

32GB Memory

2x 700GB Raid SAS (mirrored) connected via Dell PERC 6/i

Dell PowerVault 114T (with a single LTO 4 drive) connected via Dell SAS 5/E card (LSI manufacturer)

Dual Network (one for Hyper-V dedicated and one for parent OS)

Guest_1 (child partition) Software

Windows Server 2003 (legacy print server, disk to VHD used to created computer from physical)

Printer server

BE Remote agent

Trend Micro Security Agent

Guest_1 (child partition) Hardware

Setup through Hyper-V manager

80GB fixed drive

2048 MB RAM minimum memory with dynamic up to 4096

Guest_2 (child partition) Software

Windows Server 2008 R2 SP1

Trend Micro Worry Free Business Security Host

Trend Micro Security Agent

BES

BE Remote Agent

Guest_2 (child partition) Hardware

Setup through Hyper-V manager

40GB fixed drive

4096 MB RAM minimum memory with dynamic up to 6024

Guest_3 (child partition) Software

Windows SBS 2008 R2 (Domain, Exchange, OWA, File Server) (created with Disk 2 VHD and mounted the VHD on the computer to change the disk # as it is the same physical disk that is currently in use)

Trend Micro Security Agent

Guest_3 (child partition) Hardware

Setup through Hyper-V manager

700GB fixed drive (not preallocated)

13096 RAM fixed allocated

Troubleshooting Steps

1) Re-initiated all VSS and verified.  No failed status

2) Tried alternative tapes, cables, card. (same problem)

3) Adjusted Tape Drive Configuration (within Backup Exec - block size, etc) (same problem)

4) Tried smaller backups of two of the three guest machines.  When backing up two of the three (any two) it will sometimes work and sometimes fail.  I could not find a common thread as to why one would succeed and the other would fail.  Running the same job (twice) of the same two guests at least one of the two always fails.

5) High number of soft write errors per job.  I found an article through Symantec support saying this can be ignored if verified backup is successful as it is a known issue with the Dell SAS 5/E controller.

6) Uninstalled Backup Exec and reinstalled.  Have used two different licenses SBS and 2010 Full, Currently using the full license.  As a charity we have all agents available with the license I use so I do not suspect a license issue.

7) Have removed and updated the remote agents on all guests.  The version match what is being used on the host.

8) The data backed up indicates the full amount of information has been stored on the tapes however no restores work.

9) Backing up to disk works about 90% of the time through a USB connected SATA drive.  The tape drive is my preferred backup method but have been using the SATA drive to keep valid backups.  On the backup to disk I often see a error logged in windows but the backup still proceed successfully. (driver detected a controller error on \Device\Harddisk1\DR1).  Running checkdisk on the drive yields no errors.

10) Contacted Dell support about the tape drive and all testing shows no issues reading or writing to the tape drive.

11) No errors logged in the system logs aside from the USB error in #9.

Those are the trouble shooting steps I have written down although have spent many hours searching for a probable cause so may be additions things that have been attempted. 

I welcome any feedback and help anyone may be able to provide,

Andrew

[pkh]

Check that your library is listed in the Windows Device Manager as an Unknown Medium Changer with a Microsoft driver.  If it is not, change the driver using the procedure below.

How to uninstall the Original Equipment Manufacturer driver for the medium changer in a robotic libr...

[Andrew_Cameron]

Thanks pkh, the setup does not have a robotic library in it.  There is only one single backup drive and tapes are changed manually daily.

[Andrew_Cameron]

Anyone have any further suggestions? 

Updates on issues.

I have resovled the issue on all machines except the main SBS system, which really is the most important. 

I have switched the backup from Hyper-v guest to remote computer to do component testing of the backup to try to determine if there is any components of the backup (system state, storage group, physical drive) which are specifically causing an issue.  As this isn't available through Hyper-V I am doing it via remote computer.

I will post an update after the component testing has concluded.

[Andrew_Cameron]

I think I have an idea of what is occuring now.

disk to VHD was used in moving the SBS 2008 from physical to virtual. The SBS 2008 machine was added to the same computer it was previously running on which would have caused a disk id issue so I changed the disk id by adding the VHD to the 2008 R2 Host OS which would see the conflict and issue it a new disk id.  This works correctly, all that was need was a quick boot repair via the SBS 2008 disk and the server was back up and running.

It appear however whenever attempting to back up the C drive of the virtual SBS 2008 machine in backup exec it always comes back with:

Verify- \\SBS_Machine\C:

Storage device "Quantum 0001" reported an error on a request to read data from media.

Error reported:

The request could not be performed because of an I/O device error.

V-7-57344-33992 - the backup storage device has failed.

Error Code: E00084C8
 

Anyone run into this before and have an idea of how to solve it?

I am planning to move to SBS 2011 in the near future as I have a license for it anyways and I think creation of a new VHD and using of the new system will remove this issue but would like to get this operational in the mean time.  I am using windows backup without issue