CRC Errors - but changed Everything!

[Blane2000]

Hi,
After several months working alongside Dell to resolve the CRC issues, and changing every piece of hardware, I'm now thinking maybe the problem is software related!?!?

Backup Exec 9.1 - All updates (inc. SP3)
Windows 2003 Server (Was a Dell 2550 - now a HP DL380)
2 x SCSI Cards - 39160 and 29160
1 x Dell PV-132T SDLT Library (2 SDLT drives)
1 x Overland NEO SDLT Library (2 SDLT drives)

OK...

General CRC errors started around 4 months ago, where backups would fail and mark the device as offline.
I called Dell (as recommended in Veritas support documents) and they agreed to replace the drive.
We have tried changing media, replacing media and disposing problem tapes (losing our critical data).

To cut a very long story short, 4 months later and the server has been replaced, the SCSI controller card replaced, several tape drives and the SCSI cable have all been replaced. Firmware updates done by Dell on the library, drives and SCSI card (apparently Dell 39160 card requires a specific firmware version..). BIOS and even the system backplane and SCSI controllers have all been updated.

About 4 weeks ago Dell agreed to completely replace our WHOLE unit... Library chassis, drives, cable, controller card etc etc - brand new from factory - as long as we agreed to discard ALL our existing tapes and start over again (incase a damaged tape had damaged the head of the drive which in turn damaged another tape creating a fault cycle..) - Which we did.
A week later backups are back failing with CRC errors - and are worse now then before! :(
Backups are successful maybe twice a week!!!

The only common defactor here being the software.

The event ID's are: (in order)
33152 - Write Failure - Error=Error_CRC
57665 - Dell 3 Reported an error to write data to media - Data Error (crc check)
33152 - Rewind Failure (after device is offline)
58053 - Robotic Library Error
The succession between these error is about 3 minutes from the write failure to the device failing.

At first I thought medai - but after purchasing tapes from Fuji, Quantum and Dell - all giving same errors it cant be.

Also, and heres the twist; JOB1 starts at 8pm - writes 500mb to TAPE4 in DRIVE 2 and fails with CRC error.
I then choose to re-run job, which it does using the same drive, same tape and same backup and runs fine!!
Also tapes I thought were faulty have been used again without a problem in the drive that indicated an error.

Also, the NEO library uses a completley seperate SCSI card and media tapes (not used between devices) - and that gets CRC errors too, exactly the same symptoms.

Any help \ ideas - or a solution!!?

[Joshua_Small]

When you change so much, it must be very difficult to diagnose further.
My suggestions likely won't help much, but:

- Does any other backup software, such as ntbackup, work with the drive? If so, see what happens there
- Disable antivirus whilst running
- Have you applied all WIndows updates from MS?

[Blane2000]

OK - Heres where I'm at...

I built a completley new server to isolate the Dell PV-132T drive. The Dell and Neo drives are now attached to seperate servers.
The server is a Dell PE2400, new 39160 SCSI card, Backup Exec 9.1 + SP3

Recreated the backup jobs from scratch on the new server; run them as planned.

Backups Failed - CRC Errors!!
Here are some of the event log errors:
Adamm Mover Error: Write Retry!
Error = ERROR_IO_DEVICE
Drive = "DELL 2"
{2FB221A8-A917-465D-9716-CEE8E104881B}
Media = "Tuesday Wk 1"
{5EFAC3D6-355A-4FE5-A141-D4BBAEABAB65}
Read Mode: SingleBlock(0), ScsiPass(0)
Write Mode: SingleBlock(1), ScsiPass(1)


Backup Exec Alert: Job Failed
(Server: "BACKUPSERV1") (Job: "JOB 1") DB Server Backups -- The job failed with the following error: Data error (cyclic redundancy check).


Backup Exec Alert: Device Error
(Server: "BACKUPSERV1") Robotic library hardware error.

The robotic library has reported a general hardware error condition. Manual intervention, calibration, etc., may be required. This may be a device related hardware problem, or it may be some other hardware problem (i.e. SCSI card/bus etc.). The library and drive states have been set to offline. Please attend to this condition.

[Ajit_Kulkarni]

Hello,

Please refer to the following technote:

"Data error (cyclic redundancy check)" is reported in the Backup Exec job log during a failed backup/verify operation
http://support.veritas.com/docs/192216

Hope this helps you.

Regards

NOTE : If we do not receive your reply within two business days, this post would be marked "assumed answered" and would be moved to "answered questions" pool.

[Blane2000]

OK - I have an update, and what a surprise!!

Having tried everything my poor mind could stretch too (of course I read the above TID months ago when the problems started...) I decided to completely remove ALL the Backup Exec drivers.

With the 'Veritas' drivers removed, I searched for manufacturer drivers for the library and tape drives...(Dell and Quantum) and installed them through Device Manager.

I then run the Backup Exec Device Configuration Wizard, but choose to use 'plug & play' drivers and NOT the Veritas ones...

Apart from a few gliches due to old tapes etc, the backup have been running each and every night!!
Strangely enough, the seem faster and are 100% more reliable!

Don't know why the Veritas drivers caused me, Dell Tech support, Dell's Superior support, the visiting engineers and the forums I've posted on so much headache - but I'm not reinstalling them!!

Hope this helps anyone with the same sort of problem who also found document http://seer.support.veritas.com/docs/192216.htm useless!!

Thanks for the input guys.