Very fail on one drive, successful on another?!

[merkur]

Server details:

Dell PowerEdge 4600

Dual Xeon 2.8GHz

2GB ram

1x Quantum VS160 SCSI

1X HP Ultrium 448 SCSI

*Bios and firmware all current

Problem:

In the past week my nightly backups have been failing on the ultrium 448.  The error log mentions various files being different on verify check.  I went ahead and defragged the files using power defragger/contig to make sure that wasn't a source of the problem.  That didn't help.  I wanted to rule out the hardware raid 5 as a source of trouble so I used an older tape drive that is no longer part of our nightly backups, a quantum vs160.  Surprisingly, the backups complete with all files being verified.

Firmware on all devices is up to date.  Drivers being used are from veritas tapeinst.exe.  In backup exec, all default device settings are used.

I have no idea what is going on.  In 3 years, I have never had a backup job fail because of a verify issue.  Someone out there must have a solution to this weird problem.  Any help would be appreciated.

Message Edited by merkur on 02-20-2009 02:08 PM

[Ken_Putnam]

in BackupExec speak, verify does NOT compare Tape files to Disk files (in a server environment that doesn't make a lot of sense if you think about it)  What verify does is read the tape, compute checksums and compare them to checksums stored on the tape

if you are getting fairly constant verify errors, it is most likely the drive going bad

[merkur]

Thanks for the reply, I was starting to think no one was going to speak up. 

I did not mention in my original post that I have run diagnostic tests using the HP tape driver software and it did not report any issues.

To add more detail to the situation, please bare with me...

I did the defrag on the suspicious files that repeatedly failed the verify and tried doing a test backup with the HP tape drive, and it failed.  If I ran a test backup on other files that did not report previously failing the verify, the backup completes without issue.  So for some reason each time I attempt to backup the files that previously failed, they fail again.  Its a bit confusing, so hopefully you know what I mean.

The pci-x scsi card used to connect the HP tape drive is an LSI 1030 I believe.  It came bundled with the HP tape drive.  I placed an order for a new pci-x scsi card, an ADAPTEC 29160LP to test with.  Hopefully its just the cheap $80 LSI card thats failing and not the tape drive itself.

Let me know what you think, and I will certainly post back with any new information that develops.

Message Edited by merkur on 02-24-2009 08:16 AM

[Ken_Putnam]

OK

are you using AOFO?  if not, i fyou backup a file that is in use, you may get parts before and after an update. This can cause checksum errors

http://seer.entsupport.symantec.com/docs/267554.htm 

is the Symantec troubleshooting guide for verify or "Difference encountered in file data" errors

[merkur]

Thanks for the follow up.

No, to my knowledge AOFO is not enabled/installed on our server.  I have looked through the knowledge base about this and have tried erasing the tapes and redoing the backups without any luck.

At first I was thinking there must be some cache somewhere that needs to be flushed because many of the same files were coming up failed each night, or some files in the same directory as the previous nights failures.

I tried doing a hard reset on the HP tape drive, that didn't help either.

*One important note, on our Dell server, the lcd diagnotice display has shown at times an error E13F4 and E13F5 which Dell lists as a "PCI PARITY ERROR".  If do do a cold boot sometimes the diagnostic goes back to normal display as if things are fine.  All this leads me to believe that the current LSI pci-x scsi cards is the source of this issue because its the only add-in card installed in our server, the rest of our scsi and scsi raid adapters are on-board.

[Ken_Putnam]

All this leads me to believe that the current LSI pci-x scsi cards is the source of this issue

you could be right.  If you read throught the Troubleshooting Guide hardware problems are noted