Problem with corrupt/incomplete data

[JimmyU]

Background and previous errors/problems
A customer has a Windows Server 2003 Standard Edition SP2 32bit which is the backup server and has been working normally for 1.5 year. A couple of weeks ago the backup started to warn about problems with the tapes(LTO4) which were getting soft read/write errors so we exchanged them for new ones and ran cleaning on the drive.Labeling the tapes and inventorying them worked just fine but the first backup wouldn't go through properly and we did a lot of tests which resulted in the tape drive being replaced due to hardware error.

Current error/problem
With the new backup drive we could run backups but started getting the errors  "Backup- \\Servername\System?StateV-79-57344-65247 - A failure occurred reading an object." and the error code E000FE36. This is for the backup server itself, the other servers are backed up without errors.
The backup server isn't running any anitvirus software and there has been no big changes to the server other than the change of tapes and tape drive.

NTBACKUP results

I've run NTBACKUP a number of times on the server system state which have indicated problems with the following files:

Backup started on 2012-11-30 at 16:25.

WARNING: Portions of "\Boot Files\C:\WINDOWS\system32\rsmmllsv.exe" cannot be read.  The backed up data is corrupt or incomplete.

WARNING: Portions of "\Boot Files\C:\WINDOWS\system32\rtrfiltr.dll" cannot be read.  The backed up data is corrupt or incomplete.

Backup completed on 2012-11-30 at 16:26.

Backup started on 2012-12-04 at 09:21.

WARNING: Portions of "\Boot Files\C:\WINDOWS\SYSTEM32\CATROOT\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\ins.cat" cannot be read.  The backed up data is corrupt or incomplete.

Backup completed on 2012-12-04 at 09:22.

Backup started on 2012-12-05 at 13:09.

No errors/warnings

Backup completed on 2012-12-05 at 13:09.

Backup started on 2012-12-10 at 17:21.

WARNING: Portions of "\Boot Files\C:\WINDOWS\system32\perfc009.dat" cannot be read.  The backed up data is corrupt or incomplete.

WARNING: Portions of "\Boot Files\C:\WINDOWS\system32\rsmmllsv.exe" cannot be read.  The backed up data is corrupt or incomplete.

WARNING: Portions of "\Boot Files\C:\WINDOWS\system32\rtrfiltr.dll" cannot be read.  The backed up data is corrupt or incomplete.

WARNING: Portions of "\Boot Files\C:\WINDOWS\SYSTEM32\drivers\sacdrv.sys" cannot be read.  The backed up data is corrupt or incomplete.

Backup completed on 2012-12-10 at 17:22.

Backup started on 2012-12-13 at 11:24.

WARNING: Portions of "\Boot Files\C:\WINDOWS\system32\rsmmllsv.exe" cannot be read.  The backed up data is corrupt or incomplete.

WARNING: Portions of "\Boot Files\C:\WINDOWS\system32\rtrfiltr.dll" cannot be read.  The backed up data is corrupt or incomplete.

WARNING: Portions of "\Boot Files\C:\WINDOWS\SYSTEM32\drivers\videoprt.sys" cannot be read.  The backed up data is corrupt or incomplete.

Backup completed on 2012-12-13 at 11:25.

Updates
I've tried installing Windows Server 2003 SP2 again, running LiveUpdate for Backup Exec and reinstalling the device driver pack without any change.

CheckDisk
I've run CHKDSK on the C drive twice with the following result

First time

Cleaning up instance tags for file 0xaa91.

Cleaning up minor inconsistencies on the drive.

Cleaning up 1523 unused index entries from index $SII of file 0x9.

Cleaning up 1523 unused index entries from index $SDH of file 0x9.

Cleaning up 1523 unused security descriptors.

The upcase file content is incorrect.

Correcting errors in the uppercase file.

Windows has made corrections to the file system.

Second time

Cleaning up minor inconsistencies on the drive.

Cleaning up 2 unused index entries from index $SII of file 0x9.

Cleaning up 2 unused index entries from index $SDH of file 0x9.

Cleaning up 2 unused security descriptors.

Does anyone have tips/solutions for this?

[Jaydeep_S]

Looks like this is a result of some software or drivers not being cleanly uninstalled o upgraded or removed. See if you can identify any of these possibilities. Also after performing a CHKDSK did you attempt a backup with NTbackup. If so what were the results. Are the files in the warnings same as befre or have changed.

rsmmllsv.exe seems to be the removable storage application file. Make sure that if you are using BE, Removable Storage Services is Stopped and disabled.

Additionally, try to perform a complete scan of the system using an antivirus solution.

[CraigV]

...have you tried to run a repair of Windows 2003 itself?

[JimmyU]

After CHKDSK was run the first time(2012-12-05 12.37) NTBACKUP of system state ran without errors but the backup job run by Backup Exec that night still reported errors backing up system state as well as some other files.

I've attached an example from the latest backup of the errors/exceptions that occur as well as the log files from when I've run NTBACKUP. Some of the files in the logs are often the same but many change from backup to backup.
C:\WINDOWS\system32\rsmmllsv.exe and C:\WINDOWS\system32\rtrfiltr.dll seem to be the most requent files that show up with problems in the system state backup.

The "Removable Storage" service is stopped and I've set it to disabled now but it was stopped and set to manual startup so we'll see if that makes a difference.

I have tried running a full system scan with Trend Micro Housecall but it didn't find anything suspicious.

The server is a domain controller and a database server so I haven't tried running a repair of the Windows Server 2003 installation.

[pkh]

Did you turn on AOFO for your job?

[JimmyU]

With the "Removable Storage" service stopped and set to disabled the log still shows the same errors and warnings.

Did you turn on AOFO for your job?

Yes, it is turned on and according to the backup log it seems to use VSS

AOFO: Started for resource: "\\phapp01.PITEHALSAN\System?State". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

The following volumes are dependent on resource: "C:" .

The snapshot provider used by VSS for volume C: - Microsoft Software Shadow Copy provider 1.0 (Version 1.0.0.7).

[JimmyU]

I've upgraded the installation to Backup Exec 2012, which is an experience in itself with the new UI and handling of backup jobs, but the problem still remains.

All servers except the backup server itself is backed up without errors.

[pkh]

I suspect your disk is bad and is causing read/write errors.  If you run CHKDSK repeatedly and it keeps correcting error, it is time to change the disk.

[Gurvinder]

Do these files exist on the server ? If yes, have you tried to copy them on an alternate location and has that thrown some errors. If yes then I suspect the disk has Issues. If not, then I would recommend to patch W2K3 with latest VSS patches and rerun the backup (reboot after patch Install)

[pkh]

Do these files exist on the server ?

Do take note of this in the original post

Portions of "\Boot Files\C:\WINDOWS\system32\rsmmllsv.exe" cannot be read

and thank you for confirming that the disk is probably bad.