Backup Exec error: 0xe00084c7 - A backup storage read/write error has occurred.

[rmcbackupteam]

Hi Everyone,

We are getting following error with random backup jobs backing up on deduplication storage. These are intermittent failures and failed backups are completing after retry.

Job status:

Job ended: 08 August 2019 at 00:02:50
Completed status: Failed
Final error: 0xe00084c7 - A backup storage read/write error has occurred.

Final error category: Backup Media Errors

Error:

Storage device "Deduplicationdiskstorage0001:11" reported an error on a request to write data to media.

Error reported:

0x17
V-79-57344-33991 - A backup storage read/write error has occurred.

We are unable to find the root cause of this issue. There is no critical/error on event logs at the time of failure.

We have enough space available on de-dup storage. It is a local disk storage with deduplication enabled.

Reagrds,

Backup Team

[Gurvinder]

Does the error come up when doing a client side backup ? Expand this failed job log -> device and media section -> If the remote server that you are backing up shows under device hostname then the job is run with client side dedupe.

Does this come up for a particular remote server backup all the time ?

Does the error come up during the verify stage of this backup. You can expand the job log and check if these errors are reported under the backup or verify section ?

If verify is running as part of the backup , then better to run it as a seperate job. You can do this by editing the Job and from under the verify section, choose the radio button which allows for seperate verify.

In case a network connection is dropping apply this registry on BE server and remote server and check if it imporves 
KeepAliveTime registry dword. set to decimal 5000
(HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters)
(Note: reboot the server post making the change)

You can also do a health check on dedupe -
https://www.veritas.com/content/support/en_US/article.100006521.html
If the crcerrors.txt file is zero bytes, then the dcscan program did not find any corruption inside the deduplication folder.

[Greg_Fuchs]

We've been having the same issue since upgrading to 20 on two separate  BE servers.

Single threading backups helped.

In addition the dedup storage goes offline entirely sometimes.

An issue with running dcscan is that no jobs can be running during the scan. we had to kill it after 24 hours. We do not have a window in our schedule that long.

[rmcbackupteam]

Hi Gurvinder,

We are facing this issue mainly with NDMP backups however, there are instances where Flat file backups have also failed with the same error. The backups are getting completed if we retry the job after some time. VERITAS has suggested to perform following registry changes but it didn't work:

Need to create new D-Word on the registry of the BE server
Registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Misc
Value name : Disable NDMP stream handlers via MTF
Value : 1

In Backup Exec NDMP LOg I can see following error:

NDMP Log Message: DUMP: Reference time for next incremental dump is : Mon Aug 12 22:21:00 2019.
NDMP Log Message: DUMP: mapping (Pass II)[directories]
NDMP Log Message: DUMP: estimated 37547 KB.
NDMP Log Message: DUMP: dumping (Pass III) [directories]
NDMP Log Message: DUMP: dumping (Pass IV) [regular files]
NDMP Log Message: Mover: Tape write failed. NDMP Log Message: DUMP: Message from Write Dirnet: Interrupted system call NDMP Mover Halted: Media Error Storage device "Deduplicationdiskstorage0001:5" reported an error on a request to write data to media.
Error reported: 0x1712 August 2019 23:24:06 - V-79-57344-33991 - A backup storage read/write error has occurred.

NDMP Log Message: DUMP: DUMP IS ABORTED NDMP Log Message: DUMP: Total Dir to FH time spent is greater than 15 percent of phase 3 total time. Please verify the settings of backup application and the network connectivity. NDMP Log Message: DUMP: Deleting "/PLA/T2_prod_ft_lime_vol7/../snapshot_for_backup.936" snapshot.
NDMP Log Message: DATA: Operation terminated (for /PLANTATION/T2_prod_ft_lime_vol7). NDMP Data Halted: Aborted

Regards,

Abhinav.

[Gurvinder]

Disable NDMP stream handlers via MTF , this woud lower the dedupe ratio if set to 1. I would suggest to open a support case to review this.