Long Synthetic jobs fail with status 84 using OST disk storage

[matt077]

I'm just seeing if anyone else have or had the problem i have here when backing up a large filesystem from a client using synthetic backups writting to a HP OST plugin disk storage. Media and Master server are both windows boxes.

We have had this problem from version 7.7.3, after speaking with HP they pointed us towards being a NBU problem. After speaking with vertitas support and sending countless numbers of logs, they couldnt find why we are seeing this problem. We have a work around so we are backing up but its more me now wanting to know why we seeing this problem.

This was working fine in 7.6 until we upgrade to 7.7.3 and also upgraded the HP OST plugin to 4.0.2. We are now at version 8.1.1 and we still have the problem.

error in the backup when the sythetic runs - 

13-Jun-2018 11:55:04 - Error WriteVirtualSynth::handleEndpointMsg (pid=1952) received EXIT message from writer parent with error status=media write error(84)

13-Jun-2018 11:55:04 - Error WriteVirtualSynth::giveEndpointStatus (pid=1952) received end point error = media write error(84)

13-Jun-2018 11:55:04 - Error PlanAdmin::actionDone (pid=1952) action failed, error=media write error(84)

13-Jun-2018 11:55:23 - Error SynthJob::cleanup (pid=1952) end bpsynth with status = media write error(84)

 As anyone else come arcross this problem?

[Alexis_Jeldrez]

Did you redid traditional full backups after the upgrade?
Are non-synthetic backups working fine to the storage?

[matt077]

non-synthetic backups etc all work find

its just large synthetic jobs which fail. It feels like a timeout problem but hp or veritas couldnt find a fix

[Nicolai]

Using a HPE StoreOnce device ?

Instead of using synthetics with is old style by now, why not using NBU accelerator ?

[matt077]

@Nicolai

Yes we are using HPE StoreOnce device. I have been thinking of moving the job to use accelerator tho the last time we did abig filesystem with the accelerator, the jobs become to fail. When speaking with veritas about this they said it was due to the filesystem being too big and too many files for accelerator to deal with.

[Nicolai]

@matt077

Before trying accelerator on StoreOnce make sure you got the latest StoreOnce firmware and OST plugin as there are major improvements in the handling of Accelerator and StoreOnce general reliability

How big is the file system you are trying to backup ?

[matt077]

@Nicolai

Yes are at the lastest level of fm and plugin for the storeonce device

The early days when the accelerator gave us problems, the file system was around 1tb in size but the number of files was 21000000 number of files. From what i recall veritas said the number of files was the problem which is why we have kept away from using the accelerator part of things.

The filesystem we backing up via Synthetic jobs is around 3 tb in size but no where near that number of files now. I think i will need to test accelerator backups for this job being Synthetic doesnt seem to work.

[Nicolai]

@matt077

21 mill files is ridiculous many when combined size is 1TB. Your problem is not the backup type but the number of files.

Consider Flashbackup (has its dis-advantage as well)

Consider if you can exclude some file stype to to bring down total number of files

Talk to data owner why so many files are generated

Try to zip files instead and backup the zip file.

[matt077]

@Nicolai

Yes we have spoken with them about the number of files and its due to many small files which is created from the business each day. Its a pain but we deal with it

The problem of the 84 errors for the Synthetic jobs is on a different file system

[Nicolai]

Yes - you are right my bad :)

3 TB should absolutely be durable for accelerator.

[matt077]

we are testing NBU accelerator on this file system.

Would love to know why Synthetic jobs on big filesysystems have stopped working tho. small ones (500gb) are fine, its seems to be file systems over 1tb it fails on.