I have the following environment:
1 – BE2012 CASO server (2008 R2) at our corporate office, dedupe as primary storage, tape secondary
3 – BE2012 MMS’s (2008 R2) at three plant locations, dedupe as primary storage, tape secondary
These servers are up to HF 201596, but do not have SP2 applied yet.
Backups from the plant locations are duplicated to the corporate dedupe folder. Some sets are also duplicated to tape at the plant location or at the corporate office for longer term storage.
We are getting an error while trying to restore data from the dedupe storage to one of our Windows 2003 SP2 servers (the server is virtual, Hyper-V, but being backed up individually using the Windows Agent). Only selected folders were being restored, and not to the original location, but to a different drive & folder on the same server. The job failed with:
Final error: 0xe00084ca - The data being read from the media is inconsistent.
Final error category: Backup Media Errors
For additional information regarding this error refer to link V-79-57344-33994
The job failed when it was loading the second “media” from the dedupe folder. I have tried restores from both the CASO and the MMS, with different sets from 2 different dedupe folders, and also to another 2003 SP2 server (also a virtual, but backed up individually with the Windows Agent). All of these backups were also duplicated to tape with software encryption, and they all fail with the exact same error when the second “media” is loaded.
To see if the duplicate to tape stage was causing a problem, I then ran a full backup of one of the virtual 2003 servers (using the Windows Agent) and set the duplicate to tape stage on hold. On a restore of selected folders again as before, it still failed with the same error when the second “media” was loaded.
I then tried a restore of data from one of our 2008 R2 physical servers; same scenario of restore selected content from dedupe to a different location on the server. I selected enough content so it would have to load another “media” from dedupe. It completed successfully.
I am going to run some more scenarios to see if I can narrow it down if this is only occurring on restores to virtual servers and then if only 2003, but wanted to get this out in case someone else has run across this, since I have wasted so much time already trying to figure out what is causing this.
which sounds like a similar situation, but this was for 2010, so I didn’t know if the workaround registry change would apply to 2012, and I am not using encryption on the dedupe folder, only on the duplicate to tape (which I did try the same restore from the encrypted, duplicated tape, and it was successful).
The following is a section from SGMon log around the time when the second “media” is loaded on the dedupe:
BENGINE: [09/01/13 13:50:12]  2013-09-01T13:50:12.312 [loops] - RestoreBSDProcessor::TapeChange returning result = 0x00000000
BENGINE: [09/01/13 13:50:13]  2013-09-01T13:50:12.685 [loops] - stream data[DEFAULT] halted: INTERNAL ERROR
BENGINE: [09/01/13 13:50:13]  2013-09-01T13:50:12.685 [loops] - data halted: INTERNAL ERROR
BENGINE: [09/01/13 13:50:13]  2013-09-01T13:50:12.758 [loops] - notifyMoverHalted: Halt reason = HALT_CONNECT_CLOSED
BENGINE: [09/01/13 13:50:17]  2013-09-01T13:50:17.616 [tpfmt] - TF_FreeTapeBuffers: from 1 to 0 buffers
BENGINE: [09/01/13 13:50:18]  2013-09-01T13:50:17.766 [loops] - ****** deleteing *tapeServer - 0xaddf800
I have been waiting to apply SP2 to see the feedback, but going through the release notes, I don’t see anything related to this issue. Has anyone else run across this issue on restore?
9/3/14 - Update
I did not receive this error on a test restore from dedup of selected folders to a Windows 2008 R2 virtual server. It loaded a second dedupe "media" during the restore and contiued successfully. The backup of the data was also done using the Windows Agent.
It seems that the issue only occurs when you are performing a restore to Windows 2003 server.
You can try using the registry change in this technote but to be honest I do not think that would fix that either.
I would request you to narrow down the issue to whether it happening only for a virtual machine or a physical machine as well.
I see that you have taken the backup using the Agent For Windows. Are these backups GRT - enabled backups.
If yes then please check if the restores are working for NON GRT backups.
IT is strange that the restores are failing everytime on loading the second media. Ideally it seems to be an issue with the Deduplication media.
I would recommend you to enable the ShowHiddenMedia from the registry using the below technote :
Then I would like you to check the OST media and see if you see anything strange with that meda for e.g you might see duplicate instances of the same media / media may be in the retired media set etc etc.
I would ideally also like to have the debug log while you run the restore but before that we need to know whether it is a GRT or a NON GRT backup.
I don't have a physical 2003 server that I can try a restore to. Restores to physical or virtual 2008 R2 servers work correctly.
Since the registry change referred only to BE 2010, I was concerned on changing it under BE 2012 to even try.
As far as GRT, the backups are being done with the Windows Agent as if it were a physical machine, not through the Hyper-V host, so GRT/NON GRT is not even involved.
There is nothing wrong with the OST media, because as I stated, I am able to duplicate these sets to tape without any issues and even restore from those duplcated tape sets without an issue.
Thanks for your response.
I wish someone from Symantec would offer some assistance and give some indication if this is an issue they have encountered before and if it was possibly addressed in SP2.
Ok. So I see that you are able to restore the data from the tape backup sets but you are not able to restore the data from the Deduplication Folder.
The reason why I asked GRT was whether you had an Application on the VM and whether you were performing APP-GRT ( You can perform APPGRT ) using RAWS.
I agree that you are able to duplicate these backup sets to tape but I would still consider checking the OST media once by enabling showhiddenmedia.
I have faced errors like these before but not a situation like this.
Also if you could try a small backup ( like a 2-5 GB ) backup of that affected VM like backup some flat files ( C:) or D drive and try a restore and check if thats working where you just have one OST media created.
I would request you to open a Support case with Symantec so that they could collect Debugs and analyse them.
I would seriously recommend you to install SP2 because that might just fix the issue for you.
Sorry, I was thinking Hyper-V GRT. No, these backups do not have any application GRT. These are domain controllers and Exchange 2003 servers also, but the Exchange backup is run as a separate job and we are not doing Active Directory GRT.
Restores of data that are less than one OST media (50GB) work correctly. The error occurs right at the point when it loads the second media.
The OST media looks correct. Since this is happening on multiple dedupe stores from multiple backup servers, it would appear to point to an error with the application.
Due to the nightmare of trying to get through a case with Symantec support, I was hoping to avoid that.
I will try to get SP2 installed at some point to see if that may correct the problem.
Sorry for the delayed response on the thread. But seems like the discussion was heading the correct way. Let us know if you still face the issue after installing SP2. Also, if you get a support case created, let us know the case number for us to monitor.