Restore of Exchange 2010 failed because of exe files not backed up in bin folder
I am working a case with support with no luck so far on this issue and I wonder if anyone has any similar problems. We do two different backup sets for our Exchange 2010 server. One is just for the two mailstores using the exchange agent and the other is a full SDR for the whole server. Our D drive on the server had two drives fail in the RAID and we had to go to the backup to get our server back online.
The restore of the system and the edb files worked seemingly correctly, and esutil said the mailbox store was fine. But it would not start or even let us run the Exchange Management Console. We had to call an emergency support case with Microsoft to see why it wasn't working. Microsoft found that some of the exe files in the bin folder of Exchange were missing. Not all of them, just some of them including msexchnageadtopologyservice.exe and store.exe.
Microsoft support was able to help us get the files from service packs for Exchange and so we are back up and running. But why are the exe files not being backed up?
The full backups have no errors or warnings and none of the history shows these files ever being backed up in any of the catalogs. They won't backup today either. I run a separate backup of just those files and the job says it backs up 0 files in 6 directories, but no warning or error about skipped files. The security permissions on these files that don't show up are the same as the ones that do show up. There are no exclusions in BE that any support person has been able to find.
My main frustration is that I cannot restore Exchange from a full Exchange backup using Backup Exec 2014. I am out the $500 for the off-hours Microsoft call as well as making our team work an extra 10 hours over the holiday to get the system back from this bug / oddity / feature in BE that I have no explanation for yet. But I am incredibly thankful that it was on the holiday because the cost of being down during a manufacturing day would have been astronomically higher.
I have a theory that these files might be in use from antivirus running on that server, but I can copy the store.exe to another folder without problem. Once it is copied to a different folder from bin, it correctly backs up with BE 2014 from the non-bin folder.
I have the solution. First level support was totally barking up the wrong tree, but the advanced team helped us get to the answer:
exe files that are registered as a service are not backed up in the file backup, but are included in the system state backup set. (See article http://www.symantec.com/business/support/index?page=content&id=TECH92375)
We did not restore the system state when we rebuilt the D drive because the C drive was ok. But restoring the system state would have put those files back on the D drive.