New 8.2 environment backing up Ip-Less DAG on new Exchange 2019 running on Windows 2019 Core Servers.
Backup is working ok, restores not so much. We suspect that a new feature in Exchange 2019 causes the restore to fail at the commit/mount database stage. In Exchange 2019 they have implemented a SSD cache database, for quicker searches etc, that are referenced by all mail databases. When you restore an entire database, eg EXDB01, to a recovery database, eg RDB01 , the recovery database have no reference to the cache and mounting the database fails.
Progress log filename : N:\Program Files\Veritas\NetBackup\Logs\user_ops\nbadmin\logs\jbp9FAC.tmp.log Restore Job Id=145846
Restore started 08/14/2019 10:33:14
10:33:14 (145846.xxx) Found (4 608) files in (1) images for Restore Job ID 145846.xxx
10:33:25 (145847.xxx) Found (4 608) files in (1) images for Restore Job ID 145847.xxx
10:33:33 (145847.xxx) Searched ( 10) files of (4 608) files for Restore Job ID 145847.xxx
10:33:33 (145847.001) Restoring from copy 1 of image created 2019-08-14 10:27:26 from policy Silver_Exchange2019_DAG
10:33:53 (145847.001) INF - Beginning restore from server xstobup01.ref.xxx.xx to client XSTOEX41.REF.XXX.XX (client_hostname XSTOEX41.REF.XXX.XX).
10:33:55 (145847.001) TAR - Microsoft Exchange Database Availability Groups:\
10:33:55 (145847.001) TAR - Microsoft Exchange Database Availability Groups:\xstoexdag41\
10:33:55 (145847.001) TAR - Microsoft Exchange Database Availability Groups:\xstoexdag41\Microsoft Information Store\
10:33:55 (145847.001) TAR - Microsoft Exchange Database Availability Groups:\xstoexdag41\Microsoft Information Store\EXDB01\
10:33:55 (145847.001) MNR - The file was renamed to the following:
10:33:55 (145847.001) UTF - Microsoft Information Store:\RDB01\
10:34:22 (145847.001) TAR - Microsoft Exchange Database Availability Groups:\xstoexdag41\Microsoft Information Store\EXDB01\Database
10:34:23 (145847.001) MNR - The file was renamed to the following:
10:34:23 (145847.001) UTF - Microsoft Information Store:\RDB01\Database
10:34:26 (145847.001) TAR - Microsoft Exchange Database Availability Groups:\xstoexdag41\Microsoft Information Store\EXDB01\Logs_1565771088
10:34:27 (145847.001) MNR - The file was renamed to the following:
10:34:27 (145847.001) UTF - Microsoft Information Store:\RDB01\Logs_1565771088
10:34:36 (145847.001) INF - There are no jobs remaining for this restore request. Exchange database recovery will be performed.
10:34:37 (145847.001) INF - After restoring a Microsoft Exchange LCR, CCR, or DAG database, you must use the Exchange Management Shell to resume replication.
10:35:01 (145847.001) WRN - MountDatabase failed with error(0x0) for machine xstoexdag41, storage group RDB01, database RDB01
10:35:01 (145847.001) WRN - Exchange database recovery may have failed. Please check the event log for further details.
10:35:01 (145847.001) ERR - An Exchange restore error was detected. You may need to manually mount the Database stores to successfully finish the restore.
10:35:01 (145847.001) INF - TAR EXITING WITH STATUS = 5
10:35:01 (145847.001) INF - TAR RESTORED 6 OF 6 FILES SUCCESSFULLY
10:35:01 (145847.001) INF - TAR KEPT 0 EXISTING FILES
10:35:01 (145847.001) INF - TAR PARTIALLY RESTORED 0 FILES
10:35:01 (145847.001) Status of restore from copy 1 of image created 2019-08-14 10:27:26 = tar did not find all the files to be restored
10:35:01 INF - Server status = 5
10:35:02 (145846.xxx) INF - Status = the restore failed to recover the requested files.
10:35:02 INF - Server status = 2810
10:35:02 (145847.xxx) INF - Status = MS-Exchange-Server policy restore error.
FROM EXCHANGE SERVERS ERROR LOG:
msexchangerepl (5708,U,98,15.02.0330.008) RDB01\XSTOEX41: Database recovery failed with error -1216
because it encountered references to a database, C:\ExchangeMetaCacheDbs\EXDB01\EXDB01.mcdb\EXDB01-mcdb.edb', which is no longer present.
The database was not brought to a Clean Shutdown state before it was removed (or possibly moved or renamed).
The database engine will not permit recovery to complete for this instance until the missing database is re-instated.
I have just now started a case with Veritas support.
Yes, they are running 2019 CU1. Funny thing, we did some early tests with 8.1.2 software and were successfull with backup and restore. At that time there was only one node and SSD cache had not been enabled. Since it was not supported the Exchange 2019 project came to a halt until the 8.2 release. Now we have 8.2, a clustered DAG and SDD cache and it does not rock'n roll at all.
I do not have any insight into this scenario, but thank you for the heads up. I will start a conversation with the development engineers while your support case works its way through the system.
You can check your backup to confirm that your cache data files are not being backed up. With that confirmation, you can urge our support engineers to escalate it to engineering. Use my name.
I expect that it will turn out that NetBackup hasn't implemented support for the database cache. I'll ask the developers.
I have read about Exchange metacache database feature today on Microsoft pages and a 3rd party blog. The MCDB is supposed to be redundant. If the SSD fails, the database is supposed to stay up, possibly failing over to another server with healthy SSD's. There's no mention in these articles about backup and recovery, and a Google search on "Exchange MCDB RDB" gets no hits that contain all three keywords.
That's all I have until I learn more from our development engineers.
I opened a case with Microsoft. They acknowledge that it's their bug in vague terms:
>[W]e currently are aware of an issue like that and the resolution is to run a repair on the recovered database using eseutil.
I don't regularly use eseutil. I have asked them for example syntax.
We have had some success inventing workarounds performing restores with Netbackup without using commit/mount. The new version of eseutil in exchange 2019 can't repair the restored database but when we used eseutil from a older version of exchange (2016) we were at least able to repair and mount the database. From what I heard another backup software vendor could not even perform a complete backup when MCDB was enabled.
Microsoft implicitly acknowledge that it's broken, but they're not going to fix it. I asked for a KB article and repeated a question from Per that they didn't answer.
E-mail with Microsoft Support 10/24/19 follows.
I was able to have a discussion with our beta engineers today. This is being treated as by design and there are no current plans to change it. So any time a database is restored if MCDB is enabled will need a eseutil /r run on it before it will mount.
I'll pass on this answer. People won't be happy.
Please put it in a KB article and send me the link.
Did you get a response to my customer's experience that the 2019 eseutil didn't work and he had to use the 2016 version?
BTW, here is the eseutil syntax Microsoft Support provided and the explanation:
eseutil /r (log file extension, IE: E01) /d "D:\path\to\recoverydb" /i
The issue is that the restored database will have hooks into the MCDB at the time of backup. This cuts those hooks so that the database can be mounted.
Also, Per had forwarded to me an answer Microsoft gave him directly as to why the restore doesn't work. Their answer does not apply to database restores because it has to do with options for invoking Jet. Database restores are purely VSS operations. However, Veritas may need to add the suggested Jet option to GRT operations. I forwarded that information to our NetBackup and Backup Exec development engineers.
We have fallen behind in the exchange 2019 rollout plan to the point where our old exchange environment is out of support and warranty. Management has taken the decission to have the 2019 DAG uninstalled and start all over, without MCDB.
We have already verified in our reference environment that backup/restore works as expected with commit/mount restores when MCDB is not enabled. The exchange team can enable MCDB again when it is supported all the way.