cancel
Showing results for 
Search instead for 
Did you mean: 

Exchange 2013 GRT Restore fails

Malgamus
Level 3

Sorry for the long post but I want to provide as much detail as possible to make my issue easier to troubleshoot. I'm having issues restoring from a successful GRT backup. If you want to skip right to the error jump to the bottom of my post. I'll also provide the steps I took to set up the environment as well.

 

Here are the steps that I took to set up my environment to backup Exchange 2013 from: https://support.symantec.com/en_US/article.HOWTO73076.html I've marked or noted something for each step.

1. Netbackup Versions
   c. Exchange 2013 support began in 7.6.0.3 - All servers and clients are on 7.6.0.3

2. Disks supported - Granular backups MUST go to disk
Used this for storage ->   c. PDDO/MSDP
       To achieve optimal backup performance on a PureDisk/MSDP configured storage unit, the default fragment size should be decreased to a size of 15GB.
 

3. OS compatibility -
   a. Windows must be a minimum version of 2003 R2
   b. For UNIX/LINUX media servers, confirm in the compatibility guide is it is supported at the level of NetBackup you are using

Master/Media server:

Netbackup 5220 Appliance Version 2.6.0.3 -  netbackup2 - running 7.6.0.3

   
Done - 4. Permissions - The Netbackup Client Service, Netbackup Legacy Client Service, and Netbackup Legacy Network Service (if using                                                   Netbackup 7.0.1. and higher) on each server must run as an account:
Done -   a. That is unique to Exchange - 
       Use Active Directory Users and Computers to create a user account that has a mailbox with a unique name.
       A unique name is one that does not already exist within the Exchange Organization. This name cannot be contained as a set of characters in an existing name.
       For example:EXCH1is entered as the unique mailbox name, and other mailbox names such as EXCH1BACKUP or BACKUPEXCH1 exist. The backup or restore of individual mailboxes will fail.
Done -   b. The account must be a Domain Admin

Done -   c. The account must have a mailbox associated with it and cannot be hidden from the Global Address List. 
       **Verify the account can send and receive email by sending and replying to a test email.**
       If this is Exchange 2007, the account must reside on the server (or in the cluster). See TECH77721 for more information.
  Done -   d. Grant the appropriate permissions within Exchange:
       Exchange 2010/2013 - Exchange Organization Management **By default, this group contains Exchange Recipient Management, Public Folder Management, and Server Management. 
       Removing or modifying any of those roles will affect our backups.**

Done -   e. To ensure the backups are successful for Exchange 2013, EWS impersonation must be configured:
       Command from Exchange PowerShell to check whether the role exist or not :
       Get-ManagementRoleAssignment -Role "EWSImpersonationRole"
       - If the role does not exist or has not been set for the Netbackup Account, refer to below instruction. 
       Command to create a new role called EWSImpersonationRole:
       New-ManagementRole -Name EWSImpersonationRole -Parent ApplicationImpersonation
       Command to assign a user to EWSImpersonationRoleAssignment:
       New-ManagementRoleAssignment -Role EWSImpersonationRole -User svcnetbackup EWSImpersonationRoleAssignment
Done -   f.  Exchange 2013 only - Enter the domain\account and password under the Netbackup Admin Console - Host Properties - clients - Exchange 2013 server's properties (to include all mailbox nodes and CAS servers)
       Expand Windows Client - Highlight Exchange - Enter the domain\account and password under Exchange credentials
       This requires that the account is assigned to "Replace a process level token" under the server's local security policy. A pop up will remind you of this, (even if it's already in place) whenever you modify the
       Exchange Credentials in the Netbackup Admin Console.
Done -   g. Update the Distributed Application Mapping if this is a DAG, clustered, CCR or Off-host backup:
       Go to, Host properties - master servers - Properties of master server - select Distributed Application Restore Mapping
       Provide the name of the application host and the name of the component host, as follows:
       Environment         Application host                       Component host
       DAG                ExDag                                    excas1-dc1
                              ExDag                                    excas2-dc1
                              ExDag                                    exmbx1-dc1
                              ExDag                                    exmbx2-dc1

Done - 5. NFS configuration - this must be setup on media and Exchange servers (this includes CAS servers in Exchange 2010)
   c. Windows 2012 To enable services for Network File System, perform the following:
      1. Open the Server Manager.
      2. From the Manage menu, click Add Roles and Features.
      3. In the Add Roles and Features Wizard, on the Before You Begin page, click Next.
      4. On the Select installation, Click Next.
      5. On the Sever Selection page, click Select a server from the server pool and select the server. Click Next.
      6. On the Server Roles page, Expand File and Storage Services - File and iSCSI Services - Click Server for NFS role, Click Next.
      7. On the Features page, click Client for NFS. Click Next.
      8. On the Confirmation page, click Install.
      9. Open up Services, disable Server for NFS, enable Client for NFS set to automatic.
    10. Open a command prompt, run "sc config portmap start= auto"
   d. If media server is UNIX, no additional configuration is required - check the OS compatibility to ensure it is a supported platform - TECH59978 
 N/A -e. If firewall is used, open port 7394

6. Policy
   a. Attributes
       Policy Type should be MS-Exchange-Server.
       Policy Storage should be a supported disk pool See #2 for more information.
       Granular does not support backup to tape.
       Check the "Enable granular Recovery".
       If this is Exchange 2007 cluster, Exchange 2010 or Exchange 2013, check "Perform snapshot backups" and specify VSS under options.
       If this is and Exchange 2010/2013 DAG or Exchange 2007 cluster, specify the database backup source.
   b. Schedules - 
       Full schedule only - Granular backups do not support Incremental or Differential backups. 
   c. Client - 
       Exchange Server name
       If this is a Microsoft cluster (Exchange 2003/2007) use the virtual cluster name
       If this is a DAG (Exchange 2010/2013) use the DAG name
   d. Backup Selection - 
       Exchange 2003 and 2007 use - Microsoft Information Store:\*
       Exchange 2010/2013 Standalone use - Microsoft Information Store:\*
       Exchange 2010/2013 DAG use - Microsoft Exchange Database Availability Group:\*

I've backed up a single Mailbox database as a test using the exmbx1-dc1 server as the client which backed up with GRT sucessfully (I can see down to the mailbox folders/messages).
      

7. Additional notes -
 Done -  a. Netbackup must be installed on all servers in Exchange. If clustered, use the node name not the cluster (or DAG) name when installing the Netbackup Client Software. 
 Done -  b. If using Exchange 2010/2013, the CAS servers must be configured with the same version of Netbackup and NFS must be configured. Restores will fail if not configured correctly.
Backing up to Disk -   c. If duplicating to tape or a non-supported disk (i.e. OST plug-in) to store the image, you must duplicate back to a supported disk location for the granular restore.
   d. Exchange 2003 and before uses MAPI for restore of individual items. Exchange 2007 requires Collaborative Data Objects 1.2.1 to be installed and Exchange 2010 uses Exchange Web Services on the CAS servers.
  

e. If using a load balancer for your Exchange CAS server, you will need to put a host file entry on your master and media server to point the load balancer names to 1 of your CAS servers. 
       Otherwise, the restore will fail with a communication error.
       For example:
       CAS01 and CAS02 are the names of my CAS servers.
       1.2.3.4 and 1.2.3.5 are the IP addresses of my CAS servers.
       CASARRAY.COM is the name of my load balancer.
       1.2.3.6 is the IP of the load balancer.

       For the host file, choose to point the loadbalancername to CAS01 (only choose 1)
       1.2.3.4       CAS01        CAS01.domain         CASARRAY.COM

 The load balancer shouldn't apply since we have a seperate network for backup and all the Exchange nodes and Netbackup appliance are on it and resolve to the correct IP on the backup network.

 

I have successfully been able to backup a single Mail Database with GRT by using exmbx1-dc1 as the client for a test. My problem is when I try to restore I get the following:

 Restore Job Id=77871
Restore started 07/30/2015 16:39:13
 
16:39:47 (77872.001) Restoring from copy 1 of image created Mon 27 Jul 2015 04:02:12 PM EDT from policy Ex2013Test
16:39:47 (77872.001) Granular restore has started, this may take a while....
16:40:12 (77872.001) INF - Restore executed on server EXMBX1-DC1
16:41:13 (77872.001) TAR - \\EXMBX1-DC1\Microsoft Information Store\MbxDB-2GB-1\Database
16:41:14 (77872.001) ERR - unable to create object for restore: \\EXMBX1-DC1\Microsoft Information Store\MbxDB-2GB-1\Database\Brian Grell [grellb], rai error = 17
16:41:14 (77872.001) ERR - unable to create object for restore: \\EXMBX1-DC1\Microsoft Information Store\MbxDB-2GB-1\Database\Brian Grell [grellb]\Top of Information Store, rai error = 17
16:41:14 (77872.001) ERR - unable to create object for restore: \\EXMBX1-DC1\Microsoft Information Store\MbxDB-2GB-1\Database\Brian Grell [grellb]\Top of Information Store\Inbox, rai error = 17
16:41:17 (77872.001) ERR - unable to create object for restore: \\EXMBX1-DC1\Microsoft Information Store\MbxDB-2GB-1\Database\Brian Grell [grellb]\Top of Information Store\Inbox\Fwd: Request 1217819 for Page, Jessie J Closed. <00000000542190b10d4ed5438ec0e3b46ba77fb107009a92a025c1a20c4fa49783ea1f8a24ba0000001daf6900008bc28142785f65449b8c84acaf80617e00001221c2990000>, rai error = 17

16:41:32 (77872.001) INF - GRE EXITING WITH STATUS = 0
16:41:32 (77872.001) INF - GRE RESTORED 1 OF 5 FILES SUCCESSFULLY
16:41:32 (77872.001) INF - GRE KEPT 0 EXISTING FILES
16:41:32 (77872.001) INF - GRE PARTIALLY RESTORED 0 FILES
 
16:41:32 (77872.001) Status of restore from copy 1 of image created Mon 27 Jul 2015 04:02:12 PM EDT = the restore failed to recover the requested files
 
16:41:33 INF - Server status = 2810

 

Any ideas what could be causing this error or what steps I can take to track down the issue? It looks like it was able to create the Database but not create my Mailbox or anything below that.

 

Thanks,

Brian

1 ACCEPTED SOLUTION

Accepted Solutions

Malgamus
Level 3

After speaking with our Exchange Admin, I resolved our issue. I had to map the hostname of our load balancer for Exchange to the IP of one of our CAS servers. Once we did that GRT restore works like a charm. This was mentioned in the https://support.symantec.com/en_US/article.HOWTO73076.html article but I wasn't sure it was needed since everything is on a secondary backup network. The restore process however needed it to reach the EWS URL for restores.

View solution in original post

4 REPLIES 4

GulzarShaikhAUS
Level 6
Partner Accredited Certified

Can u please share following logs - If not created, then create and set maximum verbosity

bpdbm and bprd from master server

nbfsd bpbrm and bptm from appliance 

beds bpbkar nbfsd ncflbc and ncfgre from destination client

Run following commands to set the verbosty on nbfsd ncflbc and ncfgre log

vxlogcfg -a -p NB -o nbfsd -s DebugLevel=4 -s DiagnosticLevel=4

vxlogcfg -a -p NB -o ncfblc -s DebugLevel=4 -s DiagnosticLevel=4

vxlogcfg -a -p NB -o ncfgre -s DebugLevel=4 -s DiagnosticLevel=4

Malgamus
Level 3

I've attached the log files from the client. Hopefully they can shine some light on the issue.

GulzarShaikhAUS
Level 6
Partner Accredited Certified

Hi,

I think this looks bullseye. Please check this article - 

Exchange GRT restore fails with Cannot open one or more mail messages because of unknown Exchange Web Service errors

http://www.symantec.com/docs/TECH227379

Solution
The Exchange Admin needs make the AutoDiscover service accessible. once available the above Beds error seen in the EWS log will now longer be displayed. You may also need to after that take a look at  TECH225854 if you see the following in the EWS log AutoDiscover Failed in locating the url The request failed. The remote server returned an error: (401) Unauthorized.

Malgamus
Level 3

After speaking with our Exchange Admin, I resolved our issue. I had to map the hostname of our load balancer for Exchange to the IP of one of our CAS servers. Once we did that GRT restore works like a charm. This was mentioned in the https://support.symantec.com/en_US/article.HOWTO73076.html article but I wasn't sure it was needed since everything is on a secondary backup network. The restore process however needed it to reach the EWS URL for restores.

View solution in original post