Exchange 2010 GRT Recovery Error

[Yajith]

Hi All,

I'm trying to perform a item level restore on exchange 2010.

My environement is as below.

2 MailBox Servers in a DAG

2 HUB/CAS Servers

All the servers are running Windows 2008 R2 and installed with NetBackup 7.1.0.3 Client as with the Master Server.

As a test i have performed a backup with GRT enabled and it seems to be working properly. I can view individual e-mail items on the Restore interface.

The issue is, when i try to restore it fails with giving the below error.

---

4/4/2012 12:23:35 PM - begin Restore
4/4/2012 12:23:37 PM - restoring image wataniyadag.wataniya-maldives.com_1333523877
4/4/2012 12:23:40 PM - Info bpbrm(pid=7876) VHM1SVEXHC03.WATANIYA-MALDIVES.COM is the host to restore to      
4/4/2012 12:23:40 PM - Info bpbrm(pid=7876) reading file list from client        
4/4/2012 12:23:41 PM - connecting
4/4/2012 12:23:41 PM - Info bpbrm(pid=7876) start nbfsd on client         
4/4/2012 12:23:43 PM - Info bpbrm(pid=7876) start nbgre on client         
4/4/2012 12:23:44 PM - Info tar32(pid=0) Restore started           
4/4/2012 12:23:44 PM - connected; connect time: 00:00:03
4/4/2012 12:23:50 PM - Info tar32(pid=0) done. status 13          
4/4/2012 12:23:50 PM - Info tar32(pid=0) done. status: 13          
4/4/2012 12:23:50 PM - restored image wataniyadag.wataniya-maldives.com_1333523877 - (file read failed(13)); restore time 00:00:13
4/4/2012 12:23:50 PM - end Restore; elapsed time: 00:00:15
MS-Exchange-Server policy restore error(2810)
4/4/2012 12:23:55 PM - Info tar32(pid=0) done. status: 13: file read failed       
4/4/2012 12:23:55 PM - Error bpbrm(pid=7876) client restore EXIT STATUS 13: file read failed   

---

VHM1SVEXHC03 is the virtual name of the CAS nodes.

I have enabled NFS Services on both mailbox servers and CAS servers.

I have created two files inside C:\Program Files\Veritas\NetBackup\db\altnames with the names of mailbox servers, and both files contain the DAG virtual name in them.

Your advices in figuring this one out is much appreciated.

Thanks!

Yajith

[Mark_Solutions]

You need to open up the Master Servers Host Properties and go to Distributed Application Restore Mapping and add all of your components of the DAG in there

DAG appears on the left - CAS, Mailbox Servers and, if used, Media Servers acting a Proxy Restore Hosts on the right.

Probably need a service re-start on the Master to properly regsiter it all.

You also need check name resolutions all round and if you use a hardware CAS loadbalancer you may need to add host file entries for the CAS to point to a "real" CAS server that has a NBU client on it.

[Yajith]

What needs to be added to Distributed Application Restore Mapping for CAS. Is it the virtual name or the individual server names ? I tried with below, but it was not successful.

DAG - MBX1

DAG - MBX2

DAG - VIRTUAL-NAME-OF-CAS

I will try with physical names of the CAS servers also.

[Yajith]

Tested with below settings also, but so far couldn't get it to work.

DAG - MBX1

DAG - MBX2

DAG - CAS1

DAG - CAS2

DAG - VIRTUAL-NAME-OF-CAS

I have added a host file entry on the master server to point VIRTUAL-NAME-OF-CAS to CAS1. I'm attempting this restore from the master server, and there are no other media servers involved.

NetBackup client is installed on the CAS1 (CAS2 aswell).

Appreciate any ideas.

[Mark_Solutions]

Your settings now look OK presuming that they are all using FQDN (as I see in the original log file that the client is set as very FQDN!) The bpresolver logs on the CAS may give you clues here.

Also watch out when adding the client names - I see upper and lower case in play here and it can be touchy:

wataniyadag.wataniya-maldives.com
VHM1SVEXHC03.WATANIYA-MALDIVES.COM

Just check on the CAS and MBX NetBackup Client Host Properties (accessed through the Admin Console on the Master) that you dont have an Exchange Granular Restore Proxy Host set (which i guess would be the Master) as you would also need that adding to the list as well

I also had a customer that added a suitable mailbox name in that section which made his restores work - even though it is not supposed to be used any more so may have been co-incidence.

Another thing I just want to check is if you have any firewalls in place anywhere - 2008 is prone to having it enabled which stops things working.

Finally - it is snowing in the UK today - so if your domain indicates where you are then send me the tickets, book my hotel and i will come and sort it for you - LOL!

[Yajith]

I have added the FQDN and hostnames both to the list and re-tried without success.

Also verified that there is no restore proxy host set on the individual client settings.

Just to clarify, when I'm attempting the restore from the master server, i have set the source and destination both to the name of the DAG. Pls let me know if something needs to be changed.

Another thing i noticed is that there is no logging happening on the bpresolver log directory on the CAS when i attempt a new restore. Does that give some kind of a clue to what is missing here ?

And yes domain does indicate where i am, too bad the issue is unresoved yet, lot of things i could do otherwise. ;)

[Mark_Solutions]

Do you definatley not have a hardware load balancer in front of your CAS servers?

If you ping the virtual cas name does it resolve to one of the CAS Servers?

If so set up bpcd logging one the CAS Servers to see if we are at least talking to them.

From the initial log you posted it does resolve the mailbox server to do the restore to so it must be getting that information from somewhere - it just then fails when it tries to do the restore

Do all of the members have their Netbackup Client Service set to use a suitable account?

[Yajith]

Definietly do not have a hardware load balancer in front of the CAS servers.

Virtual CAS name is getting resolved from all the nodes, When I ping to virtual CAS name, it gets resolved to its own IP address, which is not the CAS Server 1 or Server 2 physical IP address. It is maintained by Windows NLB and the requests that are coming to it will be balanced by the NLB it self.

I'm checking on bpcd logging right now, do share if you have any additional ideas.

[Mark_Solutions]

Just the client accounts used - in fact I believe you need to set the NetBackup Client Service and NetBackup Legacy Client Network Service both to use the exchange admin account

[Yajith]

All the clients are running with a service account we created. Service account was created according to the guidelines in the exchange admin. guide.

I just checked to confirm and NetBackup Client, NetBackup Legacy Client, NetBackup Legacy Network all are running with that service account privileges.

[Mark_Solutions]

OK - lets have all the logging set up and see if we can pin down where it fails:

On the CAS / MBX Servers
<install dir>\netbackup\logs\beds
<install dir>\netbackup\logs\bpcd
<install dir>\netbackup\logs\ncfgre
<install dir>\netbackup\logs\nbfsd
<install dir>\netbackup\logs\ncflbc
<install dir>\netbackup\logs\bpresolver
<install dir>\netbackup\logs\tar

On the media server
<install dir>\netbackup\logs\bpbrm
<install dir>\netbackup\logs\bptm
<install dir>\netbackup\logs\nbfsd

on the master server
<install dir>\netbackup\logs\bprd

[Yajith]

I have attached the logs mentioned above. CAS2 and MBX2 didn't have any of the above logs collected.

[Mark_Solutions]

I dont see off hand in these what is causing the issue here.

A couple of things to try ...

1. Make sure the netbackup account being used has sufficient rights and is also specifically added as a local administrator on each server

2. Try doing the restore and selecting the active mailbox server as the destination client, but using the restore to original location selection during the restore - just to see if that makes a difference

[Yajith]

Added the netbackup account to the local adminstrator group of each of the systems (2 x MBX , 2 X CAS) and tried to restore with the active mailbox and to the original location, but so far did not notice any changes. Still getting the same error.

I will keep trying and update when i find something. Thanks Mark for your responces.!

[Mark_Solutions]

One other thing i read somewhere is that you should only select one backup image (little green orb) in the BAR GUI before trying the restore - so try that as well

[Yajith · Accepted solution]

Hi Mark,

We were able to finally sort this out with the tech support, and after making two changes. There was an issue with one CAS server communication with master server.

Support engineer identified it using "bpclntcmd -pn" command. After that, we added the host file entries for CAS1, CAS2, CAS-VIRTUAL-IP, MBX1, MBX2 to the host files of both CAS servers and the NetBackup master server.

The other change we performed was to create a "No.Restrictions" file in C:\Program Files\Veritas\NetBackup\db\altnames. I feel it is this change and not the host file changes that did the trick for us. Anyway, the item level restores are working now. Strange thing is this "No.Restrictions" is not mentioned in the Exchange admin guide, if it is a mandatory change in this type of a setup.

Sorry for the delay in update, I had to get on a flight and leave "where the domain indicated".

Appreciate your effort very much in helping out with this issue.

Yajith