cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to restore GRT email with error "unable to creae object for restore"

Yet
Level 5
Partner Accredited

I have EXCH backup to Appliance successful. 

However, i failed on restoring an email. below job detials,

 

all notes on this technote has been confirmed - https://www.veritas.com/support/en_US/article.HOWTO73076

i've attached logfiles as well.

 

Any idea?

18 REPLIES 18

Lowell_Palecek
Level 6
Employee

First set of observations

The Technote needs to be updated. There's nothing harmful in it but we need to eliminate references to Exchange 2003 and 2007, MAPI, CAS, and ancient Windows versions. More importantly, you don't need to execute the services mentioned as anything other than LocalSystem. It's necessary and sufficient for all supported Exchange versions, that you configure the Exchange service account in the client host properties for all the mailbox servers in the DAG. (Exception: for Windows 2016, execute the NetBackup Discovery Framework service - nbdisco - as your Exchange account until the first NetBackup release in 2019.)

Regarding the logs and what is working for you:

- The backup (bpbkar32) found your Exchange account. Otherwise it would not have done GRT processing. Your Exchange account had sufficient memberships and privileges to catalog the mailboxes. If not (unless you have Exchange 2010), you either would not have seen any mailboxes when you browsed, or the aliases would have been messed up.

- Your browse for restore (nblbc) worked. This shows that the media server and the client on which nblbc ran have NFS configured correctly, and nbfsd is working on both client and media server. Did nbgre run on the same client? (Note, the logs for nblbc and nbgre are ncflbc and ncfgre.)

- The discovery service (nbdisco) is not a problem because you got your mailboxes catalogued, and NetBackup on the master server was able to find Exchange clients on which to run the the backup, browse, and restore.

What's next:

The "not logged on" error seems to be the critical problem. I will check a couple of things in your ncfgre log. Meanwhile, please get the "monad" log and the two files with "EWS" in their names from the NetBackup\logs\beds folder.

Here's the error in the ncfgre log:

>Cannot log on to EWS with the specified credentials.

This is nbgre's "EWS Provider" trying to log onto the Microsoft EWS service.

The <hostname>-monad.log and <hostname>-EWSLog00.log files may give more information about the failed logon attempt. (Monad.exe is the process that executes PowerShell commands. The EWS Provider is a Veritas DLL within the nbgre.exe process that accesses the EWS service. It logs to the EWSLog00 file.)

You probably don't have the second "EWS" log file in your NetBackup\logs\beds folder. When it exists, its name is ewstrace.txt. This file logs the XML request/response interaction with the EWS service after we're logged into it.)

Lowell_Palecek
Level 6
Employee

I have updated the TechNote you had found. I removed obsolete clutter and tried to make some of the points clearer or more accurate. This doesn't invalidate the steps you took to try to make it work.

Nicolai
Moderator
Moderator
Partner    VIP   

Good job - outdated tech notes are so last year.

/Nicolai

Yet
Level 5
Partner Accredited

Many thanks Lowell!

Sorry got tied up lately, was working on 2 different issues with my NBU environement.Will get those monad & EWS logfiles. I'll also change the login acct for NBU services @ exch nodes to Local.

New development but not exactly progress, I tried to restore from active exch node but I cannot browse any image for restore with error - cannot find common CA Root for secure handshake. I checked CA cert & host ID via nbcertcmd and everything seems in place. Same issue with other exch node. Not sure if this is related to my original issue. I supposed can restore from exch node right?

 

Yes, you should be able to browse for restore from any Exchange client in your DAG. There are security checks. The master server won't let a client browse images catalogued for another client. In your case, the image was catalogued under the DAG name, so it looks like the node is trying to browse an image belonging to the DAG. To explicitly allow this, there is something called the Distributed Application Restore Mapping in the master server configuration. It's described in the TechNote you referenced at the beginning of this thread. (Caveat: I haven't seen the check I mention produce a certificate check failure.)

You may have an issue when the master server resolves the DAG name to a node other than the one you're using to browse the image. Find out what client the master server resolves the DAG to by looking in the NetBackup\db\discovery folder on the master server. The newest "<server> exchange.xml file wins. Either browse from that node, or add a "node1 node2" cross-reference to the Distributed Application Restore Mapping.

Yet
Level 5
Partner Accredited

I think I have Distributed Application Restore Mapping properly filled up as below. Are entries here case senstiive? Exch nodes resolved on lower case, though I already changed this to lower case and still same issue. 

Application host                    Component host

DAG01.xxxxx.com.xx      NODE1.XXXXX.COM.XX

DAG01.xxxxx.com.xx      NODE2.XXXXX.COM.XX

I've attached EWS & monad logfiles. Monad file doesn't seem to show any abnormality. However, with EWSLog00 I got this, not sure what this means. Got ewstrace.txt file as well but it's 0kb, nothing logged.

========

AutoDiscover Failed in locating the url The Autodiscover service couldn't be located.
[77dc] 01/10/19 17:31:33 Retrying using CAS server urls from Monad ......
[77dc] 01/10/19 17:31:33 RowSetHelper::Calling Monads GetWebServicesUrls
[77dc] 01/10/19 17:31:34 RowSetHelper::Returned with retval 0iCount 1
[77dc] 01/10/19 17:31:34 Weburl 0https://mail.xxxxxx.com/EWS/Exchange.asmx
[77dc] 01/10/19 17:31:34 RowSetHelper::weburl at 0 is https://mail.xxxxx.com/EWS/Exchange.asmx
[77dc] 01/10/19 17:31:34 Count is 1
[77dc] 01/10/19 17:31:34 Trying url https://mail.xxxxx.com/EWS/Exchange.asmx
[77dc] 01/10/19 17:31:34 Could not use...... webserviceurl https://mail.xxxxx.com/EWS/Exchange.asmxThe request failed. The underlying connection was closed: An unexpected error occurred on a receive.

=========

Some development, I tried to downgrade the NBU Client agent on exch nodes to 8.1.1 because Appliance media server is 3.1.1. I got rid of CA cert issue but post another issue when tried to browse for restore on exch node2 - "server (master server hostname) does not contain any backups for client (node2 hostname) using the specified type (MS-Exchange-Server) as requested by client (node2 hostname)".

Active node is node1. 

 

 

Yet
Level 5
Partner Accredited

addition, those logs were produce by attemping to restore from Master Server Admin Console with DAG hostname as source & destination server.  I presusmed since node1 is the active node hence it's trying to work from there and not in node2. 

You need to discuss this with your Exchange admins:

[77dc] 01/10/19 17:31:33  AutoDiscover Failed in locating the url  The Autodiscover service couldn't be located.

EWS Autodiscover is an Exchange feature.

When Autodiscover fails, it seems the following tends to fail as well. Focus on Autodiscover.

[77dc] 01/10/19 17:31:33  Retrying using CAS server urls  from Monad ......

[77dc] 01/10/19 17:31:34 Could not use...... webserviceurl <xxx> request failed. The underlying connection was closed: An unexpected error occurred on a receive.

Other notes:

DARM entries are not case sensitive. For browsing the DAG from node2, you may need to add entries where Application host is node1 and Component host is node2, if NetBackup resolves the DAG to node1.

Never have your client version ahead of your media server version or your media server version ahead of your master server version.

The following is because your backup client is EXDAG01, not node2.

>"server (master server hostname) does not contain any backups for client (node2 hostname) using the specified type (MS-Exchange-Server) as requested by client (node2 hostname)"

I'm not sure what you mean by "Active node is node1." A DAG doesn't have active and passive nodes. Databases have active and passive copies on mailbox servers. If NetBackup resolves EXDAG01 to node 1, then you may need the DARM entry noted above.

 

I was just reminded in another customer case of this TechNote from Backup Exec. This could apply to your case if you have TLS 1.0 and 1.1 disabled, because NetBackup and Backup Exec share the components that communicate with PowerShell and EWS.

https://www.veritas.com/support/en_US/article.100040690

Mostly it's about needing BE builds 20.x or newer to disable TLS 1.0 and 1.1. That translates to NetBackup 8.1.2. At the very bottom it says that even with the latest Veritas builds, EWS won't work. Our TechNote from November 2017 ends with "Currently Microsoft does not recommend disabling TLS 1.0 on the Exchange server even when TLS 1.1 or 1.2 is enabled."

I'm not a communications security expert. Someone else would have to advise you on whether TLS is an issue with your versions of Windows and Exchange.

Yet
Level 5
Partner Accredited

Yes, looks like my case.

If I got it right, restore is failing because TLS 1.0 is disabled at Exch node? 

Reading some blogs, looks like disabling TLS 1.0 started on Exch 2016 CU8 and is a security requirement on some environment, going for TLS 1.2 instead. 

If this is the case does NBU 8.1.2 has some workaround? 

In regards to my "restore from exch node" issue related to this, it will still fail because of EWS failure, right?

I found a Microsoft TechNet posting from last week with instructions on how to make it work. I have sent an e-mail to a Veritas development engineer asking whether BE / NetBackup works with the version of .NET cited in the Microsoft answer. I will update this thread when I learn more.

Yet
Level 5
Partner Accredited

Thnaks a lot Lowell.

I have this already open with VERITAS. AutoDiscover issue has been sorted out but EWS credential issue still remains. We closed today at the point where Support advising to open a case with Microsoft, they find all configurations in place and yet issue still exist. they can no longer point out the exact problem. 

Could you kindly share the technote you mentioned?

This is the Microsoft TechNet article, but I caution that I don't have verification that BE or NetBackup will work with it without a code change. The development engineer replied that they need to test it. At a minimum you need NetBackup 8.1.2. You may have to wait for a future release, depending on whether we need a change that's too extensive for an EEB.

https://social.technet.microsoft.com/Forums/exchange/en-US/f660c4b2-a2dc-4919-81f1-2efd46af8d4c/mana...

Yet
Level 5
Partner Accredited

Thanks Lowell.

A case with Microsoft has also been open, as advised by VERITAS Support. 

Will see what MS will reply. 

I'll keep posting...

 

Got the following reply from the development engineer:

We haven’t had a chance to test with the latest changes which the [Microsoft] Exchange team is suggesting. However going through the link and seeing that we don’t explicitly specify the protocol to use in [NetBackup / BE] code, setting the registry should work. Since we don’t target [our code] to 4.7.1, setting that key is necessary to enable TLS 1.2. The side effect of this will be that since this is a system wide setting any other app using 4.x will also use this.

My summary:

Microsoft’s solution could work but we haven’t tested it yet and it could have side effects in other applications.

Yet
Level 5
Partner Accredited

Looks like the Microsoft technote you shared is what Microsoft Support is going to. Got update that Microsoft Support is evaluating the situation as there's an internal cases related to this open by numerous customers worldwide. Will know more in the coming days.

On the side, we're evaluating the application of new registry setting with Security Team, if al party agrees we'll apply it.

Will get back soon.

Yet
Level 5
Partner Accredited

Microsoft came back updating Exchange to CU11, did some testing (attached) and confirmed everything wtih AutoDiscover and EWS working fine. However, restore still failing with same issue.

Will see what VERITAS come up with, we'd open case.

Get back soon.