cancel
Showing results for 
Search instead for 
Did you mean: 

Restore Exchange 2010 Failes - Status 2810

nocoffeeman
Level 3

Hi.

We recently upgraded from netbackup 7.6.1 to 7.7.3 and after that mail restore has stopped working.
Runnin Exchange 2010
We are getting the following error:

02/14/2018 09:29:47 - Info bpbrm (pid=8288) DAG2.HEDMARK.ORG is the host to restore to
02/14/2018 09:29:47 - Info bpbrm (pid=8288) reading file list for client
02/14/2018 09:29:48 - connecting
02/14/2018 09:29:48 - Info bpbrm (pid=8288) start nbfsd on client
02/14/2018 09:29:49 - begin Restore
02/14/2018 09:29:50 - restoring from image dag.hedmark.org_1518549437
02/14/2018 09:29:50 - Info bprd (pid=8256) Found (16 424) files in (1) images for Restore Job ID 637471.xxx
02/14/2018 09:29:50 - Info bprd (pid=8256) Restoring from copy 1 of image created 02/13/18 20:17:17 from policy dag.hedmark.org
02/14/2018 09:29:50 - Info bprd (pid=8256) Granular restore has started, this may take a while....
02/14/2018 09:29:50 - requesting resource @aaaa5
02/14/2018 09:29:50 - granted resource MediaID=@aaaa5;DiskVolume=PureDiskVolume;DiskPool=NB-media1-msdp;Path=PureDiskVolume;StorageServer=nb-media1.hedmark.org;MediaServer=nb-media1.hedmark.org
02/14/2018 09:29:52 - Info bpbrm (pid=8288) start nbgre on client
02/14/2018 09:29:54 - Info tar32 (pid=0) Restore started
02/14/2018 09:29:54 - connected; connect time: 0:00:00
02/14/2018 09:29:59 - Info bpbrm (pid=8288) Restore executed on server DAG2
02/14/2018 09:30:24 - Error bpbrm (pid=8288) from client DAG2.HEDMARK.ORG: ERR - unable to create object for restore: \\dag\Microsoft Information Store\InternDB\Database, rai error = 6
02/14/2018 09:30:27 - Error bpbrm (pid=8288) from client DAG2.HEDMARK.ORG: ERR - unable to create object for restore: \\dag\Microsoft Information Store\InternDB\Database\Larsen, Mads Melvold [madlar2], rai error = 6
02/14/2018 09:30:27 - Error bpbrm (pid=8288) from client DAG2.HEDMARK.ORG: ERR - unable to create object for restore: \\dag\Microsoft Information Store\InternDB\Database\Larsen, Mads Melvold [madlar2]\Øverst i informasjonslageret, rai error = 6
02/14/2018 09:30:27 - Error bpbrm (pid=8288) from client DAG2.HEDMARK.ORG: ERR - unable to create object for restore: \\dag\Microsoft Information Store\InternDB\Database\Larsen, Mads Melvold [madlar2]\Øverst i informasjonslageret\Innboks, rai error = 6
02/14/2018 09:30:29 - Error bpbrm (pid=8288) from client DAG2.HEDMARK.ORG: ERR - unable to create object for restore: \\dag\Microsoft Information Store\InternDB\Database\Larsen, Mads Melvold [madlar2]\Øverst i informasjonslageret\Innboks\[hedmark.org #58920] Får veldig mye spam <0000000079ea5bacdae7694aa848492b9a35c5ac070067815ec78c9c98439326eda7131f3e01000000e0869a000067815ec78c9c98439326eda7131f3e010000134be8c60000>, rai error = 6
02/14/2018 09:30:29 - Info tar32 (pid=0) done. status 0
02/14/2018 09:30:29 - Info tar32 (pid=0) done. status: 5
02/14/2018 09:30:29 - Info tar32 (pid=0) done. status: 5: the restore failed to recover the requested files
02/14/2018 09:30:29 - Error bpbrm (pid=8288) client restore EXIT STATUS 5: the restore failed to recover the requested files
02/14/2018 09:30:38 - restored from image dag.hedmark.org_1518549437; restore time: 0:00:48
02/14/2018 09:30:38 - end Restore; elapsed time 0:00:49
MS-Exchange policy restore error (2810)

Ive checked the eventlog on the server and this is what it says:

(Process w3wp.exe, PID 5480) "RBAC authorization returns Access Denied for user hedmark.org/Globale servere/Tjenester/Mail/DAG2. Reason: No role assignments associated with the specified user were found on Domain Controller DC4.hedmark.org"

Ive checked our service account and it seems to be okay using this article:

https://www.veritas.com/support/en_US/article.000085655

16 REPLIES 16

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Sometimes NBU upgrade changes NBU Client Service logon account to LocalSystem.

Have you double-checked the service account on all DAG nodes? 

I'm running NetBackup Legacy Client Service and NetBackup Client Service as the serviceuser. On both my DAGs.

So this dosent seem to be the problem.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I am out of ideas. 

Hopefully Exchange expert @Lowell_Palecek will be along soon. Smiley Tongue

(Many time zones west of @Marianne)

I've said this before. Anytime you see this in the job details:

>Error bpbrm (pid=8288) from client...

Look to the client log to see what's going on. In this case the client is nbgre.exe. For historical reasons, the log folder is ncfgre. The VxUL OID is 352. Another log that will help with your particular error is the <host>-EWSLog00.log file in the BEDS folder. This log is written by a component of NetBackup BEDS called the EWS Provider. The ewstrace.txt file doesn't apply here. That's an XML file written by the Microsoft EWS library.

With that background, I see this interesting thing on the job details you posted. The client is named DAG2.HEDMARK.ORG while the mailbox that can't be found is in a database under "dag": dag\Microsoft Information Store\InternDB\Database\Larsen, Mads Melvold [madlar2]. Is DAG2 the name of a mailbox server in a dag named "dag"?

You upgraded from 7.6.1 to 7.7.3. Already at 7.6.1, you should be putting your Exchange service account credentials in the client host properties. Then you may as well run the NetBackup services on the client as LocalSystem. At some point, it became a requirement to configure the credentials in the host properties. (Do this for every mailbox server in the DAG.)

After that, there are many possible reasons for the symptom reported, "cannot create object for restore ... dag\Microsoft Information Store\InternDB\Database\Larsen, Mads Melvold [madlar2]." Here are a few ideas that come to mind.

- The DAG or mailbox server "dag" isn't known on the server DAG2, the InternDB database doesn't exist in "dag," or the given mailbox doesn't exist in InternDB." I just dealt with a support case this week where the mailbox didn't exist. (The Exchange admins had changed the mailbox alias between backup and restore.)

- The BEDS component (monad) that makes the PowerShell calls to find the mailbox either can't get started or lacks the credentials to make the calls. Look for "monad" and "ex07" in the ncfgre log. A common problem is that the Exchange service account lacks the right to replace a process level token. That means that the credential token can't be inherited when nbgre starts monad.

- You have two DAGs in your forest with the same short name, distinguished only by using their FQDNs. There was a bug in the implementation to support IP-less DAGs, in 7.7.3, wherein NetBackup could select a node of the wrong DAG for GRT browse and restore.

Thanks for the info.

Ive now set the credentials on all the hosts.

Ive take a screenshot of our mailsystem. Our dag cluster is named dag.hedmark.org and we have 3 mailbox servers dag1, dag2 and dag3. Wich the DBs are mounted on DAG2.

Confirmed:
- Exchange credentials are defined in client host configuration for server "dag2".
- Database InternDB exists in DAG "dag" and it's mounted. What's more, the database is active on dag2.

Can you confirm that the target mailbox exists in the database, with display name "Larsen, Mads Melvold" and alias "madlar2"?

Do you see any error messages involving "ESE07" in the ncfgre log, for example, "ESE07 process not ready"? (I mistakenly said "ex07" in my earlier post.) ESE07 is an internal name for a monad.exe process.

Please post the files DAG2-EWSLog00.log and DAG2-monad.log from the NetBackup\logs\beds folder on dag2. (Change the extensions to .txt because some filters won't let .log through.)

Re-reading your original post, I notice now that the first error is not with the mailbox Larson but with the database itself:

>02/14/2018 09:30:24 - Error bpbrm (pid=8288) from client DAG2.HEDMARK.ORG: ERR - unable to create object for restore: \\dag\Microsoft Information Store\InternDB\Database, rai error = 6

Don't worry about the EWS and monad logs as in my preceding post yet. This error says we couldn't open the database, most likely the copy of the database on the backup image. We need the ncfgre log for this. As a shot in the dark, I suspect you are having troubles with the "nbfs" linking to the media server. Did your NFS configurations change since your restore last worked?

Please post the ncfgre log. It could be big. It would be helpful if you were to capture only one run.

We shouldent have done anything with the NFS config.
But I've moved the backup from one mediaserver to another.

I've ran the restore and made a new logfile. Its attached.

0,51216,352,352,983,1518682062497,10596,104,0:,149:Skipped by default because no filters match: \\dag\Microsoft Information Store\InternDB\Database\Øvregaard, Ivar [k00pivo] (../GRECallback.cpp:2509),24:GRECallback::objProduced,1
0,51216,309,352,3088,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3089,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3090,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,352,352,984,1518682062497,10596,104,0:,161:Skipped by default because no filters match: \\dag\Microsoft Information Store\InternDB\Database\Øyen, Irene Nyberget [irene.nyberget] (../GRECallback.cpp:2509),24:GRECallback::objProduced,1
0,51216,309,352,3091,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3092,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3093,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3094,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3095,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3096,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3097,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
0,51216,309,352,3098,1518682062497,10596,104,0:,72:1 non-object reasons still holding the blocker (../NcfRefCounted.cpp:79),24:NCFBlockUntil::logStatus,1
1,51216,309,352,3099,1518682062512,10596,104,0:,0:,38:ConfigDataIterator::metadataIdCallback,3,(35|A21:3:Error_Messages_Max;|A2:10|)
1,51216,309,352,3100,1518682062512,10596,104,0:,0:,38:ConfigDataIterator::metadataIdCallback,3,(35|A18:3:EXCHANGE_USERID;|A19:hedmark\backupadmin|)
1,51216,309,352,3101,1518682062512,10596,104,0:,0:,38:ConfigDataIterator::metadataIdCallback,3,(35|A20:3:EXCHANGE_PASSWORD;|A128:a5faeefb5da59eec35ed08e7e294ac9465cc530b12ff7288927390228ebf3308a0e62665ff64b9185bc2750b7633e33da151dccfd7e431ce3356a5f7dc8e08a2|)
1,51216,309,352,3102,1518682062512,10596,104,0:,0:,38:ConfigDataIterator::metadataIdCallback,3,(35|A24:3:EXCHANGE_PASSWORD_KEY;|A32:53b78d4c69da267c50521f209193281f|)
1,51216,309,352,3103,1518682062512,10596,104,0:,0:,38:ConfigDataIterator::metadataIdCallback,3,(35|A25:3:CLIENT_CONNECT_TIMEOUT;|A3:300|)
0,51216,309,352,3104,1518682062512,10596,104,0:,93:Processing ProcessDebug:nbgre; using the Window's registry (../DefaultConfigProvider.cpp:345),40:DefaultConfigProvider::getValueNonBpConf,1
1,51216,309,352,3105,1518682062512,10596,104,0:,0:,40:DefaultConfigProvider::getValueNonBpConf,3,(36|S20:\HKEY_LOCAL_MACHINE\|S61:SOFTWARE\Veritas\NetBackup\CurrentVersion\Config\ProcessDebug|S5:nbgre|D32:2|)
0,51216,309,352,3106,1518682062512,10596,104,0:,93:Processing Granular:CopyLocal; using the Window's registry (../DefaultConfigProvider.cpp:345),40:DefaultConfigProvider::getValueNonBpConf,1
1,51216,309,352,3107,1518682062512,10596,104,0:,0:,40:DefaultConfigProvider::getValueNonBpConf,3,(36|S20:\HKEY_LOCAL_MACHINE\|S57:SOFTWARE\Veritas\NetBackup\CurrentVersion\Config\Granular|S9:CopyLocal|D32:2|)
1,51216,309,352,3108,1518682063997,10596,104,0:,0:,14:nbfs_unmount(),0,(28|A33:INF - Lets terminate the event...|)
1,51216,309,352,3109,1518682063997,10596,104,0:,0:,14:nbfs_unmount(),0,(28|A25:INF - Event terminated...|)

 

Edit: The  "Server for NFS" service was running and enabled on one of our media servers. I turned this off. But still the same error.

The log is at very low verbosity. Could you do it again with the logging turned up? There is a logging level in host properties under Windows Client / Client Settings. Set it to 2. Set the global logging level to 5 for the test.

With the log provided, I find the following issue:

>2/15/2018 09:07:36.840 [ResourceChildBEDS::_attach()] SpsOverwrite:True

SPS is SharePoint. This entry shouldn't appear in an Exchange restore log. We had a bug in the master server NetBackup 7.0 to 7.7.2 that could cause this, but it was fixed in 7.7.3. I'd like to see your bprd log for a restore, with the logging level at 3 or higher.

Note: You may have a different experience between starting the restore from the NetBackup Console (the "Java GUI") and the Windows BAR application (the "BAR GUI"). In particular, if there is a bprd issue causing SPS option corruption, it's likely to affect only restores started from the Java GUI.

Here is the first error reported in the log. It comes before anything you've reported.

>2/15/2018 09:07:36.840 [Object::create()] FS_CreateObj() Failure! (0xE0000388:Cannot log on to EWS with the specified credentials. Review the resource credentials for the job, and then run the job again.)

However, it comes after the unexpected SpsOverwrite entry. Let's address that first.

Summary:
- Restore that fails started from the Java GUI may work if started from the Windows BAR GUI.
- We need a level 3 bprd log from the master server, and an ncfgre log with the Windows client logging at level 2 and the global logging level up at 5.

I'm only using the Windows BAR while restoring, have not tried via Java yet. But I can give that a shot.

Thanks for helping me out here, and sorry for the delay in response. We must be i different timezones.

So I've got the logfiles. They are huge, so i zipped them.

Bump :)

Is there any way I can restore a whole mailbox to a .pst file?

Systems_Team
Moderator
Moderator
   VIP   

Hi NoCoffeeMan,

You can, but that is an Exchange function, not NetBackup.  I'm on Exchange 2013, so you'd have to double-check if there are differences for 2010, but below are some of the Exchange powershell cmdlets you'll need:

Get-ManagementRoleAssignment -Role "Mailbox Import Export"     (You need this Role to be able to import/export)
New-MailboxExportRequest -Mailbox RecoverMe@Company.com -FilePath \\Server\share\mailbox.pst
Get-MailboxExportRequest | Get-MailboxExportRequestStatistics
(Restart Replication service if Export stuck in Queued state)

Hope this helps,

Steve

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
@nocoffeeman
Probably time to log a Support call with Veritas?

Unfortunately very few forum members have the time to sift through large log files.

Sorry for the delay. I was off-site Friday with no time to check here. Monday was a U.S. holiday, which I spent most of working on a hot support case in a completely different area. Please follow @Marianne's advice to start a support case. I may get time to look at your logs this week, I may not. Our support team in Europe is very good on Exchange, and they consult with me when they need to, so it's not really redundant.

Meanwhile, to get unstuck, you can restore the database to an RDB and extract the mail from there.

I got back to this today. (My hot support case is coming out happily.)

Never mind what I said about SpsOverwrite. That's ok. The bprd log you provided does not show the issue I was fishing for, and the more verbose ncfgre log you provided shows "[GetSpsOptions] sps_options: 0". This is what we want on an Exchange restore, and what we once had an issue with.

Which brings us back to this:

FS_CreateObj() Failure! (0xE0000388:Cannot log on to EWS with the specified credentials. Review the resource credentials for the job, and then run the job again.)

The ncfgre log shows that you have your service account configured as needed. Nbgre starts as backupadmin@HEDMARK, which is a member of Organization Management. What's more, the thread that needs access to your Exchange system successfully logs on as backupadmin@HEDMARK with the credentials you provided in the client host configuration.

The process called "monad" started up successfully and got the backupadmin credentials.

So, why didn't the EWS logon work? I think you need to take this up with your Exchange admin. Microsoft provides a tool called Test-WebServicesConnectivity. See https://technet.microsoft.com/en-us/library/aa998328(v=exchg.160).aspx. Veritas Support folks can help you with this tool.