cancel
Showing results for 
Search instead for 
Did you mean: 

Exchange 2013 GRT Restore fails- need help troubleshooting

gkman
Level 5

Hello,

I am trying to restore a few Items of a users calendar back into his mailbox (original location)
if I try triggering the restore through the "administration console" from the "backup, archive and restore tree", a error window pops up saying "Invalid command usage. a failed restore job appears in the activity monitor with either error 130 or 50 (can't figure out what's the change). no information is shown in the details status.

However if I try triggering the restore from the legacy "backup, archive and restore" (nbwin.exe), a job successfully starts but is stuck on the restore, no error message and in the details status the last update in "info bprm (pid=134244) restore executed on server..", no kilobytes written or anything.

the only bpbrm log I found was located on our media-server (appliance), and the last two lines are:

12:50:01.707 [178955.178955] <2> bpbrm write_msg_to_progress_file: (1406836.001) MNR - error writing file: %1
12:50:22.127 [134244.134244] <2> bpbrm main: Could not send keepalive to bprd

the time now is 13:30. (I canceled and restarted the job a few times)

what else can I try? this is the first time ever this has happened. previous restore attempts worked properly.

thanks in advance for the help

1 ACCEPTED SOLUTION

Accepted Solutions

Status 5 is the general code for failed restore.  These are changed into 28xx series statuses, where xx depends on the client type. For Exchange, all status 5 results become 2810.

There is no tar log for an Exchange GRT restore. The restore is performed by nbgre. Just how big did your ncfgre log get? If you look at the last several MB of it, what was it doing at the time of failure? Can you find an error in it?

At the start of this thread, you said you were trying to restore some calendar items. Then you posted a log snippet that shows Inbox messages being restored. What are you selecting for restore? GRT is a bad way to try to restore an entire mailbox. It's good for restoring a few messages, or a folder or two. For large-scale restores, you should restore the database to an RDB and use Exchange tools to pick what you need out of the RDB.

If you continue to have troubles, I suggest opening a case with Veritas support. They can help you analyze your NetBackup logs.

 

View solution in original post

12 REPLIES 12

Lowell_Palecek
Level 6
Employee

If bpbrm can't send a 1-byte message to bprd, then this isn't an Exchange question. It's why can't my media server talk to my master server?

What has changed in your network from when the restore worked until it didn't work?

Did this bpbrm pid log anything of use before that error? Did it get far enough to start nbfsd and nbgre?

Our security team did start going over firewall rules and making massive changes.
it could very well be something there, however since only this specific restore job didn't work (other restore jobs and backups work) I didn't think it had much to do with them. 

what port numbers should I check?

A list of ports for 6.x and 7.x can be found on:
https://www.veritas.com/support/en_US/article.000008025

And can be downloaded for 8.0 on:
https://www.veritas.com/support/en_US/article.000116400

Iirc, the most used ports are:
TCP 13724, 13720, 1556, 111, 3794, 13786, 13782
UDP 162

Hi again.

sorry for the late reply- I had a few urgent matters to see to and this restore problem got pushed aside.

I have yet to determined the reason for this issue- as it turns out it is not a firewall problem.
I had tried performing now a new restore. the restore job managed to restore 2 emails (out of 4) when it got stuck (the job is still running but with no progress).

the restore job (in the master server management java GUI BAR section ) showing:

 

Progress log filename : C:\Program Files\Veritas\netbackup\logs\user_ops\DOM\rinata\logs\jbp19AA.tmp.log Restore Job Id=1452702
Restore started 01/16/2017 10:27:42
10:38:54 (1452703.001) UTF - \\DAG01\Microsoft Information Store\DB30\Database\Restore Mailbox [RestoreMailbox]\Top of Information Store\Inbox\RE: ׳�׳“׳™׳” ׳’׳•׳¨׳� ׳�׳�׳� 283 <00000000c630a0b878920e4fb1aa0e60e6cc65550700ff25bdddba8f294d969563a5e43a81ef000000a3a95400009473fbba39ae674689dfcc8c79d90e2c0000da8de7ba0000>
10:38:54 (1452703.001) TAR - \\dag01\Microsoft Information Store\DB01\Database\׳�׳�׳•׳™׳�׳� ׳¨׳™׳ ׳× - AMOYAL RINAT [Rinata]\Top of Information Store\Inbox\RE: ׳�׳“׳™׳” ׳’׳•׳¨׳� ׳�׳�׳� 283 <00000000c630a0b878920e4fb1aa0e60e6cc65550700ff25bdddba8f294d969563a5e43a81ef000000a3a95400009473fbba39ae674689dfcc8c79d90e2c0000da8de7d00000>
10:38:54 (1452703.001) MNR - The file was renamed to the following:
10:38:54 (1452703.001) UTF - \\DAG01\Microsoft Information Store\DB30\Database\Restore Mailbox [RestoreMailbox]\Top of Information Store\Inbox\RE: ׳�׳“׳™׳” ׳’׳•׳¨׳� ׳�׳�׳� 283 <00000000c630a0b878920e4fb1aa0e60e6cc65550700ff25bdddba8f294d969563a5e43a81ef000000a3a95400009473fbba39ae674689dfcc8c79d90e2c0000da8de7d00000>
10:38:54 (1452703.001) TAR - \\dag01\Microsoft Information Store\DB01\Database\׳�׳�׳•׳™׳�׳� ׳¨׳™׳ ׳× - AMOYAL RINAT [Rinata]\Top of Information Store\Inbox\ERROR_VNX5400 <00000000c630a0b878920e4fb1aa0e60e6cc65550700ff25bdddba8f294d969563a5e43a81ef000000a3a95400009473fbba39ae674689dfcc8c79d90e2c0000da8de7fc0000>
10:38:54 (1452703.001) MNR - The file was renamed to the following:
10:38:54 (1452703.001) UTF - \\DAG01\Microsoft Information Store\DB30\Database\Restore Mailbox [RestoreMailbox]\Top of Information Store\Inbox\ERROR_VNX5400 <00000000c630a0b878920e4fb1aa0e60e6cc65550700ff25bdddba8f294d969563a5e43a81ef000000a3a95400009473fbba39ae674689dfcc8c79d90e2c0000da8de7fc0000>
10:38:55 (1452703.001) TAR - \\dag01\Microsoft Information Store\DB01\Database\׳�׳�׳•׳™׳�׳� ׳¨׳™׳ ׳× - AMOYAL RINAT [Rinata]\Top of Information Store\Inbox\ATLAS: Logical Disk: Disk space on C (%) is low <00000000c630a0b878920e4fb1aa0e60e6cc65550700ff25bdddba8f294d969563a5e43a81ef000000a3a95400009473fbba39ae674689dfcc8c79d90e2c0000da8de8020000>
10:38:55 (1452703.001) MNR - The file was renamed to the following:
10:38:55 (1452703.001) UTF - \\DAG01\Microsoft Information Store\DB30\Database\Restore Mailbox [RestoreMailbox]\Top of Information Store\Inbox\ATLAS: Logical Disk: Disk space on C (%) is low <00000000c630a0b878920e4fb1aa0e60e6cc65550700ff25bdddba8f294d969563a5e43a81ef000000a3a95400009473fbba39ae674689dfcc8c79d90e2c0000da8de8020000>
10:38:55 (1452703.001) MNR - error writing file: %1

 

the log seems to be is referring to a folder names ATLAS- but there is no such folder in neither the source or the destination mailboxes. also disk C: is referring to which server? 

I am still unclear on how to troubleshoot this. in what logs should I even look in?

thanks for the help

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

ATLAS seems to be a mail item inside the user's Inbox that cannot be restored due to diskspace on C:\ running out.

You need tar log folder on the destination client to troubleshoot restore - preferably level 3 or higher. 

Check disk space on C as NBU client is probably also installed on C-drive. The higher the logging level, the more space will be needed.

As Marianne said, there seems to be an issue with the C: disk on your NBU client server. The server that is referred to, is the one who is hosting DB30 in your DAG group (you should verify this on exchange console or through exchange powershell commands like get-mailboxdatabase).

There is a high probability that your restore will continue if you fix the problem with the C: disk.

how much space is required on c:? there is 10gb free.
it also restored 2 items already, all of the sudden its not enough space?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Don't shoot the messenger.... 

The amount of free space needed is unknown to me. This will hugely depend on the size that the original mailbox was. If we are talking about a mailbox of (god forbid!) 1TB, you will need more than 1 TB free space.

And yes, you have this sudden need for more space because you are doing a GRT restore: item per item (so mail per mail) is restored, not the whole mailbox in one go.

Hold on. NetBackup isn't saying your disk space is low. That's in the title of the message that was last logged.

I haven't seen any mention of what you find in the client log. The client process is nbgre.exe. The log goes in the ncfgre folder. There are two files with "EWS" in their names in the NetBackup\logs\beds folder. One is from NetBackup / BEDS, and the other is an XML trace file produced by Microsoft's EWS API.

Back to disk space. If you turn on the logs I mention, you'll use more disk space on the client. (You didn't mention which system(s) have this disk limitation, whether server or client.) 10 GB is tiny. When I let the free space on my computer get that small, it pretty much stops working, and it's not an Exchange server. Can you free up enough space so the OS will function?

I increased the disk size and now have 50GB free. the restore job was still stuck so I canceld and started a new restore.

It restored much more emails this time around, but eventualy failed with error 2810 (the parent job is status 5)

I looked at the 'tar' log folder and there is no log from today at the client, media server or the master server (I have changed the logging to 3 on all servers and clients).
the ncfgre does have a log from today but I did not find any useful information (it's so long I kind of got lost in the log). I tried uploading it, but it exceeds the size limit. 

also- I have noticed that when I started restoring the mailbox- the free disk space on c: dropped by 2.5 GB- but after the job finished it did not reclaim the space. does the clean up need to be performed manually? 

thanks again for the help everyone. 

Status 5 is the general code for failed restore.  These are changed into 28xx series statuses, where xx depends on the client type. For Exchange, all status 5 results become 2810.

There is no tar log for an Exchange GRT restore. The restore is performed by nbgre. Just how big did your ncfgre log get? If you look at the last several MB of it, what was it doing at the time of failure? Can you find an error in it?

At the start of this thread, you said you were trying to restore some calendar items. Then you posted a log snippet that shows Inbox messages being restored. What are you selecting for restore? GRT is a bad way to try to restore an entire mailbox. It's good for restoring a few messages, or a folder or two. For large-scale restores, you should restore the database to an RDB and use Exchange tools to pick what you need out of the RDB.

If you continue to have troubles, I suggest opening a case with Veritas support. They can help you analyze your NetBackup logs.