cancel
Showing results for 
Search instead for 
Did you mean: 

Exchange 2013 backup failure with status 72 after Exchange CU22 update was done

eccio
Level 4

Hi all.

Anybody faced already with a Exchange 2013 DAG backup failure after an update on Exchange was done to version CU22?

After the update the backup started to fail with error 2 

08.mar.2019 04:12:42 - Error bpbrm (pid=9564) You cannot use the backup directive for Database Availability Groups for an Exchange Standalone backup
08.mar.2019 04:12:42 - Info bpresolver (pid=44088) done. status: 72: the client type is incorrect in the configuration database

 

Client/Master = Master
NetBackup Client Platform = PC-x64, WindowsXP
NetBackup Client Protocol Level = 8.1.0
Product = NetBackup
Version Name = 8.1
Version Number = 810000
Client OS/Release = Windows2012 6

Is there any Netbackup update, fix or workaround?

According the last compatibility list released on 7th of March CU22 is not supported.

thanks for any advice

20 REPLIES 20

Systems_Team
Moderator
Moderator
   VIP   

Hi Eccio,

Unfortunately I think you're stuck.  That's why it is critical to check the compatibility lists before making environment changes.  There won't be any update, fix or workaround until CU22 is a supported configuration.

I guess your two main options are:

  • Log a call wiith Veritas support and see if you can get any information as to when CU22 will be supported.
  • Rollback to your prior Exchange CU (haven't tried that on 2013, so don't know how possible it is).

One other temporary option might be to use Windows Server backup in the meantime, until either CU22 is supported or you rollback to a prior CU (if possible).

Good Luck,

Steve

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I agree with @Systems_Team . I am not aware of a workaround if the Exch version is unsupported.

Weird thing is that this version is no longer acknowledged by NBU as a DAG, but rather as Standalone.

Hopefully @Lowell_Palecek will be along soon with thoughts/advice.... 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

The error message in NBU seems to be a result of insufficient priveleges as per this TN (although it is for different Exch version): 
https://www.veritas.com/support/en_US/article.100003563

My guess is that user/admin requirements may have changed in the Exch update.

Hi Steve. 

Thanks you for your reply.

unfortunately the update was done without notyfying me, so I had no chance to stop them to do the update.

Hi Marianne.

For me it was as well an access right issue initially, but I'm not able to find more information in the logs.

bpresolver log file is empty

eccio
Level 4

BTW: these two changes were made on the permissions, not possible to revert

Reducing permissions required to run Exchange Server by using Shared Permissions Model

https://support.microsoft.com/en-us/help/4490059/using-shared-permissions-model-to-run-exchange-serv...

and

Exchange Web Services Push Notifications can be used to gain unauthorized access

https://support.microsoft.com/en-us/help/4490060/exchange-web-services-push-notifications-can-provid...

**corrected on proofreading**

I am not able to pre-announce other teams' plans. All I have from the Exchange folks is that they are working on qualifying 2013 CU22, as well as other current versions.

I can look into the bpresolver message. The bpresolver failure is the cause of the message in bpbrm about not being able to use a DAG backup directive for a non-DAG server. Let's start by taking the bpresolver message at face value. Are you sure the policy client type is Exchange?

I generally expect any changes from Microsoft to only affect GRT backups, if at all. For example, if the EWS change in one of the links you gave matters, it would matter only for GRT restore.

The shared permissions issue is new to me. I will have to wait to see what the QA and development team's come up with to qualify the new CU versions (and Exchange 2019).

Please get a bpresolver log with the client logging levels set at 5 and 2, and the relevant part of a bpbrm log with the media server logging level set to at least 3.

As I mentioned above, bpresolver logs are empty even if the loging level on exchange client is set to 5

attached the policy settings

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@eccio 

According to the job details snippet in your opening post, bpresolver ran on one of the nodes:

08.mar.2019 04:12:42 - Info bpresolver (pid=44088) done. status: 72: the client type is incorrect ...

There should be something else in Job details to tell you on which node bpresolver ran. 

Did you check all nodes? Especially the 'Preferred nodes': 
Exchange DAG Preferred Server: SOSLMLS06 SOSLMLS05 

Bpresolver executes on the Exchange server that represents the DAG.

If the DAG has an IP address shown in the Failover Cluster Manager, the DNS resolves the DAG to the server that has that address.

From Exchange 2013 forward, IP-less DAGs are prevalent. (I think in Microsoft parlance, the DAG has no administrative access point.) To handle this case, NetBackup discovers a mapping of all the Exchange servers in a DAG. The mapping is actually reverse. The NetBackup database on the master server has and XML file for each Exchange server that reports the DAG to which the server belongs. NetBackup master server components nbjm and bprd that execute bpresolver, choose the Exchange server that has most recently sent its discovery information to the master server. (Bpbrm also can execute bpresolver, but the caller of bpbrm supplies the server name.)

To find out where bpresolver executed you can do one of the following. The first is easiest.
- Check all the Exchange servers in the DAG on which you have NetBackup client installed.
- Check the log of the process that called bpresolver - nbjm for backup and bprd for restore.
- Check which "exchange" file in NetBackup\db\discovery\ is newest at the time of the job in question.

Also check the bpbrm log for entries related to starting bpresolver. Sometimes it does so when it's been given an Exchange client name, to check whether the server belongs to a DAG.

Sometimes bpbrm doesn't check for DAG membership if it thinks it already knows the answer. That could be why you're not finding a bpresolver log. The other processes I mentioned do not call bpresolver for a standalone Exchange server.

Here's what to check in that regard. In order to not be repeatedly asked "is this Exchange server part of a DAG," bpresolver saves the answer in the Registry. Look for the value Exchange_DAG_Info under the key HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Agents\Exchange. It may not exist, but if it does its value should be 3 for a DAG, 2 for a standalone server.

If the Exchange_DAG_Info value exists, try deleting it and re-running your backup.

In addition to all that, I wonder what you're specifying as the client name in your backup policy. You should be using the DAG name to back up Exchange databases in a DAG. I think it's allowed to specify an individual server name, but you shouldn't. I don't remember the reason this was allowed.

Your backup selection should begin with "Microsoft Exchange Database Availability Groups:\" for a DAG, or for a server that's in a DAG. For a standalone Exchange server that's not in a DAG, your backup selection should begin with "Microsoft Information Store:\"

In addition:

today I found out, that is not possible to perform GRT restore from the previously successfuly taken backups.

It is possible to browse the DB and mailbox structure, but it is not showing any content.

Getting error: no entity was found

Let's look into the error you're getting. First, let's make sure I understand.

You can open the backup image in either the BAR window of the NetBackup console (the "Java GUI") or the Windows BAR application (the "BAR GUI").

You can navigate as far as the mailboxes, but when you click to drill into a mailbox, you get an error. Probably "Database system error" 220.

This means the backup catalogued the mailboxes successfully. Before continuing, look at the mailbox aliases. Are they correct, or do you see GUIDs where the aliases should be. Correct would look like "Jill User [juser]". Incorrect would look like "Jill User [<long hex number>]".

The mailbox names you see, whether good or bad, were catalogued at backup time. Drilling into them requires something called "live browse." In this, bpdbm on the master server invokes nblbc on the destination client. The error you got means this didn't work.

Start with the bpdbm log. It should be at level 3 or higher. Look for occurrences of Q_IMAGE_LIST_FILES. Any bplist action, whether internal or external will produce such a query to bpdbm. As you browse through the image, every click you make produces this query. For each one, you will see a "pattern" logged on a line that begins with ImageListFiles::executeQuery. Follow the progression until the pattern is the mailbox name. Focus on that pid.

When you get that far, post what you find.

BTW, Veritas updated the Software Compatibility List yesterday to claim support for Exchange 2013 CU22 and Exchange 2016 CU12. Start of support is given as NetBackup 8.0, but 8.0 needs an EEB for Exchange 2016 as stated in the footnote.

Correct, I'm able to open the backup images in Netbackup console.
I'm able to navigate up to the mailbox level, but than not possible to drill into a mailbox.
Actually I'm getting two types of error messages, depending on what image I open.
1. Database system error (see attached bpdbm log file 1)
2. no entity was found (see attached bpdbm log file 2)

The mailbox names and aliases are OK, I see the correct names not GUIDs.

I saw the new compatibility list as well, but it is still Exchange CU22 with NBU 8.0. No mention about 8.1

The compatibility list entry means CU22 is supported from NetBackup 8.0 forward.

In the first bpdbm log, these lines show nbfsd being started on the media server and nblbc on a client:

>dbm_start_nbfsd_and_open_nblbc_connection: Invoking nbfsd on media server soslbck001

>bpcr_get_fileinfo_rqst: bpcr read status 0 from bpcd

The client on which nblbc is executed is soslmls07.internal.financecorp.biz:

>fl_peername soslmls07.internal.financecorp.biz

But then it failed:

>bpcr_get_string_pentry: Error detected: ERROR 13

This error from nblbc becomes the 220 error from bpdbm. Look for an ncflbc log on the client to investigate further.

The second bpdbm log is similar, with these differences:

- The backup client name is soslmlc01 rather than soslmls07. (The destination client is still soslmls07. That's ok. Look for the ncflbc log on soslmls07 for this case, too.)

- The backup time is from Feb 28 rather than Mar 19.

- A range of backup times were given. After nblbc failed for the first image, other images were checked and found not to contain that database. It's because of the additional checks that the bpdbm status changed to 227.

Next step, look at your ncflbc log. It's in VxUL format.

I think your next step really is to open a support case.

Hi Lowell.

I checked the ncflbc log and found some errors inside.

Sample attached. If you don't mind to take a quick look.

thanks

Marian

 

File vxul_readable.txt is not very readable. I recommend you try vxlogview. I generally cd to NetBackup\bin, then:

>vxlogview -i 351 >..\logs\ncflbc\nblbc.txt

This command needs the filename to be in its original VxUL form and the logs to be in their expected location relative to your cwd. For logs that are not in NetBackup\logs folders, use the -G option. If you need to preserve pid and tid info at the cost of a somewhat less readable output, use "-d all".

All that notwithstanding, here's your problem:

>1,51216,309,351,787,1553253521739,15392,22360,0:,0:,12:nbfs_mount(),2,(28|A138:INF - running command - "C:\Program Files\Veritas\NetBackup\bin\nbfs" mount -server soslbck001 -port 7394 -timeout 60 -cred 093468E7A6...|)

>1,51216,309,351,791,1553253583098,15392,22360,0:,0:,12:nbfs_mount(),2,(28|A61:INF - response - aborting connection, timeout period exceeded|)

1553253521739 and 1553253583098 are timestamps in milliseconds. There's a 61 second difference.

You can look at the nbfsd log on the client for hints why the mount failed. My guess is that either you don't have Client for NFS running on the client, or you haven't configured NFS on the media server.

I do recommend that you work with Veritas Support to sort out your multiple issues.

// Casual followers note that the topic has drifted from backups failing to GRT browse of previously successful backups failing.