Unable to expand mailboxes with GRT restores

[CadenL]

e have a single Exchange 2016 server running as a virtual machine which we backup using a vmware policy with Exchange GRT and log truncation enabled. The backup completes fine as does the ASC but we're unable to expand the database to see the individual mesages when we request the restore (either from the master server or directly from the client)

The Exchange server has the NFS client, version 8.1 of the NBU client and the Veritas VSS installed and the Master server is an NBU Appliance running 3.1. All the Exchange impersonation permissions have been set and the credentials have been added to both the NBU client services on the Exchange server and in the client properties in the NBU Java console.

When we attempt to expand a users mailbox to see the messages we get and error saying 'System Database Error' and we can't see any messages. We get the same error when on the exchange client BAR or the master server BAR.

Below is an error we see in the nbflc log which we feel is relevent but we cannot find out anything more as to how to troubleshoot this error or what file it's looking for.

0,51216,158,351,2352,1519136187215,11312,16848,0:,94:<from Producer::iterateResource> VDiskMounter::UnMountAllDisks called (../BEDSContext.cpp:159),20:[mounter] ,4
0,51216,158,351,2353,1519136187215,11312,16848,0:,92:<from Producer::iterateResource> VDiskMounter::~VDiskMounter called (../BEDSContext.cpp:159),20:[mounter] ,4
0,51216,158,351,2354,1519136187215,11312,16848,0:,94:<from Producer::iterateResource> VDiskMounter::UnMountAllDisks called (../BEDSContext.cpp:159),20:[mounter] ,4
0,51216,158,351,2355,1519136187215,11312,16848,0:,100:FS_AttachToDLE() Failed! (0x2:The system cannot find the file specified.) (../ResourceChild.cpp:674),28:ResourceChildBEDS::_attach(),1
0,51216,158,351,2356,1519136187215,11312,16848,0:,56:--- EXITING 00000000002FE348 --- (../RAIProducer.cpp:78),23:RAIChildContext::attach,6
0,51216,309,351,846,1519136187215,11312,16848,0:,52:+++ ENTERING 00000000002FE988 +++ (../CFEObj.cpp:49),32:CFEObj::CFEObj(producer, marker),6
0,51216,309,351,847,1519136187215,11312,16848,0:,51:--- EXITING 00000000002FE988 --- (../CFEObj.cpp:49),32:CFEObj::CFEObj(producer, marker),6
0,51216,309,351,848,1519136187215,11312,16848,0:,60:+++ ENTERING 00000000002FEA98 +++ (../CFEObjProducer.cpp:67),34:CFEObjProducer::objProducerHandoff,6
0,51216,309,351,849,1519136187215,11312,16848,0:,68:1 non-object reasons added to the blocker

I think the NFS client is perhaps mounting the database ok but NBU then can't browse it as we also see the following warning in the event viewer on the exchange client.

Client for NFS requested a mount with file locking enabled, and the remote Network File System (NFS) server does not support file locking.

Client for NFS mounted the shared resource anyway. File locking may not be available.

To avoid this warning in the future, enable file-locking suppport on the remote NFS server.

One final point of note is that the Exchange databases are installed on GPT disks and so we do get a warning within the ASC job saying that GRT can't be used with GPT disks - BUT - we believe this is now supported and this error can be ignored following a Veritas update in the following post (please scroll to the bottom of the post)

https://vox.veritas.com/t5/NetBackup/Configuring-VMware-that-protect-Exchange-2010-DAG/m-p/824661

[Lowell_Palecek]

To see if it's a problem mounting the backup with NFS protocol, check all occurrences of "nbfs" in the ncflbc log.

The snippet of log is a narrow window. The fact that it's mostly UnMountAllDisks and the VDiskMounter::~VDiskMounter destructor suggests that you are looking quite a ways downstream from the actual error.

I can't answer for the significance of the NFS file locking issue. I suggest you use vxlogview on your ncflbc log to make it easier to compare timestamps with your Event Viewer log. Here's what I do:

>cd <Netbackup\bin>
>vxlogview -d all -i 351 >..\logs\ncflbc\nblbc.txt

Yes, we do believe we can support Exchange GRT from VMware backup with the database on a GPT disk.

If you don't find a solution quickly, I suggest engaging Veritas support. Those folks can give you more holistic help than I can.

[CadenL]

Thanks - I'll run the command as you've suggested.

Attached is a full copy of the log if you have time to review that would be greatly appreciated.

thanks

[Systems_Team]

Hi Caden_L,

I can confirm that the NFS warnings you see in the Event Viewer on Windows for clients with GRT can be ignored.  We see this on both GRT backups of Active Directory and Exchange, and have done every since we started using GRT.

You'll likely be getting two warnings - one in the Application event log and one in the System event log, and it only happens during GRT backup.

Application Event Log Warning

Windows(R) Lightweight Directory Access Protocol (LDAP) failed a request to connect to Active Directory Domain Services(R) for Windows user <DOMAIN\UserName>.

Without the corresponding UNIX identity of the Windows user, the user cannot access Network File System (NFS) shared resources.

Verify that the Windows user is in Active Directory Domain Services and has access permissions.

System Event Log Warning

Client for NFS requested a mount with file locking enabled, and the remote Network File System (NFS) server does not support file locking.

Client for NFS mounted the shared resource anyway. File locking may not be available.

To avoid this warning in the future, enable file-locking suppport on the remote NFS server.

Hope this helps,

Steve

[Lowell_Palecek]

The attempt to open the database quits after hitting this:

UpdateExchangePdi: unable to find any files in '\\?\E:\TEMP\NetBackup\MP00\E\Exchange Server\ACME Mailbox Database\E01*.log'

The database is on D:. The log doesn't say that's a problem. What can you tell me about E:?

Try browsing the image as a VMware image rather than Exchange. Can you browse the E: volume? I don't know whether you should see the path in the error message, but you should be able to find files in E:\Exchange Server\ACME Mailbox Database\.

[CadenL]

Thanks for looking into the logs for me - 

I can in fact browse the E:\ all ok when selecting a VMware type restore and can see log files resident in the E:\Exchange Server\ACME Mailbox Database\ folder.

The drive itself is the same as the D:\ drive which NBU seems to be able to read and mount ok. Both drives are GPT but I understand this to be ok now.

Are there any other logs that may perhaps show why NetBackup can't see any files on the E:\

thanks

[Lowell_Palecek]

We could check the bpfis log from the ASC job or cat_convert -dump of the Exchange image would show everything the ASC job catalogued. It can be done with bplist, too but it would need options for recursion and showing hidden entries. We would be looking to see that all the expected files and folders got catalogued under Exchange.

Otherwise, I would have to spend time with logs, code, and the developer who implemented mounting the VMware image for Exchange GRT. I won't get to that for a few days.

I'll ask the test engineer who tested Exchange GRT on GPT disks with VMware backups for the log files for comparison with yours.

[CadenL]

OK - thanks

I'll capture a copy of the bpfis log (I assume this is from the Exchange client) and also try a cat_convert -dump on the Exchange backup image - which I assume I can pipe out to a file.

I have noticed that our old Exchange 2010 server (from which all the mailboxes were migratedand only has a handful of old test mailboxes left on it) was configured the same as the new Exchange 2016 server and is still backed up via a vmware policy. It uses the same Exchange credentials but has MBR disks instead of GPT. I'll test this again, but the last time I checked it all worked ok for GPT restores.

Would there be any issues with the fact that we have two Exchange servers (not in a DAG) that use the same credentials and hence a common NetBackup mailbox for backups? The mailbox for the NetBackup Exchange user is hosted on the new Exchange 2016 server.

thanks

[Lowell_Palecek]

The test engineer is going to get me verbose logs over the weekend that I will be able to compare with yours.

Regarding using the same netbackup account for both your Exchange 2010 and 2016 servers, I expect it to work on the system that hosts the netbackup account's mailbox, which is your 2016 server. I would not have been able to tell you whether to expect the GRT restore to work on the 2010 server. The step that would fail if there were a problem here would be logging onto EWS. I think your Exchange 2016 GRT restore fails before it gets to that step.

Yes, the bpfis log for the ASC job is on the Exchange client. To be sure I'm correct on the mailbox question, please include the <clientname>-EWSLog00.log and <clientname>-monad.log files from the NetBackup\logs\beds folder on the client. I expect there to be nothing entered in them at the time of the failure.

[CadenL]

Brilliant - thanks very much.

I'll gather the logs you've requested over the next couple of days and post them up for you.

kind regards

[Lowell_Palecek]

Meanwhile, I asked the developer who knows this area best to look at your nblbc log. He found confirmation that nbfsd successfully mounted the C, D, and E drives from your backup image. That seems to indicate that the issue is not with D and E being GPT disks.

He reminded me of another recent case involving AD. (LIke Exchange, AD uses a JET database.) What Windows version is your Exchange server?

[CadenL]

Hi

The Exchange version is 2016 CU7 and it's running as a virtual machine on Windows 2012 R2

thanks