Configuring VMware that protect Exchange 2010 DAG

[Dtrace_225]

Hi All, 

I have to implement same configuration as describe in this post: https://vox.veritas.com/t5/NetBackup/Configuring-GRT-based-Exchange-Backup-using-vmware-backup-polic...

Note that I have implement tranditional backup of exchange and it works. But for perfomance issue, I want to implement VM backup that protects exchange and disable traditional backup.

I have an error 13. I'm troubleshooting... Can you help?

About DAR configuration, must I use FQDN or short name?

I will post job details in fex time.

Tnahk for your help.

[manatee]

how far have you gone with regards to following the answer in that link?

[sdo]

Maybe this will help:

VMware Exchange Application Capture backup fails with status 13 (156, 130) when backing up DAG passive mailbox databases
http://www.veritas.com/docs/000025499

[Dtrace_225]

HI All,

I found that costumer MBX DB are on GPT disk, that is not supported by this king of configuration.

01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Direction' is not supported because volume <F:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Direction' is not supported because volume <F:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Informaticiens' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Informaticiens' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\NBU' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\NBU' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs3' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs3' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs5' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs5' is not supported because volume <G:\> is on a GPT disk and is not supported.

I can I deal with this?

[sdo]

This next linked post is regarding Backup Exec, so take it with a pinch of salt:

https://vox.veritas.com/t5/Backup-Exec/GRT-Enabled-Backups-on-GPT-Partitioned-VMware-VMDKs/td-p/7020...

...but it does seem to indicate that the issue of GRT functionality with GPT disks is a limitation of the VMware VDDK.

Can I ask two questions:

1) Exact version of NetBackup on the Media Server (backup proxy) ?

2) Version of VMware vSphere ESXi ?

[Dtrace_225]

Master/Media OS 2012 R2 Netbackup version 7.7.3

Client 7.7.3

Vsphère 5.5

[sdo]

NetBackup v7.7.3 uses VMware VDDK v6.0.

.

The VMware VDDK release notes for v5.5.1 notes that there was a bug/limitation in VDDK v5.0 around GPT disks:

https://www.vmware.com/support/developer/vddk/vddk-551-releasenotes.html

...which says:

"VixMntapi now works better with GPT partitions.
The VDDK 5.0 Release Notes said that the VixMntapi library did not support GPT. After a root cause analysis, VMware determined that the problem was partition ordering, with the shadow copy partition appearing first although it was index numbered later. So VixMntapi was modified to put the shadow copy partition in its expected order. This change enables file-level restore on Windows Server 2012."

.

You say you are using VMware vSphere ESXi v5.5.  Do you really mean v5.5 GA, or has it been patched.  I'd be curious to know "exactly" which version of ESXi you are using.  But I'm not asking because I know of a solution.  I mean, I don't know of a solution either way.... but telling us exactly which version of ESXi you are running might help others forum users provide you with better advice.

.

Now then, I can't help but think that... if your ESXi really is v5.5 GA, and not v5.5.1 or later, then maybe the v6.0 VDDK in NetBackup v6.0 is not enough... and so maybe... if your ESXi was v5.5.1 or above, then you might be ok... but then this is proabably nonsense from me... and all because I don't know which exact version of ESXi you are running.  Do you see why we always need to be very specific with forum posts about exactly which versions we are running.

.

Anyway, enough of all that conjecture.   Personally, IMO, I think you need to open an official proper Veritas Support case, and reference this forum post... because there could be a little nugget in this forum post (and the forum posts linked from this post itself) which might help the Veritas Support technician pinpoint exactly what the problem is.

[Dtrace_225]

Hi Sdo

Thank for your help. My exact version is Vsphère server 5.5.0. I will open a case.

Thank again

[sdo]

Sounds like a plan.

.

My 2 cents... your ESXi hosts are vSphere 5.5.0 and therefore contain VDDK v5.5.0... and your NetBackup version is v7.7.3 (i.e. contains VMware VDDK v6.0)... and we know that the VDDK v5.5.1 contained some kind of fix to handle GPT partitions in some way... so, on the face of it, it would seem to me that your vSphere versions do not contain the GPT fix which is in VDDK v5.5.1 onwards.

[sdo]

What I mean is... we can think of a VMware vSphere host as containing a version of ESXi plus a version of VDDK, and we can also think of the backup host/proxy as containing a version of NetBackup plus a version of the VDDK.

So, an ESXi host can be thought of as:           ESXi          plus VDDK server...

...and NetBackup host can be thought of as:  NetBackup plus VDDK client...

...do you see how your VDDK versions are different.  They do and should work together for most / all documented features, but v5.5.0 of the VDDK API "server" (within your v5.5.0 ESXi hosts) doesn't know about the GPT disk fix (from VDDK v5.5.1 onwards) within v6.0 of the VDDK API "client" (within your NetBackup v7.7.3 host).

Please do come back with the solution (or more questions).

[Lowell_Palecek · Accepted solution]

** See update in 25 January 2018 post ** We have demonstrated that we can support this, although you will get a warning from the ASC job.

Whether or not VMware has improved its ability to snapshot a GPT disk, the NetBackup ASC job for Exchange 8.0 still rejects the GPT disk. The log message you found still applies:

Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Direction' is not supported because volume <F:\> is on a GPT disk and is not supported.

[sdo]

@Lowell_Palecek - I take it that you've just tested that, so that was NetBackup v8.0 which contains VDDK v6.0.2, so that should support VDDK interop with ESXi v5.5.0.  So, it is as I feared that the "GPT issue" fixed in VDDK v5.5.1 has nothing to do with @Dtrace_225 actual underlying issue.

[Lowell_Palecek]

Actually, I looked at the ASC code. We explicitly check for and exclude databases on GPT disks from Exchange cataloguing.

[Disclosure: I was a developer for the NetBackup Exchange agent from 6.5.1 until just before 7.6. For the last few years, I have worked in the Customer Focus Team, meaning that if a problem gets escalated all the way back to engineering, I'm one of the people who gets to solve it. My touch has broadened beyond Exchange, but it's still my core area. I try to explain in this forum how things work operationally without giving away internal IP.]

[sdo]

We are all fortunate to have you on here.  Thank you Lowell.

[sdo]

@Dtrace_225 - I think the message at this stage is... unfortunately... that you may well have to investigate and check whether you can change from using GPT partitioned psuedo LUNs inside your VMDK, to instead use MBR partitioned psuedo LUNs inside your VMDK.

[Lowell_Palecek]

Clarification

The log message from the ASC job is that GRT restore is not supported on GPT disks. It does not say that Exchange database restore is not supported. If this were the only issue found, the ASC job would end in status 1, not 156 (snapshot error) or 130 (system error).

There are two snapshot jobs involved here. The ASC job takes a system provider snapshot in order to catalog the Exchange data files. It also catalogs VSS metadata and leaves it in the NetBackup\online_util\fi_cntl folder to be backed up with the VM. When the ASC job is done, it releases the snapshot it created.

After all the ASC jobs for Exchange, SharePoint, and SQL Server, NetBackup backs up the VM itself. For this, it uses the Symantec VSS Provider (seen as the Backup Exec provider in the Event log) to snap the system. Use of this provider is critical to pause the databases so that an application consistent backup can be taken. For Exchange, it's also needed to selectively truncate logs only for the databases that are protected in the VM backup.

To investigate the 156 or 130 failure, you need to look more closely at the job details and bpfis log to see which snapshot failed. Was it the ASC job, or was it the VM snapshot? Then you need to find the error that produces the hard failure. As I have noted, the step in Exchange ASC processing that produces the log message you found only produces status 1 when it finds a GPT disk. Something else produced the hard failure.

[Dtrace_225]

Hi Lowell

Thank for your help. Below my complete job's details:

01/24/2017 09:02:45 - Info nbjm (pid=4772) starting backup job (jobid=3386) for client WINSRVBF-DB1.BOABurkina.local, policy BKPVM-EXCH, schedule sch-vmexch-Dfull
01/24/2017 09:02:45 - Info nbjm (pid=4772) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=3386, request id:{113D7640-D24A-448A-AFA6-2A5F254270B5})
01/24/2017 09:02:45 - requesting resource STU_DP_DD2500
01/24/2017 09:02:45 - requesting resource winsrvbf-ntbkp.NBU_CLIENT.MAXJOBS.WINSRVBF-DB1.BOABurkina.local
01/24/2017 09:02:45 - granted resource  winsrvbf-ntbkp.NBU_CLIENT.MAXJOBS.WINSRVBF-DB1.BOABurkina.local
01/24/2017 09:02:45 - estimated 0 kbytes needed
01/24/2017 09:02:45 - begin Parent Job
01/24/2017 09:02:45 - begin Application State Capture: Step By Condition
Operation Status: 0
01/24/2017 09:02:45 - end Application State Capture: Step By Condition; elapsed time 0:00:00
01/24/2017 09:07:16 - Info ascc (pid=9048) Exchange: snapshot preparation successful, with the following information: Found the Symantec VSS Provider installed on this VM.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Direction' is not supported because volume <F:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Direction' is not supported because volume <F:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Informaticiens' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Informaticiens' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\NBU' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\NBU' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs3' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs3' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs5' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\Utilisateurs5' is not supported because volume <G:\> is on a GPT disk and is not supported.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: Database <NBUTEST> is not mounted and cannot be backed up.
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: At least one of the Exchange databases cannot be backed up because the database is not mounted.
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: 
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: 
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: snapshot processing failed, status 156
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: snapshot creation failed - SNAPSHOT_NOTIFICATION::PostSnapshot failed., status 130
01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: Microsoft Information Store:\* is not frozen
Operation Status: 13
01/24/2017 09:07:16 - end Parent Job; elapsed time 0:04:31
Operation Status: 13
file read failed  (13)

It's great an good case that need investigation. From my side I stop investigation once GRT is not supported. So I have not testing environment for next step.

Thank again for your precious help.

[Lowell_Palecek]

Seems that things went along with at most status 1 until it came time to take snapshots of the volumes containing the databases. (Besides the messages about the GPT disks F: and G:, I also see two messages about unmounted databases being skipped. These also produce status 1.)

I would need to see the bpfis log to advise you further. All I would need is whatever happened at 9:07:16:

01/24/2017 09:07:16 - Warning ascc (pid=9048) Exchange: snapshot preparation successful, with the following condition: At least one of the Exchange databases cannot be backed up because the database is not mounted.
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange:
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange:
01/24/2017 09:07:16 - Critical ascc (pid=9048) Exchange: snapshot processing failed, status 156

I'd like a log with the log levels turned up, but the error cause ought to come through at any logging level. The Windows Application Event log from the same time would be useful.

[Dtrace_225]

@Lowell_Palecek

I have noted some error on VSS Writer. I asked Windows team to chek. If you want, I can provide logs from client side.

BR.

[Lowell_Palecek]

I am updating this post to say that Exchange GRT operations for databases on GPT disks are supported on a VMware backup. In the following document, we have back-dated that support to NetBackup 7.7.3.

https://www.veritas.com/content/support/en_US/doc/NB_70_80_VE

The ASC job creates a warning that it won’t work and ends with status 1. This warning should be ignored. We will remove the warning from a future release. As always, you should investigate the status 1 to make sure there are not other issues, such as a database being skipped because it isn't mounted.

We will update the NetBackup for Exchange Administration Guide to say that it is supported.