07-20-2017 04:15 AM
Hello Netbackup team
we have 3 node Servers with DAG Exchange 2016
Polisy type MS_exchage _Server and Enable granular recovery
2 nodes almost always failed in Backup with error "add DB snap to file list failed, status 69 and invalid filelist specification (69)"
and
ERR - C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.Leumit-DAG-2K16.leumit.co.il_150
0535113.1.0.VSS_Writer_File_List.txt: invalid format
no errors in vssadmin list writers but
Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {d4abea0b-7d29-465e-918b-a22d3dfa9f81}
State: [5] Waiting for completion
Last error: No error
could you advice please how to resolve the problem about backup our Exchange 2016
07-20-2017 07:03 AM
Could you provide more info about your policy using bpplist?
Has this backup ever ran successfully?
Have you tried to restart the Exchange VSS writer, to set it back to a stable state and rerun the backup job?
07-20-2017 08:16 AM
07-20-2017 08:28 AM
Hi,
Below there is a TN with the Best practices for Exchange Granular Recovery..Other thing, as far as I know you need to solve this issue with VSS.
One question: This backup was working before?
Everything you need to know to get Exchange Granular backups to work. - https://www.veritas.com/support/en_US/article.000068163
Regards,
Thiago
07-24-2017 12:07 AM
Hi
yes we have IPless DAG and clinet name =DAG name
DAG consist with 3 members. we have rebooted problem members and got it Writer name: 'Microsoft Exchange Writer' stable but after trying to backup getting again State: [5] Waiting for completion, for the same nodes
acctually we being in migration mailbox process from Exchange 2010 to Exchange 2016
before massive migration all backups worked good and problem started after massive migration
every node keep 7 active databases
07-24-2017 12:18 AM
by the way only 2 nodes from DAG making problems with "writer" an 1 node backup is good
07-24-2017 08:23 AM
Is that migration complete?
You still haven't given us any information about the backup policy, please try to do that.
07-24-2017 12:30 PM
Hi,
Have you opportunity to check this TN?
https://www.veritas.com/support/en_US/article.TECH223843 - Backing up an Exchange 2013 or Exchange 2016 IP less DAG pre-NetBackup 7.7.3
Regards,
Thiago Ribeiro
07-25-2017 01:56 AM
atached 3 files
07-27-2017 02:14 AM
Hi
I am attaching policy file and I have some news from testing. I have found out that only 2 DBs from 21 DB's making problem with snapshot. Once I am canceling granular recovery in the Exchange Policy those 2 problem DBs getting backing up as well. I have moved 2 problems DB's to the 3 rd node and now looks like backup working good. My question how is possible on the same 1 Exchage Server node 6 Db's getting backup good and only one failed ? The same on the Second Exchage Server ....
thank you in advance
07-27-2017 06:56 AM
Hi,
From what I understand, It seems to me that the problem is with GRT enabled..Can you check these steps on all DAG nodes?
From NetBackup Admin Guide for Exchange
Step 1
Verify that you have a supported Exchange Server configuration and have a media server platform that supports GRT.
Step 2
Ensure that requirements are met for the NetBackup server and the Exchange Server software.
NetBackup server requirements for NetBackup for Exchange -->pags 19,20.
Exchange server software requirements for NetBackup for Exchange --> pag. 21
Step 3
Determine which clients require configuration and ensure that requirements are met for the NetBackup clients.
See Exchange granular clients and non-VMware backups --> pags 49,50.
See NetBackup client requirements for NetBackup for Exchange --> pag 21
In a cluster or replicated environment, perform the steps on each database node in the cluster. For an Exchange Database Availability Group (DAG), perform the steps on each database node in the DAG -
Did you check all nodes? You said about two problematics nodes, can you check then?
Step 4
On all granular clients, ensure that each node has an unassigned drive letter on which to mount the backup image.
-If you are using Windows/Unix media server.
Step 5
Enable or configure NFS for your environment on the following:
All granular clients.The NetBackup media server
See About configuring Services for Network File System (NFS) on Windows 2012 and Windows 2012 R2 (NetBackup for Exchange) --> pag 56
See About configuring Services for Network File System (NFS) on Windows 2008 and Windows 2008 R2 (NetBackup for Exchange) --> pag 64
See Configuring a UNIX media server and Windows clients for backups and restores that use Granular Recovery Technology (GRT) (NetBackup for Exchange) --> pag 73
Step 6
On all Exchange mailbox servers, create an account for Exchange operations (a unique mailbox) for NetBackup.
See About configuring the account for NetBackup Exchange operations --> 34
https://www.veritas.com/support/en_US/article.id-SF930541453_nbu_exchange_web
Step 7
On all Exchange mailbox servers, configure the Exchange credentials.Use the credentials for the account for NetBackup Exchange operations.
See About the Exchange credentials in the client host properties --> 32
Did you check the credential on all exchange servers?
Alternatively for Exchange 2013 and 2016, you can add "Exchange Servers" to the "View-Only Organization Management" role group. Perform this configuration in the Exchange Administration Center (EAC) or in Active Directory. See the following Microsoft article for more information: http://technet.microsoft.com/en-us/library/jj657492
Step 8
Create a MS-Exchange-Server policy as follows:
Select a supported disk storage unit.
Select Enable granular recovery on the Attributes tab.
See the NetBackup Hardware Compatibility List.
For more information on how to create a policy with GRT, see the following:
See About configuring snapshot backups of Exchange Server --> pag 100
Step 9
On the NetBackup master server, configure the list of Exchange hosts.
For backups in a DAG, cluster, private network, or if you use a proxy host or off-host client, you must create a mapping of host names. For example, each DAG node must be able to access a backup image using the DAG name. Configure this mapping in the Distributed Application Restore Mapping host property on the master server.
See “Configuring the Exchange hosts ” --> pag 43
Regards,
Thiago Ribeiro
07-28-2017 12:15 PM
Please, what is the NetBackup version? Do you have any EEBs in place on the mailbox servers that have the problem?
The file C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\ ... .VSS_Writer_File_List.txt is generated and consumed within NetBackup. It is the "snap file list" to which the "add db" function failed. This makes me wonder about database names, filenames, and paths. What are the names of the databases that work and don't work on an affected mailbox server?
The bpfis log at a high logging level will tell us the answers to these and other questions.
The Exchange VSS writer left "waiting for completion" suggests that bpfis did not complete its VSS transaction. Look for all occurrences of IVSS in the bpfis log to see the commands and notifications that bpfis sent to the VSS API. There should be something like BackupComplete or BackupStatus, but I don't remember it exactly.
I had a question about active vs. passive database copies, but I see in one of your screenshots that you are specifying active only. Nevertheless, could we see the "Work list" from the tail of the bpresolver log?