12-30-2016 11:33 AM
I have VM backups set to backup my Exchange server, and was working fine for the past couple of years.
Now, I am getting consistent failures, which look identical to the article in https://www.veritas.com/support/en_US/article.000025499. However, I have no DAG, and therefore, I have no passive databases to capture. I also confirmed the permissions, and nothing has changed.
I have since reinstalled the symantec VSS provider, and confirmed that the VMWare VSS provider is not installed.
I also removed the lock files, as suggested in https://www.veritas.com/support/en_US/article.000024052
I am at a total loss of how to get these going again.
12/19/2016 6:01:18 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Found the Symantec VSS Provider installed on this VM
12/19/2016 6:01:18 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database02>
12/19/2016 6:01:19 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database03>
12/19/2016 6:01:20 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database04>
12/19/2016 6:01:21 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database05>
12/19/2016 6:01:21 PM - Critical ascc(pid=23118) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
12/19/2016 6:01:22 PM - Critical ascc(pid=23118) Exchange:
12/19/2016 6:01:22 PM - Critical ascc(pid=23118) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
12/19/2016 6:01:23 PM - Critical ascc(pid=23118) Exchange:
12/19/2016 6:01:23 PM - Critical ascc(pid=23118) Exchange: snapshot processing failed, status 156
12/19/2016 6:01:24 PM - Critical ascc(pid=23118) Exchange: snapshot creation failed - Error attempting to gather metadata., status 130
12/19/2016 6:01:24 PM - Warning ascc(pid=23118) Exchange: Microsoft Information Store:\* is not frozen
12/19/2016 6:01:25 PM - end Application State Capture, Application State Capture; elapsed time: 0:00:48
12/19/2016 6:01:25 PM - end Parent Job; elapsed time: 0:00:48
file read failed (13)
12-31-2016 09:57 PM
hey, this is just a shot in the dark as i too am suffering from VSS snapshot errors and the TN i found is to create a separate disk just for the VSS snapshots. also not to include this VSS snapshot disk in the backup.
haven't tried it yet but sounds crazy it might work! i will today start on it though.
Happy New Year! :)
01-02-2017 02:59 AM - edited 01-02-2017 03:01 AM
Need more info:
1) exact versions of client O/S, client MS Exchange version, client NetBackup version
2) exact version of vSphere ESXi
3) bppllist POLICY_NAME -U
01-02-2017 03:21 AM
Please check event viewer on exchange system and try to find out any alert related to MS Exchange around "12/19/2016 6:01:21"
Post that alert here,
01-03-2017 08:11 AM - edited 01-03-2017 08:16 AM
Are your working backups Exchange 2010 and the problematic ones 2013 or 2016? What version of NetBackup? If 2013, is it the infamous lump-of-coal CU11 that Microsoft gave us last Christmas? See https://vox.veritas.com/t5/NetBackup/FYI-Exchange-2013-changes/m-p/823247#U823247.
We could use the following verbose client logs to tell more: bpfis, nbdisco, monad, Windows Application Event log. (Work with Support to get the monad log, or I can post instructions here. I think we published a TechNote on it.) Also, please provide any files with names ending in "exchange.dat" or "exchange.xml" in the master server NetBackup/db/discovery directory.
This failure has nothing to do with the Symantec VSS Provider. It's happening during the ASC job, which uses the Microsoft VSS system provider to take a temporary snapshot.
The following happens only when the NetBackup Framework Discovery service (nbdisco) did not send mailbox information for the databases to the master server in something called "memento" files:
>... Need to gather mailbox information for database <database04>
There can be various reasons for this, some legitimate. When the memento file isn't there, bpfis tries to get the information directly. That will succeed unless it hits some problem that causes nbdisco to fail.
However, failure to get the mailbox information isn't fatal. It only means you wouldn't be able to do GRT (mailbox level) restores. The 156 and 130 errors are fundamental errors in snapshot processing. (156 is the general snapshot failure, 130 is a general system error.) The bpfis log should tell more about why it failed.
Finally, if it used to work, what changed?
01-03-2017 02:49 PM
That's the thing - I cannot find anything that changed. This started between our monthly windows server patching cycles.
I am trying to dig into the logs for the other folk's questions.
01-04-2017 06:51 AM
There are two separate issues here:
1. Why isn't the NetBackup Discovery Framework service (nbdisco) on the client providing the mailbox lists? A simple reason may be that your maintenance changed which Exchange server is backing up the databases, and you haven't had a successful backup since. As an optimization, because the PowerShell commands are expensive to your CPU, nbdisco only gathers the mailbox information for the databases that were most recently backed up on the server. The nbdisco log would tell whether this is the case. If so, you can get nbdisco to gather mailbox lists for all the databases by removing the value LastBacked_Up_Databases in the Registry key HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Agents\Exchange.
(To get a new nbdisco log, do the following: 1. Stop the NetBackup Discovery Framework service. 2. Increase both client logging levels to the max. 3. Start the service. 4. Wait a few minutes. Upon startup, nbdisco regenerates the Exchange discovery files and pushes them to the master server.)
2. Your bigger problem is that your ASC snapshots are failing. These are simple System Provider snapshots that bpfis uses to harvest metadata about the Exchange databases. For this problem, rino19ny's suggestion isn't so crazy. After your server maintenance, do you have adequate space allocated to VSS snapshots? I suggest you poke around a bit with vssadmin or equivalent. Take a system provider snapshot of your Exchange volumes to prove it works.
01-05-2017 03:13 PM
It appears the problem may have been unrelated to all of the things we were thinking.
I had an anti-malware program updated the day before this problem appeared, and completely missed this fact when I was starting to triage. I disabled it last night, and finally got one successful backup.
I am hoping tonight's is also successful.