Forum Discussion

PaulLaVigne's avatar
PaulLaVigne
Level 4
9 years ago

Exchange VM Snapshot backups failing, but there's no DAG: status 13 (156, 130)

I have VM backups set to backup my Exchange server, and was working fine for the past couple of years.

Now, I am getting consistent failures, which look identical to the article in https://www.veritas.com/support/en_US/article.000025499. However, I have no DAG, and therefore, I have no passive databases to capture. I also confirmed the permissions, and nothing has changed.

I have since reinstalled the symantec VSS provider, and confirmed that the VMWare VSS provider is not installed.

I also removed the lock files, as suggested in https://www.veritas.com/support/en_US/article.000024052

I am at a total loss of how to get these going again.


12/19/2016 6:01:18 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Found the Symantec VSS Provider installed on this VM

12/19/2016 6:01:18 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database02>
12/19/2016 6:01:19 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database03>
12/19/2016 6:01:20 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database04>
12/19/2016 6:01:21 PM - Info ascc(pid=23118) Exchange: snapshot preparation successful, with the following information: Need to gather mailbox information for database <database05>
12/19/2016 6:01:21 PM - Critical ascc(pid=23118) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
12/19/2016 6:01:22 PM - Critical ascc(pid=23118) Exchange:
12/19/2016 6:01:22 PM - Critical ascc(pid=23118) Exchange: vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit
12/19/2016 6:01:23 PM - Critical ascc(pid=23118) Exchange:
12/19/2016 6:01:23 PM - Critical ascc(pid=23118) Exchange: snapshot processing failed, status 156
12/19/2016 6:01:24 PM - Critical ascc(pid=23118) Exchange: snapshot creation failed - Error attempting to gather metadata., status 130
12/19/2016 6:01:24 PM - Warning ascc(pid=23118) Exchange: Microsoft Information Store:\* is not frozen
Status 13
12/19/2016 6:01:25 PM - end Application State Capture, Application State Capture; elapsed time: 0:00:48
Status 13
12/19/2016 6:01:25 PM - end Parent Job; elapsed time: 0:00:48
file read failed (13)

7.6
Windows Server (2003/2008)

7 Replies

Sort By
  • Lowell_Palecek's avatar
    Lowell_Palecek
    Level 6
    9 years ago

    Are your working backups Exchange 2010 and the problematic ones 2013 or 2016? What version of NetBackup? If 2013, is it the infamous lump-of-coal CU11 that Microsoft gave us last Christmas? See https://vox.veritas.com/t5/NetBackup/FYI-Exchange-2013-changes/m-p/823247#U823247.

    We could use the following verbose client logs to tell more: bpfis, nbdisco, monad, Windows Application Event log. (Work with Support to get the monad log, or I can post instructions here. I think we published a TechNote on it.) Also, please provide any files with names ending in "exchange.dat" or "exchange.xml" in the master server NetBackup/db/discovery directory.

    This failure has nothing to do with the Symantec VSS Provider. It's happening during the ASC job, which uses the Microsoft VSS system provider to take a temporary snapshot.

    The following happens only when the NetBackup Framework Discovery service (nbdisco) did not send mailbox information for the databases to the master server in something called "memento" files:

    >... Need to gather mailbox information for database <database04>

    There can be various reasons for this, some legitimate. When the memento file isn't there, bpfis tries to get the information directly. That will succeed unless it hits some problem that causes nbdisco to fail.

    However, failure to get the mailbox information isn't fatal. It only means you wouldn't be able to do GRT (mailbox level) restores. The 156 and 130 errors are fundamental errors in snapshot processing.  (156 is the general snapshot failure, 130 is a general system error.) The bpfis log should tell more about why it failed.

    Finally, if it used to work, what changed?

    • PaulLaVigne's avatar
      PaulLaVigne
      Level 4
      9 years ago

      That's the thing - I cannot find anything that changed. This started between our monthly windows server patching cycles.

      I am trying to dig into the logs for the other folk's questions.

      • Lowell_Palecek's avatar
        Lowell_Palecek
        Level 6
        9 years ago

        There are two separate issues here:

        1. Why isn't the NetBackup Discovery Framework service (nbdisco) on the client providing the mailbox lists? A simple reason may be that your maintenance changed which Exchange server is backing up the databases, and you haven't had a successful backup since. As an optimization, because the PowerShell commands are expensive to your CPU, nbdisco only gathers the mailbox information for the databases that were most recently backed up on the server. The nbdisco log would tell whether this is the case. If so, you can get nbdisco to gather mailbox lists for all the databases by removing the value LastBacked_Up_Databases in the Registry key HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Agents\Exchange.

        (To get a new nbdisco log, do the following: 1. Stop the NetBackup Discovery Framework service. 2. Increase both client logging levels to the max. 3. Start the service. 4. Wait a few minutes.  Upon startup, nbdisco regenerates the Exchange discovery files and pushes them to the master server.)

        2. Your bigger problem is that your ASC snapshots are failing. These are simple System Provider snapshots that bpfis uses to harvest metadata about the Exchange databases. For this problem, rino19ny's suggestion isn't so crazy. After your server maintenance, do you have adequate space allocated to VSS snapshots?  I suggest you poke around a bit with vssadmin or equivalent. Take a system provider snapshot of your Exchange volumes to prove it works.

  • manatee's avatar
    manatee
    Level 6
    9 years ago

    hey, this is just a shot in the dark as i too am suffering from VSS snapshot errors and the TN i found is to create a separate disk just for the VSS snapshots. also not to include this VSS snapshot disk in the backup.

    haven't tried it yet but sounds crazy it might work! i will today start on it though.

    Happy New Year! :-)

  • sdo's avatar
    sdo
    Moderator
    9 years ago

    Need more info:

    1) exact versions of client O/S, client MS Exchange version, client NetBackup version

    2) exact version of vSphere ESXi

    3) bppllist POLICY_NAME -U

  • NBU35's avatar
    NBU35
    Level 6
    9 years ago

    Please check event viewer on exchange system and try to find out any alert related to MS Exchange around "12/19/2016 6:01:21"

    Post that alert here,

  • PaulLaVigne's avatar
    PaulLaVigne
    Level 4
    9 years ago

    It appears the problem may have been unrelated to all of the things we were thinking.

    I had an anti-malware program updated the day before this problem appeared, and completely missed this fact when I was starting to triage. I disabled it last night, and finally got one successful backup.

    I am hoping tonight's is also successful.

    Thanks!