Exchange snapshot takes 2 hours

If you have "perform consistency check before backup w/Microsoft VSS, this could explain the 2 hour delay.

Since this is a stand alone server, this is required.

The sysadmin guide reads : "NetBackup performs the consistency check during the snapshot importstep."

Thanks Dyneshia

Customer has recently migrated to new Exchange server and divided 3 large stores into 15 smaller stores (+- 200 GB each).

Where Daily Full backups used to complete overnight, backups now take DAYS to complete... (with multiplexing and multistreaming.)
2 hours for snapshot and consistency check for relative small db's seems a bit excessive.
I understand that there is nothing we can do from NBU point of view to speed up this process.

Things are actually worse this morning....

We have disabled the consistency check in an attempt to actually get some backups through.

Backup for 3 DBs was still busy creating snapshot after 3 hours!

bpfis showed success snapshot preparation, still no successful snapshot creation after 3 hours:

07:54:30.562 [14332.13344] <4> bpfis: INF - Preparing freeze of NEW_STREAM0 using snapshot method VSS_Writer.
07:54:30.562 [14332.13344] <4> bpfis: INF - Created mount point ......
.....
,,,,,
07:54:58.565 [14332.13344] <4> bpfis: TRV - snapshot preparation successful

No errors in Evt logs - just the following:

Log Name:      System
Source:        Service Control Manager
Date:          2013/05/21 07:54:44 AM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      EXCH01
Description:
The Volume Shadow Copy service entered the running state.

Log Name:      System
Source:        Service Control Manager
Date:          2013/05/21 07:54:51 AM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      EXCH01
Description:
The Microsoft Software Shadow Copy Provider service entered the running state.

My colleague found this post: 

https://www-secure.symantec.com/connect/forums/exchange-2010-backup-using-netbackup-issue#comment-5420431  

Yes, millions of files in online_util\fi_cntl folder (Had to close Explorer when the file count got to 2 million and Admin complained that my Explorer session was consuming 3 GB of memory).  Seems deleting these files will take hours as per above post....  netbackup\temp folder did not have that many entries.

I guess we have no hope of getting good backups as long as these orphaned files are present?

Dyneshia - would you happen to know WHY NBU is not clearing out these folders after completion of backups? Or even with log file maintenance? 

hi marianne,

see this ms hotfix for vss

http://support.microsoft.com/kb/2748349/en-us

Hello Marianne,

I worked on same issue with Symantec more than 3 days with 4 Symantec engineer, and didn’t get any solution from Symantec.

The solution that I found that bpfis was taking approx 5 hours or more than that to create snapshot...I reinstalled the client and very first i started disk backup on the same server...This was simple OS backup.. After this bpfis completed in just 10 min.. 

Till then I didn't see any bpfis issue in my environment...

Thanks Ankit!

Which NBU version and patch did you install?

No VSS hotfixes were installed?

Hi Marianne,

We are struggling with the same problems with Exchange 2010 snapshot taking a looooong time. We are running NBU 7.5.0.5.

We have disabled consistency check as this is a DAG

I read somewhere (can't remember if it was a howto that the cleanup of files in the netbackup\temp has been solved in recent releases.

I have manually cleaned up files generated when the client was at 7.1.0.4.
I have yet to check whether the cleanup is handled correctly in 7.5.0.5 (which don't help you much at 7.0.1 but could give you an indication whether an upgrade of NBU will help)

I will check if the hotfix n.ludwig linked to is applied.

--jakob;

Dude - that is a ugly bug 

I am having a similar issue using Backup Exec 2012, current patch-level.  Are you referring to DAG backups or server image snapshots (looks like you are, just confirming)?

DAG backups here take about 20 hours, and that is backup to disk with granular level restore enabled.  A copy to tape runs another few hours, which I have to do manually, since otherwise the Exchange backup/duplicate operations would interfere with my other servers' backups. 

I worked with support on this, and there is no resolution at this time for BE 2012 for Exchange 2010 SP3.  Supposedly a suport pack is/was due in May but I have not seen it yet.  I am amazed that this is a "known issue".  I am sure to take that into consideration when my renewals are due.

Because of these backup issues, I"m looking to replicate my Exchange environment to another of our sites and at least hope for a DAG backup to work several times a week. 

Thanks Jakob

We have found the same issue as per my post above. 

There were not that many files in netbackup\temp but millions in online_util\fi_cntl.

Customer renamed the folder, created a new, empty folder and then deleted the renamed folder.

We have also disabled consistency check to see if we can just get backups going.

Customer kicked off another backup but the snapshot process is once again stuck after 'snapshot preparation successful'. Elapsed time right now is 1hr 31 min.

We have also noticed these errors in Evt Application log - See this BE TN: http://www.symantec.com/docs/TECH169638

NO IDEA why "Get-PublicFolderDatabase" query is executed by NBU when 3 Information Stores are explicitly mentioned in Backup Selection.

Not sure if this is playing a role in the snapshot delay?