Exchange snapshot takes 2 hours

[Marianne]

Environment:

Exchange 2010 Standalone server
NBU 7.0.1 on master/media server and client
W2008 R2 SP1

Exchange policy with 3 Information Stores in Backup Selection with multiplexing and multistreaming enabled.

Backup starts and snapshots are created on Exchange Client.

bpfis log:

12:00:08.142 [9876.17364] <4> bpfis: INF - Preparing freeze of NEW_STREAM0 using snapshot method VSS_Writer.

12:00:08.142 [9876.17364] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_NEW_STREAM0_9876

12:00:08.157 [9876.17364] <4> bpfis: INF - Preparing freeze of Microsoft Information Store:\Archive DB3\ using snapshot method VSS_Writer.

12:00:08.157 [9876.17364] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_Microsoft Information Store_Archive DB3__9876

12:00:34.335 [9876.17364] <4> bpfis: INF - Preparing freeze of NEW_STREAM1 using snapshot method VSS_Writer.

12:00:34.335 [9876.17364] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_NEW_STREAM1_9876

12:00:34.335 [9876.17364] <4> bpfis: INF - Preparing freeze of Microsoft Information Store:\Archive DB9\ using snapshot method VSS_Writer.

12:00:34.335 [9876.17364] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_Microsoft Information Store_Archive DB9__9876

12:00:36.363 [9876.17364] <4> bpfis: INF - Preparing freeze of NEW_STREAM2 using snapshot method VSS_Writer.

12:00:36.363 [9876.17364] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_NEW_STREAM2_9876

12:00:36.363 [9876.17364] <4> bpfis: INF - Preparing freeze of Microsoft Information Store:\Archive DB10\ using snapshot method VSS_Writer.

12:00:36.363 [9876.17364] <4> bpfis: INF - Created mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_Microsoft Information Store_Archive DB10__9876

12:00:38.438 [9876.17364] <4> bpfis: TRV - snapshot preparation successful

13:50:33.580 [9876.17364] <2> bpfis main: snapshot creation successful

Have a look at the last two lines - just short of 2 hours between 'snapshot preparation' and 'snapshot creation'.

bpbkar log starts at 13:56.

Unfortunately nothing in Event Viewer Application log as the log got recreated last night (first entry about 23:30).

Can anyone tell me what is happening in the 2 hours?

Does this delay include database validation?

Thanks!

[Dyneshia]

If you have "perform consistency check before backup w/Microsoft VSS, this could explain the 2 hour delay.

Since this is a stand alone server, this is required.

The sysadmin guide reads : "NetBackup performs the consistency check during the snapshot importstep."

[Marianne]

Thanks Dyneshia

Customer has recently migrated to new Exchange server and divided 3 large stores into 15 smaller stores (+- 200 GB each).

Where Daily Full backups used to complete overnight, backups now take DAYS to complete... (with multiplexing and multistreaming.)
2 hours for snapshot and consistency check for relative small db's seems a bit excessive.
I understand that there is nothing we can do from NBU point of view to speed up this process.

[Marianne]

Things are actually worse this morning....

We have disabled the consistency check in an attempt to actually get some backups through.

Backup for 3 DBs was still busy creating snapshot after 3 hours!

bpfis showed success snapshot preparation, still no successful snapshot creation after 3 hours:

07:54:30.562 [14332.13344] <4> bpfis: INF - Preparing freeze of NEW_STREAM0 using snapshot method VSS_Writer.
07:54:30.562 [14332.13344] <4> bpfis: INF - Created mount point ......
.....
,,,,,
07:54:58.565 [14332.13344] <4> bpfis: TRV - snapshot preparation successful

No errors in Evt logs - just the following:

Log Name:      System
Source:        Service Control Manager
Date:          2013/05/21 07:54:44 AM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      EXCH01
Description:
The Volume Shadow Copy service entered the running state.

Log Name:      System
Source:        Service Control Manager
Date:          2013/05/21 07:54:51 AM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      EXCH01
Description:
The Microsoft Software Shadow Copy Provider service entered the running state.

My colleague found this post: 

https://www-secure.symantec.com/connect/forums/exchange-2010-backup-using-netbackup-issue#comment-54...  

Yes, millions of files in online_util\fi_cntl folder (Had to close Explorer when the file count got to 2 million and Admin complained that my Explorer session was consuming 3 GB of memory).  Seems deleting these files will take hours as per above post....  netbackup\temp folder did not have that many entries.

I guess we have no hope of getting good backups as long as these orphaned files are present?

Dyneshia - would you happen to know WHY NBU is not clearing out these folders after completion of backups? Or even with log file maintenance? 

[n_ludwig]

hi marianne,

see this ms hotfix for vss

http://support.microsoft.com/kb/2748349/en-us

[Ankit_Maheshwar]

Hello Marianne,

I worked on same issue with Symantec more than 3 days with 4 Symantec engineer, and didn’t get any solution from Symantec.

The solution that I found that bpfis was taking approx 5 hours or more than that to create snapshot...I reinstalled the client and very first i started disk backup on the same server...This was simple OS backup.. After this bpfis completed in just 10 min.. 

Till then I didn't see any bpfis issue in my environment...

[Marianne]

Thanks Ankit!

Which NBU version and patch did you install?

No VSS hotfixes were installed?

[jandersen1]

Hi Marianne,

We are struggling with the same problems with Exchange 2010 snapshot taking a looooong time. We are running NBU 7.5.0.5.

We have disabled consistency check as this is a DAG

I read somewhere (can't remember if it was a howto that the cleanup of files in the netbackup\temp has been solved in recent releases.

I have manually cleaned up files generated when the client was at 7.1.0.4.
I have yet to check whether the cleanup is handled correctly in 7.5.0.5 (which don't help you much at 7.0.1 but could give you an indication whether an upgrade of NBU will help)

I will check if the hotfix n.ludwig linked to is applied.

--jakob;

[Nicolai]

Dude - that is a ugly bug 

[SFG]

I am having a similar issue using Backup Exec 2012, current patch-level.  Are you referring to DAG backups or server image snapshots (looks like you are, just confirming)?

DAG backups here take about 20 hours, and that is backup to disk with granular level restore enabled.  A copy to tape runs another few hours, which I have to do manually, since otherwise the Exchange backup/duplicate operations would interfere with my other servers' backups. 

I worked with support on this, and there is no resolution at this time for BE 2012 for Exchange 2010 SP3.  Supposedly a suport pack is/was due in May but I have not seen it yet.  I am amazed that this is a "known issue".  I am sure to take that into consideration when my renewals are due.

Because of these backup issues, I"m looking to replicate my Exchange environment to another of our sites and at least hope for a DAG backup to work several times a week. 

[Marianne]

Thanks Jakob

We have found the same issue as per my post above. 

There were not that many files in netbackup\temp but millions in online_util\fi_cntl.

Customer renamed the folder, created a new, empty folder and then deleted the renamed folder.

We have also disabled consistency check to see if we can just get backups going.

Customer kicked off another backup but the snapshot process is once again stuck after 'snapshot preparation successful'. Elapsed time right now is 1hr 31 min.

We have also noticed these errors in Evt Application log - See this BE TN: http://www.symantec.com/docs/TECH169638

NO IDEA why "Get-PublicFolderDatabase" query is executed by NBU when 3 Information Stores are explicitly mentioned in Backup Selection.

Not sure if this is playing a role in the snapshot delay?

[Marianne]

This is standalone Exch server - no DAG, no GRT.

Exch 2010 backup policy must have snapshot option selected in the policy attributes as Exch2010 does not support streaming backups.

Weird thing is that backups worked perfectly on previous server with 3 large Information Stores. All Daily Full backups completed successfully overnight. Only difference is that the stores were split into 15 smaller stores. Now a policy that lists 3 (smaller) stores explicitly cannot complete overnight.
Stores were split into different policies in an attempt to stagger backups.

[Dyneshia]

Since this is a stand alone DAG, you must have consistency check enabled.

snippet from exchange admin guide :

NetBackup is configured to run consistency checks for Exchange snapshot backups.

Consistency checks are required for Exchange 2007 and earlier or for a stand alone

Exchange 2010 server.

As far as  online_util\fi_cntl not clearing up...I am surprised you are not seeing anything more in the system event viewer.  Anything in the application event viewer ? How are the writers ?

vssadmin list writers

vssadmin list providers

I know this is an exchange box, but if you need a backup ASAP, have you tried stopping services and clear up online_util\fi_cntl or reboot the exchange server ?

Is this exchange server a VM ?

If you have a bpfis at Verbose 5 could you upload it for review ?

[uuontario]

did you open a case with Microsoft? maybe it is not related to NBU

[Nicolai]

http://technet.microsoft.com/en-us/library/aa998827(v=exchg.141).aspx

I think NBU uses that comand to view public folder database settings, even if they are not there.

Just like the NBU probes NFS mount even if they they are not in the file list.

[Marianne]

Dyneshia, I hear what you are saying about consistency check.

We are trying to figure out what is causing the long delay between successful snap prepare and snap create.

There are no VSS errors in Event Viewer - just the messages as per my post above.  

As per this post, we have cleaned out the fi_cntl folder and started another backup. No difference - just the event log errors as per TECH169638 which according to Symantec can be 'safely ignored'. 
Still no snapshot created after more than 2 hours.

Not sure if I have mentioned this, but all VSS writers are in 'Stable, No error' state.
We have unfortunately only level 0 bpfis logs. Will ask the customer if we can increase logging level.
 

Now that I have everyone's attention  -  I am sure I have seen a TN or post with cmd method to test VSS snapshot of Information Store. My Google skills are seriously letting me down....

If we can run this test, it will completely take NBU out of the picture and focus on the Microsoft side of the snapshot creation.

Does anyone know where I can find this command/TN/post?

[Dyneshia]

hmmm... here is the tech note to trace the exchange writers...

http://technet.microsoft.com/en-us/library/ff597980%28v=exchg.80%29.aspx

Still looking for The Information store directly.

[Nicolai]

http://www.symantec.com/docs/HOWTO43657

This T/N explain how to use bpfis for creating snapshots. I think example two can be used for "template:

install path\bin\bpfis create -rhost hostB -fim MS-Exchange-Server  "Microsoft Information Store:\"

I do not have access to exchange server so I can't verify it this command work. The command reference has a little more information about qualifiers compared to the T/N.

[Marianne]

THANKS!

[Ankit_Maheshwar]

Marianne,

Version-7.1.0.4

We didn't install any HOTFIX as Microsoft updated that server is running on latest one...

One more thing we rebooted the server before installing the client...