Stopping Centera Replication

[Energy99]

We have paused replication between are two Centera's while the 2nd Centera is being moved to our DR site.  All archiving has been stopped during this period, only recalls and deletes are taking place.  We have noticed a huge hit to our recall performance but were being told it's not related to the pause of replication and downing of the 2nd Centera.  All Sql databases have regular online maintenance and we just performed an offline one to ensure fragmentation isn't the issue.  Anyone have any idea's... this issue is effecting both FSA and Exchange archiving..

Thanks..

[TonySterling]

Can you run a dtrace on StorageOnlineOpns when retrieving an archived item?  This will verify it isn't looking for the replica centera.

[Energy99]

I did the dtrace and I do see it looking for the replica centera.  In fact the dtrace shows the same results of the following article ...http://www.symantec.com/business/support/index?page=content&id=TECH55736&key=50996&actp=LIST.  I brought this to the attention of support but they don't feel this would impact us to the extent were seeing.  

I should note we recently upgrade to 8.0 SP5, we were running fine for a week and then we proceeded with pausing replication and moving the centera.

[TonySterling]

Well the dtrace should show how long it is spending on each item.

How long til the replica is back up?

[JesusWept3]

Are you absolutely sure archiving has stopped? It's most likely just archiving but because you use replication, you are in an after backup scenario so it's not changing the pending shortcuts to full shortcuts

[Energy99]

Replica is back online, but with no load everything is quick, Monday morning will be the true test if having the replica offline was the issue. 

Thanks for all the feedback..

[Bruce_Cranksh1]

Another consideration is around Deletions .EV will always delete from the Replica first ,so if this was Offline then it wouldn't be able to remove the Clip . That would impact performace ,did you get loads of errors in the EV Event Viewer around deletions ?

[Energy99]

Yes, we were seeing loads of deletion errors in the log.  We were told these are to be expected.  Apparently once the replica is back online these deletions (Exchange) will be flushed from the JournalDelete table so these items will no longer be seen by the users in Archive Explorer. 

Would anyone know the SQL query to be used to check what items are in the JournalDelete table?

[AndrewB]

how about something like:

USE yourVaultStoreDB
SELECT COUNT(*) FROM JournalDelete

[Bruce_Cranksh1]

Yes ,once the Replica is back online EV will delete the item

If you look at the actual deletion error in the Event View you should see its the same items it tries to delete . It doesn't move on until those items are removed .

So I would just wait until the Replica is back online ,or disable Deletions temporarily .

I have attached included the Centera deletion process so you can see how its the Replica it will delete from first

[Energy99 · Accepted solution]

Thank you everyone for your comments... The Replica is backonline and everything is running at peak performance again.  This issue appears to be resolved by putting the replica back online.  Lesson learned, pausing replication and downing your replica does more than just effect your performance a little (TECH55736), it could bring your Exchange and FSA down to a crawl.

[AndrewB]

Was Exchange itself impacted or was it just user experience when recalling archived emails?

[Energy99]

Just the user experience when recalling archived email was effected.  For FSA both archived and unarchived files were effected, in fact the entire file cluster was bouncing around from node to node.

[AndrewB]

i'm going to have to do the same thing very soon so i really appreciate you sharing your experience

[Energy99]

In our environment when a user deletes an archived item it will also delete the item from the vault.  We were told that these deletes would remain pending until the replica was back online and replication resumed.  Once the replication was up to date the deteletd mail items were never removed from the vault.  Symantec is investigating why.