cancel
Showing results for 
Search instead for 
Did you mean: 

How to clean the watchfile table?

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

Hello all,

We are fighting with an issue that prevents us from upgrading to EV9 and EV10 consequently.

We took over support for an EV8 setup that was configured with single Centera system without replication and therefore it has the watchfile table with 54M items (~24M items for the Journal Vault Store DB and ~30M items for the Mailbox Vault Store DB).

As it was already learned, these tables should be cleaned up before the upgrade otherwise the upgrade could take days or even weeks with EV locked out from users. So, the technote http://www.symantec.com/docs/TECH68494 was implemented.

It works fine for the Journaling (it's slow, though) process - mails are journaled and table processing is going on in the same time.

Unfortunately, it doesn't work well for the Mailbox database. Queue A1 is piling up with items and table processing is going on, archived mails are accessible but new mails can not be archived (new requests correctly are being correctly placed to A2 queue, but never processed because A1 queue never drains).

So, the questions are:

1. Is there any way to clean up the table in one shot using some SQL query? We definitely don't need to go through each and every item, because neither of those will have a replica.

2. If option 1 is not available, is there any way to keep the cleanup process running, but at lower priority comparing to mailbox archiving and keep the EV system fully available to users? 

1 ACCEPTED SOLUTION

Accepted Solutions

JesusWept3
Level 6
Partner Accredited Certified
Check your connect inbox, just sent you a message
https://www.linkedin.com/in/alex-allen-turl-07370146

View solution in original post

15 REPLIES 15

LCT
Level 6
Accredited Certified

WOW....54M items....I have sen around 10M items...what's been happening with your exchange databases?

To clean up the watchfile table it's not difficult, as you can just write a delete SQL query and wipe out the entire table, however, it is something that one should NEVER do manually. this watchfile table is linked with other databases/tables, if you wipe this manually you will get more problems than you bargain for.

As your issue is more serious than a lot of issues that I have seen on this forum I would strongly suggest that you log a case with SYMC Tech Support in order for them to resolve the problem for you or at least point you in the right direction without causing further damage to your environment.

MichelZ
Level 6
Partner Accredited Certified

I would do the same as LCT suggests... ask support if you can delete those items in SQL. The deletion itself should not be complicated (DELETE FROM Watchfile).

Your second suggestion is not possible AFAIK, as the queues are prioritized A1 to A7.


cloudficient - EV Migration, creators of EVComplete.

JesusWept3
Level 6
Partner Accredited Certified
Blimey, what is it with people on this forum goin delete happy on the watch file? Regardless of its size I agree with LCT, troubleshoot the problem else the issue will keep reoccurring
https://www.linkedin.com/in/alex-allen-turl-07370146

MichelZ
Level 6
Partner Accredited Certified

I think they already "solved" the issue itself by implementing the Technote referenced in the OP's post.
The immediate problem is waiting for the table to clean up, and archiving effectively coming to a halt it seems. In this case, I can see why they want to delete the watchfile contents.

And I agree, usually, you should try to fix problems, and not circumvent them :)


cloudficient - EV Migration, creators of EVComplete.

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

I would love to let SYMC support to resolve the problem.

We we turned away from support with the following "Non-replicated Centera is not supported with EV8.0SP4 and earlier versions, clean the watchfile table, upgrade it to EV8.0SP5 at least and then we will support you".

Now I'm trying to figure out a way to clean up the watchfile table without interrupting access to EV for undefinite amount of time while the IgnoreBackup workaround slowly cleans the table.

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

Support says other way round, clean up the table, upgrade the EV and we'll support you afterwards :)

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

Michel, unfortunately we know how to fix the root of the problem. First, we need to set IgnoreBackup for VaultStore DB and wait for the table to clean-up. Next we need to upgrade EV to the next version where non-replicated Centera is supported. Now we are looking for a creative way to workaround slowness of the IgnoreBackup workaround :)

MichelZ
Level 6
Partner Accredited Certified

Unfortunately, I think you're out of options here..
Either you delete the table (unsupported then), or you wait until it cleans up...


cloudficient - EV Migration, creators of EVComplete.

LCT
Level 6
Accredited Certified

Ouch!!! That is harsh from support, to be fair, this environment was probably been built incorrectly by using a non-support storage type to begin with, hence you are inheriting a badly built environment as a consequent of 54M items waiting for backup. I was once a SYMC Tech Support so I know it's a vicious circle unfortunately.

What I would do in your case is increase and give the EV server as much RAM and disk space to beef up MSMQ memory and let the A1 queue process as much items as quick possible and wait and hope for the best. :)

Deleting the items in watchfile table will get you to the point of upgrading after it has been cleared but you WILL get problems in the long run, could be wosre, you may get problems during the Vault Store database upgrade which will not help.

JesusWept3
Level 6
Partner Accredited Certified

hang on a second, if its a centera and there is no replica then it simply will not go away, period.
Call symc support and ask for a registry key to get it moving

https://www.linkedin.com/in/alex-allen-turl-07370146

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

EV server (surprisingly) is not loaded at all. The old SQL server has been already maxed out, we have provisioned a new server and maxed out its resources too :) 

Unfortunately, hope is not a strategy, that's why I'm looking for a creative solution :)

Thanks anyway!

Mouse
Moderator
Moderator
Partner    VIP    Accredited Certified

With IgnoreBackup option set for the VaultStore it goes away, but slow (2000 items/minute).

We requested support to help us, but were turned away with reference to a technote: http://www.symantec.com/docs/TECH156575 that says our version 8.0SP4 is not supported in this configuration.

Can you please elaborate, which registry key we need to request support to set in the system? 

JesusWept3
Level 6
Partner Accredited Certified
Check your connect inbox, just sent you a message
https://www.linkedin.com/in/alex-allen-turl-07370146

MichelZ
Level 6
Partner Accredited Certified

On another note... You could try to up the "Number of concurrent connections to Exchange Server", which may clean the A1 queue faster and thus allows for other queues to be processed.

Cheers


cloudficient - EV Migration, creators of EVComplete.

John_Chisari
Level 6
Partner Accredited

You know, just thinking, upgrading to EV8 SP5 does not include the changes which will take the upgrade ages to complete if you have millions of items in JournalArchive, JournalDelete and JournalUpdate tables.  I think that came through in EV9 SP2.  So you could upgrade to EV8 SP5 and then call them back ;)  But I think JW may have fixed you up already.

Once you do fix the watchfile table - you will still have millions of items in the JournalArchive tables.  You will need to set the transaction history to a very low number (0 is good) - then restart the EV services and let this process go through to clear the JournalArchive tables etc.  Then do the upgrade to EV9 SP3.