Journal Message De-Duplication - auto-forward the mail from a mailbox on a round-robin basis
Hi,
Just a quick question to see if anyone has implemented any work-arounds to the duplication caused by having multiple journal mailboxes in Exchange.
An idea that has been passed around (from Symantec consultant) is to point Exchange to a single mailbox and auto-forward the mail from this mailbox on a round-robin basis to multiple journal mailboxes in turn. This does mean that the active journal mailbox will be receiving more mail than EV can handle, but when the forwarding moves on to the next mailbox EV finish the backlog before it's next turn.
I've over 1,000,000 messages per hour to journal & must use 30 active EV servers to ingest from 200 journal mailboxes.
We have a high usage of DA and CA. almost 500 CA reviewers. Duplicates are killing our reviewers. In some legal cases 90 % of items discovered in DA / CA are duplicates. Meaning almost if you export 1000 items, it is actually 100 items that shows like 1000.
We can’t shrink number of journal mailboxes because then EV servers can’t keep of with the rate of items coming in.
The solution is to journal only to 1 mailbox (I like to call it proxy mailbox), do not archive directly from proxy mailbox but instead auto forward round robin to other mailboxes (ex. every 20 minutes auto-forward to a new journal mailbox) then have EV to archive from these journal mailboxes. So in this scenario I will forward only 20 minutes to each journal mailbox but I have hours to ingest items from that mailbox.
I tested this in the lab, it eliminated CA duplicates, DA duplicates, Storage duplicates, shrunk SQL Vault Store database size and shrunk indexes size.
The question is that I don’t have a automatic process to auto-forward messages to different mailboxes. I can write a script but this will takes months. Any suggestions? Have anybody used any tool to do that?
Have you considered increasing the number of threads per journal archive target/task as AndrewB advised? If so has it helped? Obviously this will impact your system so you need to make sure you have the resources to cope with it and that you wont run into any Mapi session limitations.
From a reviewer perspective CA/DA can remove duplicate message for reviewers providing the messages meet the minimum criteria for de-duplication purposes, in DA the reviewer has three option to choose from in order to reduce the messages displayed.
If you are absolutely certain that it is the department tagging in the Journal connector that is creating the delay then it would need some more investigation to try and determine the root of the performance degradation. In order to do that it would be necessary to see a JournalTask dtrace and assess where the biggest delta's are between the various functions.