We currently use Managed folders in exchange, let me be clear not Exchange manged folders but folders that we created through a utility that we then use EVPM to set a retention period on.
We have setup the Managed folder utility so that it will create default folders based on your department/title. Some of the defaults are general correspondence, 1 year - Project updates, 2 years. The default folders are automatically pushed to the users mailbox, if they have additional requirements they can request additional folders with different retention through the managed folder utility. Once the folders are created a EVPM file is created and automatically ran that will assign the proper retention to the folders.
This way when a user moves email around their mailbox they can change the retention based on what folder they place the email in. If the user does not move the item and they leave it in their mailbox it will be archived at 30 days with a 90 day retention. IN short if the user wants to keep emails longer than 90 days they MUST take action and move it to the proper folder.
Here in lies the problem that I am curious if anyone out that has noticed.
We have over 200 millions emails archived in the vault and all of the corresponding shortcuts in the users mailboxes. The users constantly have access to move the emails around and thus change the retention. the fact that we have so many shortcuts in the mailboxes means that it takes 2-3 days to process 3500 mailboxes per exchange server. when expiration comes around it DOES NOT check where the shortcut is and it just expires based on what the Database says. so the end result is that items which should be retained are being expired. Orphan shortcut cleanup process then comes around and within 2-3 days removes the orphaned shortcut.
Has anyone noticed this behavior in their environment?
We are currently running enterprise Vault 9.3. in January we are upgrading to 10.2.
This has occured in every version that we have been on. We started on EV 8.0, that was the first version that retention management was released.
What you are describing - i.e that the expiry is based on the database rather than the location of the shortcut - is true. When you do a scheduled archiving run, when items are identified as having moved within a folder, that folder is added to the A6 queue for processing post the archiving run (the messages on the A5 queue...the process mailbox requests...take priority). There have been issues in the past where retention was not properly updating when EVPM was used to set a filter on the folder, but your version is past that, and your description does not suggest this is the issue anyway.
So ultimately your problem/observation seems to be that it takes too long to do a complete archiving run -i.e not all mailboxes are archived during the archiving run and therefore you are having issues because the folders are only going to be added to the 'to process' list once you have archived them and there is a significant window for storage expiry to get in there first.
Either you are
1) Doing too many items per pass to go through all mailboxes
2) Not allowing enough time during the archiving window
How many items are you processing per pass ?
What does the archiving schedule look like?
Do you have thousands of messages typically left on the A5 queue after an archiving run?
The thing is though, even if you WERE to reduce the archiving window to a single pass and run expiry directly afterwards you would STILL have a window of opportunity for shortcuts to be moved before the item is reflected within the archive.
If the shortcut was to be expired from the mailbox due to shortcut expiry BEFORE storage expiry, then you would minimise this problem further. In other words, set the shortcut retention to be 90 days. The ACTUAL retention of the item then needs to be set slightly longer than this so that there is time for the shortcut to be processed. There is only one shortcut time deletion policy that can be applied to the whole mailbox so the assumption here is that once the item is filed, that it is then typically not moved to another folder and retention changed again. And now you are using a shortcut-less strategy for working with archived items, which means you will need to use archive explorer or virtual vault in order to regain client side access to the archive heirarchy.
A big drawback will therefore be the way that users interface with archived items changes.
One of the benefits of working like this is that because you are no longer using shortcuts, the number of objects in exchange will be reduced and so archiving runs will be faster.
If you need to change retention again after shortcuts are removed, then this can still be achieved using virtual vault drag and drop - but again you will be facing the small possibility that the item is deleted before the location change can be uploaded....
You are exactly correct, we do not get through the A5 queue on a nightly basis. We have been at this for three years and have tried changing many settings, items per pass, number of Exchange threads. While we have been able to whittle the A5 queue down to 2 days, we have never been able to process all mailboxes in a single archviing run.
Here is our current schedule.
Monday - Friday
Virtual Vault is a great idea but it also has short comings when it comes to expiration and the implemented use of managed folders. If a user is working offline say during a vacation and they move items around in their Virtual Vault the items that were moved will not be sync'd with the Vault and could potentially be expired before the user returns and syncs.
That is a fairly hefty archiving window. 8hrs x 2 means you are barely getting through a couple of hundred mailboxes per hour.
A typical archiving run may see 20 or so mails per user per day, with a 'heavy' environment as much as 100.
This puts my guess of your archiving rate somewhere between 4000 and 20000 items an hour - which is pretty poor as a total. Especially if you are using reasonable kit. How many exchange servers are you processing in this environment per EV server.
Have you got any idea what is the bottleneck in this environment?
Yes - there are still expiry issues with virtual vault too....although you could make it 'read only' and not allow changes, so that only during the first 90 days f the shortcut existence can the home of an item be fixed (without restoring and re-archiving).