1 Million Objects in A1 queue in Enterprise Vault 8 Sp4

[Ronen_Tzimbel]

Our environment working with Safety copy (Remove After Backup)

and we working without OSIS (Optimize Single instance), we stay with Default Upgrade group when we Upgrade from version 2007.

I run the archive task in report mode   for all of the users  to reset  the pending item

And after that I see that all of the item out of pending.

My question Is,   why the relevant DVS  File not delete  from the partition ?

note : the size in all of the Item that we reset pending is approximately 80 Giga.

Is it by design?

Many Thanks

Ronen

[JesusWept3]

When an item is stored by Enterprise Vault in an After Backup mode , the following occurs

1. The item goes in to the A2 queue to be processed
2. The item is turned in to a Pending Item in the users mailbox
3. An item is put in to Vault Store database in to two tables, JournalArchive and WatchFile
4. In the JournalArchive table, it has a column named "BackupComplete" and set to 0
5. In the WatchFile table is a full path to the physical DVS file that is created
6. Every several hours or when the storage server is restarted, StorageFileWatch scans all of the items in the WatchFile table
7. If the ArchiveBit has been removed it then Deletes the WatchFile entry to the item
8. The process then changes the BackupComplete column from 0 to a 1
9. Once its been confirmed as Backed up, it puts the item in to the A1 queue
10. The archive task then changes the Pending Item in to a full shortcut

Once an item has been fully stored, the DVS files are written and the items are added to the Indexes, if you run a cancel pending against the items, it will only turn the items back in to regular mail and will not be removed from Enterprise Vault.

Once the items are archived again you will have duplicate items and when you search for the items you will get multiple results for the same item.

Most likely what happened in your place, items had not been backing up for a long long time, then after a while they got fully backed up, and added to the A1 queue...

Either the Archive Tasks weren't running and able to keep up or your MSMQ breached its quota. By default in Windows 2003 SP2 and above, the quota is 1GB, so as soon as it contains more than 1GB of data, nothing will be processed against that MSMQ

[Ronen_Tzimbel]

hi,

yes, the item had not been backing up for a long time.

How can I delete duplicate items in automatically procedure ?

many thanks Ronen

[JesusWept3 · Accepted solution]

Technically you can't
Your best bet would actually be better off doing a DR to the time before you did the cancel pending, including a DR of the storage and databases, there was at one time a dedupe cleaner but was strictly used by support and may not even be applicable any more.

So you have several choices

1. Live with the duplicates and have your users delete them as and when they see them
2. Contact support and see if they can help automating the de-duplication
3. Contact Symantec Consultancy to see if they can help
4. Contact a third party such as Globanet to develop a tool to deduplicate the items
5. Run a Disaster Recovery before the cancel pending occured

None are particularly great choice but there really isn't anything you can do

[Ronen_Tzimbel]

Thanks,

Ronen

[Bruce_Cranksh1]

@ JesusWept

This is an interesting thread but there is something I am missing that maybe you can explain  ,why would he see duplicates ?

My understanding is that prior to Version 8 , EV used SIS within the Vault Store Partition.

Ronen  never allowed EV to complete the steps you mentioned as he reversed the Archive Pending items.

In the case above the original item has not  been  fully stored so therefore not Indexed .

A search is always against the Index ...so wouldn't EV  when it now archives the same item again determine the Item exists within the Partition and then Index it the first time ?

So he should only see one item as all copies are references to the original  within the same Partition ?

[JesusWept3]

nope, so what happens is as soon as the item goes to the A1, the items already been added to the saveset tables and such, shared out and what not, its been committed to the stores already, when it goes to A1 its 99% complete, all there is left to do is to convert it to a full shortcut

When you cancel pending the *only* time it won't make a duplicate is if the item has not left the A2 queue, so say the archiving task is failing to read that particular message or what not and you have a bunch of pending items, then you can find that you can cancel pending and all it will do is remove the item from A2 and turn it back from an IPM.NOTE.EnterpriseVault.PendingArchive to a regular IPM.NOTE message

However when you cancel pending on an item, it will just clear the Transaction ID and treat it as a newly archived item, the only time it will "re-archive" an item and treat is as already being archived is when you restore the item and it leaves saveset information and restored dates and what not in the mapi metadata and then re-archives it

Its an issue as old as EV itself

[Bruce_Cranksh1]

@  JesusWept

That is interesting ...I knew I could rely on your insight to explain it :)