Problem archiving a particular mailbox

[djbrown71]

Hi,

I have just noticed one of my mailboxes had old emails present that should have been archived. Coincidentally, the last email in the inbox was an EV message informing of the completion of a PST migration? 

The mailbox is provisioned properly and the correct policy applied (Archive everything over 3mths and create shortcuts).

So far I have disabled archiving and re-enabled for this mailbox, re-synched and ran another provisioning task.

If i try a 'Run Now' report nothing gets logged either.

Any ideas?

Darryl.

[TonySterling]

Can you check the modified date of the old messages?  If the messages were restored they won't be archived until the modified date reaches the Archive rules threshold.

If that is the case then you can use IgnoreEVDates to force it to archive right away.

If 32bit HKLM\Software\KVS\Enterprise Vault\Agents

or If the Windows Servers operative system is 64bit, add the key to the following path:

HKEY_LOCAL_MACHINE\Software\Wow6432Node\KVS\Enterprise Vault\Agents

IgnoreEVDates

0 = Off
1 = Ignore Archived date
2 = Ignore Restored date
3 = Ignore both Archived and Restored date

Set to 3, do a run now and archive the mailbox then set to 0.

[djbrown71]

Sorry i should have said that i had checked the modified dates and they do met the archiving policy criteria. I had come across this on a previous occasion but i wasn't aware of the IgnoreEVDates registry entry so thanks for that.

Any other ideas?

[TonySterling]

Next thing would be to check the message class for the items.  Also, can you manually archive them?

Lastly, do a dtrace of the Archive Task whilst doing a run now against that mailbox.

[Rob_Wilcox1]

Is the mailbox "suspended" from archiving?  You can check in the directory database, or do a run now in report mode.

[djbrown71]

The message class is IPM.Note which is set to be archived in the policy.

I can manually archive the items as the icon changes to pending.

The mailbox isn't suspended. If i perform a 'Run Now' in report mode i get no output in the reports folder.

I am planning to run a DTrace but i find it hard to decipher these sometimes :)

[KarlW]

What quota's do you use?  I assume it's within the send/receive limit?

I'm guessing the Dtrace may be pretty short - feel free to post here so we can take a look.

-Karl

[MarkBarefoot]

Are you doing a Run Now? I'm guessing the DTRACE wasn't a Run Now against the user Graham Topley? If so, these jobs go onto the A3 MSMQ. The A5 is scheduled jobs and can be purged (it 'should' goto 0 if there is enough time to do the work overnight etc)

Looking at the trace I would say that shortcuts in the mailbox TCEJ have invalid references and this is causing a delay in your archiving window. The particular thread ran for 170 mins against a number of mailboxes, trying to do the shortcut processing. It's trying to contact an EV server with an alias of DIUSSITE_ALIAS - can you ping this? I suspect it's not contactable.

I would suggest looking into this mailbox, if you can, or disable it temporarily from archiving.

Looking at the Graham Topley mailbox we did the following:-

1  Mailboxes processed for archiving (Process User)
1  Mailboxes detected as being enabled.
60  Folders scanned
11252  Items checked for eligibility for archiving (IsEligible)
1000  Items selected for archiving and sent for storing. (MakeitemPending...)
1  Items that are Archive Pending but not old enough to be cancelled yet.
4099  Shortcuts found but have not been moved.
1000  MSMQ Items picked up and processed by the agents
 

so we sent 1000 items to storage

you could also look at turning off shortcut processing to just get the mailboxes archived. Which version of EV are you running?

[djbrown71]

I was doing a 'Run Now' against Graham Topley and it was reflected in the A3 queue.Interestingly at the moment it says 1 when i view via the Private queues but when i look at the queue itself there is nothing there. I tried purging this but it still says 1.

 I mention the other queue as i saw things in it.The issue with the user TCEJ is worrying i thought DIUSSITE_ALIAS was something to do with EV. I did remember awhile back having a problem with someone who had been using a google connector in Outlook which stored IMAP information in a PST file. This PST was migrated to the EV and it caused a lot of issues with archiving. I will exclude this mailbox from archiving until i've worked out what is going on.

What impact will it have if i turn off shortcut processing?

Thanks for your help.

Darryl.

[MarkBarefoot]

If you exclude this user then you won't need to worry about shortcut processing, as their mailbox won't be scanned and therefore won't cause the RPC errors.

When I mentioned shortcut processing, I was referring to when you do a Manual RUN NOW and you can choose to do everything, or just archive. Shortcut Processing does a number of different operations which include checking for shortcuts that have moved, items that are ready for deletion, orphaned links etc.

I, personally, would purge the A5 MSMQ (whenever you purge you also need to purge the relevant ADMIN MSMQ too as the purged jobs drop into here). MSMQ also needs refreshing in the screen as they are not dynamic, and they can also show "ghost" entries - there are tools out there that are better than the Windows MSMQ viewer but I can't recall the names, Google MSMQ viewers and see what you find.

This TN, http://www.symantec.com/docs/TECH47392, shows how you can setup monitoring on the Storage Queue and get an idea of how your archiving is going with regard to your time window. It looks like you have a big enough window to be honest but anything that causes delays like this dodgy mailbox will have an impact.

[KarlW]

From what I can see in the log the mailbox Graham.Topley was processed three times.  For the first two passes the task reports 1000 items sent to storage and 6668 left to process.  What stands out as wrong here is that the second time there is still the same number of items left to process.

What stood out to me was the number of Operation Failed calls being processed in the trace (1998 in total). 

These indicate an issue with the Storage Service.  The archiving task appears to be archiving and sending items to storage which are then not being archived.  It would appear the mailbox is potentially stuck in a loop - sending the items to be archived, getting back the failures then sending the same items back again and so on.

Are there any errors in your EV logs regarding Storage?  Perhaps tracing StorageArchive at the same time might help.

Regards

Karl

[djbrown71]

Hi all, thanks to all of you who have offered advice on the issue i'm having.

I wanted to summarise where i have got to with my problem:

1 users mailbox which dosen't seem to archive during the nightly scheduled run. When i look at the A5 queue there is 1 record which corresponds to this mailbox.

If i run a Dtrace on the ArchiveTask when only the A5 shows only 1 entry (for the mailbox which isn't archiving) and the A6 queue shows 1 but doesn't actually have entry, i get the following:

EV:M CQMonitor::MQ Queue:EVSVR1\Private$\Enterprise Vault Storage Archive Status:1
EV:H VaultCoCreateInstanceEx: An error occurred - RequestedServerName = [DIUSSITE_ALIAS], UsedServerName = [DIUSSITE_ALIAS], hrCCI = [0x800706BA], hrResultsQI = [0x80004005], NumTried = [6], bLocalMachine = [False]
EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() failed.|Hr: 0x800706ba|Retries: 6|TryLocal: false|Directory Name: DIUSSITE_ALIAS|
EV:M HRXEX fn trace :|Source     : .\DirectoryServiceWrapper.cpp (CBaseDirectoryServiceWrapper::GetInterface) [lines {656,696,722}] built Nov  2 19:42:12 2009|Description: Call error 0x800706ba |ref[1]: DIUSSITE_ALIAS
EV:M HRXEX fn trace :|Source     : .\DirectoryServiceWrapper.cpp (CBaseDirectoryServiceWrapper::Invoke) [lines {401,407}] built Nov  2 19:42:12 2009|Description: Call error 0x800706ba
EV:M HRXEX fn trace :|Source     : .\DirectoryServiceWrapper.cpp (CReadDirectoryServiceWrapper::Invoke) [lines {1003,1019}] built Nov  2 19:42:12 2009|Description: Call error 0x800706ba
EV:M CArchivingAgent::ProcessShortcutItem - Could not scan user mailbox /o=National Physical Laboratory/ou=NPL/cn=Recipients/cn=TCEJ, cannot list messages [0x800706BA] [(null)]
EV:M CArchivingAgent::processChunkOfMessages - Call to ProcessShortcutItem Failed [0x800706BA] but we are carrying on to the next item
EV:H :CArchivingAgent::ProcessChunkOfShortcuts() |Entering routine |
EV:M CArchivingAgent::ProcessShortcutItem - processing shortcut '[UNCLASSIFIED] Annual Innovations Report ', size 5811 bytes
EV:M CArchivingAgent::ProcessShortcutItem - offline vault user F, shortcutexpiryperiod 0, deleteexpireditems T, orphanedshortcut T
EV:M CArchivingAgent::ProcessShortcutItem - Using default ExpiryDateBasis: Archived Date
EV:M CArchivingAgent::ProcessShortcutItem - Overriding ExpiryDateBasis: Calculated Date
EV:M CArchivingAgent::ProcessShortcutItem - '[UNCLASSIFIED] Annual Innovations Report ' this is a moved item that has been poison pilled so do not try and update its metadata again
EV:M CArchivingAgent::ProcessShortcutItem - '[UNCLASSIFIED] Annual Innovations Report ' has not moved so do checks to see if it is eligible for shortcut deletion
EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
225693076 11:19:35.085  [8044] (ArchiveTask) <10036> EV:M CQMonitor::MQ Queue:EVSVR1\Private$\Enterprise Vault Storage Archive Status:1 

The really wierd thing is that the mailbox TCEJ is referencing an EV server which doesn't exist! Interestingly our company is associated to a goverment body which was called DIUS. I disabled this user from archiving but am still getting info in the Dtrace.

Is it likely that the event is related to the record in the A6 queue which i can't see info on?

Where is this EV server information for TCEJ being picked up?

I am probably going to log this as a support case soon but from recent experience i think indepth issues like this take a long time to get resolved.

All advice appreciated.

Darryl.

[KarlW]

Darryl

The issue with shortcut processing referencing another server quite likely relates to a shortcut in the mailbox the user has received from the other system.  Unfortunately there is no current solution other than deleting the shortcut from the user mailbox.

Regarding the mailbox not processing did you trace storagearchive so we can look as to why it's generating lots of Operation Failed callbacks to the archiving task?

Regards

Karl

[KarlW]

I can spot the error in the trace around GetMsgProperty - this rings a bell.  I'm having a quick search around to see what I can find.  Will let you know what I find.

Thanks

Karl

[KarlW]

Sorry quick question I can't spot in the thread (and it's not in the trace) which EV version you are running.

Thanks

Karl

[JesusWept3 · Accepted solution]

based on the length of time, the evidence provided etc, i would suggest calling in a case with symantec support, have them reference this thread and they can go through the comments and dtraces and get up to speed pretty quickly....

[KarlW]

The issue I had in mind was similar but probably not related - I agree with JW2 that it would be best to open a support case.

Sorry I couldn't help further.

Regards

Karl

[djbrown71]

Running the DTrace at the moment. I've attached a zip of the results so far. The task is taking forever for this one mailbox which might account for why it never outputed to the when run as 'Report'. The mailbox is about 1.6GB.

An error which i noticed that didn't make sense was a reference to a different users mailbox during the Run Now archive task.The mailbox i am working on is 'Graham Topley' and the other mailbox referenced is 'TCEJ'.

I am concerned that my nightly archive runs are not finishing. The schedule is from 8pm to 7am. When i looked in the message queue a5 i had 244 items there. I hoping this isn't part of a much bigger issue. Is there any way to cancel anything pending before i run the task?

I believe i might have an issue with our exchange server related to performance due to mailboxes with large item counts. The idea with EV was to reduce mailbox sizes but because we haven't started deleted shortcuts the performance of exchange is suffering. The server only has 400 mailboxes, an average mailbox size of 700MB, about 50 mailboxes around a 1.5GB and a lot mailboxes with single folder item counts well over the MS recommended.

Help!

[djbrown71]

I purged the queues and was thinking about running another 'Run Now' for a single mailbox but before i did so i wanted to check if the archive task is doing anything. I have attached a Dtrace which was building a large log file so it looks like it's still doing something.

Any chance you could have a quick look?

[djbrown71]

Karl,

Thanks, i did suspect the reference to an other EV site was possibily related a shortcut in the persons mailbox. I have seen something similar when a PST with IMAP stubs was migrated to the vault.

I have done a 'Run Now' selecting Archive only on the first 500 items and ran a Dtrace with the ArchiveTask and StorageArchive trace options. See attached. Can you have a look to see if you can spot anything?

Many thanks

Darryl.