EV - Mail Archiving Event ID 2262

[sherb]

Hi,

Got an ongoing problem with archiving our exchange system.    

System Setup

EV Server 1 – Mail Archiving and Directory Server.

                Windows 2003 Enterprise SP2

                Enterprise Vault  8.0.5.1048 – SP5

EV Server 2 – File Archiving

                Windows 2003 Enterprise SP2

                Enterprise Vault  8.0.5.1048 – SP5

SQL Server –

                Windows 2003 Enterprise 64-bit SP2

                MS SQL 2005

Mail Server 1 –

                Windows 2008

                Exchange 2007

Mail Server 2

                Windows 2008

                Exchange 2007

The vault stores are backed up by 2 centera devices which replicate to each other.

Issues:

Back on the 14th of Feb, came in to find that all of the EV services had stopped.  Checked and EV Event view was showing hundreds of 2273 events linking to 2 mailboxes on the same Exchange Server.  This was causing the services to stop when an excessive number of error messages was reached – Event ID 4183.  To try to resolve this I disabled both mailboxes and restarted the services and the archive tasks.  The same problem occurred again linked to the 2 mailboxes.  I tried to archive only one mailbox on that server, and the same problem occurs – it still generated 2273 errors linked to the problem mailboxes.  I checked the Application log, and we were getting a number of different Events in there.  The first Entry is an error 2216.  It generates this event relating to alternative Exchange servers every 5 minutes or so until 5 in the morning, when the Archive Task Service is restarted.  This then occurs again at 11:55, and keeps happening every 5 minutes until 17:00 which is when the archive task is scheduled to start.  At this point it starts generating 2262 errors every second for both servers, until the Archive Schedule ends at 21:00.  It continues to generate 2216 errors during this time, until  the services restart at 05:00 again.  The system just continued in this pattern.  

When investigating this, I found that Starting Processing on the archive jobs wasn't working.  On investigation, I'd noticed that the EV message queues were very full and didn't seem to be reducing.  I attempted a purge but wasn't able to clear a couple of the logs.  I tried to restart the MSMQ Service but it was not starting up.  At this point, we rebooted the server and the MSMQService still failed to start.  We'd had this issue before, so we did our usual fix (removing lg1, lg2, and QMlog and amending the registry to create a new log).  This cause the service to start ok, and the queues were clear.  We then ran archiving and it seemed to work successfully. We had a successful archive on Thursday and Friday, but on Sunday we seem to have run into problems again.  In the EV event Log, we are getting ID 3156 and 2262 every few moments while the archive task is scheduled.  We also have occasional 8390 errors - this is quite usual.  Once the Archive task schedule is over, we instead get 6190, 2216 and 2217's all tied to the Retrieval Task.  These errors repeat every 5 minutes, until the services do a scheduled restart at 5am.  Users were able to open archived emails ok, but trying to retrieve mails from the archive doesn't work at all. 

The next attempted fix was to reinstall MSMQ.  Did this, and the archive jobs ran fine for 2 days before running into the same probems and errors for the Archiving Tasks.  However, users are now able to retrieve emails perfectly and we are not getting any errors regarding the Retrieval Task. 

It has been suggested that the EV System accounts have been moved or are on the wrong exchange server.  Have checked this, and it doesn't appear to be the case. 

We have a call logged with EV but were wondering if anyone else has come across this issue before as we don't seem to be getting anywhere fast. 

[AndrewB]

have you checked the size of your MSMQ storage limit? the default is 1gb and it's recommended to increase that to 8gb.

[sherb]

Going to try that this evening as it is set to 1gb at the moment. 

Have managed to find a bit more information.  Came this morning to find that the Archive had ran ok up to about 19:00, and then the queue started to get 8390 RPC errors.  These stopped when the archiving schedule was finished.  Tried a manual archive, and started getting the 3156/2262 errors.  Previously when this problem has occurred, the log is so full of 3156 and 2262 errors we've not been able to see the time or instance that the problem occurred.  Managed to find this article - https://www-secure.symantec.com/connect/forums/event-id-8390-enterprisevaultdirectoryconnection-obje... so I'll try that as well. 

[sherb]

Amended the MSMQ storage size didn't help.  Over the weekend MSMQ seemed to have become extremely damaged which was preventing retrievals.  Reinstalled MSMQ again, and amended settings, etc.  Have turned off shortcut processing, and we have had successfully archived for the past couple of days.  Am going to run shortcut processing alone today and see how that gets on. 

[sherb]

Have ran the archive processing with a Dtrace and got some rather interesting results which seem to point to one mailbox.  I've pasted these comments below:

12194329 11:04:01.106  [4664] (ArchiveTask) <8340> EV:M CQMonitor::MQ Queue:MDS-SAN-EV1\Private$\Enterprise Vault Storage Archive Status:1
12194330 11:04:02.043  [4664] (ArchiveTask) <780> EV:H VaultCoCreateInstanceEx: An error occurred - RequestedServerName = [1B13044424E94AF41A503], UsedServerName = [1B13044424E94AF41A503], hrCCI = [0x800706BA], hrResultsQI = [0x80004005], NumTried = [6], bLocalMachine = [False]
12194331 11:04:02.043  [4664] (ArchiveTask) <780> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() failed.|Hr: 0x800706ba|Retries: 6|TryLocal: false|Directory Name: 1B13044424E94AF41A503|
12194332 11:04:02.043  [4664] (ArchiveTask) <780> EV:M HRXEX fn trace :|Source     : .\DirectoryServiceWrapper.cpp (CBaseDirectoryServiceWrapper::GetInterface) [lines {656,696,722}] built Oct  6 11:23:02 2010|Description: Call error 0x800706ba |ref[1]: 1B13044424E94AF41A503
12194335 11:04:02.043  [4664] (ArchiveTask) <7684> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12194337 11:04:02.043  [4664] (ArchiveTask) <780> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12194338 11:04:02.043  [4664] (ArchiveTask) <780> EV:M CFolderHelper::GetFolderSettings() - Error 0x800706BA|Internal:..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010
12194339 11:04:02.043  [4664] (ArchiveTask) <780> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolder) [lines {2605,2615}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12194340 11:04:02.043  [4664] (ArchiveTask) <780> EV:M CFolderHelper::GSF() - Error getting 'special' folder Ids for mailbox, 0x800706BA
12194341 11:04:02.043  [4664] (ArchiveTask) <780> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolder) [lines {...,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12194342 11:04:02.043  [4664] (ArchiveTask) <780> EV:M CFolderHelper::GSF() - Error getting 'special' folder Ids for mailbox, 0x800706BA
12194343 11:04:02.043  [4664] (ArchiveTask) <780> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolder) [lines {...,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12194344 11:04:02.043  [4664] (ArchiveTask) <780> EV:M CFolderHelper::GSF() - Error getting 'special' folder Ids for mailbox, 0x800706BA
12194345 11:04:02.043  [4664] (ArchiveTask) <780> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolders()) [lines {1879,1902,1906,1907,1913}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12194346 11:04:02.043  [4664] (ArchiveTask) <780> EV:M CFolderHelper::SynchronizeFolders() - Error 0x800706BA|Internal:..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolders()) [lines {1879,1902,1906,1907,1913}] built Oct  6 11:22:25 2010
12194347 11:04:02.043  [4664] (ArchiveTask) <780> EV:M CArchivingAgent::ProcessUserEx - Mailbox is [Not Enabled] or is [Suspended]
12194348 11:04:02.043  [4664] (ArchiveTask) <7684> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
12194350 11:04:02.043  [4664] (ArchiveTask) <9716> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]

11:06:26.840  [4664] (ArchiveTask) <9368> EV:H VaultCoCreateInstanceEx: An error occurred - RequestedServerName = [1B13044424E94AF41A503], UsedServerName = [1B13044424E94AF41A503], hrCCI = [0x800706BA], hrResultsQI = [0x80004005], NumTried = [6], bLocalMachine = [False]
12710381 11:06:26.840  [4664] (ArchiveTask) <9368> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() failed.|Hr: 0x800706ba|Retries: 6|TryLocal: false|Directory Name: 1B13044424E94AF41A503|
12710382 11:06:26.840  [4664] (ArchiveTask) <9368> EV:M HRXEX fn trace :|Source     : .\DirectoryServiceWrapper.cpp (CBaseDirectoryServiceWrapper::GetInterface) [lines {656,696,722}] built Oct  6 11:23:02 2010|Description: Call error 0x800706ba |ref[1]: 1B13044424E94AF41A503
12710385 11:06:26.840  [4664] (ArchiveTask) <7684> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12710387 11:06:26.840  [4664] (ArchiveTask) <9368> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12710388 11:06:26.840  [4664] (ArchiveTask) <9368> EV:M CFolderHelper::GetFolderSettings() - Error 0x800706BA|Internal:..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010
12710389 11:06:26.840  [4664] (ArchiveTask) <9368> EV:L :CArchivingAgent::PF_SetFilterAndRetention() |There is no current folder filter for the folder HPV_core_messages, so the parent folder filter will be used |
12710390 11:06:26.840  [4664] (ArchiveTask) <9368> EV:L :CArchivingAgent::PF_SetFilterAndRetention() |The VaultId value applied to the folder HPV_core_messages is (null) |
12710391 11:06:26.840  [4664] (ArchiveTask) <9368> EV:L :CArchivingAgent::PF_SetFilterAndRetention() |The Retention Category value applied to the folder HPV_core_messages is (null) |
12710392 11:06:26.840  [4664] (ArchiveTask) <9368> EV:H :CArchivingAgent::PF_SetFilterAndRentention() |Exiting routine  |
12710393 11:06:26.840  [4664] (ArchiveTask) <9368> EV:H :CArchivingAgent::ProcessFolders() |Exiting routine |
12710394 11:06:26.840  [4664] (ArchiveTask) <9368> EV:H :CArchivingAgent::PF_ProcessSubFolders() |Exiting routine  |
12710395 11:06:26.840  [4664] (ArchiveTask) <9368> EV:H :CArchivingAgent::ProcessFolders() |Exiting routine |
12710396 11:06:26.840  [4664] (ArchiveTask) <9368> EV:H :CArchivingAgent::PF_ProcessSubFolders() |Exiting routine  |
12710397 11:06:26.840  [4664] (ArchiveTask) <9368> EV:H :CArchivingAgent::ProcessFolders() |Exiting routine |
12710398 11:06:26.840  [4664] (ArchiveTask) <9368> EV:M ExMbxArcFolderHelper - Starting EMAF Sync
12710399 11:06:26.840  [4664] (ArchiveTask) <7684> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
12710401 11:06:26.840  [4664] (ArchiveTask) <9716> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12710402 11:06:26.840  [4664] (ArchiveTask) <9368> EV:M Updating ExMbxArcFolder records

Turned on the Shortcut and Archive Processing last night, and got 8 RPC failures in towards the end of the job.  Ran this again with Dtrace against it, and have found the following 2 issues.  These repeat at appropriate times to tie in with the RPC Failures.  

12758250 11:07:17.028  [4664] (ArchiveTask) <7832> EV:H VaultCoCreateInstanceEx: An error occurred - RequestedServerName = [1B13044424E94AF41A503], UsedServerName = [1B13044424E94AF41A503], hrCCI = [0x800706BA], hrResultsQI = [0x80004005], NumTried = [6], bLocalMachine = [False]
12758251 11:07:17.028  [4664] (ArchiveTask) <7832> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() failed.|Hr: 0x800706ba|Retries: 6|TryLocal: false|Directory Name: 1B13044424E94AF41A503|
12758252 11:07:17.028  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : .\DirectoryServiceWrapper.cpp (CBaseDirectoryServiceWrapper::GetInterface) [lines {656,696,722}] built Oct  6 11:23:02 2010|Description: Call error 0x800706ba |ref[1]: 1B13044424E94AF41A503
12758255 11:07:17.028  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12758256 11:07:17.028  [4664] (ArchiveTask) <7832> EV:M CFolderHelper::GetFolderSettings() - Error 0x800706BA|Internal:..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010
12758257 11:07:17.028  [4664] (ArchiveTask) <7832> EV:L :CArchivingAgent::PF_SetFilterAndRetention() |There is no current folder filter for the folder HPV_core_messages, so the parent folder filter will be used |
12758258 11:07:17.028  [4664] (ArchiveTask) <7832> EV:L :CArchivingAgent::PF_SetFilterAndRetention() |The VaultId value applied to the folder HPV_core_messages is (null) |
12758259 11:07:17.028  [4664] (ArchiveTask) <7832> EV:L :CArchivingAgent::PF_SetFilterAndRetention() |The Retention Category value applied to the folder HPV_core_messages is (null) |
12758260 11:07:17.028  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::PF_SetFilterAndRentention() |Exiting routine  |
12758261 11:07:17.028  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::ProcessFolders() |Exiting routine |
12758262 11:07:17.028  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::PF_ProcessSubFolders() |Exiting routine  |
12758263 11:07:17.028  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::ProcessFolders() |Exiting routine |
12758264 11:07:17.028  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::PF_ProcessSubFolders() |Exiting routine  |
12758265 11:07:17.028  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::ProcessFolders() |Exiting routine |
12758266 11:07:17.028  [4664] (ArchiveTask) <7832> EV:M ExMbxArcFolderHelper - Starting EMAF Sync
12758267 11:07:17.028  [4664] (ArchiveTask) <7832> EV:M Updating ExMbxArcFolder records
12758269 11:07:17.028  [4664] (ArchiveTask) <7832> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12758270 11:07:17.043  [4664] (ArchiveTask) <7832> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
12758271 11:07:17.059  [4664] (ArchiveTask) <7832> EV:M Updating ExMbxArcFolder records
12758273 11:07:17.059  [4664] (ArchiveTask) <7832> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12758274 11:07:17.074  [4664] (ArchiveTask) <7832> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
12758275 11:07:17.090  [4664] (ArchiveTask) <7832> EV:M Performing post-processing on ExMbxArcFolder Records
12758277 11:07:17.090  [4664] (ArchiveTask) <7832> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12758278 11:07:17.106  [4664] (ArchiveTask) <7832> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
12758279 11:07:17.106  [4664] (ArchiveTask) <7832> EV:M ExMbxArcFolderHelper - Finished EMAF Sync
12758280 11:07:17.106  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::ProcessUser() |Exiting routine |
12758281 11:07:17.106  [4664] (ArchiveTask) <7832> EV:H CArchivingAgent::ProcessUserEx Time Taken to process mailbox - /o=The University of Birmingham/ou=Medicine/cn=Recipients/cn=warminsa: 50.138851 secs
12758282 11:07:17.106  [4664] (ArchiveTask) <7832> EV:M CMailboxUsage::RemoveUserFromList - Removing MbxDN to List:[/O=,<info removed for Security>| /CN=RECIPIENTS/CN=WARMINSA]
12758283 11:07:17.106  [4664] (ArchiveTask) <7832> EV:M :AgentMessageDispenser::ActivateObject() |An error that we do not specifically recognize has occurred, so we'll just increment this messages retry count |
12758284 11:07:17.106  [4664] (ArchiveTask) <7832> EV:H :AgentMessageDispenser::ActivateObject() |Exiting routine at point A |
12758285 11:07:17.106  [4664] (ArchiveTask) <7832> EV:M :AgentMessageDispenser::ProcessNextMessage() |ActivateObject has returned failure |
12758286 11:07:17.106  [4664] (ArchiveTask) <7832> EV:M :AgentMessageDispenser::ProcessNextMessage() |Activate object returned with status AGENTS_E_DISPRETRY. The retry count will be incremented, and the message reposted to the end of the queue |

12728325 11:06:49.981  [4664] (ArchiveTask) <7832> EV:H VaultCoCreateInstanceEx: An error occurred - RequestedServerName = [1B13044424E94AF41A503], UsedServerName = [1B13044424E94AF41A503], hrCCI = [0x800706BA], hrResultsQI = [0x80004005], NumTried = [6], bLocalMachine = [False]
12728326 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() failed.|Hr: 0x800706ba|Retries: 6|TryLocal: false|Directory Name: 1B13044424E94AF41A503|
12728327 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : .\DirectoryServiceWrapper.cpp (CBaseDirectoryServiceWrapper::GetInterface) [lines {656,696,722}] built Oct  6 11:23:02 2010|Description: Call error 0x800706ba |ref[1]: 1B13044424E94AF41A503
12728330 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12728331 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CFolderHelper::GetFolderSettings() - Error 0x800706BA|Internal:..\AgentsCommon\FolderHelper.cpp (CFolderHelper::GetFolderSettings) [lines {375,424,429,434,437,672}] built Oct  6 11:22:25 2010
12728332 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolder) [lines {2605,2615}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12728333 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CFolderHelper::GSF() - Error getting 'special' folder Ids for mailbox, 0x800706BA
12728334 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolder) [lines {...,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12728335 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CFolderHelper::GSF() - Error getting 'special' folder Ids for mailbox, 0x800706BA
12728336 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolder) [lines {...,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668,2661,2668}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12728337 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CFolderHelper::GSF() - Error getting 'special' folder Ids for mailbox, 0x800706BA
12728338 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M HRXEX fn trace :|Source     : ..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolders()) [lines {1879,1902,1906,1907,1913}] built Oct  6 11:22:25 2010|Description: Call error 0x800706ba
12728339 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CFolderHelper::SynchronizeFolders() - Error 0x800706BA|Internal:..\AgentsCommon\FolderHelper.cpp (CFolderHelper::SynchronizeFolders()) [lines {1879,1902,1906,1907,1913}] built Oct  6 11:22:25 2010
12728340 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CArchivingAgent::ProcessUserEx - Mailbox is [Not Enabled] or is [Suspended]
12728342 11:06:49.981  [4664] (ArchiveTask) <7832> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12728343 11:06:49.981  [4664] (ArchiveTask) <7832> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
12728344 11:06:49.996  [4664] (ArchiveTask) <7832> EV:M CFilterHelper::StampDefaults - calling SetProps
12728345 11:06:49.996  [4664] (ArchiveTask) <7832> EV:M CFilterHelper::StampDefaults - calling SaveChanges
12728346 11:06:49.996  [4664] (ArchiveTask) <7832> EV:M MAShortcutUpdate: Checking whether shortcut update is required for VaultID: 1B13044424E94AF41A5036559274D327A1110000mds-evsite.adf.bham.ac.uk
12728348 11:06:49.996  [4664] (ArchiveTask) <7832> EV:M CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
12728349 11:06:50.012  [4664] (ArchiveTask) <7832> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
12728350 11:06:50.012  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::ProcessUser() |Reading users quota limit and current usage |
12728351 11:06:50.106  [10124] (w3wp) <8344> EV:L CFederatedSearchBroker::WaitForResults - Elapsed: 39 (sec)|Active: 1 ;|Failed: 0|
12728352 11:06:50.481  [4664] (ArchiveTask) <7832> EV:H A storage limit has been found on mailbox CN=warminsa,<info removed for Security>| Quota Type: QUOTA_NO_SEND_RECEIVE| Quota Limit (K): 2411520| Quota Used (K): 800207| (Note: This is the information store default for all mailboxes) |
12728353 11:06:50.481  [4664] (ArchiveTask) <7832> EV:H :CArchivingAgent::ProcessUser() |We have an enabled mailbox that is ready for processing. |

These also seem to point to the same mailbox.  I've disabled the MB and ran the provisioning task, but it still keeps trying to archive the inbox.  Any thoughts? 

[MarkBarefoot]

RPC errors normally relate to not being able to communicate with the Directory Service on a particular EV server. In the trace we are trying to contact an EV server with an alias of 1B13044424E94AF41A503 - is this correct?

It's prob worth zipping the DTRACE up as the times above are out of sync. Was this trace a RUN NOW against a single mailbox?

[sherb]

These were run nows against all enabled mailboxes.  The problem is that the issue only seems to occur if all maiboxes are processed.  Unfortuantly, this generates huge dtrace logs, so we are having to run them manually for approx an hour at a time and hope that we catch the point where the error occurs. 

In terms of the alias for the Directory server - where can I check this?  I'm assuming this is held in the DB but can't find an appropriate table. 

[MarkBarefoot]

It's the EV Server alias that is shown in my last post, the 1B130.... but I don't think this is right, as this chunk of characters seem to be part of a Vault ID ( 1B13044424E94AF41A5036559274D327A1110000mds-evsite.adf.bham.ac.uk)

Ths ServerName is normally the alias of the EV server that is running the Directory Service, each server should be running this service and the task will try to use a local service where necessary. I'm confused as to why yours is trying to contact this odd ID.

You can do a Run Now against that particular mailbox to keep the DTRACE down, but this may warrant a call to Support.

[sherb]

Hi,

Sorry, I was assuming that EV must use a alpha-numeric code as its alias for the server.  Assuming that it just uses the usual server name alias, then that is definitly not correct.  As advised by EV I've zapped the problem mailbox and will see how we get.