cancel
Showing results for 
Search instead for 
Did you mean: 

Cannot force vault mailbox

dorlow615
Level 4

I'm stumped over a mailbox vault again.  I have a user that somehow got a 120 GB mailbox.  I have to get it vaulted.  I've tried to force vault it many times and nothing.  Their vault store is empty.  I've disabled/re-enabled them for vault.  I disabled, deleted the vault store, re-enabled.  Changed where the vault store is at.  Zapped the archive and mailbox.  Deleted the mapi profiles for the server.  

In the dtrace, I do see some errors like this...

ProcessNextMessage() |ActivateObject has returned failure

 

Goggled the error.  States to make sure the service account for the task isn't hidden or disabled.  It's not.  The mailbox was just moved to a different database last week which should've gotten rid of corruption in the mailbox.  I've restarted the server.  

 

When I run this sql command, I get no results.

 

select * from VaultStoreEntry where VaultStoreEntryId = '1E174C048BCECD94BB48273648BF79B5A1110000EVSARCHIVE01'

 

I should get results (this is her EV archive ID.)

 

 

 

 

16 REPLIES 16

dorlow615
Level 4
 

daveoflave
Level 4
Employee

Hi dorlow,

   Really tough to say, based on just that one line.  Usually here in support we'll hone in on a thread in a dtrace and read several lines above to see where it goes wrong.  So since I'm a little blind here, I'll just throw out some of the easier points to rule in/out:

 

-Do you know for sure the task works? As in, have you successfully archived any other mailboxes with it?  You mentioned a vault store is empty, so maybe if you archived a mailbox in another vault store with that task previously, did it work?

-Are you sure that what the mailbox is full of is items that your Site settings and policy settings are configured to archive?  Meaning:  Do they have a bunch of attachments larger than your max attachment size limit? Are they mostly calendar items when you don't archive calendar items, etc?

-Might be worth creating a new system mailbox all together, since you mentioned that it was moved.  It's easy enough to do, and there's nothing special about the mailbox that you need to set; just give the VSA full permission and send as permission.

-Nothing stuck in the msmq's, you have an open partition, you're not in backup mode?

-When you do a "Run now" archiving run against that mailbox, do you get the start and stop events for it? I think 3442 and 3448 (or 3446 if finished with warnings) are the ones to look for, though that's just off the top of my head.

-Other than that, I'd suggest providing more of the dtrace.  The things I mentioned are all low hanging fruit; and likely it would be more complex.  But this is a good starting point to rule out.

Thanks,
Daveoflave

Attached is a dtrace.

I've been able to get some things to go to the vault.  I've opened up Outlook on the server as the user and can select the button to move emails to vault.  That works.  The archive task doesn't work for the user.  msmq queues are empty.

Also we have about 30,000 mailboxes vaulted, so 1/8 of them are on this server.  The vault store is not in backup mode.

Pradeep-Papnai
Level 6
Employee Accredited Certified

As per the dtrace lines, I could imagine the issue is similar as per TN https://www.veritas.com/support/en_US/article.000083503

EV:L       :AgentMessageDispenser::ActivateObject() |Exiting routine at point A |

EV:M     :AgentMessageDispenser::ProcessNextMessage() |ActivateObject has returned failure |

EV:M     AgentMessageDispenser::ProcessNextMessage - AGENTS_E_DISPABORTITEM, agent will sleep and abort the the current MSMQ Transation

EV:M     :AgentMessageDispenser::ProcessNextMessage() |Retrieved a message successfully from the queue |

EV:M     :AgentMessageDispenser::ProcessNextMessage() |Read new message or a message that is within its retry limit (). |About to process the message body |

EV:L       :AgentMessageDispenser::ActivateObject() |Entering routine |

EV:L       :AgentMessageDispenser::ActivateObject() |Called server side object, with arguments: |m_pIArchiveAction->ArchiveMarkedItems(ExchangeServer = "outlook.chs.net",| MailboxDn = "/o=CHS Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=gnieszcz",| FolderPath = " ",| PathSeparator = "AAAAAAAAKNCAFOPLINMNLCEOLCKIMIDCMKNAONLDABAADLNAOIPIGHLLEOEKLMHIMCCBEMBFCICDAAAAAAFPKHOCAAAA",| TimeStamp = "2017-07-25T18:52:37",| RetentionCategoryId = "",| AuthToken = "S-1-1-0 S-1-*****",| RetryCount = "0") |

EV:L       {CArchivingAgent::ArchiveMarkedItems()} (Entry)

EV:M     CAA::AMI() - Another thread is currently processing mailbox /o=CHS Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=gnieszcz, this request will be put to the back of the queue

EV:L       {CArchivingAgent::ArchiveMarkedItems()} (Exit) Status: [Success]

Regards

Pradeep Papnai

Hi dorlow,

   I'd absolutely agree with Pradeep here.  Just this line alone says enough:

 

88605813 14:23:28.959  [3236] (ArchiveTask) <10436> EV:H :CArchivingAgent::ProcessUser() |Archive/Journal Task ignored a Process Mailbox Message because the Process Mailbox mutex was locked and the mailbox is being processed by another thread |

 

  I'd have been surprised if adding the DS Server reg key fixed it (what the article lists as the solution).  However, you're working on an EV version that's so far out of support I barely remember what worked and what didn't with it, and it's easy enough to try.  Plus that article was designed for 9 and 10, so that's right in your range.  Give that a shot, and upgrade when you can!

 

Thanks,
Daveoflave

Our company will not upgrade.  Their goal is to replace Vault with a different solution.  It has never worked reliably since install.  Enterprise Vault is 15 letter swear word at my company.  I have an IT manager that stops me often and asks on my progress of replacing EV with anything else.  EV has an incredibly bad rep within our organization.

So, I got a Wintel guy involved.  He looked at the DNS server the server is pointing to.  He said he found a problem with the NIC.  All the Exchange and Vault servers have a pool of DNS servers they use.   He said this particular Vault server was pointing to one in the pool that had a different NIC and he suspected that.  He failed it over and I'm still having the problem.  

13916335 20:21:13.648 [7440] (ArchiveTask) <12500> EV:H :CArchivingAgent::ProcessUser() |Archive/Journal Task ignored a Process Mailbox Message because the Process Mailbox mutex was locked and the mailbox is being processed by another thread |

Googling more, another article stated it might be a mapi profile issue.  I stopped the archiving tasks.  Then Deleted all the mail profiles and also deleted the registry keys under windows messaging subsystem profiles.  

I'm still getting the mutex error in dtraces.  Googling more, it sounds like maybe there are bad messages in her mailbox that need to be removed.  If so, how do I know which messages they are?  What other ideas are there?

Hi Dorlow,

   I don't want to send you into another list of colorful descriptions of our product, but in newer versions of EV, the message subject would typically show in the dtrace when it starts processing them, which would be a way to determine what the messages are.

    The article Pradeep offered mentioned adding the DS Server key.  I didn't see you mention if you tried this, any luck with it? I know I had doubts about it, but it's easy enough to do and worth a try. I searched our case history again, and found 2 other cases in which adding the DS Server registry key fixed this issue.  Of course the Exchange targets were 2010... so is this mailbox in question on a 2010 server?  Just google "DS Server" and you should get Microsoft's article about this key.  Remember to use the CAS Array as the value, and not the GC.

Thanks,
Daveoflave

Ok, still no luck.  So, what I did now was I have the emails that need to be put into vault.  None of the PSTs will import directly into vault.  They all error.  So, I created a test user.  I imported the emails into the test user via the new-mailboximportrequest ps command.  It imported no problem.  I setup the test user for Vault.  I cannot get one email from the mailbox to move to Vault.  The theory is we're going to vault the PSTs for a test user.  Then export the vault to pst.  The new PSTs will be generated by vault so theres no reason those PSTs will import directly into the user's vault seeing the PSTs were generated by Vault.

Attached is a dtrace.

 

I just checked too... the exchange server the user is on versus the exchange server where the test user are different so they use two different vault tasks on two different servers.  So, the only thing in common is the mail content.  I checked the mailbox and it's full if IPM.Note emails.  They didn't start transitioning to Vault Pending type emails.

Does this message mean anything to anyone?  Dtrace shows error...

[17712] (StorageArchive) <14920> EV:H {CMAPIMsgTraverser::GetMsgProperty} (Exit) Status: [The parameter is incorrect.  (0x80070057)]

Sounds like a MAPI issue.

I just stopped the two archive tasks on the server and then checked both the registry and mail icon in control panel for profiles and none showed up.  So, I didn't have any bad mapi profiles it appears.  

Pradeep-Papnai
Level 6
Employee Accredited Certified

Hi,

The existing issue may be passed with last suggestion/troubleshooting, now I could see the issue similar as TN https://www.veritas.com/support/en_US/article.000082706 (event id may not match but exception & call looks similar).

EV:L    CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
EV:H    CFolderAgent::UpdateArchiveFolderRecord() - HRXEX fn trace : Error 0xc0041bf0, ..\AgentsCommon\FolderHelper.cpp [lines {2374,2405}], built May 14 19:52:50 2012.
EV:M    CFolderHelper::UAFR() - Error updating record 1B150EFDABC476F42AA07A2D16C38B5861110000EVSARCHIVE01, error = 0xC0041BF0|Internal:CFolderAgent::UpdateArchiveFolderRecord() ..\AgentsCommon\FolderHelper.cpp [lines {2374,2405}], built May 14 19:52:50 2012
EV:H    CFolderHelper::GetFolderSettings - HRXEX fn trace : Error 0xc0041bf0, ..\AgentsCommon\FolderHelper.cpp [lines (16){...,442,502,792,793,806,807,808,809,810,811,812,962}], built May 14 19:52:50 2012.
EV~E    Event ID: 2243 Could not scan user mailbox /o=CHS Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=CHS Test2d8, cannot list messages |
EV:M    {CArchivingAgent::processMessages:#5534} Could not scan [user mailbox] [/o=CHS Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=CHS Test2d8]. This problem is commonly fixed on the next scan. Cannot list messages: [0xc0041bf0] [(null)]

Other thing to notice, the registry key “IgnoreEVDates” is set with 1, my suggestion to make it as 3 as per TN https://www.veritas.com/support/en_US/article.000040806 , ArchivingTask seems skipping many items due to date issue. The dtrace of last post was mentioning mailbox “gnieszcz”, now the mailbox looks “CHS Test2d8”, Is that mailbox resolved OR am I missing anywhere?

Regards
Pradeep Papnai

Hi dorlow,

   Pradeep is quite right.  Here are a few more lines from just above the snippet he referenced.  The bottom line is key:

30080597 11:30:10.290  [12424] (ArchiveTask) <14200> EV:L {CFolderHelper::GetFolderSettings:#778} Current folder [Easton] is using parent folder's archiveVEID
30080599 11:30:10.290  [12424] (ArchiveTask) <14200> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
30080600 11:30:10.290  [12424] (ArchiveTask) <14200> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
30080601 11:30:10.290  [12424] (ArchiveTask) <14200> EV:M CDirectoryVaultObject::GetAttributeListFromType Attribute list for type ArchiveFolderView : RootIdentity,FolderName,FolderPath,ParentFolderRootIdentity,FolderIcon,CustomId,CustomQual,SID,VaultEntryId,Type,Hidden,ContainerRootIdentity,OwningTrusteeIdentity,AutoSecurityDesc,ManualSecurityDesc,ArchiveVEID,ParentFolderVEID
30080602 11:30:10.290  [12424] (ArchiveTask) <14200> EV:L {CFolderHelper::GetFolderSettings:#959} Okay to synchronise folder [Easton]. Parent folder is new.
30080603 11:30:10.290  [12424] (ArchiveTask) <14200> EV:M CFolderHelper::UAFR() - Updating folder record 1B150EFDABC476F42AA07A2D16C38B5861110000EVSARCHIVE01 with following properties|DisplayName = Easton|FolderPath = �******************)�Folders�Miscelllaneous�Easton|ParentFolderVEID = ***********************************************RetentionCategoryID = (null)|ExistsAtSource = True
30080605 11:30:10.290  [12424] (ArchiveTask) <14200> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
30080606 11:30:10.306  [12424] (ArchiveTask) <14200> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
30080607 11:30:10.306  [5756] (DirectoryService) <18428> EV~E Event ID: 8391 The EnterpriseVault.DirectoryService object reported an error.| |An attempt was made to move an ArchiveFolder beneath itself (hierarchy inversion)| |

 

I censored your user's name and Archive ID. 

 

So it seems like a folder was dragged into one of its child folders; specifically "Easton."  The article Pradeep gave is the way to go in that case, https://www.veritas.com/support/en_US/article.000082706.

   Also I'd ask the same thing, which is: Are we targeting the same mailbox in this dtrace? We didn't see these same events in your last one, and these events would have occurred in this situation independent of the samll changes you've made since starting this thread.  Anyway, hopefully the export/import to new archive method helps. 

Thanks,
Daveoflave