Journal Backlog from HELL!

[blop]

heeeeeeellllllllllllllppppppppppp!

Hey all,

I have just been tasked with Enterprise Vault and have very little experience of it.

I noticed whilst doing some Exchange administration that the Journalling mailbox had approximatley 65 GB of mail in it, and that the Journalling task on the Enterprise Server is reporting errors. I urgently need to get it fixed but dont really know where to start, all services are working and tasks running.

Many thanks for any help.

Event:

Event Type:    Warning
Event Source:    Enterprise Vault
Event Category:    Journal Task
Event ID:    2270
Date:        13/10/2008
Time:        12:00:07
User:        N/A
Computer:    EV-SRV02
Description:
A queued operation exceeded the retry count and has been discarded



m_pIExchangeSynch->SynchronizeMailboxEntryEx(LegacyMailboxDn = "/O=UNIVERSITY-OF-DERBY/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=JOURNALING",
 adMbxDn = "CN=UoD Journaling,OU=Service Accounts,OU=Management,OU=UoD,DC=university,DC=ds,DC=derby,DC=ac,DC=uk",
 adMsgStoreDn = "CN=Mailbox Database,CN=First Storage Group,CN=InformationStore,CN=mkt-mbx01,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=University-of-Derby,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=ds,DC=derby,DC=ac,DC=uk",
 SynchModes = "3")
HRESULT: 0x8004060C

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp
 

[Matthew_Bennett]

First thing first is to find the little twit that was looking after EV and give them a good kicking.

The backlog is a massive amount of data. Most amount I have had my Journal back log has been 10 gb

Have you checked to see if services are running on the ev server, including the tasks especially the Journal Task.

 It also could be that the journal vault store has reached its limit :-

 Check by going into the vault console on the ev server and drilling  - console root-Enterprise Vault-Directory "server name" - "Vault name" - Archives - Exchange Journal. Look at the staus of the current Journal store, make sure its available

or it may be the  index volume status is set as failed.

Best idea would be to log a support call with Symantec.

[MarkB_2]

0x8004060C is MAPI error for a full mailbox MAPI_E_STORE_FULL. Have you got a limit set or is there a problem with disk resources at all?

[blop]

Thank you very much for the replies.

I have to admit it is causing me a fair bit of hassle; so your help is appreciated.

I can confirm that both of our EV servers have all their Services and Tasks running...especially the Journal Task. Also the current status of the Journal Status is Available.

There is also no limit for the mailbox size which is partly why it is now 60gb!! 

I have disabled the Exchange Journalling from Exchange Management Console so as to not add anymore to the EV journalling mailbox i want to clear the backlog before reenabling.

Is my next best course of action speaking to Symantec?

[MichelZ]

Hi there

I hope you know that disabling journaling will lead to lost messages in journaling?

Messages won't get re-journaled!

Is there anything more in the eventlog?

Cheers

Michel

[blop]

Hi Micheal,

many thanks...

yes i realised that the disabling of Exchange Journalling would mean no more Journaled mail for EV....but the 60gb mailbox was about to cripple the Exchange Storage Group....i did not have an option.

I have done some work on error checking the permissions but it seems to all be fine, i have rebooted the box and now it is saying the journal task has started successfully and then this error at 10 minute intervals.

Event Type:    Warning
Event Source:    Enterprise Vault
Event Category:    Journal Task
Event ID:    2216
Date:        14/10/2008
Time:        11:16:58
User:        N/A
Computer:    MKT-EV002
Description:
Message dispenser will suspend processing for 10 minutes due to a recoverable error

Task: Exchange Journaling Task for MKT-MBX01

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp

[MichelZ]

Have a look here

Cheers

Michel

[blop]

Hi Micheal,

thanks again for the help, really appreciate it.

I have done a dtrace from the EV Admin Console and then a search inside it for some syntax that is the same as that link you gave me...and i cant find any.

Do you propose that i follow the links instructions and delete / move / recreate the Journal MBX? As with it being 65GB the only solution i have is to delete it and try and make a new empty one and get EX / EV journalling to use that.....

Would it be usefull for me to post the dtrace?

its big!!!!

Message Edited by blop on 10-14-2008 05:58 AM

[MichelZ]

Yes, it may be useful to review it.

If it's really big, you may want to send it to me via e-mail:

michel.zehnder (at) vcare-it.com

Was the MBX moved?

What you also could try:

Are you having safety copies in place?

Are you able to open the Journal Mailbox with the EV Service Account? (i.e. attach it in Outlook)

Do you see there the pending archive symbol, or just "normal" messages?

Cheers

Michel

[blop]

I'll defintley take advantage of your genorisity there....beaming the dtrace to you now!

The MBX was not moved as far as i can tell, by that i mean it is still where it was intially...

safety copies? We archive with EV on a seperate server...if thats what you mean so the normal mailbox archive is safe.

I can open the EV journal account but due to the huge amount of mail in the box its taking ages to open...the message count reads at 1149077! due to that i cant see any message symbols at all.

cheerS!

[Matthew_Bennett]

Best idea would be to contact Symantec Support. I would hope that Derby university would have a valid support contract. This is your best option. They would be able to connect up using web ex and check the system out.

At this point I would not advise to delete the journal mail box and starting again, There is a reason why the University has a journal mailbox. We have ours for FOI purposes and look after it with great care.

Did you check to see if the store was available as described in my post? and the journal index had not failed?

Matthew.

[MichelZ]

OK, from the trace it shows the same findings as MarkB did,

342    11:26:58.957     [2248]    (JournalTask)    <1404>    EV:H    :CArchivingAgent::IsEligibleForArchive() |Entering routine  |
343    11:26:58.957     [2248]    (JournalTask)    <1404>    EV:L    :CArchivingAgent::IsEligibleForArchive(/O=UNIVERSITY-OF-DERBY/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=JOURNALING) |The message titled: Your mailbox is full is eligible for archive  |
344    11:26:58.957     [2248]    (JournalTask)    <1404>    EV:M    CJournalPart::MakeJournalPartMsg - Considering Message [Your mailbox is full]
345    11:26:58.957     [2248]    (JournalTask)    <1404>    EV:M    CJournalPart::MakeJournalPartMsg - Opening The Message
346    11:26:58.957     [2248]    (JournalTask)    <1404>    EV:M    CJournalPart::MakeJournalPartMsg - Is a E2007 Journal Message? [False]
347    11:26:58.957     [2248]    (JournalTask)    <1404>    EV:H    :CArchivingAgent::IsEligibleForArchive() |Exiting routine |

As well as:

2097    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:L    CJournalPart::GroupExchangeJournalParts - Previous message group ready for submission
2098    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:M    HRXEX fn trace :|Source     : .\JournalPart.cpp (CJournalPart::ProcessMessageGroup) [lines {452,516}] built Jan  2 11:36:32 2008|Description: Call error 0x8007000e
2099    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:M    CJournalPart::GroupExchangeJournalParts - Checking Time Msg TimeStamp [FW: Car booking for Buxton Tues, 7 Oct 9am - 4pm]
2100    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:M    CJournalPart::IsEarlierTime - TargetTime [14-10-2008 10:21]
2101    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:M    CJournalPart::IsEarlierTime - TestTime [06-10-2008 08:08]
2102    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:M    CJournalPart::IsEarlierTime - TestTime is Earlier Than Target Time? [TRUE]
2103    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:M    HRXEX fn trace :|Source     : .\JournalPart.cpp (CJournalPart::GroupExchangeJournalParts) [lines {...,375,321,331,335,340,375,321,331,335,340,375,380}] built Jan  2 11:36:32 2008|Description: Call error 0x8007000e
2104    11:26:59.332     [2248]    (JournalTask)    <1404>    EV:H    :CArchivingAgent::ProcessFolders() |Exiting routine |
 

This is out of memory exception.

Are you really sure that there is no limit on the mailbox?

For the OOM exception:

Maybe the journal backlog really is too much.

I'd suggest the following (if there is really no limit):

- Set up a new Mailbox

- Set up exchange journaling to use this mailbox

- Set up EV to use this as the "new" journaling mailbox (delete journal task, and recreate)

(Leave the current journaling mailbox in place)

See if EV then picks up archiving from the new journal mailbox.

If yes, then you would have a functional journaling again, and you can take the next steps to get the other data into the journaling vault again with less pressure...

(Or contact Symantec support, as Matthew suggests ;) )

Cheers

Michel

[MichelZ]

Another thing just popped into my mind:

Could you try reducing the "concurrent connections to exchange server" in the Journaling Task and restart it?

Try setting it to 1..

Cheers

Michel

[Steve_C_Blair]

Have you tried checking to make sure the Vault Store partition where EV writes to is not full? Additionally, I'm surprised no one has pointed this out to you to try:

1) Stop the Archiving task

2) Created a "hold folder" in the JNL MBX

3) remove the oldest 10 messages in your Journal MBX and moving them to the "hold folder" for now

4) Start the Archiving task

It could be you have a corrupted message (s) at the top of the JNL queue and that the reason you are here in the first place is because of that. There are messages that Exchange (or poorly written apps) can send that might in very rare cases not be able to be processed, and hence archived. Unfortunately there is no list that anyone, including MSFT has that can point you to those *exact* messages / causes.

If you do the above and you start processing your backlog, then you'll know you've got one of the selected messages could be bad. Using a tool like OutlookSpy you might be able to determine why a message is bad, and then one-by-one place the others back onto the JNL MBX queue to archive.

Failing all of this, contact EV Support at your first instance is well needed!

[MichelZ]

Steve

I agree, this could help.

But how should he do this, given the fact that the journal mailbox is so big, that he can't even open it in Outlook? ;)

/Michel

[blop]

Hi,

I have just setup a new Journal Task using a new Exchange Mailbox and a new task.

Sadly it does not seem to be doing anything...

The messages are flowing in from Exchange but they do not leave, the messages all have the normal envelope sign on them. 

The EV server reports that the new Journal Task has started but then does not report anything else.

It has been allegedly going for 45 mins but no events have been reported since...

I think im going to log a call with Symantec....

[MichelZ]

Yes, do this.

I would be very interested in the solution, it has to be something obvious *sigh*

Cheers

Michel