1 Mailbox, lots of Exchange Transaction logs

[Korbyn]

I'm seeing a weird issue.  One particular mailbox being processed by EV, yet nothing is being archived, BUT, and here's the big problem, about 1.5 GB of Exchange transaction logs per hour are being generated for the database that this user is in.

Nothing is going into the pending backup, the users mailbox items archived or archived size is not increasing at all.

I ran a DTrace for 5 mins and got a 218 Mb file, but don't really know what to go searching for.  Nothing obvious stands out.

Runnig EV 8.0 SP4.  Any suggestions on codes to look for, or ideas on why something might be archving but not....

[JesusWept3]

open a support case if its urgent, but really open up the trans logs and find out what its doing, last time i saw that it was a corrupt item

[Korbyn]

I grabbed a tlog, and though I can't decifer much of what's happening, the users name comes up alot. 

Have you found some magical tool that can actually read and translate the Exchange transaction log?

Any code suggesstions to look for in Dtrace like [ERROR] or something?

From looking through the DTrace, it's processing through different messages, it doesn't appear to be stuck on any one message.

[JesusWept3]

well honestly if you're in a situation where its just causing massive transaction logs to occur etc and you're pretty much down then i would open a case, as it will get you fixed a lot quicker than here.

If i were the support tech on the end of the phone i'd ask for the trans logs and the DTrace of the archiving task.

I'd also ask if its happening for one specific user or is it happening for all users.
I'd probably ask you to disable that user and try archiving other users and see if its just something specifically on that mailbox or whether its multiple mailboxes.

In the dtrace, the first thing i'd look for is EV~E (Errors) and EV~W (Warnings, these are events that are thrown in the event log, usually they're the first place you'd look for as EV throws errors for everything.

If the error appears to match what may be happening (may be it throws an error about reading or writing exchange properties) then i'd use TextPad and do a search for that particular thread and follow it through from beginning through to the the error

also another common error indicator is 0x800 as most errors will begin with this

MAPI errors will be 0x8004xxxx and Windows errors will tend to be 0x8007xxxx

Another thing i'd also check on is whether the Exchange server itself is throwing errors, as i seem to recall this may also be due to the maximum amount of custom mapi properties limit being reached

But honestly though, Troubleshooting is difficult enough, DTraces are a dark art that I could probably write a good 20 page article on and still only cover the very basics

[Korbyn]

Those were the basic's I was looking for, EV~E and EV~W.

Unfortunately, they were no help in this case and hopefully I'll be able to file away and remember in the future.  No errors, no warnings, and the only 0x800 codes are 0x80030002

1342970 09:30:19.824  [8000] (ArchiveTask) <8212> EV:M CSavesetOnIStg::get_IndexableItemSize (Entry)
1342971 09:30:19.824  [8000] (ArchiveTask) <8212> EV:M CSavesetOnIStg::OpenIndexableItem (Entry)
1342972 09:30:19.824  [8000] (ArchiveTask) <8212> EV:M CSavesetOnIStg::OpenSubStg (Exit). hr=%1 could not be found.  [0x80030002]
1342973 09:30:19.824  [8000] (ArchiveTask) <8212> EV:M CSavesetOnIStg::OpenIndexableItem (Exit). hr=%1 could not be found.  [0x80030002]
1342974 09:30:19.824  [8000] (ArchiveTask) <8212> EV:M CSavesetOnIStg::get_IndexableItemSize (Exit). hr=False [0x1]
 

Which will might be of interest to the Tech Support, or mean nothing.  80030002 is like a file not found error, but from similar posts, this isn't unusual to see.

Thanks JW.

[JesusWept3]

0x80030002 is a common red herring, its just looking for the content handler for the message, its why it isn't thrown as an event as it would kill your event logs, 99% of environments see this code being thrown in the background, just simply ignore it

[Rob_Wilcox1]

Last time I saw anything like this it was a corrupt message in the mailbox store too.  Some "ideas" (you should discuss with Microsoft as a first port of call) :-

* Move the mailbox to a different mailbox store (same or different machine)

* Run isinteg -test alltests on the store (having dismounted it)

[Korbyn]

0x80004004 from the StorageArchive process pretty much a red herring as well?

We're batting the idea of moving the affect users mailbox, however, it's 4 GB and she's a director...  And so far, this is the only mailbox we've found causing this issue.

We're going to get a case going this afternoon.  So far though, nothing in the Event Logs on either the Exchange or EV server, and the DTrace shows bupkis.

[JesusWept3]

0x80004004 is Operation aborted, so really it depends on what the lead up to that is, Storage Archive would suggest that the item has actually gone in to pending and being passed to the storage service.

It's possible that the storage service is then connecting to the mailbox and an error is occuring there maybe reading or trying to copy the item.

What really should happen is determine the item that its trying to get that on and see if its literally just looping over the same item over and over and if it is, that may be your problem child.

But as Rob pointed out, it could be something more to do with the mailbox, and indicative of bigger problems, it just happens to be that at this moment Enterprise Vault is the only thing that is showing the symptoms.

if you can run an integrity check on the mailbox store that would be optimal

[Korbyn]

I think it's just part of the same, normal archiving process and waiting for the saveset to be replicated to the secondary Centera.

Looks like we also found a 2nd affected mailbox (out of 10,000).  Hopefully support can sort through the logs and figure out the why.  Might be as simple as moving the mailboxes, just were wanting to avoid that kind of activity.

I might suggest to the client that they move the 2 users to an isolated database, and we'll see if we can reproduce the issue or if it's resolved.  If it's not resolved, then they only bring down themselves...

Just an aside, if they are using custom forms, EV should be ignoring them unless we actually added the message class in.

[Ameen]

Hi,

I had similar issue in my environment and that was because of virus in one of my user's system. We did remove the system from the network and solved the issue. Later we did re-imaged the system before connecting back.

Ameen.

[Korbyn]

a co-conspiritor working with Symantec support trouble shooting the issue, ran EVPM (I don't know the ini settings at this time) but the issue seems to be resolved, but now we have two Vaults for this one user.  We don't know if this was true before the EVPM was run or after, but suspect for after.

Is there a best way to recombine these two archives back into one? 

[Korbyn]

Apparently I need the caffine...

IF I export the disconnected Vault to PST and deleting from the Vault as it goes, and run now the shortcut processing, with the Remove Orphaned short cuts enabled, those stub messages should all clear out then, and we could reimport the PST directly into the users vault, creating the new stubs as we go, correct?

A little more work, but should do the trick right.  Was hoping that there was a better way.

[JesusWept3 · Accepted solution]

I really would ask support to clean up their mess if I were you, really though I'd disable the user, export the smaller archive, delete the smaller archive, import the pst tile to the old archive then enable the user and attach their mailbox to the old archive