Error 0x8007000e (Outlook 2007 SP2 + HF2475891)

[-Maurice]

Hi all,

we have one customer which is getting Error Messages EventID 3411 (Not enough Storage available...) followed by EventID 2216 (Message dispenser will suspend processing...) , during the archiving window. This results in really poor archiving throuput/performance and thousends of messages stucking in MSMQ.

Environment:

• 7x EV Servers running on 9.0.2
• Outlook 2007 SP2 + HF2475891
• EV Servers on Windows Server 2008 R2
• Exchange 2003 latest SP2 & Exchange 2010 SP1 (2003 and 2010 SP1 Archiving Tasks are located on seperate EV Servers)
• 6x Ex2010 Mailbox Archiving Tasks on EV003
• 5x Ex2003 Mailbox Archiving Tasks on EV001

Exact Error Message:

An error has been reported by ConfigureMsgService.  It may be due to a problem contacting a Global Catalogue Server.

Exchange Server: XXXXXXXX011
Mailbox: SMTP:EnterpriseVault-XXXXXXXX011@customer.com
Error: 0x8007000e   
 

The Customer is in an on going exchange 2010 migration and centralization of ~16k mailboxes (2k mailboxes are already migrated). We are dealing more than 5 month with support to solve that issue with no solution. Hopefully someone has an Idea which helps to fix that issue.

We tried already the the following to solve the issue:

• Set the "DS Server" key to CAS array and also to a specific CAS
• Set the "DS Server" key to specific GC server
• Re-Installed Outlook Hotfix 2475891 and also Re-Installed Outlook 2007 SP2
• After advice from syma support we implemented NSPI regkey to 0xffffffff on every server in CAS array
• Lowering the concurrent connections to 1 or 2 did "solved" the error message, but not the archiving performance, what is logical in result of the reduced connections
• Set ProfileExpiry to 0 - after restart of task controller the servers are operating more or less well for 1 or 2 hours and then the error occurs again.

Thanks for reading

Best Regards, Maurice

[JesusWept3]

are you absolutely 100% 1000% 10000000% sure its  KB2475891?
The only reason why i ask is the behavior you are seeing, we saw on the first release of the hotfix, that caused Outlook.exe to crash.

We then had to run TraceTimeTool and send the logs to microsoft and they fixed the memory leak/performance issues.

If you can , can you tell me the file and product version of Outlook.exe?

Also what storage device do you use? NAS or SAN or NetApp or Centera or iSCSI etc?

[-Maurice]

yes, i am definitely sure. I reinstalled Outlook and the installed the hotfix by myself two times.

The file version of Outlook.exe is 12.0.6554.5000

Customer is using NetApp Storage accessed cia CIFS

[JesusWept3]

Yup, that is indeed the correct version , hmmm

OK So here is my suggestion, First disable the task controller before doing the following

Download and install Debug Diag and the windows resource Kits
1. Download and install DebugDiag to your EV Server from here
http://www.microsoft.com/downloads/en/details.aspx?familyid=28bd5941-c458-46f1-b24d-f60151d875a3&dis...

2. Note that although there is an x64 MSI for debugdiag its just to read the outputs, not to capture them, to capture them you must use the x86 versions

Configure DebugDiag
3. Open up DebugDiag
4. From the wizard choose "Crash" and press Next
5. Choose "A specific process" and press Next
6. From Selected Processes type "ArchiveTask.exe" and press Next
7. From Action Type , choose "Log Stack Trace" and then put something like 10 instances
8. Press the "Exceptions" button and then press "Add Exception"
9. In Exception Code put in "8007000e"
10. For Action Type put "Full User Dump"
11. For Action Limit put in a number such as 10
12. Press OK and then press Save & Close
13. Press Next and then choose a suitable place for the memory dumps to be saved to
14. Press Next and then press "Do Not Activate The Rule At This " and press Finish

Configure DTrace
15. Open up another Command Prompt
16. CD to your \Program Files\Enterprise Vault directory
17. Type DTrace and press enter
18. Type "set ArchiveTask v" and press Enter
19. Type "log <yourDebugDiagLocation>\EVDtrace.txt" and press enter

Then start your TaskController service and allow for the errors to reproduce, each time the error occurs, you will then end up with crash dumps that you can then pass to Microsoft and Symantec, as well as the Dtrace which shows how its leading up to that point too.

Also just to note, the attachment of the DebugDiag can actually lead to memory exhaustion as well, so dont keep it on too long as it can become a question mark itself.

And lastly Microsoft may have you run the TimeTraceTool which they can guide you through but its pretty straight forward, it pretty much records everything happening on the machine at a low level, last time when we had this issue, it was purely microsoft and it was just EV that was suffering.

If you feel symantec aren't being responsive enough, you can always ask for it to be pushed to backline and have them create an ETrack for assistance from the Customer Focus Team, and also to create any TSANet cases with Microsoft so they can work together to resolve the issue

[-Maurice]

thanks for your suggestion. I configured the DebugDiag and DTrace tool to gather the logfiles and started archiving. DebugDiag did not created any Dumps, because the ArchiveTasks did not crash in our case. They just dropping the error, but the task is still running and did not crash.

The Dtrace logs are round 5GB in size and are now posted to symantec support and we wait for the analysis.

Has someone may another idea how to get more information (logs/traces etc.) to find out what's causing the error?

Cheers Maurice

[JesusWept3 · Accepted solution]

Well regardless of a crash once it traps 0x8007000e it should still generate the debug logs

But really your best bet is to continue to work with symantec, get them to work with microsoft to see if they can get this resolved for you

[ZeRoC00L]

Maurice,

Did you ever get a solution for this issue ?
I seem to have the same issue.

[JesusWept3]

Follow the advice above, collect the relevant data a d submit a support case

[-Maurice]

You can try to log a case at symantec, but for my customer the support did not find a solution. The only helpful things were: Implementing regkeys, which reduced the number of events, but not completly solved. - AttachmentMax - RecipientMax And restart the taskcontroller at least daily and if necessary multiple times a day. Regards Maurice

[ZeRoC00L]

Thanks, case is already created, but I was hopeing to get this issue resolved by this forum. Will let you know when I get a response.

[AndrewB]

about the NSPI reg key, i was wondering if that's supposed to be done on all the domain controllers or on the CAS servers as you describe?

[-Maurice]

The support wasn't sure about this. In exchange 2010, the cas act as proxy for the mapi interface and is for the client the nspi endpoint. So we were adviced to set the nspi key on the cas array. But i have no idea if it's effective or not.

[ZeRoC00L]

My customer do not have 2008 DC's yet, so I will change the setting on the Exchange 2010 CAS server. And see if it makes any difference.