05-25-2011 01:07 AM
Hi all,
we have one customer which is getting Error Messages EventID 3411 (Not enough Storage available...) followed by EventID 2216 (Message dispenser will suspend processing...) , during the archiving window. This results in really poor archiving throuput/performance and thousends of messages stucking in MSMQ.
Environment:
Exact Error Message:
An error has been reported by ConfigureMsgService. It may be due to a problem contacting a Global Catalogue Server.
Exchange Server: XXXXXXXX011
Mailbox: SMTP:EnterpriseVault-XXXXXXXX011@customer.com
Error: 0x8007000e
The Customer is in an on going exchange 2010 migration and centralization of ~16k mailboxes (2k mailboxes are already migrated). We are dealing more than 5 month with support to solve that issue with no solution. Hopefully someone has an Idea which helps to fix that issue.
We tried already the the following to solve the issue:
Thanks for reading
Best Regards, Maurice
Solved! Go to Solution.
05-27-2011 06:06 AM
Well regardless of a crash once it traps 0x8007000e it should still generate the debug logs
But really your best bet is to continue to work with symantec, get them to work with microsoft to see if they can get this resolved for you
05-25-2011 08:13 AM
are you absolutely 100% 1000% 10000000% sure its KB2475891?
The only reason why i ask is the behavior you are seeing, we saw on the first release of the hotfix, that caused Outlook.exe to crash.
We then had to run TraceTimeTool and send the logs to microsoft and they fixed the memory leak/performance issues.
If you can , can you tell me the file and product version of Outlook.exe?
Also what storage device do you use? NAS or SAN or NetApp or Centera or iSCSI etc?
05-25-2011 09:09 AM
yes, i am definitely sure. I reinstalled Outlook and the installed the hotfix by myself two times.
The file version of Outlook.exe is 12.0.6554.5000
Customer is using NetApp Storage accessed cia CIFS
05-25-2011 09:27 AM
Yup, that is indeed the correct version , hmmm
OK So here is my suggestion, First disable the task controller before doing the following
Download and install Debug Diag and the windows resource Kits
1. Download and install DebugDiag to your EV Server from here
http://www.microsoft.com/downloads/en/details.aspx?familyid=28bd5941-c458-46f1-b24d-f60151d875a3&dis...
2. Note that although there is an x64 MSI for debugdiag its just to read the outputs, not to capture them, to capture them you must use the x86 versions
Configure DebugDiag
3. Open up DebugDiag
4. From the wizard choose "Crash" and press Next
5. Choose "A specific process" and press Next
6. From Selected Processes type "ArchiveTask.exe" and press Next
7. From Action Type , choose "Log Stack Trace" and then put something like 10 instances
8. Press the "Exceptions" button and then press "Add Exception"
9. In Exception Code put in "8007000e"
10. For Action Type put "Full User Dump"
11. For Action Limit put in a number such as 10
12. Press OK and then press Save & Close
13. Press Next and then choose a suitable place for the memory dumps to be saved to
14. Press Next and then press "Do Not Activate The Rule At This " and press Finish
Configure DTrace
15. Open up another Command Prompt
16. CD to your \Program Files\Enterprise Vault directory
17. Type DTrace and press enter
18. Type "set ArchiveTask v" and press Enter
19. Type "log <yourDebugDiagLocation>\EVDtrace.txt" and press enter
Then start your TaskController service and allow for the errors to reproduce, each time the error occurs, you will then end up with crash dumps that you can then pass to Microsoft and Symantec, as well as the Dtrace which shows how its leading up to that point too.
Also just to note, the attachment of the DebugDiag can actually lead to memory exhaustion as well, so dont keep it on too long as it can become a question mark itself.
And lastly Microsoft may have you run the TimeTraceTool which they can guide you through but its pretty straight forward, it pretty much records everything happening on the machine at a low level, last time when we had this issue, it was purely microsoft and it was just EV that was suffering.
If you feel symantec aren't being responsive enough, you can always ask for it to be pushed to backline and have them create an ETrack for assistance from the Customer Focus Team, and also to create any TSANet cases with Microsoft so they can work together to resolve the issue
05-27-2011 12:38 AM
thanks for your suggestion. I configured the DebugDiag and DTrace tool to gather the logfiles and started archiving. DebugDiag did not created any Dumps, because the ArchiveTasks did not crash in our case. They just dropping the error, but the task is still running and did not crash.
The Dtrace logs are round 5GB in size and are now posted to symantec support and we wait for the analysis.
Has someone may another idea how to get more information (logs/traces etc.) to find out what's causing the error?
Cheers Maurice
05-27-2011 06:06 AM
Well regardless of a crash once it traps 0x8007000e it should still generate the debug logs
But really your best bet is to continue to work with symantec, get them to work with microsoft to see if they can get this resolved for you
08-19-2011 07:27 AM
Maurice,
Did you ever get a solution for this issue ?
I seem to have the same issue.
08-19-2011 07:42 AM
08-19-2011 09:10 AM
08-19-2011 11:52 AM
Thanks, case is already created, but I was hopeing to get this issue resolved by this forum. Will let you know when I get a response.
08-20-2011 10:29 PM
about the NSPI reg key, i was wondering if that's supposed to be done on all the domain controllers or on the CAS servers as you describe?
08-21-2011 01:34 AM
08-21-2011 01:59 AM
My customer do not have 2008 DC's yet, so I will change the setting on the Exchange 2010 CAS server. And see if it makes any difference.