02-17-2016 10:50 AM
We are running EV 10.0.4 CH3 on Windows 2008 R2 Enterprise Edition. I've got 32gb of RAM on the VM. This server is used for Exchange 2013 archiving. When we look at running processes there is nothing that adds up to 31GB of RAM in use. The highest amount of memory used is 407mb for EV IndexQueryServer and the next highest is 231mb EVIndexVolumesProcessor. If I stop all EV Services the memory drops to 28gb in use. Something is wrong with this box. Additionally I cannot enable a user archive because I get an out of storage error. Event ID 3475, 3188, 3196
3196 - Error: Not enough storage is available to complete this operation. (0x8007000e)
3188 - An error occurred performing a request to synchronize the mailbox
3475 - One or more errors occurred during the creation of a profile to connect to an Exchange Server.
Targeted Exchange Server:
ConfigureMsgService failed with the following errors:
Out of memory error has occurred for the following connection points:
This issue occurs every few days. We haven't been able to identify what is using the memory. Any help would be appreciated.
02-17-2016 11:34 AM
The EV Archive server has 7000 user and shared mailbox archives on it. We have 8 Exchange 2013 servers each with about 1000 user mailboxes and shared mailboxes.
02-17-2016 12:17 PM
How many users do you have on this EV server? You might have outgrown the server and its starting to run out of resources.
02-17-2016 12:19 PM
And you are targeting all these exchange server swith one EV server? Are you also journaling? All this could be the reason for your issues
02-17-2016 12:21 PM
your server is simply running out of resources. could be a combination of mapi, iis, msmq, sql, storage, network connections, etc. two main issues i see:
1. 8 Exchange servers is too many for a single EV server to archive
2. 7000 users is beyond what i would recommend for 1 EV server
02-17-2016 12:21 PM
I have 8 archive tasks setup from one EV server. The archiving runs from 6pm to 12am. In that time it hits all mailboxes. The Exchange Journaling is done on a different EV server.
02-17-2016 12:29 PM
The server is sized pretty good. 16cpu and 32gb of Ram. Hitachi SAN Storage. I have 23 mount point partitions at 1TB each.
If the archiving is running at night why am I out of resources during the day time when its just taking iis requests.
02-17-2016 01:20 PM
Next steps would be to ensure AV exclusions are correctly set per this article.
02-17-2016 01:21 PM
the truth is we see this all the time. cpu and ram are great but there are other factors too. if veritas or an authorized partner designed your EV architecture like this then i'd go back to them and let them know it's not working. what typically happens is the system is designed to accomodate a certain environment and then the environment changes but the system isnt redesigned to accomodate because everything seems to work fine until a random issue like this pops up.
as far as best practices go, you're pretty far outside of them and i would recommend adding a second EV mailbox archiving server, splitting the 8 archiving tasks to 4 on each server, and eventually moving the related archives (3500 or so assuming even distribution in Exchange) to the second EV mailbox server.
another option would be to increase your archiving window and stagger your 8 archiving tasks' schedules. also look at the number of connections per exchange server per task (in the task properties) and make sure it's not above 5.
lastly, if you upgrade to EV 11 and Windows 2012 you can take advantage of Outlook 2013 (instead of 2007) and an OS with better resource management.
by the way, 23 mount points and all that HBA traffic are also eating up resources outside of cpu and ram.
02-17-2016 02:12 PM
We were running into severe resource (memory) issues when doing searches as well. Upgrade of the SQL server and higher performance SAN solved our problems.
02-17-2016 03:17 PM
Might want to look into SQL fragmentation and statistics then if upgrading is not an option:
02-18-2016 07:49 AM
Why would the 23 mount points be an issue? We use that many on the EV Journal server as well. Once the partition is full we move on to the next one. Our SAN performance is showing 8ms latency which is within recommended range.
As far as Archive tasks go would I be better off consolidating to 4 Archive tasks, each task containing 2 servers Exchange servers?
I attached 2 screenshots of the memory utilization on the EV Archive server. Currently there is no archiving, or backup running. Memory usage is very high and there is no process showing what is using it all.
Our EV Journal server which uses the same SQL server doesn't have memory issues like the archive server.
02-18-2016 09:43 AM
archiving tasks are 1 per Exchange server. if you have 8 Exchange servers then you'd have 8 archiving tasks. hence the recommendation for 2 EV servers with 4 archiving tasks on each. again, 7000 users is beyond the best practice for a single EV mailbox server. you're running into this issue now but if you didnt, you'd run into other issues later anyway.
since you took the time to post the screenshots, it is odd that your AuthServer.exe is using so much memory.
02-18-2016 01:17 PM
To me this sounds like the MetaFile taking up all the memory, you can use RAMMAP by SysInternals to free the system working set and probably see it drop quite dramatically
you can also use DynamicCache to set the ceiling of how much the MetaFile can take up in memory
It's really more prevalent after the switch to 64bit Converters that were introduced
02-22-2016 01:46 AM
I agree with AdnrewB, I've only used 4000 max users with one EV server.
You should consider to build another EV server.
02-22-2016 05:21 AM
Honestly the number of users isn't really a factor, EV has pretty decent throttling on the Storage side.
The biggest concern would be mapi utilization, 8 exchange archiving tasks at 5 threads each would be 40 mapi connections.
On top of that for retrieval, vault cache builds, pst imports and exports, you could quickly hit that limit.
but just looking at what you posted, AuthServer is extremely high, but it doesn't account for all the memory usage.
There could be a strong likelihood that you go to a lot of effort to build another EV server and have the exact same issue, you need to determine whats causing that high memory usage, because I have servers far busier than that with more users than that and maybe utilize 60% of memory after configuring the dynamic cache