cancel
Showing results for 
Search instead for 
Did you mean: 

EV 10.0.4 CH3 Mailbox Archive Server memory 98% utilized

jbernstein
Level 3

We are running EV 10.0.4 CH3 on Windows 2008 R2 Enterprise Edition.  I've got 32gb of RAM on the VM. This server is used for Exchange 2013 archiving.  When we look at running processes there is nothing that adds up to 31GB of RAM in use.  The highest amount of memory used is 407mb for EV IndexQueryServer and the next highest is 231mb EVIndexVolumesProcessor.  If I stop all EV Services the memory drops to 28gb in use.  Something is wrong with this box.  Additionally I cannot enable a user archive because I get an out of storage error. Event ID 3475, 3188, 3196

3196 - Error: Not enough storage is available to complete this operation.  (0x8007000e) 

3188 - An error occurred performing a request to synchronize the mailbox 

3475 - One or more errors occurred during the creation of a profile to connect to an Exchange Server. 

Targeted Exchange Server: 
Mailbox: SMTP: 

ConfigureMsgService failed with the following errors: 

Out of memory error has occurred for the following connection points:
 

This issue occurs every few days. We haven't been able to identify what is using the memory.  Any help would be appreciated.

15 REPLIES 15

jbernstein
Level 3

The EV Archive server has 7000 user and shared mailbox archives on it.  We have 8 Exchange 2013 servers each with about 1000 user mailboxes and shared mailboxes.

JimmyNeutron
Level 6
Partner Accredited

How many users do you have on this EV server? You might have outgrown the server and its starting to run out of resources.

JimmyNeutron
Level 6
Partner Accredited

And you are targeting all these exchange server swith one EV server? Are you also journaling? All this could be the reason for your issues

AndrewB
Moderator
Moderator
Partner    VIP    Accredited

your server is simply running out of resources. could be a combination of mapi, iis, msmq, sql, storage, network connections, etc. two main issues i see:

1. 8 Exchange servers is too many for a single EV server to archive

2. 7000 users is beyond what i would recommend for 1 EV server

 

jbernstein
Level 3

I have 8 archive tasks setup from one EV server.  The archiving runs from 6pm to 12am.  In that time it hits all mailboxes.  The Exchange Journaling is done on a different EV server.

jbernstein
Level 3

The server is sized pretty good.  16cpu and 32gb of Ram.  Hitachi SAN Storage.  I have 23 mount point partitions at 1TB each. 

If the archiving is running at night why am I out of resources during the day time when its just taking iis requests.

JimmyNeutron
Level 6
Partner Accredited

Next steps would be to ensure AV exclusions are correctly set per this article.

https://www.veritas.com/support/en_US/article.000032085

 

 

AndrewB
Moderator
Moderator
Partner    VIP    Accredited

the truth is we see this all the time. cpu and ram are great but there are other factors too. if veritas or an authorized partner designed your EV architecture like this then i'd go back to them and let them know it's not working. what typically happens is the system is designed to accomodate a certain environment and then the environment changes but the system isnt redesigned to accomodate because everything seems to work fine until a random issue like this pops up.

as far as best practices go, you're pretty far outside of them and i would recommend adding a second EV mailbox archiving server, splitting the 8 archiving tasks to 4 on each server, and eventually moving the related archives (3500 or so assuming even distribution in Exchange) to the second EV mailbox server.

another option would be to increase your archiving window and stagger your 8 archiving tasks' schedules. also look at the number of connections per exchange server per task (in the task properties) and make sure it's not above 5.

lastly, if you upgrade to EV 11 and Windows 2012 you can take advantage of Outlook 2013 (instead of 2007) and an OS with better resource management.

by the way, 23 mount points and all that HBA traffic are also eating up resources outside of cpu and ram.

WiTSend
Level 6
Partner

We were running into severe resource (memory) issues when doing searches as well.  Upgrade of the SQL server and higher performance SAN solved our problems.

JimmyNeutron
Level 6
Partner Accredited

Might want to look into SQL fragmentation and statistics then if upgrading is not an option:

https://www.veritas.com/support/en_US/article.TECH168905

 

 

 

 

jbernstein
Level 3

Why would the 23 mount points be an issue?  We use that many on the EV Journal server as well.  Once the partition is full we move on to the next one.  Our SAN performance is showing 8ms latency which is within recommended range. 

As far as Archive tasks go would I be better off consolidating to 4 Archive tasks, each task containing 2 servers Exchange servers?

I attached 2 screenshots of the memory utilization on the EV Archive server.  Currently there is no archiving, or backup running.  Memory usage is very high and there is no process showing what is using it all.

Our EV Journal server which uses the same SQL server doesn't have memory issues like the archive server.

AndrewB
Moderator
Moderator
Partner    VIP    Accredited

archiving tasks are 1 per Exchange server. if you have 8 Exchange servers then you'd have 8 archiving tasks. hence the recommendation for 2 EV servers with 4 archiving tasks on each. again, 7000 users is beyond the best practice for a single EV mailbox server. you're running into this issue now but if you didnt, you'd run into other issues later anyway.

since you took the time to post the screenshots, it is odd that your AuthServer.exe is using so much memory.

JesusWept3
Level 6
Partner Accredited Certified

To me this sounds like the MetaFile taking up all the memory, you can use RAMMAP by SysInternals to free the system working set and probably see it drop quite dramatically
you can also use DynamicCache to set the ceiling of how much the MetaFile can take up in memory

It's really more prevalent after the switch to 64bit Converters that were introduced

https://www.linkedin.com/in/alex-allen-turl-07370146

CConsult
Moderator
Moderator
Partner    VIP   

I agree with AdnrewB, I've only used 4000 max users with one EV server.

You should consider to build another EV server.

 

JesusWept3
Level 6
Partner Accredited Certified

Honestly the number of users isn't really a factor, EV has pretty decent throttling on the Storage side.
The biggest concern would be mapi utilization, 8  exchange archiving tasks at 5 threads each would be 40 mapi connections.
On top of that for retrieval, vault cache builds, pst imports and exports, you could quickly hit that limit.

but just looking at what you posted, AuthServer is extremely high, but it doesn't account for all the memory usage.
There could be a strong likelihood that you go to a lot of effort to build another EV server and have the exact same issue, you need to determine whats causing that high memory usage, because I have servers far busier than that with more users than that and maybe utilize 60% of memory after configuring the dynamic cache


https://support.microsoft.com/en-us/kb/976618

http://blogs.technet.com/b/mspfe/archive/2012/12/06/lots-of-ram-but-no-available-memory.aspx

https://prashantd.wordpress.com/2013/11/14/solved-high-memory-usage-by-metafile-on-windows-2008-r2/

https://www.linkedin.com/in/alex-allen-turl-07370146