Memory problems by EV9.0.2

[sandrine_goetha]

Hello,

Last week I updated all ( 7 virtual machines) my EV servers from EV9.0 to EV9.0.2. I didn't change anything else to our environment.

We are only archiving Exchange mailboxes no Domino, file s or sharepoint.

Since the change I have a HUGE memory problem on the servers.I tried a lot of things and the only one which helps is to restart the machines which is for me not a solution but a workaround. Could any of you tell me how I could figure out where the leak is and how to solve it ?
Any help is welcome

Thank you So much 
 

Sandrine

[Rob_Wilcox1]

Have you taken a look at the articles on the forums with regards to troubleshooting performance/memory issues?

Have you identified which process(es) are consuming the memory, and over what period of time?

[sandrine_goetha]

Hallo Rob thank you for your answer .
Yes I read your articles but they don't really speak about memory problems. And from a CPU point of view everything looks " normal"
To you second question: I can identify the process consumming the memory this is exactly my problem otherwise I would nknow what to solve . And During the Archving windows I of course expect such consumption but the memory should be given back after which is not the case.If I manage I will join a jpg from the SCOm about the memory consumtion as exemple, the fall down of memory during the night is normal as it is the running windows so... but I expect to get the memory back after.

Anyway thank you for your time

Sandrine

[Rob_Wilcox1]

Hmm the picture doesn't really show a lot.  

I would suggest using good old fashioned performance monitor type of things first of all.

Add in :

Process -> All counters & all instances

Memory -> All counters & all instances

Processor -> All counters and all instances

Set the sampling interval to 5 minutes.. restart your EV Admin service, and then watch the chart for a while.

Or if you do it to a file for later review ...

Objects -> Process, and Memory, and Processor

Again sampling at 5 minute intervals, and let it run for 24 hours or so.

High memory usage on it's own does not indicate a problem - take a look at your Exchange server for example, it'll have high memory usage all day every day.

On top of that are you getting any errors in the event log on the Enterprise Vault server?  Is it just an Enterprise Vault server, it doesn't have any other services on it, eg SQL, etc?

[sandrine_goetha]

Thank for your input, my concern is not about the high memory usage during the run. This is normal I think but the fact that I don't get it back after the run so in time do I get less and less memory and I really don't know where it disappears.

I'll try the recommended monitoring on one of the machine.

What I can already add is that the worse behavior is on the machine with the most "Vault Stores".My Stores are on Centera but linked to an EV server if this makes sense to you. And I also checked all the msmques and they are empty for each machine.

And no I can't see anything abnormal in the Etvlogs but with the amount we have (~40.000 mailboxes ) it is easy to oversee something.And yes my machines are just EV servers, I have a dedicated SQL server.

Sandrine

[Rob_Wilcox1]

Sandrine,

The Enterprise Vault Admin Service will shut the EV Services down, and log to the event log if/when memory is almost exhausted.

If you're not seeing that, then you might not have a problem.

Data collection as described above, will help you.

[sandrine_goetha]

Rob,

I don't have a problem "really" yet I just saw the pattern and I reacted before. I realised that I have less and less available memory and I didn't want my servers to run into a problem. I should perhaps open a case  as I really can't find anything anymore ( it archives the CPU is "normal" for the operations,...)even the perfmon looks normal to me ( I confess I'm not a specialist) and I don't think that restarting the servers every morning is a solution.

Sandrine

[Rob_Wilcox1]

Okay... yes if you're not comfortable in setting things up, and reviewing the output your best bet is to log a call with Support.

[JesusWept3]

Ok I've read the entire thread and the really most important question that rob asked is *what* exactly is consuming the memory and supposedly not releasing it? Storage? Archivetask? Journaltask? Indexing? Evconverter? Etc Also was the kind of monitoring you performed ever done before the upgrade? What alerted you to the fact that it's only just started? But really it's impossible to have any idea without knowing what's actually happening on your server

[jim_leggett]

You mentioned rebooting is not a solution for you, but have you tried recycling individual task and services and monitoring memory usage in between in order to see which process releases a large amount of memory? You probably want to do this outside of your archving window.

[sandrine_goetha]

Jesus this is exactly my point  I don't manage to find out what is using this memory and I don't know how to do it ( this is why I called for help). And what alerted me is just luck as I by chance saw it on the SCOM monitoring ( I'm checking the memory available Megabytes 2008). And as I told it started right after the upgrade to EV9.0.2
 Jim I tried restarted the task but all together and never one after each other, I'll give it a try.
Thank you all so much for your inputs as I'm totally lost here.

Rob thank you so much for all your help and time, the perfmon is really good if you know exactly know what you are looking for otherwise you just get lost in all those information's.

Sandrine

[Rob_Wilcox1]

Sandrine, yes it's a very, very, VERY, complex area to get in to... I know when I worked at Microsoft there were dedicated people to look at these sort of issues, and every single one of them was "different".  So, I think your best port of call, if you don't want to spend an inordinate amount of time scratching your head and saying "hmmm, but..." is to start off with logging a call with Symantec Support.  They'll at least start things going, though it then may branch out in to talking to Microsoft, hardware vendors and the like.