Enterprise Vault for Lotus Domino Sessions

[LukeMih]

I'd just like to pose the question.  How many established sessions does a EVDG establish and leave open on a per provisioned user.  For instance, 1000 users provisioned, 660 EVDG sessions to the primary domino server.

I've run seen, multiple times now, where the number of ports available on my primary domino server is exhausted and the sessions start recycling and inbound / outbound new session requests fail.  At this same time, I might notice 3000+ sessions from my EVDG to the primary domino server.

Just wondering if anyone has seen this before.

LM

[Arjun_Shelke]

Well, if you are talking about the Domino Archiving then the number of sessions to primary mail domino server is configurable. From Vault Admin console on EV Server, goto properties of the Domino Mailbox Archiving task. Settings tab allow you to control number of threads this task can start to connect to primary domino server.

By default its 10 threads. So while archiving is running, EV will make 10 concurrent connections to Domino Server to archive the data.

Can you please elaborate more on your environment, such as number of EV Servers, EVDG Servers, Domino target Servers, and thier versions such as Domino, Notes and EV.

[Arjun_Shelke]

While I was waiting for your response, I thought of sharing this information with you.

As I described above, Archiving threads can be controlled by using the task properties, and this should not be the problem you are facing because archiving task wont use those many sessions to Domino as you have mentioned in the post.

This must be users retrieval job which causes those sessions/connections to Mail Domino Servers.

If you are running EV 9.0.1 or higher, then you could expect this sudden rise in connections because of architecture changes with regards to Domino Retrieval of archived items from Notes client and DWA/iNotes.

Prior EV 9.0.1, we use to create temp database on EVDG Server when user requests for retrieval of an item. Using which the item was first restored to this temp database and shown to the user. This architecture had some known issues which were affecting the operations.

Now since 9.0.1 onwards, when user requests retrieval of an archived item, the item is directly retrieved to users mailfile on Domino Server and then shown to the user. This process requires additional connections/sessions to Mail Domino Server in order to access users mailfile to retrieve the item.

You can have a look at this demo of changes to Domino Retrieval.

http://www.symantec.com/docs/HOWTO77220

So the number of sessions/connections you see from EV/EVDG to Mail Domino Server are for Archiving + Retrieval.

I hope this helps.

[LukeMih]

Thank you for the reply. 

I'll describe my environment a little.  I have a single EVDG, and 2 target Domino servers.  Everything involved is on version 9.0.4.  The two domino servers are in a domino cluster.

I had noticed that setting for threads when we implimented this a few years ago, and thought that wasn't involved in the retrieval of messages that were archived as it was an archival task setting.  Either way that is set at the default of 10.

The part that is particularly interesting is that the target domino server runs out of memory on the port and starts recycling sessions after about a week or 2 of uptime and eventually the domino server crashes.  Just recently I started becoming more interested in resolving this and ran a show sessions >c:\sessions-<date>.txt to review what sessions existed and found the majority of these sessions were from the EVDG, also that the count continually grows over time.

I did notice that 9.0.4 restores the messages into the mailfile, which was well accepted by the users of the archiving system.  The weekly crashes have been a pain, and this weeks crash was avoided by restarting the EVDG server prior to the target server crashing.  This didn't make the sessions disappear either, it just prevented additional new ones and eventually the old sessions timed out returning the target server back to normal operation pretty much immediately.  A side effect of this is that the sessions limit parameter is lowered to a number below what it was prior to the error condition which is displayed on the console and shown in a console command like 'show stat NET.<portname>.Limit' on the affected tcpip port name.  Also the port having the recycled connections because of the limit being reached shows under the statistic of net.<portname>.recycled.

For instance these are the current values I have:

  NET.TCPIP.Sessions.Limit = 2721

  NET.TCPIP.Sessions.Recycled = 4759
 

I can provide more insight if needed, I was just looking for baseline numbers to determine if I need to provide a tcpip port just for EVDG connections.

LM

[Arjun_Shelke · Accepted solution]

Ok, so for example if you have 1000 EV enabled users accessing thier mailfiles and symultaneously they are accessing archived items and EV is also archiving items from mailfiles at the same time then it would look like:

Notes users mailfile access: 1000 sessions.

Notes users accessing archived items: EVDG to Mail Domino - 1000 Sessions.

EV Archiving Task: Trawling mailfiles, using 10 sessions at a time.

Total TCP/IP sessions used = 2010 sessions.

Is this correct? If yes, then the EVDG sessions to Mail Domino would be 1010 at that time. This should be the theoritical boundary to the max sessions. But we forget to add EV Domino Provisioning Task which runs twice a day to update few things in users mailfile. So this task will also eat some sessions, not sure how many sessions will be used at a time by this task.

Looking at the scenario, I feel like idle sessions are not killed in time?

I have not seen this behaviour ever, not sure if you can change the TCP IP port for EVDG sessions. I think the sessions are not killed properly, when user retrieves archived item and closes the open note window, the EVDG session for that item/mailfile should be killed or kept idle for some time for next retrieval and then killed if not used.

[LukeMih]

I think I have instituted a workaround at least for the problem I was seeing when the server would throw console errors, recycle existing connections and reset the connection limit to lower numbers.  The setting that I changed was the notes.ini parameter Server_Session_Timeout which was by default 240 minutes.  I reset this to 30 minutes and have noticed that the session count, so far today is reasonable for our workload.

The console command for showing that statistics:

show stat net.tcpip.sessions.peak

Currently: NET.TCPIP.Sessions.Peak = 638

Previously: NET.TCPIP.Sessions.Peak = 3946

I have a suspicion that the timeout parameter being 4 hours by default is a bit high if the applications using domino are relying on the domino server to terminate the idle connections in a 1000+ user environment.  However, the strange part I see is that we almost never see more than 140 notes/inotes users signed into this domino server.

LM

[LukeMih]

Thanks for the insight by the way.  I appreciate the help and venture to say this thread prevented me from opening a support case which would have occupied a considerably larger amount of time than I had available to dedicate to a single issue.

Thanks again.

LM

[Arjun_Shelke]

Thanks for your compliments. I am glad that we (as Forum Support) were able to help you in understanding the issue and provide insight of the technological process.

I hope we continue resolving such issues in less time to avoid load on Support team and your inconvenience.

I am glad to know that you were able to work around this situation with the help of this thread.

Thanks once again.