EV-CA 10.0.3 searches for group returns zero hits

[BigAnvil]

I searched the forums for a while and can't find a simlar issue but feel free to send just a link if there is a topic that addresses this already...

We are running EV & CA 10.0.3 on Windows Server 2008 R2 Standard.  We have a scheduled daily search for four departments that contains employees.  For some time now the scheduled searches return zero hits against hotword sets.   To troubleshoot, I did the following:

• I tried running an immediate search for one of the problem departments with the same hotword sets selected for the subject & content (same as the schedule searches).  For this search, I specified a five-day timeframe rather than a previous-day timeframe.  The search is between the department and internal and/or external to the organization.
• I tried running an immediate search for only a small subset of the employees in the department (group) by removing the checkmark from the box that selects all the employees and selecting just a few.  The rest of the parameters are the same as the above.
• I tried running another search the same way as above except I manually selected each of the employees in the department (group) - so all employees were selected but not by selecting the whole group.

The 2nd and 3rd test searches returned the expected results (however many thousands of hits it was).  The 1st test search returned zero hits as the scheduled search does.

So it seems like the bottom line is that if the department (group) is selected, I will get zero results.  These departments (groups) are mapped to groups in Active Directory.  Obviously, this creates a problem with making sure that the search runs against a group whose members may change rather than having to manually check the members before each search runs.

I'm going to review the sync properties but could use guidance, especially if someone has seen this before and knows where to point me.

Thanks!

[EV_Ajay]

Hi,

Just want to know whether have you installed the "Journaling Connector" on the Enterprise Vault Server which is running "Enterprise Vault Journal Task". The "Journaling Connector" is perform the Department Level Taggin.

Installing the Journaling Connector
http://www.symantec.com/docs/HOWTO58525

In the Search when we selemct Department / Group , will it reflect in front of Department <All Employees> appeared ?

[BigAnvil]

Ev_Ajay,

Yes, the journaling connector is installed on the Enterprise Vault server.  It's the same version (10.0.3) as the EV product and EV-CA.

If I put a checkmark in the box for the department, <All Employees> appears like you said.

[Kenneth_Adams · Accepted solution]

Greetings, BigAnvil;

Please look on the EV Journaling server at the registry values that are set in HKEY_LOCAL_MACHNE\SOFWARE\Wow6432node\KVS\External Filters\Journaling.

Look to see if you have values similar to the following:

Name          Type                Data

(Default)                      REG_SZ              (value not set)

1                                REG_SZ              KVS.Accelerator.Plugin.Filter

MoveOnFilterFailure     REG_DWORD     0x00000001 (1)

Override                      REG_DWORD     0x00000001 (1)

I suspect your Journal Connector has been disabled as searches for the DepartmentID tag return 0 hits while searching for the SMTP addresses of the individual Monitored Employees in a Department returns hits.

The default search behavior in CA is to search for the DepartmentID tag after the Journal Connector has been installed.  When you uncheck the check box by the Department name and check the check box of individual Monitored Employees within the Department, you are telling the CA search behavior to use the SMTP addresses of those selected Employees instead of the DepartmentID tag.

The only way I know to prevent the DepartmentID tagging to be applied is to disable the Journal Connector.  Disabling the JC is accomplished by taking the Name (in the example above, the name is the number 1) and changing it to the number 0 or anything other than the last number of all external filters.

To explain further, all External Filters are Custom Filters that are used by the Journal Archiving process to perform actions on all items in the Journal Inbox during the archiving operations.  These External Fitlers must have a name that is a number, with the number designating the order in which the External Filters are to be applied.  If the JC is the only External Filter, it's name must be the number 1.  If there is another External Filter, the JC must be the last External Filter to be processed, to the other External Filter will have its name as the number 1 and the JC will have its name as the number 2.

So, let us know if you find your JC has been disabled and, if it has, were you able to enable it.  Remember that you'll have to restart the Journal Task to fully enable the JC if you make a change to its name in the registry.

[EV_Ajay]

Hi Ken,

Very informational and helpfule explanation.

Thanks for the valuable explanation.

[BigAnvil]

@Kenneth Adams

Thank you for the helpful information.  I remember that sort of thing happening in the past (many years ago) - where the journal connector was updated or reinstalled and was assigned an ID that was not the last in the list of filters.  Since it's not something I've had to check in so long, I didn't think of it.

I need to wait until later today to verify (I made the change yesterday morning but our backup guys have had the primary Vault Store group in backup mode for the last 24 hours, and now that I've cleared it we need to wait until EV can process all the itesm sitting in the journal mailbox).

I'll follow up and give credit after verification.  Thanks Kenneth!

[Kenneth_Adams]

Greetings, BigAnvil;

Heres is a trick you can use to confirm the department tagging is working without having to wait until the Random Sampling processing completes tomorrow.

1) Launch an advanced browser search (http://EVServer/EnterpriseVault/search.asp?advanced=3) and select the journal archive (depending on the account you use for this, you may have to grant Read access to the archive through the Vault Admin Console).

2) In the 'Other Result Attributes' line, enter the following as I have it typed: KVSCA.Deptartment KVSCA.DeptRecips KVSCA.DeptAuthor

3) In the 'Date' line, enter into the "From:" field a date known to not return hits in the scheduled search (i.e, 07/10/2013) and in the "To:" field, there the same date or the next day's date (i.e., 07/11/2013).  Run the sesarch and look at the first page of the output to see if any of the items returned also show lines that begin with the labels in the 'Other Result Attributes' line.  If you don't see any such lines, then no department tagging was done.

4) Run the same search except with yesterday's date in the 'Date' line's "From:" and "To:" fields.  Depending on when in the day you made the change, you may have to scroll through a few pages of returned items to see any with the department tagging; however, just seeing any from yesterday (or today if you want to check with today's date) will be proof that the JC is now doing the department tagging.

I hope this helps.  I look forward to your update on the results of enabling the JC yesterday.

[EV_Ajay]

Hi,

Some information about the "KVSCA.Deptartment KVSCA.DeptRecips KVSCA.DeptAuthor" :

 Some of the tags that CA looks for in the message metadata are :

• KVSCA.Department - the DepartmentID of a sender, recipient, or exception.
• KVSCA.DeptRecips - DepartmentID associated with the recipients.
• KVSCA.DeptAuthor - the DepartmentID associated with the author.

[BigAnvil]

Kenneth,

The name of the string prior to changing it was "0" and all our journal messages were being archived properly (converting to pending then being removed from the journal mailbox per our policy).  

I deleted the entry for "LVS.Accelerator.Plug.Filter" and recreated it with a name of "1" as below.  It was the only filter.

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\External Filtering\Journaling]

"Override"=dword:00000001

"MoveOnFilterFailure"=dword:00000001

"1"="KVS.Accelerator.PlugIn.Filter"

Since doing this, and despite clearing backup mode from the vault stores and indexes, items in the journal mailbox are no longer being archived.  The journal task was restarted and the EV server itself was restarted but the items in the journal mailbox are still not being archived.

I've confirmed that the stores and indexes are NOT in backup mode.

Changing the name of the filter back to "0" and restarting the journal task causes journal mailbox items to be archived properly.

[Kenneth_Adams]

Ah.  So the JC was disabled intentionally.  Interesting.

Look in the Vault Admin Console to see if the journal task (JT) is in a failed state. If it is, there is something wrong with the JC that is causing the task to fail.

If the JT has failed, look in the Symantec Enterprise Vault Event Log on the journaling server to see if there are any errors thrown related to the external filter (which would be the JC).

Possible causes of the JC to stop journaling are:

1) Two or more CA Customers exist in the environment where these Customers share the same CA Configuration database AND more than 1 Customer's properties has a blank for the VaultID(s) entry.  To check for this, the EV Event Log will have an error entry with the description stating the default customer could not be determined.  Also, looking at the Customers in the EVBAAdmin site on the CA server, clicking on each customer in the left pane will display the properties in the right pane where the VaultID(s): field can be easily viewed.  In environemnts where multiple CA Customers exist within the same CA Configuration database, one and ony one Customer must be considered the default customer into which all items not targeted for other customers are placed.  The default Customer is noted by the VaultID(s): field being blank.  All other Customers MUST have something in their VaultID(s): field, and the entries must not contain blanks.

2) An upgrade to CA has been started, but not fully completed.  When CA is upgraded, the CA Configuration database is the first database to be upgraded after the upgraded binaries (the exe, dll, etc files) are installed.  After the CA Configuration database upgrade has completed, each CA Customer must be upgraded.  After all CA Customer upgrades have been completed, the Journal Connector must be upgraded.  All of the databases and the JC must be at the same major and minor release of the product (i.e., all must be at CA 10.0 SP3).

3) The CA Configuration database has been moved from one SQL Server or SQL Server instance to another SQL Server or SQL Server instance.  When a CA Configuration database is moved, there are certain files on the CA server and the EV journaling server that must be updated to reflect that move.  On the journaling server, there is a file in the EV installation folder (default folder is 'C;\Program Files (x86)\Enterprise Vault') named JournalTask.exe.config for Exchange and EVLotusDominoJournalTask.exe.config that must be edited in Notepad (or some other plain text editor) to change the 'server=' entry from the old SQL Server or SQL Server instance to the new SQL Server or SQL Server instance.  Once the file has been edited with the correct SQL information, the JT must be started (the edit should be done with the JT stopped).

Since you can run CA searches, we know that at least the CA Configuration database and one CA Customer are running at the same CA version.  From this information, we know that the SQL Server name or SQL Server instance information in the CA server's CA configuration files is correct.  To confirm the SQL Server or SQL Server instance name that should be in the JC configuration file (i.e., JournalTask.exe.config), look on the CA server in the CA installation folder (default is 'C:\Program Files (x86)\Enterprise Vault Business Accelerator) for the file named AcceleratorManager.exe.config.  Open this file in Notepad and look for the line with the name of the SQL Server or SQL Server instance in the 'server=' entry.  Note that the same line will have an entry labeled 'initial Catalog=' that contains the name of the CA Configuration database.  Note this database name as well as the SQL Server / SQL Server instance name. Take that information and match it to the information in the JC configuration file.  If either the SQL Server / SQL Server instance name or the CA Configuration database name are incorrect, change them and save the JC configuration file, then restart the JT.

Let us know what you find when you go through these troubleshooting steps, please.

[BigAnvil]

Kenneth,

I could see the Event Log entries showing the journal task as failed. But rather than review those entries I think the JournalTask.exe.config file shows where the issue is.  Though we didn't change SQL servers or instances, it doesn't seem to contain the correct info - our SQL server is kvs-sql.domain.com, the CA server is evault-ca.domain.com, and the EV server is entvault.domain.com

<?xml version="1.0" encoding="utf-8"?>

<configuration>

  <appSettings>

    <add key="DSN" value="server=EVServer;Integrated Security=true;Initial Catalog=evaccelerator;Connection Timeout=500" />

    <add key="DSNConfiguration" value="server=evault-ca.domain.com;Integrated Security=true;Initial Catalog=EVConfiguration;Connection Timeout=500" />

    <add key="DirectoryDSN" value="packet size=4096;integrated security=SSPI;data source=&quot;.&quot;;persist security info=False;initial catalog=EnterpriseVaultDirectory" />

    <add key="Event Application Log Source Name" value="EV Compliance Accelerator" />

    <add key="Stop Journaling On Error" value="Yes" />

  </appSettings>

  <runtime>

    <generatePublisherEvidence enabled="false" />

    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">

      <dependentAssembly>

        <assemblyIdentity name="KVS.EnterpriseVault.Common" publicKeyToken="26c5e2ccf2b9267c" />

        <codeBase version="10.0.3.0" href="file:///C:\Program Files (x86)\Enterprise Vault\KVS.EnterpriseVault.Common.dll" />

        <bindingRedirect oldVersion="0.0.0.0-63555.63555.63555.63555" newVersion="10.0.3.0" />

      </dependentAssembly>

      <dependentAssembly>

        <assemblyIdentity name="KVS.Accelerator.Sampling" publicKeyToken="26c5e2ccf2b9267c" />

        <codeBase version="10.0.3.0" href="file:///C:\Program Files (x86)\Enterprise Vault\KVS.Accelerator.Sampling.dll" />

        <bindingRedirect oldVersion="0.0.0.0-63555.63555.63555.63555" newVersion="10.0.3.0" />

      </dependentAssembly>

    </assemblyBinding>

  </runtime>

</configuration>

I've changed the config file to contain the correct settings with the Journal Task stopped and also updated the registry entry for the KVS.Accelerator.PlugIn.Filter.  Restarted the Journal Task and everything looks good from an archiving perspective - actually, noticably faster.  I guess that's what happens when you have a config file with invalid information and fix it.  I worked with Symantec months ago on another case and I can't remember if we ended up reinstalling the connector or not but based on the info I provided, we must have and then failed to edit it.

I'll follow up later today on the status of the CA searches - gotta go get something else done now!  Thanks for all your help Kenneth!

[Kenneth_Adams]

Ah.  So the JC was configured with the CA server name instead of the SQL Server name.  That's a common mistake, so it's easy to fix.

Thank you for letting us know the root cause of the problem.  Have you tried the advanced browser search that I noted a few comments ago?  That would tell you if you're getting the department tagging now.  I expect you will be, but it never hurts to be sure.

FYI, the work you did to troubleshoot the issue can be found in our Technical Article TECH50455, "Journaling Stops when the using Journal Connector and how to properly configure the Journal Connector", available at http://www.symantec.com/docs/TECH50455.  I'll be updating this article soon to add the versions of CA that are not currently listed as this applies to all versions of CA.

[EV_Ajay]

Hi Ken,

Thanks a lot for providing the Possible causes of the JC to stop journaling. Got more information about the issue and cause.

[BigAnvil]

Kenneth,

I wear many hats here as most of us do, so I did not have an opportunity to test as you had asked.  However, when I arrived this morning, the scheduled search returned results so all is good again.

Thank you for all the helpful information and your quick responses!

[EV_Ajay]

Hi Ken,

I really learn many things from this issue. It's happy to know that this issue is resolved.

[Kenneth_Adams]

I understand being too busy to run some tests and having to wait for the normal processing to complete to confirm a fix.  I, too, was busier than expected yesterday after the general availability release of EV/DA/DA 10.0.4 this past Wednesday.

I'm glad that I was able to assist in resolving your issue.