01-14-2020 01:00 AM
Hi Team,
We are classifying by share by share, but sometimes the request is getting stuck and not getting resumed. Sometime after the restart of the classificaiton server it is getting resumed.
--- Some logs where request suddenly stopped. and after restart it is good and resumed.
--- Some logs from VIC.log , where even after the restart it is same there is no increase in the classification request number in the console. After restart I can see that the polices are not loaded,
and some files are still there in content folder, request ID.
- We tried to do a new classificaiton request and i am not sure that request is running good. BUt the stucked once are still showing as inprogress.
VIC0.0 Log:
2020-01-13 08:51:39 SEVERE: #{166} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_124943.docx
javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 19 more
2020-01-13 08:52:50 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] E:\DataInsight\data\classification\content\272\124\2214_125062.file->E:\DataInsight\data\classification\content\272\124\2214_125062.pdf
2020-01-13 08:52:50 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] Submitting path: E:\DataInsight\data\classification\content\272\124\2214_125062.pdf for classification
2020-01-13 08:52:55 SEVERE: #{149} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_125049.pdf
com.veritas.vic.client.ClassifierClientException: Service Unavailable
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:244)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1035)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
... 8 more
2020-01-13 08:52:55 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] E:\DataInsight\data\classification\content\272\124\2214_125065.file->E:\DataInsight\data\classification\content\272\124\2214_125065.pdf
2020-01-13 08:52:55 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] Submitting path: E:\DataInsight\data\classification\content\272\124\2214_125065.pdf for classification
2020-01-13 08:52:55 SEVERE: #{163} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_125048.pdf
com.veritas.vic.client.ClassifierClientException: Service Unavailable
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:244)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1035)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
... 8 more
vic.log
2020-01-14 02:43:43 ERROR: #{PolicyMonitor-1} Failed to GET policies. Status=[null]
2020-01-14 02:43:43 ERROR: #{PatternMonitor-1} Failed to GET patterns. Status=[null]
2020-01-14 02:43:43 ERROR: #{PolicyMonitor-1} Failed to check for policies changes, or notification failed. Will try again later.
javax.ws.rs.ProcessingException: java.lang.IllegalStateException: Connection pool shut down
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681)
at com.veritas.vic.classifier.policiesClient.PoliciesClient.getPolicies(PoliciesClient.java:61)
at com.veritas.vic.classifier.policiesClient.PolicyChangeMonitor.checkForPolicyChanges(PolicyChangeMonitor.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:268)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 18 common frames omitted
01-14-2020 06:02 AM
I have no solution, but have a similar issue.
After this in the vic0.0.log the classification stops:
2020-01-14 14:26:04 SEVERE: #{1355} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 15_108943.pdf
javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 19 more
After a reboot of the classification server, the classification continues normaly...
01-14-2020 02:21 PM
What version of Data Insight are you running and is it the same across the various components (collector, indexer, classifier)?
01-14-2020 11:20 PM
We are on the latest version on both mangement and classifcation servers.
01-15-2020 11:42 AM
We are having the same issue. It is also taking much longer for the classification to work since upgrading. It used to take 10 or so minutes to classify 2 documents before the upgrade. It took 7 hours for two files to be classified post 6.1.5.
01-19-2020 10:04 PM
Changing the Java Heap memory seems to make things better. But I don't know if this is the solution...
But do test and see if it runs better.
Problem
Classification jobs on large shares stayed "in progress" for several days.
Cause
Out of JavaHeap memory
Solution
Modify the memory settings for VICSERVER.vmoptions and VICServerService.vmoptions settings on each Classification Node. Navigate to C:\Program Files\DataInsight\bin
Open the files VICSERVER.vmoptions and VICServerService.vmoptions and change the values below:
-Xmx2500m to -Xmx8000m
-XX:MaxPermSize=128m to -XX:MaxPermSize=2000m
Restart all the DataInsight Services on the Classification nodes.
02-04-2020 11:36 PM - edited 02-04-2020 11:37 PM
Version is 6.1 rp5 and we are seeing this after the upgrade aswell and is more.
Also seeing this issue on all classification servers and requests.
can Anyone help?
02-05-2020 12:48 AM
@Bengt_H is correct, changing the java memory options in the vmoptions files on the classificatiaon server should resolve this issue. There are two file to alter VICServer.vmoptions & VICServerService.vmoptions - make the changes to the -Xmx parameter (I don't think the MaxPermSize parameter is used anymore with the current java version installed).
If your DI installation is large, you might also want to look at equivalent settings in the vmoptions files on the management server and indexer as well. I had a situation of my communication service and watchdog service on the managemenbt server fail due to memory issues - best bet is to search for the string "outofmemory" in the log directory of each of your servers to see if anything is failing (use a case insensitive search - e.g. findstr /i outofmemory *.log).
FInally, if after altering the java memory settings you still get Veritas support involved - you pay the money for support, so might as well use it.