cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Classification request getting stuck - Inprogress

SRGP
Level 2

 

Hi Team,

We are classifying by share by share, but sometimes the request is getting stuck and not getting resumed. Sometime after the restart of the classificaiton server it is getting resumed.

--- Some logs where request suddenly stopped. and after restart it is good and resumed.

--- Some logs from VIC.log , where even after the restart it is same there is no increase in the classification request number in the console. After restart I can see that the polices are not loaded,

and some files are still there in content folder, request ID.

    - We tried to do a new classificaiton request and i am not sure that request is running good. BUt the stucked once are still showing as inprogress.

 

VIC0.0 Log:

2020-01-13 08:51:39 SEVERE: #{166} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_124943.docx
javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 19 more

 

2020-01-13 08:52:50 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] E:\DataInsight\data\classification\content\272\124\2214_125062.file->E:\DataInsight\data\classification\content\272\124\2214_125062.pdf
2020-01-13 08:52:50 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] Submitting path: E:\DataInsight\data\classification\content\272\124\2214_125062.pdf for classification
2020-01-13 08:52:55 SEVERE: #{149} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_125049.pdf
com.veritas.vic.client.ClassifierClientException: Service Unavailable
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:244)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1035)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
... 8 more
2020-01-13 08:52:55 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] E:\DataInsight\data\classification\content\272\124\2214_125065.file->E:\DataInsight\data\classification\content\272\124\2214_125065.pdf
2020-01-13 08:52:55 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] Submitting path: E:\DataInsight\data\classification\content\272\124\2214_125065.pdf for classification
2020-01-13 08:52:55 SEVERE: #{163} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_125048.pdf
com.veritas.vic.client.ClassifierClientException: Service Unavailable
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:244)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1035)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
... 8 more

 

 

vic.log

 

2020-01-14 02:43:43 ERROR: #{PolicyMonitor-1} Failed to GET policies. Status=[null]
2020-01-14 02:43:43 ERROR: #{PatternMonitor-1} Failed to GET patterns. Status=[null]
2020-01-14 02:43:43 ERROR: #{PolicyMonitor-1} Failed to check for policies changes, or notification failed. Will try again later.
javax.ws.rs.ProcessingException: java.lang.IllegalStateException: Connection pool shut down
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681)
at com.veritas.vic.classifier.policiesClient.PoliciesClient.getPolicies(PoliciesClient.java:61)
at com.veritas.vic.classifier.policiesClient.PolicyChangeMonitor.checkForPolicyChanges(PolicyChangeMonitor.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:268)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 18 common frames omitted

7 REPLIES 7

Bengt_H
Level 3
Partner Accredited

I have no solution, but have a similar issue.
After this in the vic0.0.log the classification stops:

2020-01-14 14:26:04 SEVERE: #{1355} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 15_108943.pdf
javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 19 more

After a reboot of the classification server, the classification continues normaly...

What version of Data Insight are you running and is it the same across the various components (collector, indexer, classifier)?

Bengt_H
Level 3
Partner Accredited

We are on the latest version on both mangement and classifcation servers.

Version.PNG

We are having the same issue. It is also taking much longer for the classification to work since upgrading. It used to take 10 or so minutes to classify 2 documents before the upgrade.  It took 7 hours for two files to be classified post 6.1.5.

Bengt_H
Level 3
Partner Accredited

Changing the Java Heap memory seems to make things better. But I don't know if this is the solution...
But do test and see if it runs better.

Problem

Classification jobs on large shares stayed "in progress" for several days.

 

Cause

Out of JavaHeap memory

 

Solution

Modify the memory settings for VICSERVER.vmoptions and VICServerService.vmoptions settings on each Classification Node.  Navigate to C:\Program Files\DataInsight\bin

Open the files VICSERVER.vmoptions and VICServerService.vmoptions and change the values below:

-Xmx2500m to -Xmx8000m

-XX:MaxPermSize=128m to -XX:MaxPermSize=2000m

 

Restart all the DataInsight Services on the Classification nodes.

Version is 6.1 rp5 and we are seeing this after the upgrade aswell and is more.

Also seeing this issue on all classification servers and requests.

 

can Anyone  help?

davidmoline
Level 6
Employee

@Bengt_H is correct, changing the java memory options in the vmoptions files on the classificatiaon server should resolve this issue. There are two file to alter VICServer.vmoptions & VICServerService.vmoptions - make the changes to the -Xmx parameter (I don't think the MaxPermSize parameter is used anymore with the current java version installed). 

If your DI installation is large, you might also want to look at equivalent settings in the vmoptions files on the management server and indexer as well. I had a situation of my communication service and watchdog service on the managemenbt server fail due to memory issues - best bet is to search for the string "outofmemory" in the log directory of each of your servers to see if anything is failing (use a case insensitive search - e.g. findstr /i outofmemory *.log).

FInally, if after altering the java memory settings you still get Veritas support involved - you pay the money for support, so might as well use it.