Classification request getting stuck - Inprogress

Hi Team,

We are classifying by share by share, but sometimes the request is getting stuck and not getting resumed. Sometime after the restart of the classificaiton server it is getting resumed.

--- Some logs where request suddenly stopped. and after restart it is good and resumed.

--- Some logs from VIC.log , where even after the restart it is same there is no increase in the classification request number in the console. After restart I can see that the polices are not loaded,

and some files are still there in content folder, request ID.

    - We tried to do a new classificaiton request and i am not sure that request is running good. BUt the stucked once are still showing as inprogress.

VIC0.0 Log:

2020-01-13 08:51:39 SEVERE: #{166} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_124943.docx
javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 19 more

2020-01-13 08:52:50 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] E:\DataInsight\data\classification\content\272\124\2214_125062.file->E:\DataInsight\data\classification\content\272\124\2214_125062.pdf
2020-01-13 08:52:50 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] Submitting path: E:\DataInsight\data\classification\content\272\124\2214_125062.pdf for classification
2020-01-13 08:52:55 SEVERE: #{149} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_125049.pdf
com.veritas.vic.client.ClassifierClientException: Service Unavailable
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:244)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1035)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
... 8 more
2020-01-13 08:52:55 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] E:\DataInsight\data\classification\content\272\124\2214_125065.file->E:\DataInsight\data\classification\content\272\124\2214_125065.pdf
2020-01-13 08:52:55 INFO: #{48} [ClassifyJob$ClassifierThread.processsSnapshotFile] Submitting path: E:\DataInsight\data\classification\content\272\124\2214_125065.pdf for classification
2020-01-13 08:52:55 SEVERE: #{163} [ClassifyJob$ClassifierThread.classify] Failed to classify the following file: 2214_125048.pdf
com.veritas.vic.client.ClassifierClientException: Service Unavailable
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:244)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:173)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:146)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.classify(ClassifyJob.java:837)
at com.symc.matrix.classification.vic.scheduler.ClassifyJob$ClassifierThread.lambda$processsSnapshotFile$0(ClassifyJob.java:676)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1035)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
at com.veritas.vic.client.ClassifierClient.post(ClassifierClient.java:267)
at com.veritas.vic.client.ClassifierClient.classify(ClassifierClient.java:242)
... 8 more

vic.log

2020-01-14 02:43:43 ERROR: #{PolicyMonitor-1} Failed to GET policies. Status=[null]
2020-01-14 02:43:43 ERROR: #{PatternMonitor-1} Failed to GET patterns. Status=[null]
2020-01-14 02:43:43 ERROR: #{PolicyMonitor-1} Failed to check for policies changes, or notification failed. Will try again later.
javax.ws.rs.ProcessingException: java.lang.IllegalStateException: Connection pool shut down
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681)
at com.veritas.vic.classifier.policiesClient.PoliciesClient.getPolicies(PoliciesClient.java:61)
at com.veritas.vic.classifier.policiesClient.PolicyChangeMonitor.checkForPolicyChanges(PolicyChangeMonitor.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.veritas.vic.common.threads.ClassifierThreadFactory.lambda$newThread$0(ClassifierThreadFactory.java:44)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:268)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
... 18 common frames omitted