Symantec Connect Community] VCS CRITICAL V-16-1-50086 CPU usage is 97%

[David_Arnn1]

Hi,

 We receive VCS CRITICAL CPU usage errors very often:

Jan  8 01:08:45 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:08:45 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:08:52 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:08:52 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:08:57 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:08:57 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:30:43 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:30:43 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:51:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 06:10:01 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 06:10:01 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 08:17:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 08:24:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 08:34:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 08:45:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 08:47:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 09:00:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 09:01:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 09:05:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 09:26:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 09:28:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 09:40:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 09:54:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 09:55:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 09:58:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 10:09:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 10:12:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 10:17:30 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 10:23:31 LHDBEDB01 last message repeated 2 times
Jan  8 10:24:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 10:28:12 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:12 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:28:15 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:15 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:28:18 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:18 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:28:49 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:49 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:30:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 10:53:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 98%
Jan  8 10:54:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 10:59:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 11:22:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 11:44:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 11:49:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 11:56:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 12:25:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 12:26:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 12:41:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 12:47:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 12:49:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 13:02:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 13:04:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 13:12:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 13:16:02 LHDBEDB01 last message repeated 2 times
Jan  8 13:36:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 13:42:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 13:45:32 LHDBEDB01 last message repeated 1 time
Jan  8 13:47:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 13:51:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 14:02:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 14:03:47 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 14:03:47 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 14:07:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 14:11:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 14:20:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 14:24:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 14:25:31 LHDBEDB01 last message repeated 1 time
Jan  8 14:32:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 14:35:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 14:42:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 14:45:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 98%
Jan  8 14:47:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 14:48:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 14:51:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 14:53:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:02:01 LHDBEDB01 last message repeated 2 times
Jan  8 15:10:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:12:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 15:13:33 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 15:20:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 15:22:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 15:25:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:29:01 LHDBEDB01 last message repeated 4 times
Jan  8 15:33:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:38:31 LHDBEDB01 last message repeated 2 times

Jan  8 15:41:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 15:44:02 LHDBEDB01 last message repeated 1 time
Jan  8 15:50:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 15:51:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:51:24 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 15:51:24 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 15:51:30 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 15:51:30 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 15:57:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
bash-2.05$

root@LHDBEAP01 # pkginfo -l NTAPnfsa
   PKGINST:  NTAPnfsa
      NAME:  NetApp NFS Client Agent for Veritas Cluster Server by Symantec
  CATEGORY:  optional
      ARCH:  sun4u
   VERSION:  5.0.0
   BASEDIR:  /
    VENDOR:  Network Appliance, Inc.
      DESC:  NetApp NFS Client Agent for Veritas Cluster Server by Symantec
    PSTAMP:  NTAPnfsa:693884:20070816.1459
  INSTDATE:  Dec 18 2013 03:54
    STATUS:  completely installed
     FILES:       26 installed pathnames
                   7 shared pathnames
                  10 directories
                   2 executables
                 242 blocks used (approx)

 

bash-2.05$ sar 1 10

SunOS LHDBEDB01 5.9 Generic_117171-17 sun4u    01/08/2015

16:11:27    %usr    %sys    %wio   %idle
16:11:28      71       2       0      27
16:11:29      56       5       0      39
16:11:30      51       4       0      45
16:11:31      78       9       0      13
16:11:32      52       0       0      48
16:11:33      51       2       0      47
16:11:34      56       5       0      39
16:11:35      50       1       0      49
16:11:36      58       6       0      36
16:11:37      50       1       0      49

Average       57       3       0      39
bash-2.05$

[mikebounds]

In the very short 10 seconds  you ran sar, the CPU ranged from 51% to 87%, so sounds feasible that your CPU is getting to 97% and this is causing Monitor timeouts and LLT issues.

You should investigate CPU usage over larger time span and see what processes are using CPU (use prstat).

Mike

[David_Arnn1]

Dear Mike

I've tried the PRSTAT but everything show normal. When DMESG i got CPU usage sometime up to 97 to 100%.

[mikebounds]

I would run sar with 1 minute or less interval and run for a period of time until you see a few 97 - 100% CPU usage messages in the logs and if output of sar is not also reporting high usage the post results here.

Also have a look at /var/VRTSvcs/log/HostMonitor_A.log as this gives more information about the CPU usage - post enties in this log for the timespan you run sar if as above sar is not reporting the same high usage as in the logs.

Mike

[Marianne]

You may also want to download and install 'top' utility and keep it running in a terminal window to see which processes are high on CPU usage.