cancel
Showing results for 
Search instead for 
Did you mean: 

Symantec Connect Community] VCS CRITICAL V-16-1-50086 CPU usage is 97%

David_Arnn1
Level 2

Hi,

 We receive VCS CRITICAL CPU usage errors very often:

Jan  8 01:08:45 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:08:45 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:08:52 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:08:52 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:08:57 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:08:57 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:30:43 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 01:30:43 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 01:51:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 06:10:01 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 06:10:01 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 08:17:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 08:24:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 08:34:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 08:45:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 08:47:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 09:00:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 09:01:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 09:05:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 09:26:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 09:28:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 09:40:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 09:54:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 09:55:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 09:58:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 10:09:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 10:12:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 10:17:30 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 10:23:31 LHDBEDB01 last message repeated 2 times
Jan  8 10:24:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 10:28:12 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:12 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:28:15 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:15 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:28:18 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:18 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:28:49 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 10:28:49 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 10:30:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 10:53:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 98%
Jan  8 10:54:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 10:59:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 11:22:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 11:44:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 11:49:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 11:56:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 12:25:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 12:26:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 12:41:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 95%
Jan  8 12:47:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 12:49:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 13:02:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 94%
Jan  8 13:04:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 13:12:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 13:16:02 LHDBEDB01 last message repeated 2 times
Jan  8 13:36:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 13:42:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 13:45:32 LHDBEDB01 last message repeated 1 time
Jan  8 13:47:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 13:51:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 14:02:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 14:03:47 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 14:03:47 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 14:07:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 14:11:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
Jan  8 14:20:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 14:24:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 14:25:31 LHDBEDB01 last message repeated 1 time
Jan  8 14:32:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 14:35:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 14:42:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 14:45:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 98%
Jan  8 14:47:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 14:48:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 14:51:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 14:53:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:02:01 LHDBEDB01 last message repeated 2 times
Jan  8 15:10:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:12:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 91%
Jan  8 15:13:33 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1
3027 (LHDBEAP01) Resource(app) - monitor procedure did not complete within the expected time.
Jan  8 15:20:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 93%
Jan  8 15:22:32 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 97%
Jan  8 15:25:02 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:29:01 LHDBEDB01 last message repeated 4 times
Jan  8 15:33:31 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:38:31 LHDBEDB01 last message repeated 2 times

Jan  8 15:41:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 96%
Jan  8 15:44:02 LHDBEDB01 last message repeated 1 time
Jan  8 15:50:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 92%
Jan  8 15:51:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 100%
Jan  8 15:51:24 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 15:51:24 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 15:51:30 LHDBEDB01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (ce0) node 1 in trouble
Jan  8 15:51:30 LHDBEDB01 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 2 (ce0) node 1 active
Jan  8 15:57:01 LHDBEDB01 Had[377]: [ID 702911 daemon.notice] VCS CRITICAL V-16-1-50086 CPU usage on LHDBEDB01 is 99%
bash-2.05$

root@LHDBEAP01 # pkginfo -l NTAPnfsa
   PKGINST:  NTAPnfsa
      NAME:  NetApp NFS Client Agent for Veritas Cluster Server by Symantec
  CATEGORY:  optional
      ARCH:  sun4u
   VERSION:  5.0.0
   BASEDIR:  /
    VENDOR:  Network Appliance, Inc.
      DESC:  NetApp NFS Client Agent for Veritas Cluster Server by Symantec
    PSTAMP:  NTAPnfsa:693884:20070816.1459
  INSTDATE:  Dec 18 2013 03:54
    STATUS:  completely installed
     FILES:       26 installed pathnames
                   7 shared pathnames
                  10 directories
                   2 executables
                 242 blocks used (approx)

 


bash-2.05$ sar 1 10

SunOS LHDBEDB01 5.9 Generic_117171-17 sun4u    01/08/2015

16:11:27    %usr    %sys    %wio   %idle
16:11:28      71       2       0      27
16:11:29      56       5       0      39
16:11:30      51       4       0      45
16:11:31      78       9       0      13
16:11:32      52       0       0      48
16:11:33      51       2       0      47
16:11:34      56       5       0      39
16:11:35      50       1       0      49
16:11:36      58       6       0      36
16:11:37      50       1       0      49

Average       57       3       0      39
bash-2.05$

4 REPLIES 4

mikebounds
Level 6
Partner Accredited

In the very short 10 seconds  you ran sar, the CPU ranged from 51% to 87%, so sounds feasible that your CPU is getting to 97% and this is causing Monitor timeouts and LLT issues.

You should investigate CPU usage over larger time span and see what processes are using CPU (use prstat).

Mike

David_Arnn1
Level 2

Dear Mike

I've tried the PRSTAT but everything show normal. When DMESG i got CPU usage sometime up to 97 to 100%.

mikebounds
Level 6
Partner Accredited

I would run sar with 1 minute or less interval and run for a period of time until you see a few 97 - 100% CPU usage messages in the logs and if output of sar is not also reporting high usage the post results here.

Also have a look at /var/VRTSvcs/log/HostMonitor_A.log as this gives more information about the CPU usage - post enties in this log for the timespan you run sar if as above sar is not reporting the same high usage as in the logs.

Mike

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

You may also want to download and install 'top' utility and keep it running in a terminal window to see which processes are high on CPU usage.