VCS for Solaris10 Swap used out issue
Hi, Experts there
I encounterd a problem confused me onsite. Swap useage on the master server is 100%. Although we already have 128g physical memory and 80g Swap. Asscociated Info as below.
Any suggestion? Thanks very much!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
VCS version
root@HOSTNAME> pkginfo -l VRTSvcs
PKGINST: VRTSvcs
NAME: Veritas Cluster Server by Symantec
CATEGORY: system
ARCH: sparc
VERSION: 5.0
BASEDIR: /
VENDOR: Symantec Corporation
DESC: Veritas Cluster Server by Symantec
PSTAMP: Veritas-5.0MP3RP3-11/27/09-09:17:00
INSTDATE: Jan 29 2011 12:09
STATUS: completely installed
FILES: 176 installed pathnames
26 shared pathnames
4 linked files
50 directories
90 executables
149372 blocks used (approx)
OS
root@HOSTNAME> uname -a
SunOS HOSTNAME 5.10 Generic_142900-03 sun4u sparc SUNW,SPARC-Enterprise
SWAP INFO
root@ df -lh | grep swap //////// I add 30g swap
swap 30G 2.3M 30G 1% /etc/svc/volatile
swap 30G 65M 30G 1% /tmp
swap 30G 96K 30G 1% /var/run
swap 30G 0K 30G 0% /dev/vx/dmp
swap 30G 0K 30G 0% /dev/vx/rdmp
root@ swap -l
swapfile dev swaplo blocks free
/dev/md/dsk/d20 85,20 16 524976 524976
/dev/vx/dsk/ossdg/swap 337,102024 16 22527984 22527984
/dev/vx/dsk/ossdg/swap-2 337,102000 16 67108848 67108848
/dev/vx/dsk/ossdg/swap-3 337,102002 16 41943024 41943024
/dev/vx/dsk/ossdg/swap-4 337,102003 16 62914544 62914544
Engine Log
root@ pwd
/var/VRTSvcs/log
root@ tail -50 engine_A.log
2011/02/02 13:28:25 VCS INFO V-16-6-0 (yginmaster) postonline:Executing /ericsson/hacs/scripts/postonline.sh with arg0=yginmaster, arg1=Oss
2011/02/02 13:28:25 VCS INFO V-16-6-0 (yginmaster) postonline.sh:Oss:Nothing done for group Oss
2011/02/02 13:28:25 VCS INFO V-16-6-0 (yginmaster) postonline:Completed execution of /ericsson/hacs/scripts/postonline.sh for group Oss
2011/02/02 13:28:26 VCS INFO V-16-6-15002 (yginmaster) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postonline yginmaster Oss successfully
2011/03/06 03:08:32 VCS ERROR V-16-2-13027 (yginmaster) Resource(cluster_maint) - monitor procedure did not complete within the expected time.
2011/03/08 14:31:10 VCS ERROR V-16-2-13027 (yginmaster) Resource(vcsweb) - monitor procedure did not complete within the expected time.
2011/03/13 11:01:10 VCS ERROR V-16-2-13027 (yginmaster) Resource(vcsweb) - monitor procedure did not complete within the expected time.
2011/03/15 09:45:31 VCS ERROR V-16-2-13027 (yginmaster) Resource(cluster_maint) - monitor procedure did not complete within the expected time.
2011/03/21 07:10:13 VCS INFO V-16-1-50086 Swap usage on yginmaster is 63%
2011/03/21 09:10:13 VCS INFO V-16-1-50086 Swap usage on yginmaster is 63%
2011/03/21 11:10:13 VCS INFO V-16-1-50086 Swap usage on yginmaster is 63%
2011/03/21 13:10:13 VCS INFO V-16-1-50086 Swap usage on yginmaster is 68%
2011/03/21 15:10:13 VCS INFO V-16-1-50086 Swap usage on yginmaster is 68%
2011/03/21 17:10:13 VCS INFO V-16-1-50086 Swap usage on yginmaster is 68%
2011/03/21 18:36:13 VCS NOTICE V-16-1-50086 Swap usage on yginmaster is 72%
2011/03/21 20:36:13 VCS NOTICE V-16-1-50086 Swap usage on yginmaster is 72%
2011/03/21 22:36:13 VCS NOTICE V-16-1-50086 Swap usage on yginmaster is 72%
2011/03/22 00:36:13 VCS NOTICE V-16-1-50086 Swap usage on yginmaster is 76%
2011/03/22 02:36:13 VCS NOTICE V-16-1-50086 Swap usage on yginmaster is 76%
2011/03/22 04:36:13 VCS NOTICE V-16-1-50086 Swap usage on yginmaster is 79%
2011/03/22 05:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 80%
2011/03/22 07:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 81%
2011/03/22 09:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 82%
2011/03/22 11:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 84%
2011/03/22 13:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 85%
2011/03/22 15:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 87%
2011/03/22 17:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 88%
2011/03/22 19:19:43 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 89%
2011/03/22 21:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 91%
2011/03/22 23:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 92%
2011/03/23 01:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 93%
2011/03/23 03:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 95%
2011/03/23 05:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 96%
2011/03/23 07:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 97%
2011/03/23 09:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 99%
2011/03/23 11:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 100%
2011/03/23 13:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 100%
2011/03/23 15:10:13 VCS WARNING V-16-1-50086 Swap usage on yginmaster is 89%
2011/03/23 17:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 91%
2011/03/23 19:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 92%
2011/03/23 21:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 93%
2011/03/23 23:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 95%
2011/03/24 01:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 96%
2011/03/24 03:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 97%
2011/03/24 05:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 98%
2011/03/24 07:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 99%
2011/03/24 09:10:13 VCS CRITICAL V-16-1-50086 Swap usage on yginmaster is 100%
2011/03/24 09:26:47 VCS WARNING V-16-1-10023 Agent Monitor not sending alive messages since Thu Mar 24 09:24:35 2011
2011/03/24 09:26:47 VCS WARNING V-16-1-53025 Agent Monitor has faulted; ipm connection was lost; restarting the agent
2011/03/24 09:26:47 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Monitor/MonitorAgent for resource type Monitor successfully started at Thu Mar 24 09:26:47 2011
