need urgent help
While booting below errors
dmesg
Sun Oct 14 16:25:41 UTC 2012
Oct 14 14:22:18 rmgdbp09 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x710 belonging to the dmpnode 286/0xc8
Oct 14 14:23:52 rmgdbp09 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x710 belonging to the dmpnode 286/0xc8
Oct 14 14:29:16 rmgdbp09 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x700 belonging to the dmpnode 286/0x50
Oct 14 14:33:18 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,32 (ssd223):
Oct 14 14:33:18 rmgdbp09 offline
Oct 14 14:33:52 rmgdbp09 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x700 belonging to the dmpnode 286/0x50
Oct 14 14:38:18 rmgdbp09 vxdmp: [ID 917986 kern.notice] NOTICE: VxVM vxdmp V-5-0-112 disabled path 118/0x6f8 belonging to the dmpnode 286/0x80
Oct 14 14:38:52 rmgdbp09 vxdmp: [ID 736771 kern.notice] NOTICE: VxVM vxdmp V-5-0-148 enabled path 118/0x6f8 belonging to the dmpnode 286/0x80
Oct 14 14:39:45 rmgdbp09 vxesd[141]: [ID 360244 daemon.notice] Event Source daemon started
Oct 14 14:39:45 rmgdbp09 syseventd[75]: [ID 617319 daemon.error] SIGHUP caught - reloading modules
Oct 14 14:39:46 rmgdbp09 syseventd[75]: [ID 661968 daemon.error] Daemon restarted
Oct 14 14:41:14 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: laner0
Oct 14 14:41:14 rmgdbp09 genunix: [ID 936769 kern.info] laner0 is /pseudo/laner@0
Oct 14 14:46:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:46:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 14:49:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:49:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 14:51:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:51:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 14:53:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:53:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 14:56:18 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 14:56:18 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 14:58:21 rmgdbp09 llt: [ID 144861 kern.notice] LLT INFO V-14-1-10009 LLT Protocol available
Oct 14 14:58:32 rmgdbp09 lmx: [ID 504452 kern.notice] LMX Multiplexor available
Oct 14 14:58:34 rmgdbp09 root: [ID 702911 daemon.error] CAPTURE_UPTIME ERROR: /var/opt/SUNWsrsrp missing
Oct 14 14:58:34 rmgdbp09 savecore: [ID 686316 auth.error] no dump device configured
Oct 14 14:58:34 rmgdbp09 last message repeated 1 time
Oct 14 14:58:34 rmgdbp09 genunix: [ID 454863 kern.info] dump on /dev/dsk/c1t0d0s1 size 32793 MB
Oct 14 14:58:34 rmgdbp09 ntpdate[1207]: [ID 558275 daemon.notice] adjust time server 193.88.157.124 offset -0.367765 sec
Oct 14 14:58:35 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: tod0
Oct 14 14:58:35 rmgdbp09 genunix: [ID 936769 kern.info] tod0 is /pseudo/tod@0
Oct 14 14:58:35 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: pm0
Oct 14 14:58:35 rmgdbp09 genunix: [ID 936769 kern.info] pm0 is /pseudo/pm@0
Oct 14 14:58:35 rmgdbp09 gab: [ID 872190 kern.notice] GAB INFO V-15-1-20021 GAB available
Oct 14 14:58:35 rmgdbp09 gab: [ID 222459 kern.notice] GAB INFO V-15-1-20026 Port a registration waiting for seed port membership
Oct 14 14:58:36 rmgdbp09 gab: [ID 222459 kern.notice] GAB INFO V-15-1-20026 Port d registration waiting for seed port membership
Oct 14 14:58:36 rmgdbp09 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 1 (ce5) node 0 active
Oct 14 14:58:36 rmgdbp09 llt: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 0 (ce1) node 0 active
Oct 14 14:58:36 rmgdbp09 xntpd[1314]: [ID 702911 daemon.notice] xntpd 3-5.93e Mon Sep 20 15:47:11 PDT 1999 (1)
Oct 14 14:58:36 rmgdbp09 xntpd[1314]: [ID 301315 daemon.notice] tickadj = 5, tick = 10000, tvu_maxslew = 495, est. hz = 100
Oct 14 14:58:37 rmgdbp09 xntpd[1314]: [ID 798731 daemon.notice] using kernel phase-lock loop 0041
Oct 14 14:58:37 rmgdbp09 last message repeated 1 time
Oct 14 14:58:40 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port d gen 2d7c05 membership 01
Oct 14 14:58:40 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port a gen 2d7c06 membership 01
Oct 14 15:00:01 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 15:07:12 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:07:12 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 15:15:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 15:22:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:22:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 15:30:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 15:32:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:32:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 15:38:17 rmgdbp09 scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060e8006cfa802,0 (ssd226):
Oct 14 15:38:17 rmgdbp09 SCSI transport failed: reason 'timeout': giving up
Oct 14 15:45:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 16:00:00 rmgdbp09 vcsmm: [ID 743917 kern.notice] VCS RAC ERROR V-10-1-15013 vcsmm_ioctl: driver not configured
Oct 14 16:08:22 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port b gen 2d7c0a membership 01
Oct 14 16:08:22 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port o gen 2d7c09 membership 01
Oct 14 16:08:22 rmgdbp09 vcsmm: [ID 357760 kern.notice] VCS RAC INFO V-10-1-15047 mmpl_reconfig_ioctl: dev_ioctl failed, vxfen may not be configured
Oct 14 16:08:22 rmgdbp09 vxfen: [ID 725375 kern.notice] NOTICE: VCS FEN INFO V-11-1-35 Fencing driver going into RUNNING state
Oct 14 16:08:42 rmgdbp09 snmpXdmid: [ID 723131 daemon.error] Error in Adding Row for Subscription Table Entry
Oct 14 16:08:42 rmgdbp09 snmpXdmid: [ID 132663 daemon.error] Failed to add filter to SP for Event delivery
Oct 14 16:09:49 rmgdbp09 syslog[4934]: [ID 702911 daemon.notice] VCS INFO V-16-1-11240 Command Server: running with security OFF
Oct 14 16:09:49 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10619 'HAD' starting on: rmgdbp09
Oct 14 16:09:50 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10620 Waiting for local cluster configuration status
Oct 14 16:09:51 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10624 Local cluster configuration stale
Oct 14 16:09:51 rmgdbp09 last message repeated 1 time
Oct 14 16:09:51 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-11034 Registering for cluster membership
Oct 14 16:09:51 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-11035 Waiting for cluster membership
Oct 14 16:09:56 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port h gen 2d7c19 membership 01
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS INFO V-16-1-10077 Received new cluster membership
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10086 System (Node '0') is in Regular Membership - Membership: 0x3
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10086 System rmgdbp09 (Node '1') is in Regular Membership - Membership: 0x3
Oct 14 16:09:56 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10075 Building from remote system
Oct 14 16:09:58 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:09:58 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-10066 Entering RUNNING state
Oct 14 16:09:58 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS NOTICE V-16-1-50311 VCS Engine: running with security OFF
Oct 14 16:10:00 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:10:04 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:10:09 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port u gen 2d7c1b membership ;1
Oct 14 16:10:09 rmgdbp09 gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port u gen 2d7c1b k_jeopardy 0
Oct 14 16:10:09 rmgdbp09 gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port u gen 2d7c1b visible 0
Oct 14 16:10:14 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen 2d7c1a membership 01
Oct 14 16:10:15 rmgdbp09 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Oct 14 16:10:15 rmgdbp09 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Oct 14 16:10:20 rmgdbp09 vxvm:vxconfigd: [ID 456610 daemon.notice] V-5-1-7900 CVM_VOLD_CONFIG command received
Oct 14 16:10:20 rmgdbp09 vxvm:vxconfigd: [ID 699813 daemon.notice] V-5-1-7899 CVM_VOLD_CHANGE command received
Oct 14 16:10:20 rmgdbp09 vxvm:vxconfigd: [ID 322665 daemon.notice] V-5-1-7961 establishing cluster
Oct 14 16:10:25 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen 2d7c20 membership 01
Oct 14 16:10:31 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:11:16 rmgdbp09 last message repeated 40 times
Oct 14 16:11:25 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(srvm_vol_dg) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(RMAN_VOL01) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:11:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:13:35 rmgdbp09 vxio: [ID 535583 kern.warning] WARNING: VxVM vxio V-5-0-164 Failed to join cluster rmgprod, aborting
Oct 14 16:13:35 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port u closed
Oct 14 16:13:35 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port v closed
Oct 14 16:13:35 rmgdbp09 vxvm:vxconfigd: [ID 955251 daemon.error] V-5-1-8765 cluster join for node 1 failed: Connection timed out
Oct 14 16:13:39 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port w closed
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 741513 daemon.notice] V-5-1-7901 CVM_VOLD_STOP command received
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 778436 daemon.error] V-5-1-4109 -1 returned from volcvm_establish
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 886039 daemon.warning] V-5-1-4852 cluster_establish: timed out
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 453237 daemon.error] V-5-1-11178 kernel_fail_join() : join timed out during reconfiguration (12, -1)
Oct 14 16:13:39 rmgdbp09 vxvm:vxconfigd: [ID 565473 daemon.notice] V-5-1-9543 Timeout is not reset: another reconfig in progress
Oct 14 16:14:10 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:14:10 rmgdbp09 reason: request to join failed
Oct 14 16:15:00 rmgdbp09 vcsmm: [ID 832650 kern.notice] VCSMM: mm_deblog_sz = 1048576
Oct 14 16:15:00 rmgdbp09 vcsmm: [ID 706457 kern.notice] VCSMM: mm_msglog_sz = 128
Oct 14 16:15:00 rmgdbp09 vcsmm: [ID 838746 kern.notice] VCSMM: mm_slave_max = 1024
Oct 14 16:15:10 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:15:10 rmgdbp09 reason: request to join failed
Oct 14 16:15:12 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13066 (rmgdbp09) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Oct 14 16:15:13 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:15:17 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port u gen 2d7c22 membership ;1
Oct 14 16:15:17 rmgdbp09 gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port u gen 2d7c22 k_jeopardy 0
Oct 14 16:15:17 rmgdbp09 gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port u gen 2d7c22 visible 0
Oct 14 16:15:22 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen 2d7c1c membership 01
Oct 14 16:15:28 rmgdbp09 vxvm:vxconfigd: [ID 456610 daemon.notice] V-5-1-7900 CVM_VOLD_CONFIG command received
Oct 14 16:15:28 rmgdbp09 vxvm:vxconfigd: [ID 699813 daemon.notice] V-5-1-7899 CVM_VOLD_CHANGE command received
Oct 14 16:15:28 rmgdbp09 vxvm:vxconfigd: [ID 322665 daemon.notice] V-5-1-7961 establishing cluster
Oct 14 16:15:33 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen 2d7c22 membership 01
Oct 14 16:15:39 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:16:44 rmgdbp09 last message repeated 23 times
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(RMAN_VOL01) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:16:45 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:16:52 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:17:02 rmgdbp09 last message repeated 16 times
Oct 14 16:17:25 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(srvm_vol_dg) - monitor procedure did not complete within the expected time.
Oct 14 16:18:43 rmgdbp09 vxio: [ID 535583 kern.warning] WARNING: VxVM vxio V-5-0-164 Failed to join cluster rmgprod, aborting
Oct 14 16:18:43 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port u closed
Oct 14 16:18:43 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port v closed
Oct 14 16:18:43 rmgdbp09 vxvm:vxconfigd: [ID 955251 daemon.error] V-5-1-8765 cluster join for node 1 failed: Connection timed out
Oct 14 16:18:47 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port w closed
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 741513 daemon.notice] V-5-1-7901 CVM_VOLD_STOP command received
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 778436 daemon.error] V-5-1-4109 -1 returned from volcvm_establish
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 886039 daemon.warning] V-5-1-4852 cluster_establish: timed out
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 453237 daemon.error] V-5-1-11178 kernel_fail_join() : join timed out during reconfiguration (12, -1)
Oct 14 16:18:47 rmgdbp09 vxvm:vxconfigd: [ID 565473 daemon.notice] V-5-1-9543 Timeout is not reset: another reconfig in progress
Oct 14 16:19:18 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:19:18 rmgdbp09 reason: request to join failed
Oct 14 16:20:18 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:20:18 rmgdbp09 reason: request to join failed
Oct 14 16:20:19 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13066 (rmgdbp09) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Oct 14 16:20:20 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:20:25 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port u gen 2d7c24 membership ;1
Oct 14 16:20:25 rmgdbp09 gab: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port u gen 2d7c24 k_jeopardy 0
Oct 14 16:20:25 rmgdbp09 gab: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port u gen 2d7c24 visible 0
Oct 14 16:20:30 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port v gen 2d7c1e membership 01
Oct 14 16:20:36 rmgdbp09 vxvm:vxconfigd: [ID 456610 daemon.notice] V-5-1-7900 CVM_VOLD_CONFIG command received
Oct 14 16:20:36 rmgdbp09 vxvm:vxconfigd: [ID 699813 daemon.notice] V-5-1-7899 CVM_VOLD_CHANGE command received
Oct 14 16:20:36 rmgdbp09 vxvm:vxconfigd: [ID 322665 daemon.notice] V-5-1-7961 establishing cluster
Oct 14 16:20:41 rmgdbp09 gab: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port w gen 2d7c24 membership 01
Oct 14 16:20:47 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:21:44 rmgdbp09 last message repeated 20 times
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPUB_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_oracle) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PPTL_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(PAVS_VOL_arch) - monitor procedure did not complete within the expected time.
Oct 14 16:21:46 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(RMAN_VOL01) - monitor procedure did not complete within the expected time.
Oct 14 16:21:50 rmgdbp09 vxfen: [ID 214757 kern.notice] NOTICE: VCS FEN INFO V-11-1-34 The ioctl VXFEN_IOC_CLUSTSTAT returned 0
Oct 14 16:22:10 rmgdbp09 last message repeated 19 times
Oct 14 16:22:24 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13027 (rmgdbp08) Resource(srvm_vol_dg) - monitor procedure did not complete within the expected time.
Oct 14 16:23:20 rmgdbp09 sshd[8133]: [ID 800047 auth.error] error: PAM: System error for illegal user sumitsg from 193.88.157.75
Oct 14 16:23:25 rmgdbp09 last message repeated 1 time
Oct 14 16:23:29 rmgdbp09 sshd[8180]: [ID 800047 auth.error] error: ssh_msg_send: write
Oct 14 16:23:51 rmgdbp09 vxio: [ID 535583 kern.warning] WARNING: VxVM vxio V-5-0-164 Failed to join cluster rmgprod, aborting
Oct 14 16:23:51 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port u closed
Oct 14 16:23:51 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port v closed
Oct 14 16:23:51 rmgdbp09 vxvm:vxconfigd: [ID 955251 daemon.error] V-5-1-8765 cluster join for node 1 failed: Connection timed out
Oct 14 16:23:55 rmgdbp09 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port w closed
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 741513 daemon.notice] V-5-1-7901 CVM_VOLD_STOP command received
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 778436 daemon.error] V-5-1-4109 -1 returned from volcvm_establish
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 886039 daemon.warning] V-5-1-4852 cluster_establish: timed out
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 453237 daemon.error] V-5-1-11178 kernel_fail_join() : join timed out during reconfiguration (12, -1)
Oct 14 16:23:55 rmgdbp09 vxvm:vxconfigd: [ID 565473 daemon.notice] V-5-1-9543 Timeout is not reset: another reconfig in progress
Oct 14 16:24:26 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:24:26 rmgdbp09 reason: request to join failed
Oct 14 16:25:26 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:25:26 rmgdbp09 reason: request to join failed
Oct 14 16:25:27 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-13066 (rmgdbp09) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Oct 14 16:25:28 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (rmgdbp09) CVMCluster:???:monitor:node - state: out of cluster
Oct 14 16:25:28 rmgdbp09 reason: request to join failed
Oct 14 16:25:29 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10303 Resource cvm_clus (Owner: unknown, Group: cvm) is FAULTED (timed out) on sys rmgdbp09
Oct 14 16:25:30 rmgdbp09 arp: [ID 994748 kern.notice] ar_query_xmit: Could not find the ace
Oct 14 16:25:30 rmgdbp09 last message repeated 1 time
Oct 14 16:25:31 rmgdbp09 Had[4933]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group cvm is faulted on system rmgdbp09
root@rmgdbp09 #
server acme up , however no service group is online and its stuck at cvm service group
root@rmgdbp09 # /etc/vx/bin/vxclustadm nodestate
state: out of cluster
reason: request to join failed
root@rmgdbp09 # root@rmgdbp09
root@rmgdbp08 # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen 2d7c06 membership 01
Port b gen 2d7c0a membership 01
Port d gen 2d7c05 membership 01
Port f gen 2d7c18 membership 0
Port f gen 2d7c18 visible ;1
Port h gen 2d7c19 membership 01
Port o gen 2d7c09 membership 01
Port q gen 2d7c16 membership 0
Port q gen 2d7c16 visible ;1
Port v gen 2d7c1f membership 0
Port v gen 2d7c1f visible ;1
Port w gen 2d7c25 membership 0
Port w gen 2d7c25 visible ;1
root@rmgdbp08 #
Any clue from anybody ?
duplicate post:
https://www-secure.symantec.com/connect/forums/need-help-17