error messages

105 Topics

services not started after node reboot in VCS 5.1
Hello, we have a 4 node VCS 5.1 cluster (EngineVersion 5.1.10.40) configured with I/O fencing. 2 of the nodes are for the applications and the other 2 for the DB. Applications and the DB are configured to run only on the designated nodes. We are testing a complete heartbeat failure by shutting down the switch. The fencing works properly and three nodes are rebooted. We have observed a situation where 2 of the services was not started after both application nodes have rebooted. The service was in PARTIAL state. mmsoap-rg State lpdmc1p |PARTIAL| mmsoap-rg State lpdmc2p |OFFLINE| smppc-rg State lpdmc1p |PARTIAL| smppc-rg State lpdmc2p |OFFLINE| Here it is their configuration: group mmsoap-rg ( SystemList = { lpdmc1p = 0, lpdmc2p = 1 } AutoStartList = { lpdmc1p, lpdmc2p } ) IP mmsoap-lh-res ( Device = bond0 Address = "10.40.248.199" NetMask = "255.255.255.224" ) LVMLogicalVolume opt-mmsoap-lv-res ( VolumeGroup = mmsoap-vg LogicalVolume = opt-mmsoap-lv ) LVMLogicalVolume var-opt-mmsoap-lv-res ( VolumeGroup = mmsoap-vg LogicalVolume = var-opt-mmsoap-lv ) LVMVolumeGroup mmsoap-vg-res ( VolumeGroup = mmsoap-vg EnableLVMTagging = 1 ) Mount opt-mmsoap-mnt-res ( MountOpt = "rw,noatime,nodiratime,nosuid,nodev" FsckOpt = "-y" BlockDevice = "/dev/mapper/mmsoap--vg-opt--mmsoap--lv" MountPoint = "/opt/mmsoap" FSType = ext3 ) Mount var-opt-mmsoap-mnt-res ( MountOpt = "rw,noatime,nodiratime,nosuid,nodev" FsckOpt = "-y" BlockDevice = "/dev/mapper/mmsoap--vg-var--opt--mmsoap--lv" MountPoint = "/var/opt/mmsoap" FSType = ext3 ) NIC mmsoap-nic-res ( Device = bond0 ) SicapApplication mmsoap-app-res ( AppUser = mmsoap ) requires group smppc-rg online global firm mmsoap-app-res requires mmsoap-lh-res mmsoap-app-res requires opt-mmsoap-mnt-res mmsoap-app-res requires var-opt-mmsoap-mnt-res mmsoap-lh-res requires mmsoap-nic-res opt-mmsoap-lv-res requires mmsoap-vg-res opt-mmsoap-mnt-res requires opt-mmsoap-lv-res var-opt-mmsoap-lv-res requires mmsoap-vg-res var-opt-mmsoap-mnt-res requires var-opt-mmsoap-lv-res // resource dependency tree // // group mmsoap-rg // { // SicapApplication mmsoap-app-res // { // IP mmsoap-lh-res // { // NIC mmsoap-nic-res // } // Mount opt-mmsoap-mnt-res // { // LVMLogicalVolume opt-mmsoap-lv-res // { // LVMVolumeGroup mmsoap-vg-res // } // } // Mount var-opt-mmsoap-mnt-res // { // LVMLogicalVolume var-opt-mmsoap-lv-res // { // LVMVolumeGroup mmsoap-vg-res // } // } // } // } group smppc-rg ( SystemList = { lpdmc1p = 0, lpdmc2p = 1 } AutoStartList = { lpdmc1p, lpdmc2p } ) IP smppc-lh-res ( Device = bond0 Address = "10.40.248.200" NetMask = "255.255.255.224" ) LVMLogicalVolume opt-smppc-lv-res ( VolumeGroup = smppc-vg LogicalVolume = opt-smppc-lv ) LVMLogicalVolume var-opt-smppc-lv-res ( VolumeGroup = smppc-vg LogicalVolume = var-opt-smppc-lv ) LVMVolumeGroup smppc-vg-res ( VolumeGroup = smppc-vg EnableLVMTagging = 1 ) Mount opt-smppc-mnt-res ( MountOpt = "rw,noatime,nodiratime,nosuid,nodev" FsckOpt = "-y" BlockDevice = "/dev/mapper/smppc--vg-opt--smppc--lv" MountPoint = "/opt/smppc" FSType = ext3 ) Mount var-opt-smppc-mnt-res ( MountOpt = "rw,noatime,nodiratime,nosuid,nodev" FsckOpt = "-y" BlockDevice = "/dev/mapper/smppc--vg-var--opt--smppc--lv" MountPoint = "/var/opt/smppc" FSType = ext3 ) NIC smppc-nic-res ( Device = bond0 ) SicapApplication smppc-app-res ( AppUser = smppc ) requires group mmg-rg online global firm opt-smppc-lv-res requires smppc-vg-res opt-smppc-mnt-res requires opt-smppc-lv-res smppc-app-res requires opt-smppc-mnt-res smppc-app-res requires smppc-lh-res smppc-app-res requires var-opt-smppc-mnt-res smppc-lh-res requires smppc-nic-res var-opt-smppc-lv-res requires smppc-vg-res var-opt-smppc-mnt-res requires var-opt-smppc-lv-res // resource dependency tree // // group smppc-rg // { // SicapApplication smppc-app-res // { // Mount opt-smppc-mnt-res // { // LVMLogicalVolume opt-smppc-lv-res // { // LVMVolumeGroup smppc-vg-res // } // } // IP smppc-lh-res // { // NIC smppc-nic-res // } // Mount var-opt-smppc-mnt-res // { // LVMLogicalVolume var-opt-smppc-lv-res // { // LVMVolumeGroup smppc-vg-res // } // } // } // } The state of its resources: #Resource Attribute System Value mmsoap-lh-res State lpdmc1p OFFLINE mmsoap-lh-res State lpdmc2p OFFLINE opt-mmsoap-lv-res State lpdmc1p ONLINE opt-mmsoap-lv-res State lpdmc2p OFFLINE var-opt-mmsoap-lv-res State lpdmc1p ONLINE var-opt-mmsoap-lv-res State lpdmc2p OFFLINE mmsoap-vg-res State lpdmc1p OFFLINE mmsoap-vg-res State lpdmc2p OFFLINE opt-mmsoap-mnt-res State lpdmc1p OFFLINE opt-mmsoap-mnt-res State lpdmc2p OFFLINE var-opt-mmsoap-mnt-res State lpdmc1p OFFLINE var-opt-mmsoap-mnt-res State lpdmc2p OFFLINE mmsoap-nic-res State lpdmc1p ONLINE mmsoap-nic-res State lpdmc2p ONLINE mmsoap-app-res State lpdmc1p OFFLINE mmsoap-app-res State lpdmc2p OFFLINE smppc-lh-res State lpdmc1p OFFLINE smppc-lh-res State lpdmc2p OFFLINE opt-smppc-lv-res State lpdmc1p ONLINE opt-smppc-lv-res State lpdmc2p OFFLINE var-opt-smppc-lv-res State lpdmc1p ONLINE var-opt-smppc-lv-res State lpdmc2p OFFLINE smppc-vg-res State lpdmc1p OFFLINE smppc-vg-res State lpdmc2p OFFLINE opt-smppc-mnt-res State lpdmc1p OFFLINE opt-smppc-mnt-res State lpdmc2p OFFLINE var-opt-smppc-mnt-res State lpdmc1p OFFLINE var-opt-smppc-mnt-res State lpdmc2p OFFLINE smppc-nic-res State lpdmc1p ONLINE smppc-nic-res State lpdmc2p ONLINE smppc-app-res State lpdmc1p OFFLINE smppc-app-res State lpdmc2p OFFLINE In the engine log I had the following messages related to them: 2016/04/05 02:27:45 VCS NOTICE V-16-1-10181 Group mmsoap-rg AutoRestart set to 1 2016/04/05 02:27:46 VCS INFO V-16-1-10304 Resource mmsoap-lh-res (Owner: Unspecified, Group: mmsoap-rg) is offline on lpdmc1p (First probe) 2016/04/05 02:27:46 VCS INFO V-16-1-10297 Resource opt-mmsoap-lv-res (Owner: Unspecified, Group: mmsoap-rg) is online on lpdmc1p (First probe) 2016/04/05 02:27:46 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group mmsoap-rg on all nodes 2016/04/05 02:27:46 VCS INFO V-16-1-10297 Resource var-opt-mmsoap-lv-res (Owner: Unspecified, Group: mmsoap-rg) is online on lpdmc1p (First probe) 2016/04/05 02:27:46 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group mmsoap-rg on all nodes 2016/04/05 02:27:46 VCS INFO V-16-1-10304 Resource opt-mmsoap-mnt-res (Owner: Unspecified, Group: mmsoap-rg) is offline on lpdmc1p (First probe) 2016/04/05 02:27:46 VCS INFO V-16-1-10304 Resource var-opt-mmsoap-mnt-res (Owner: Unspecified, Group: mmsoap-rg) is offline on lpdmc1p (First probe) 2016/04/05 02:27:47 VCS INFO V-16-1-10304 Resource mmsoap-app-res (Owner: Unspecified, Group: mmsoap-rg) is offline on lpdmc1p (First probe) 2016/04/05 02:27:48 VCS INFO V-16-1-10304 Resource mmsoap-vg-res (Owner: Unspecified, Group: mmsoap-rg) is offline on lpdmc1p (First probe) 2016/04/05 02:27:48 VCS NOTICE V-16-1-10438 Group mmsoap-rg has been probed on system lpdmc1p if I manually onlined the group that was OK but I had to separately online both: 2016/04/05 05:58:03 VCS INFO V-16-1-50135 User root fired command: hagrp -online -any smppc-rg localclus from localhost 2016/04/05 05:58:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource smppc-lh-res (Owner: Unspecified, Group: smppc-rg) on System lpdmc1p 2016/04/05 05:58:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource smppc-vg-res (Owner: Unspecified, Group: smppc-rg) on System lpdmc1p 2016/04/05 05:58:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource opt-smppc-mnt-res (Owner: Unspecified, Group: smppc-rg) on System lpdmc1p 2016/04/05 05:58:03 VCS NOTICE V-16-1-10301 Initiating Online of Resource var-opt-smppc-mnt-res (Owner: Unspecified, Group: smppc-rg) on System lpdmc1p 2016/04/05 05:58:04 VCS ERROR V-16-10031-14001 (lpdmc1p) LVMVolumeGroup:smppc-vg-res:online:Activation of volume group failed. 2016/04/05 05:58:05 VCS INFO V-16-1-10298 Resource opt-smppc-mnt-res (Owner: Unspecified, Group: smppc-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 05:58:05 VCS INFO V-16-1-10298 Resource var-opt-smppc-mnt-res (Owner: Unspecified, Group: smppc-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 05:58:06 VCS INFO V-16-1-10298 Resource smppc-vg-res (Owner: Unspecified, Group: smppc-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 05:58:16 VCS INFO V-16-1-10298 Resource smppc-lh-res (Owner: Unspecified, Group: smppc-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 05:58:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource smppc-app-res (Owner: Unspecified, Group: smppc-rg) on System lpdmc1p 2016/04/05 05:58:16 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Running preonline for resource smppc-app-res 2016/04/05 05:58:16 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Preonline for resource smppc-app-res finished 2016/04/05 05:58:16 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Starting resource smppc-app-res 2016/04/05 05:58:21 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Resource smppc-app-res is started 2016/04/05 05:58:34 VCS INFO V-16-1-10298 Resource smppc-app-res (Owner: Unspecified, Group: smppc-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 05:58:34 VCS NOTICE V-16-1-10447 Group smppc-rg is online on system lpdmc1p 2016/04/05 05:59:52 VCS INFO V-16-1-50135 User root fired command: hagrp -online -any mmsoap-rg localclus from localhost 2016/04/05 05:59:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource mmsoap-lh-res (Owner: Unspecified, Group: mmsoap-rg) on System lpdmc1p 2016/04/05 05:59:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource mmsoap-vg-res (Owner: Unspecified, Group: mmsoap-rg) on System lpdmc1p 2016/04/05 05:59:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource opt-mmsoap-mnt-res (Owner: Unspecified, Group: mmsoap-rg) on System lpdmc1p 2016/04/05 05:59:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource var-opt-mmsoap-mnt-res (Owner: Unspecified, Group: mmsoap-rg) on System lpdmc1p 2016/04/05 05:59:53 VCS ERROR V-16-10031-14001 (lpdmc1p) LVMVolumeGroup:mmsoap-vg-res:online:Activation of volume group failed. 2016/04/05 05:59:53 VCS INFO V-16-1-10298 Resource opt-mmsoap-mnt-res (Owner: Unspecified, Group: mmsoap-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 05:59:53 VCS INFO V-16-1-10298 Resource var-opt-mmsoap-mnt-res (Owner: Unspecified, Group: mmsoap-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 05:59:54 VCS INFO V-16-1-10298 Resource mmsoap-vg-res (Owner: Unspecified, Group: mmsoap-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 06:00:02 VCS INFO V-16-1-10298 Resource mmsoap-lh-res (Owner: Unspecified, Group: mmsoap-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 06:00:02 VCS NOTICE V-16-1-10301 Initiating Online of Resource mmsoap-app-res (Owner: Unspecified, Group: mmsoap-rg) on System lpdmc1p 2016/04/05 06:00:02 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Running preonline for resource mmsoap-app-res 2016/04/05 06:00:03 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Preonline for resource mmsoap-app-res finished 2016/04/05 06:00:03 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Starting resource mmsoap-app-res 2016/04/05 06:00:09 VCS INFO V-16-1-0 (lpdmc1p) SicapApplication:???:???:Resource mmsoap-app-res is started 2016/04/05 06:00:21 VCS INFO V-16-1-10298 Resource mmsoap-app-res (Owner: Unspecified, Group: mmsoap-rg) is online on lpdmc1p (VCS initiated) 2016/04/05 06:00:21 VCS NOTICE V-16-1-10447 Group mmsoap-rg is online on system lpdmc1p What can I do to have this service started automatically in such a case when the HB is lost and the nodes are rebooted due to fencing panic? Thank you in advance, Laszlo
Laszlo_Budai
9 years ago Place Cluster Server
1.1KViews
0likes
1Comment
Unable to boot into the OS
When trying to boot Solaris 10 server with veritas volume manager 4.0 Iget the following error. NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk Oct 21 01:16:57 svc.startd[7]: svc:/system/power:default: Could not interpret user property. [ system/power:default failed (see 'svcs -x' for details) ] LLT INFO V-14-1-10009 LLT Protocol available GAB INFO V-15-1-20021 GAB available Oct 21 01:17:01 vxvm:vxconfigd: V-5-1-7601 vxvm:vxconfigd: ERROR: Could not open file /etc/vx/array.info for writing Oct 21 01:17:01 vxvm:vxconfigd: V-5-1-11219 vxvm:vxconfigd: ERROR: Failed to open file /etc/vx/disk.info.new Oct 21 01:17:01 vxvm:vxconfigd: V-5-1-7601 vxvm:vxconfigd: ERROR: Could not open file /etc/vx/array.info for writing [ system/sysevent:default failed repeatedly (see 'svcs -x' for details) ] Requesting System Maintenance Mode (See /lib/svc/share/README for more information.) Console login service(s) cannot run Can anyone provide any inight on these errors.
Phoenix80
10 years ago Place Cluster Server
1.1KViews
0likes
3Comments
V-16-20062-55 SVCCopyServices:online:Failed to get the cluster id
I am getting the following error when trying to online the SVCCopyService on windows, this comes up immediately there is no delay in receiving the message in the Status window. V-16-20062-55 (SERVERNAME) SVCCopyServices:RES-SVCCopyServices:online:Failed to get the cluster id We have exactly the same resource configuration on our UNIX system and this connects without any issues. We have also been able to run the plink command in an elevated command prompt window and this can connect to the SAN and run queries so I am a bit stumped. I have attached screenshots of the testing and error logs. Any suggestions would be greatly appreciated: OS - Windows 2012 R2 Standard VCS - 6.1 SAN - IBM V840
Solved
robhull
10 years ago Place Cluster Server
2.1KViews
1like
7Comments
lltconfig ERROR V-14-2-15121 LLT unconfigure aborted,
Hello, I've a 2 node cluster, with on CP server and 2 fencing disks. But think something is wrong on my fencing configuration. OS : RHEL6.6 VCS 6.2.1 # gabconfig -a GAB Port Memberships =============================================================== Port a gen 816e15 membership 01 Port a gen 816e15 jeopardy ;1 Port b gen 816e1b membership 01 Port b gen 816e1b jeopardy ;1 Port h gen 816e1e membership 01 Port h gen 816e1e jeopardy ;1 # lltconfig -a list Link 0 (bond0): Node 0 Node01 : 00:XX:XX:XX:XX:YY Node 1 Node02 : 00:XX:XX:XX:XX:YX permanent # cat /etc/llttab set-node Node02 set-cluster 229 link bond0 bond0 - ether - - link eth2 eth-00:11:AA:77:BB:22- ether - - I don't have a eth2 on my server, I tried to remove the second line of my/etc/llttab and stop and restart # /etc/init.d/llt stop Stopping LLT: LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 4 port(s) LLT:Warning: lltconfig failed. Retrying [1] LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 4 port(s) LLT:Warning: lltconfig failed. Retrying [2] LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 4 port(s) LLT:Warning: lltconfig failed. Retrying [3] LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 4 port(s) LLT:Warning: lltconfig failed. Retrying [4] LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 4 port(s) LLT:Warning: lltconfig failed. Retrying [5] LLT:Error: lltconfig failed Any suggestions will be welcome ... thanks Cédric
Solved
cedric_tours
10 years ago Place Cluster Server
4.7KViews
0likes
4Comments
New fileshare ressource fail on the second node
Hello guys, I need your help about a new fileshare that I create on a cluster ressource. I created the new fileshare as the old ones already excist on the same ressource. When I try to do a failover to the second node, the old fileshares bring online. But the new one no, and that's make the ressource in faulted state. I transfer the ressource to the first node, and all fileshares bring online including the new created. Conclusion : the fileshare can brings online on the first node but fails on the second node. I checked the configuration between the olds fileshare and this new one, and I didn't find any difference. Can you please help me about this issue. Thank you in advance.
Solved
Jimb2k
10 years ago Place Cluster Server
1.4KViews
0likes
5Comments
VCS Database Service Group Service Group Fails After Initiating Offline
Hi, We have 2 Storage Foundation for Oracle RAC 5.1SP1 serversrunning on AIX 5.3 Servers. We have Application Service Groupand DatabaseService Groupwhich is both active active. I encountered a problem when i initiated manual offline of the Service group. Application Service Group is ok. But in the Database Service Group it failedand the 2 servers unpectedly restarted and i encountered split-brain condition. Here's the engineA.log: ServiceGroup_PRD Application successfull offiline 2015/06/24 01:17:14 VCS INFO V-16-1-50135 User root fired command: hagrp -offline ServiceGroup_PRD Server1 from localhost 2015/06/24 01:17:14 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_PRD on system SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:17 VCS INFO V-16-1-10305 Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:19 VCS INFO V-16-1-10305 Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:19 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:19 VCS INFO V-16-1-10305 Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:19 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:20 VCS INFO V-16-1-10305 Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:20 VCS INFO V-16-1-10305 Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:24 VCS INFO V-16-1-10305 Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:24 VCS INFO V-16-1-10305 Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:32 VCS INFO V-16-1-10305 Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:32 VCS INFO V-16-1-10305 Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1 2015/06/24 01:17:33 VCS INFO V-16-1-10305 Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:34 VCS INFO V-16-1-10305 Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:17:35 VCS NOTICE V-16-1-10446 Group SERVICEGROUP_PRD is offline on system SERVER1 2015/06/24 01:17:35 VCS INFO V-16-6-15002 (SERVER1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/lvmvg_postoffline SERVER1 SERVICEGROUP_PRD successfully 2015/06/24 01:17:35 VCS INFO V-16-6-0 (SERVER1) postoffline:Invoked with arg0=SERVER1, arg1=SERVICEGROUP_PRD 2015/06/24 01:17:36 VCS INFO V-16-6-15002 (SERVER1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline SERVER1 SERVICEGROUP_PRD successfully 2015/06/24 01:17:41 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_PRD on system SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2 2015/06/24 01:17:45 VCS INFO V-16-1-10305 Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:45 VCS INFO V-16-1-10305 Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:47 VCS INFO V-16-1-10305 Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:47 VCS INFO V-16-1-10305 Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated) 2015/06/24 01:17:47 VCS NOTICE V-16-1-10446 Group SERVICEGROUP_PRD is offline on system SERVER2 2015/06/24 01:17:47 VCS INFO V-16-6-15002 (SERVER2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/lvmvg_postoffline SERVER2 SERVICEGROUP_PRD successfully 2015/06/24 01:17:47 VCS INFO V-16-6-0 (SERVER2) postoffline:Invoked with arg0=SERVER2, arg1=SERVICEGROUP_PRD2015/06/24 01:17:48 VCS INFO V-16-6-15002 (SERVER2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline SERVER2 SERVICEGROUP_PRDsuccessfully ServiceGroup_DBPRD Database fails tooffline 2015/06/24 01:18:51 VCS INFO V-16-1-50135 User root fired command: hagrp -offline SERVICEGROUP_DBPRD SERVER1 from localhost 2015/06/24 01:18:51 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_DBPRD on system SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ocr_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource quorum_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog1_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog2_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u02_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u03_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u04_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u05_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u06_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u07_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource voting_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:52 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:redolog1_mnt:offline:Umount Failed : Mount Point : /redolog1 2015/06/24 01:18:52 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:quorum_mnt:offline:Umount Failed : Mount Point : /quorum 2015/06/24 01:18:52 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog1_mnt:offline:Attempting fuser TERM : Mount Point : /redolog1 2015/06/24 01:18:52 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:quorum_mnt:offline:Attempting fuser TERM : Mount Point : /quorum 2015/06/24 01:18:53 VCS INFO V-16-1-10305 Resource ocr_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:18:53 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmocr (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1 2015/06/24 01:18:53 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:redolog2_mnt:offline:Umount Failed : Mount Point : /redolog2 2015/06/24 01:18:53 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog2_mnt:offline:Attempting fuser TERM : Mount Point : /redolog2 2015/06/24 01:18:55 VCS INFO V-16-1-10305 Resource cvmocr (Owner: unknown, Group: SERVICEGROUP_DBPRD) is offline on SERVER1 (VCS initiated) 2015/06/24 01:18:55 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u03_mnt:offline:Umount Failed : Mount Point : /u03 2015/06/24 01:18:55 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u03_mnt:offline:Attempting fuser TERM : Mount Point : /u03 2015/06/24 01:18:57 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u02_mnt:offline:Umount Failed : Mount Point : /u02 2015/06/24 01:18:57 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u02_mnt:offline:Attempting fuser TERM : Mount Point : /u02 2015/06/24 01:18:58 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u04_mnt:offline:Umount Failed : Mount Point : /u04 2015/06/24 01:18:58 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u04_mnt:offline:Attempting fuser TERM : Mount Point : /u04 2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog1_mnt:offline:Attempting fuser TERM : Mount Point : /redolog1 2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog2_mnt:offline:Attempting fuser TERM : Mount Point : /redolog2 2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:quorum_mnt:offline:Attempting fuser TERM : Mount Point : /quorum 2015/06/24 01:40:15 VCS NOTICE V-16-1-11022 VCS engine (had) started 2015/06/24 01:40:15 VCS INFO V-16-1-10196 Cluster logger started 2015/06/24 01:40:15 VCS NOTICE V-16-1-11050 VCS engine version=5.1 2015/06/24 01:40:15 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.00.0 2015/06/24 01:40:15 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.1-10/06/09-14:37:00 2015/06/24 01:40:15 VCS NOTICE V-16-1-10114 Opening GAB library 2015/06/24 01:40:16 VCS NOTICE V-16-1-10619 'HAD' starting on: SERVER1 2015/06/24 01:40:16 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms 2015/06/24 01:40:16 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms 2015/06/24 01:40:16 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message 2015/06/24 01:40:26 VCS INFO V-16-1-10077 Received new cluster membership 2015/06/24 01:40:26 VCS NOTICE V-16-1-10112 System (SERVER1) - Membership: 0x3, DDNA: 0x0 2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System (Node '1') changed state from UNKNOWN to INITING 2015/06/24 01:40:26 VCS NOTICE V-16-1-10086 System SERVER1 (Node '0') is in Regular Membership - Membership: 0x3 2015/06/24 01:40:26 VCS NOTICE V-16-1-10086 System (Node '1') is in Regular Membership - Membership: 0x3 2015/06/24 01:40:26 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'SERVER2' 2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER2 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT 2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER1 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD 2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER2 (Node '1') changed state from CURRENT_DISCOVER_WAIT to CURRENT_PEER_WAIT 2015/06/24 01:40:28 VCS NOTICE V-16-1-52006 UseFence=SCSI3. Fencing is enabled 2015/06/24 01:40:28 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/06/24 01:40:43 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/06/24 01:40:58 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/06/24 01:41:13 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/06/24 01:41:28 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/06/24 01:41:43 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/06/24 01:41:58 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing The solution i did here was to run vxfenclearpre. Is there any logs i need to check why the servers faults and restarted unexpectedly after service group offline? Any help will appreciate. Thanks.
Solved
H_A_N_B_U
10 years ago Place Cluster Server
3.7KViews
0likes
9Comments
Note able to configure the HA Proxy with Application Agent
Hi Guys, I am not able to configure the HA proxy with application AGENT. its not giving the status after probed completed. Application HAPROXY ( Critical = 0 StartProgram = "/usr/src/haproxy-1.5.12/HAPROXY start" StopProgram = " /usr/src/haproxy-1.5.12/HAPROXY stop" CleanProgram = "/usr/src/haproxy-1.5.12/HAPROXY stop" PidFiles = { "/var/run/haproxy.pid" } MonitorProcesses = { haproxy } ) ------------------------------------------------------------------------------------------------------------------ ApplicationAgent logs -- 2015/07/06 15:57:15 VCS DBG_AGDEBUG Calling monitor for resource HAPROXY VCSAgType.C:call_monitor[1492] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:monitor program <> before squeeze() Application.C:isMonitorProgramConfigured[1286] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:inside function squeeze(),arguments passed is <> Application.C:squeeze[1339] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Exiting from function squeeze, no of tokens are <0> , returning <-1> Application.C:squeeze[1379] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:monitor program <> after squeeze() Application.C:isMonitorProgramConfigured[1298] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Exiting from isMonitorProgramConfigured, return code <0> Application.C:isMonitorProgramConfigured[1299] 2015/07/06 15:57:15 VCS DBG_1 Application:HAPROXY:monitor:UseSUDash:<0>. Application.C:application_monitor[167] 2015/07/06 15:57:15 VCS DBG_1 Application:HAPROXY:monitor:State=<0>, IState=<1> Application.C:application_monitor[188] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Validating and retrieving the information for user <root> Application.C:application_monitor[190] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Retrieving information for User <root> Application.C:getuserinfo[1233] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Retrieved information for: User <root> Home Directory </root> Login Name <root> Default Shell </bin/bash> Application.C:getuserinfo[1252] 2015/07/06 15:57:15 VCS DBG_AGDEBUG Monitor-Info is 5 VCSAgPrimitives.C:VCSAgGetMonitorLevel[4588] 2015/07/06 15:57:15 VCS DBG_AGDEBUG LevelTwoMonitorFreq is 1 VCSAgPrimitives.C:VCSAgIsL2MFSet[4504] 2015/07/06 15:57:15 VCS DBG_3 Application:HAPROXY:monitor:level_one_monitor = [1], level_two_monitor = [2], l2mf set [1] Application.C:application_monitor[211] 2015/07/06 15:57:15 VCS DBG_1 Application:HAPROXY:monitor:Total number of Pid Files specified:1. Application.C:application_monitor[226] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Checking for PidFile:</var/run/haproxy.pid> Application.C:application_monitor[235] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:inside function squeeze(),arguments passed is </var/run/haproxy.pid> Application.C:squeeze[1339] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Exiting from function squeeze, no of tokens are <1> , returning <0> Application.C:squeeze[1379] 2015/07/06 15:57:15 VCS DBG_5 Application:HAPROXY:monitor:Checking for pid in PidFile:</var/run/haproxy.pid> Application.C:application_monitor[249] 2015/07/06 15:57:15 VCS DBG_4 Application:HAPROXY:monitor:Got pid-file:/var/run/haproxy.pid to check for the pid. Application.C:pidExists[419] 2015/07/06 15:57:15 VCS DBG_3 Application:HAPROXY:monitor:Found pid (17961) from pid-file (/var/run/haproxy.pid). Application.C:pidExists[457] 2015/07/06 15:57:15 VCS DBG_2 Application:HAPROXY:monitor:Pid-file:</var/run/haproxy.pid>; checking the existence of pid:<17961> Application.C:pidExists[467] 2015/07/06 15:57:15 VCS DBG_3 Application:HAPROXY:monitor:Error occured. Pid-file:</var/run/haproxy.pid>; Pid (17961) does not exists. Application.C:pidExists[472] Application.C:pidExists[472] 2015/07/06 15:57:15 VCS DBG_4 Application:HAPROXY:monitor:Status got after checking Pid File (/var/run/haproxy.pid):OFFLINE. Application.C:application_monitor[255] 2015/07/06 15:57:15 VCS DBG_AGINFO Resource(HAPROXY) - monitor entry point exited with a confidence value 0. VCSAgType.C:call_monitor[1584] 2015/07/06 15:57:15 VCS DBG_AGINFO HAPROXY reported state (Offline) & conf_level (0) VCSAgRes.C:call_entry_point[1599] 2015/07/06 15:57:15 VCS DBG_AGDEBUG Canceling timer for (HAPROXY) op(1608) VCSAgTimer.C:_cancel[947] ============================================== Engine_A_logs. VCS_CONF=/etc/VRTSvcs VCS_DIAG=/var/VRTSvcs VCS_HOME=/opt/VRTSvcs VCS_LOG_AGENT_NAME= VCS_LOG_CATEGORY=6 VCS_LOG_SCRIPT_NAME=hatrigger VCS_LOG=/var/VRTSvcs ########## Other Environment Variables ########## CONSOLETYPE=vt LANG=en_US.UTF-8 LANGSH_SOURCED=1 LD_LIBRARY_PATH=/opt/VRTSvcs/lib: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/opt/VRTSvcs/bin previous=N PREVLEVEL=N PWD=/var/VRTSvcs/diag/had runlevel=5 RUNLEVEL=5 SHLVL=5 SYSTEMCTL_SKIP_REDIRECT=true TERM=linux UPSTART_EVENTS=runlevel UPSTART_INSTANCE= UPSTART_JOB=rc _=/usr/bin/env 2015/07/06 15:36:40 VCS INFO V-16-6-15002 (DTPJBS01) hatrigger:hatrigger executed /opt/VRTSvcs/bin/internal_triggers/dump_tunables DTPJBS01 1 successfully 2015/07/06 15:36:42 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system DTPJBS01 2015/07/06 15:36:43 VCS INFO V-16-1-10304 Resource JBOSS (Owner: Unspecified, Group: JBOSS_APP) is offline on DTPJBS01 (First probe) 2015/07/06 15:36:43 VCS NOTICE V-16-1-10438 Group JBOSS_APP has been probed on system DTPJBS01 2015/07/06 15:36:43 VCS INFO V-16-1-50007 Initiating auto-start online of group JBOSS_APP 2015/07/06 15:36:43 VCS INFO V-16-1-10493 Evaluating DTPJBS01 as potential target node for group JBOSS_APP 2015/07/06 15:36:43 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group JBOSS_APP on all nodes 2015/07/06 15:36:43 VCS NOTICE V-16-1-10301 Initiating Online of Resource JBOSS (Owner: Unspecified, Group: JBOSS_APP) on System DTPJBS01 2015/07/06 15:36:44 VCS NOTICE V-16-1-10438 Group VCShmg has been probed on system DTPJBS01 2015/07/06 15:36:44 VCS NOTICE V-16-1-10435 Group VCShmg will not start automatically on System DTPJBS01 as the system is not a part of AutoStartList attribute of the group. 2015/07/06 15:36:56 VCS INFO V-16-1-10298 Resource JBOSS (Owner: Unspecified, Group: JBOSS_APP) is online on DTPJBS01 (VCS initiated) 2015/07/06 15:36:56 VCS NOTICE V-16-1-10447 Group JBOSS_APP is online on system DTPJBS01 2015/07/06 15:52:41 VCS INFO V-16-1-55029 Resource HAPROXY in offline state received recurring offline message on system DTPJBS02 2015/07/06 15:52:55 VCS NOTICE V-16-1-10460 Clearing start attribute for resource HAPROXY of group HAProxy on node DTPJBS01 2015/07/06 15:52:55 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group HAProxy on all nodes 2015/07/06 15:52:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource HAPROXY (Owner: Unspecified, Group: HAProxy) on System DTPJBS02 2015/07/06 15:52:55 VCS INFO V-16-1-50159 User fired command: hares -online HAPROXY DTPJBS02 from node DTPJBS01 2015/07/06 15:54:56 VCS ERROR V-16-2-13066 (DTPJBS02) Agent is calling clean for resource(HAPROXY) because the resource is not up even after online completed. 2015/07/06 15:54:57 VCS INFO V-16-2-13068 (DTPJBS02) Resource(HAPROXY) - clean completed successfully. 2015/07/06 15:54:57 VCS INFO V-16-2-13071 (DTPJBS02) Resource(HAPROXY): reached OnlineRetryLimit(0). 2015/07/06 15:54:57 VCS ERROR V-16-1-54031 Resource HAPROXY (Owner: Unspecified, Group: HAProxy) is FAULTED on sys DTPJBS02 2015/07/06 15:54:57 VCS NOTICE V-16-1-10446 Group HAProxy is offline on system DTPJBS02 2015/07/06 15:54:57 VCS INFO V-16-6-15015 (DTPJBS02) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed 2015/07/06 15:55:09 VCS INFO V-16-1-50159 User fired command: hares -clear HAPROXY DTPJBS02 from node DTPJBS01 2015/07/06 15:55:09 VCS INFO V-16-1-10307 Resource HAPROXY (Owner: Unspecified, Group: HAProxy) is offline on DTPJBS02 (Not initiated by VCS) 2015/07/06 15:55:13 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group HAProxy on all nodes 2015/07/06 15:55:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource HAPROXY (Owner: Unspecified, Group: HAProxy) on System DTPJBS02 2015/07/06 15:55:13 VCS INFO V-16-1-50159 User fired command: hares -online HAPROXY DTPJBS02 from node DTPJBS01 2015/07/06 15:57:14 VCS ERROR V-16-2-13066 (DTPJBS02) Agent is calling clean for resource(HAPROXY) because the resource is not up even after online completed. 2015/07/06 15:57:15 VCS INFO V-16-2-13068 (DTPJBS02) Resource(HAPROXY) - clean completed successfully. 2015/07/06 15:57:15 VCS INFO V-16-2-13071 (DTPJBS02) Resource(HAPROXY): reached OnlineRetryLimit(0). 2015/07/06 15:57:15 VCS ERROR V-16-1-54031 Resource HAPROXY (Owner: Unspecified, Group: HAProxy) is FAULTED on sys DTPJBS02 2015/07/06 15:57:15 VCS NOTICE V-16-1-10446 Group HAProxy is offline on system DTPJBS02 2015/07/06 15:57:15 VCS INFO V-16-6-15015 (DTPJBS02) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed 2015/07/06 16:02:38 VCS INFO V-16-1-50159 User fired command: hares -clear HAPROXY DTPJBS02 from node DTPJBS01 2015/07/06 16:02:38 VCS INFO V-16-1-10307 Resource HAPROXY (Owner: Unspecified, Group: HAProxy) is offline on DTPJBS02 (Not initiated by VCS) 2015/07/06 16:02:43 VCS INFO V-16-1-55029 Resource HAPROXY in offline state received recurring offline message on system DTPJBS02 2015/07/06 16:03:29 VCS INFO V-16-1-50159 User fired command: haconf -dump from node DTPJBS01 2015/07/06 18:48:51 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/07/06 22:48:52 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/07/07 02:48:53 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2015/07/07 06:48:54 VCS INFO V-16-1-53504 VCS Engine Alive message!! Please hlep.
lazaur
10 years ago Place Cluster Server
1.9KViews
1like
8Comments
LUNs are not coming automatically under veritas after reboot
Hi Community, I have installedSFCFSHA clustersolution on RHEL 1.6, storage is connected through Fiber Channel Over Ethernet (FCoE) interace and dmp is used as multipathing solution. The issue is while i reboot the system, I/O fecning got failed, disks are not coming automatically under veritas control. When i scan the disks and configure IO fecning, cluster starts working. But i am unable to understand why disks are not coming under veritas control automatically. Error - MMTE03:~# tail -100f /var/VRTSvcs/log/engine_A.log 2015/07/08 20:59:55 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:00 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:05 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:10 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:15 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:25 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:30 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... 2015/07/08 21:00:35 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... Thanks Pranav
Solved
Pranav_Tyagi
10 years ago Place Cluster Server
2.3KViews
0likes
7Comments
VCS Setup - "Install A Clustered Master Server" Greyed Out
I originally posted this in the NetBackup forum but it seems to be a VCS issue that is the root of the problem: https://www-secure.symantec.com/connect/forums/install-clustered-master-server-greyed-out I am trying to set up a test NBU 7.6.0.1 master server cluster with VCS on Windows Server 2008 R2but the option to "install a clustered master server" is greyed out. I have built2 new servers, installed SFHA / VCS 6.0.1and configured disks / volumes / disk groups / VVR replication. The VCS installation has not been modified or customised in any way as from my understanding the NBU installation will create its own service group. I have carried out the initial configuration and added the remote cluster node. Within cluster explorer, the defaultClusterService service group is all up and running, the global cluster option is enabled andthe remote cluster status also shows that the other cluster is up. Marianne has alreadyasked me to run two commands with the following output: 'gabconfig -a' - "GAB gabconfig ERROR V-15-2-25022" lltconfig -nvv - "LLT lltconfig ERROR V-14-2-15000 open \\.\llt failed: error number: 0x2" I've probably just made a silly mistake somewherebut I can't figure outwhat piece of the puzzle I have missed - any ideas?
Solved
Shaun_Taylor
10 years ago Place Cluster Server
4.6KViews
1like
37Comments
vcs resource status abnormal
I am facing a problem,i found a resource become online monitor timeout , and i don't know how to fix it ; could anybody give me some suggestion. thanks; #hastatus .............. group resource system message --------------- -------------------- -------------------- -------------------- cfsmount1 CCCG09BER ONLINE cfsmount1 CCCG10BER ONLINE cfsmount2 CCCG09BER ONLINE cfsmount2 CCCG10BER ONLINE ------------------------------------------------------------------------- cvmvoldg1 CCCG09BER ONLINE cvmvoldg1 CCCG10BER ONLINE group resource system message --------------- -------------------- --------------- -------------------- cvmvoldg1 CCCG09BER |MONITOR TIMEDOUT|UNABLE TO OFFLINE| cvmvoldg2 CCCG09BER ONLINE cvmvoldg2 CCCG10BER ONLINE cvmvoldg2 CCCG09BER |MONITOR TIMEDOUT|UNABLE TO OFFLINE| --------------------------------------------------------------------------------------------------------------------------------------------- # hares -state cvmvoldg1 #Resource Attribute System Value cvmvoldg1 State CCCG09BER ONLINE|MONITOR TIMEDOUT cvmvoldg1 State CCCG10BER ONLINE # # vxdisk list DEVICE TYPE DISK GROUP STATUS c0t0d0s2 auto:SVM - - SVM c0t1d0s2 auto:SVM - - SVM c0t2d0s2 auto:none - - online invalid c0t3d0s2 auto:none - - online invalid c3t0d0 auto:cdsdisk - - online c3t0d1 auto:sliced mmdatadgemcpower4 mmdatadg online shared c3t0d2 auto:cdsdisk - - online c3t0d3 auto:cdsdisk - - online c3t0d4 auto:sliced mmdbdgemcpower2 mmdbdg online shared c3t0d5s2 auto:sliced mmdatadgc3t0d5 mmdatadg online shared c3t0d6s2 auto:sliced mmdatadgc3t0d6 mmdatadg online shared c3t0d7s2 auto:sliced mmdatadgc3t0d7 mmdatadg online shared c3t0d8s2 auto:sliced mmdatadgc3t0d8 mmdatadg online shared c3t0d9s2 auto:sliced mmdatadgc3t0d9 mmdatadg online shared c3t0d10s2 auto:sliced mmdatadgc3t0d10 mmdatadg online shared # vxprint Disk group: mmdbdg TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg mmdbdg mmdbdg - - - - - - dm mmdbdgemcpower2 c3t0d4 - 419348384 - - - - v vol01 fsgen ENABLED 419346432 - ACTIVE - - pl vol01-01 vol01 ENABLED 419346432 - ACTIVE - - sd mmdbdgemcpower2-01 vol01-01 ENABLED 419346432 0 - - - Disk group: mmdatadg TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg mmdatadg mmdatadg - - - - - - dm mmdatadgc3t0d5 c3t0d5s2 - 1467866928 - - - - dm mmdatadgc3t0d6 c3t0d6s2 - 1887246288 - - - - dm mmdatadgc3t0d7 c3t0d7s2 - 1836919616 - - - - dm mmdatadgc3t0d8 c3t0d8s2 - 1887246288 - - - - dm mmdatadgc3t0d9 c3t0d9s2 - 1887246288 - - - - dm mmdatadgc3t0d10 c3t0d10s2 - 1417511040 - - - - dm mmdatadgemcpower4 c3t0d1 - 1887354784 - - - - v vol01 fsgen ENABLED 12271390720 - ACTIVE - - pl vol01-01 vol01 ENABLED 12271390720 - ACTIVE - - sd mmdatadgemcpower4-01 vol01-01 ENABLED 1887354784 0 - - - sd mmdatadgc3t0d5-01 vol01-01 ENABLED 1467866928 1887354784 - - - sd mmdatadgc3t0d6-01 vol01-01 ENABLED 1887246288 3355221712 - - - sd mmdatadgc3t0d7-01 vol01-01 ENABLED 1836919616 5242468000 - - - sd mmdatadgc3t0d8-01 vol01-01 ENABLED 1887246288 7079387616 - - - sd mmdatadgc3t0d9-01 vol01-01 ENABLED 1887246288 8966633904 - - - sd mmdatadgc3t0d10-01 vol01-01 ENABLED 1417510528 10853880192 - - - messages in engine_A log: 2015/03/06 13:44:21 VCS ERROR V-16-2-13027 (CCCG09BER) Resource(cvmvoldg2) - monitor procedure did not complete within the expected time. 2015/03/06 13:44:22 VCS ERROR V-16-2-13027 (CCCG09BER) Resource(cvmvoldg1) - monitor procedure did not complete within the expected time. 2015/03/06 13:46:28 VCS INFO V-16-1-50086 CPU usage on CCCG09BER is 61% 2015/03/06 13:50:21 VCS ERROR V-16-2-13210 (CCCG09BER) Agent is calling clean for resource(cvmvoldg2) because 4 successive invocatio ns of the monitor procedure did not complete within the expected time. 2015/03/06 13:50:21 VCS ERROR V-16-2-13210 (CCCG09BER) Agent is calling clean for resource(cvmvoldg1) because 4 successive invocatio ns of the monitor procedure did not complete within the expected time. 2015/03/06 13:50:39 VCS INFO V-16-2-13068 (CCCG09BER) Resource(cvmvoldg2) - clean completed successfully. 2015/03/06 13:50:53 VCS INFO V-16-2-13068 (CCCG09BER) Resource(cvmvoldg1) - clean completed successfully. 2015/03/06 13:51:40 VCS ERROR V-16-2-13077 (CCCG09BER) Agent is unable to offline resource(cvmvoldg2). Administrative intervention m ay be required. 2015/03/06 13:51:55 VCS ERROR V-16-2-13077 (CCCG09BER) Agent is unable to offline resource(cvmvoldg1). Administrative intervention m ay be required.
Solved
ebotang
10 years ago Place Cluster Server
1.1KViews
3likes
1Comment