node freeze v/s service group freeze
Hi Team, I came across a DIMM replacement activity in one of our solars servers which are in cluster. Please assist before proceeding for teh activity shall i have to freeze teh service group OR DO i need to freeze the node? ALso, kindly guide the scenario, when we should have toproceed for freezing the SG or freezing the node? Thanks..Solved6.8KViews1like1CommentChange the Host IP address in the Veritas cluster
Hello All, We have a Veritas cluster server setup (VCS-HA,VCS-CFS & VERITAS-RAC) where on few setup we required to change the data IP address of some host(node). I refer few notes but not sure except /etc/hosts is there any file need to update/edit. Please help me if you have any process/technote to make those change & make that changed IP persistant. Also would like to know the impact of this activity. The systems are Linux 6.5 & 6.6 & cluster versions are VCS 6.2 & 6.1.Solved3.6KViews0likes3Commentssystem state FAULTED
> Hi, > > I am having VCS 6.0 running on solaris 10on a 2 node cluster. All the > nodes were rebooted as part of scheduled job, during which i got number > of messages in engine log which i am trying to understand. The sequence > of events in log are as: > > 1) all nodes showed as jeopardy state after boot for a moment. Why ? Is > it that one of the link was down for a moment just after booting > 2) After that log says system changed state from RUNNING to FAULTED. I > have never seen that system goes to FAULTED state, why it went to > faulted state. > 3) After this log shows that service groups became autodisabled on all > these nodes. > 4) after this, System (hostname) is in Down State - Membership: 0x4a > 5) VCS:10451:Cleared attribute-'autodisabled' for Group on node, does > the autodisabled flag gets cleared on its own. Many times i have faced > situations where i have cleared the autodisable flag for servicegroup > manually. > > Does somebody know about these error messages & what could be the reason > behind this, specifically the system going to FAULTED state and service > groups getting autodisabled. > > Thanks >Solved3.5KViews0likes1CommentResource group in STARTING|PARTIAL
Can anyone explain step by step what we usually do when we have a resource group in STARTING|PARTIAL phase in vcs See below log: 2014/11/25 16:58:51 VCS ERROR V-16-2-13066 (localhost) Agent is calling clean for resource(cfsmount3) because the resource is not up even after online completed. 2014/11/25 16:58:52 VCS INFO V-16-2-13068 (localhost) Resource(cfsmount3) - clean completed successfully. 2014/11/25 16:58:52 VCS INFO V-16-2-13071 (localhost) Resource(cfsmount3): reached OnlineRetryLimit(0). 2014/11/25 16:58:52 VCS ERROR V-16-1-10303 Resource cfsmount3 (Owner: unknown, Group: vrts_vea_cfs_int_cfsmount2) is FAULTED (timed out) on sys (localhost) 2014/11/25 16:58:52 VCS INFO V-16-6-15004 (localhost) hatrigger:Failed to send trigger for resfault; script doesn't existSolved3.2KViews0likes1Commentconcurrency violation
Hi Team, This alert came in my environment in one of the AIX server 6.1 ofconcurrency violation. Subject: VCS SevereError for Service Group sapgtsprd, Service group concurrency violation Event Time: Wed Dec 3 06:46:54 EST 2014 Entity Name: sapgtsprd Entity Type: Service Group Entity Subtype: Failover Entity State: Service group concurrency violation Traps Origin: Veritas_Cluster_Server System Name: mapibm625 Entities Container Name: GTS_Prod Entities Container Type: VCS ================================================================================== engineA.log are, 2014/12/03 06:46:54 VCS INFO V-16-1-10299 Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) is online on mapibm625 (Not initiated by VCS) 2014/12/03 06:46:54 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group sapgtsprd 2014/12/03 06:46:54 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group sapgtsprd on all nodes 2014/12/03 06:46:55 VCS WARNING V-16-6-15034 (mapibm625) violation:-Offlining group sapgtsprd on system mapibm625 2014/12/03 06:46:55 VCS INFO V-16-1-50135 User root fired command: hagrp -offline sapgtsprd mapibm625 from localhost 2014/12/03 06:46:55 VCS NOTICE V-16-1-10167 Initiating manual offline of group sapgtsprd on system mapibm625 ====================================================================================================== What isConcurrency Violation in VCS? What are the steps we should have to take to resolve this?Kindly explain in detail. Thanks, AllaboutunixSolved2.8KViews2likes4CommentsService group concurrency violation
Hi Team, We have alerts ofconcurrency violation, we have two servers in cluster mapibm625, mapibm626 Logs are, 2014/12/26 19:37:03 VCS INFO V-16-1-10299 Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) is online on mapibm625 (Not initiated by VCS) 2014/12/26 19:37:03 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group sapgtsprd 2014/12/26 19:37:03 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group sapgtsprd on all nodes 2014/12/26 19:37:04 VCS WARNING V-16-6-15034 (mapibm625) violation:Offlining group sapgtsprd on system mapibm625 2014/12/26 19:37:04 VCS INFO V-16-1-50135 User root fired command: hagrp -offline sapgtsprd mapibm625 from localhost 2014/12/26 19:37:04 VCS NOTICE V-16-1-10167 Initiating manual offline of group sapgtsprd on system mapibm625 2014/12/26 19:37:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) on System mapibm625 2014/12/26 19:37:04 VCS INFO V-16-6-15002 (mapibm625) hatrigger:hatrigger executed /opt/VRTSvcs/bin/internal_triggers/violation mapibm625 sapgtsprd successfully 2014/12/26 19:37:04 VCS INFO V-16-10011-306 (mapibm625) Application:App_saposcol:offline:Execution of Stop Program (/opt/VRTSvcs/bin/Saposcol/offline) returned (0). 2014/12/26 19:37:05 VCS INFO V-16-2-13716 (mapibm625) Resource(App_saposcol): Output of the completed operation (offline) ============================================== 2014/12/26 19:37:06 VCS INFO V-16-1-10305 Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) is offline on mapibm625 (VCS initiated) 2014/12/26 19:37:06 VCS NOTICE V-16-1-10446 Group sapgtsprd is offline on system mapibm625 ======================================================================================== I have asked the application team to look out as whether they are working on the servers because the resource is of SAP(Resource App_saposcol) However, application team has replied that they are not working on it and might theApp_saposcol is online on both of servers which causes the issue. Then, I have checked the status of resources in both the servers and it says, [root@mapibm626]: # hares -state #Resource Attribute System Value App_saposcol State mapibm625 OFFLINE App_saposcol State mapibm626 ONLINE [root@mapibm625]: # hares -state #Resource Attribute System Value App_saposcol State mapibm625 OFFLINE App_saposcol State mapibm626 ONLINE and also checked the current logs of the server however found only, 2014/12/27 13:03:42 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/27 17:03:43 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/27 21:03:44 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 01:03:45 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 05:03:46 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 09:03:47 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 10:56:14 VCS INFO V-16-1-50086 CPU usage on mapibm625 is 61% 2014/12/28 11:26:14 VCS INFO V-16-1-50086 CPU usage on mapibm625 is 61% 2014/12/28 13:03:48 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 14:26:14 VCS INFO V-16-1-50086 CPU usage on mapibm625 is 60% 2014/12/28 17:03:49 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 21:03:50 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 01:03:51 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 05:03:52 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 09:03:53 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 13:03:55 VCS INFO V-16-1-53504 VCS Engine Alive message!! ========================================================================== Please assist what could be the possible reasons for this and in future how to avoid this? Thanks, AllaboutunixSolved2.6KViews1like7CommentsService group shows online and cluster service offline
Hi TEam, I have an output of the hastatus which is attached, It shows ClusterService online in 35th system and rest of the system shows Clusterservice Offline however, Service group POWERCENTERSERVICEMANAGER shows online in all systems 35,36,37,38,39,40. I am unable to get it why the Service groupshows online in all the other systems when only 35th system ClusterService is online and rest others are offline. Do that the scenario of Active-Active Cluster? Please help me to understand this scenario.Solved2.2KViews1like2CommentsRoot cause for resource fault in vcs 6.0
Hi Team, We have a AIX 6.0 server in which vcs 6.0 is runningresource got faulted and now came online automatically. Submitting some details, We only know that there is something wrong with database listener configuration, however unable to find the root cause of it why resource faulted and came online automatically Can you please help to understand about this? Before: [root@cylibm004 /]# hastatus -sum -- SYSTEM STATE -- System State Frozen A cylibm003 RUNNING 0 A cylibm004 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService cylibm003 Y N ONLINE B ClusterService cylibm004 Y N OFFLINE B DB_GLSCENR5 cylibm003 Y N OFFLINE B DB_GLSCENR5 cylibm004 Y N PARTIAL B DB_GLSCYL cylibm003 Y N ONLINE B DB_GLSCYL cylibm004 Y N OFFLINE -- RESOURCES FAILED -- Group Type Resource System C DB_GLSCENR5 Netlsnr lsnr_glscyl cylibm004 After: [root@cylibm004 /]# hastatus -sum -- SYSTEM STATE -- System State Frozen A cylibm003 RUNNING 0 A cylibm004 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService cylibm003 Y N ONLINE B ClusterService cylibm004 Y N OFFLINE B DB_GLSCENR5 cylibm003 Y N OFFLINE B DB_GLSCENR5 cylibm004 Y N ONLINE B DB_GLSCYL cylibm003 Y N ONLINE B DB_GLSCYL cylibm004 Y N OFFLINE -----Original Message----- From: Notifier Sent: Friday, November 07, 2014 2:15 PM Subject: VCS Error for Resource lsnr_glscyl, Resource has faulted Event Time: Fri Nov 7 14:15:06 CST 2014 Entity Name: lsnr_glscyl Entity Type: Resource Entity Subtype: Netlsnr Entity State: Resource has faulted Traps Origin: Veritas_Cluster_Server System Name: cylibm004 Entities Container Name: DB_GLSCENR5 Entities Container Type: Service Group Entities Owner: unknown ============================================================================================================================== EngineA.log - 2014/11/07 14:14:54 VCS WARNING V-16-10011-8 (cylibm004) Netlsnr:lsnr_glscyl:LsnrTest.pl: File /oracle/.profile is not a valid text file 2014/11/07 14:14:55 VCS INFO V-16-20002-211 (cylibm004) Netlsnr:lsnr_glscyl:monitor:Monitor procedure /opt/VRTSagents/ha/bin/Netlsnr/LsnrTest.pl returned the output: Cannot get "LOGNAME" variable. 2014/11/07 14:14:55 VCS ERROR V-16-2-13067 (cylibm004) Agent is calling clean for resource(lsnr_glscyl) because the resource became OFFLINE unexpectedly, on its own. 2014/11/07 14:14:55 VCS NOTICE V-16-20002-42 (cylibm004) Netlsnr:lsnr_glscyl:clean:Listener(LISTENER_GLSCENR5) kill TERM 12845176 2014/11/07 14:15:06 VCS INFO V-16-2-13068 (cylibm004) Resource(lsnr_glscyl) - clean completed successfully. 2014/11/07 14:15:06 VCS WARNING V-16-20002-226 (cylibm004) Netlsnr:lsnr_glscyl:monitor:getargs for process tnslsnr failed with return code 0 2014/11/07 14:15:06 VCS INFO V-16-1-10307 Resource lsnr_glscyl (Owner: unknown, Group: DB_GLSCENR5) is offline on cylibm004 (Not initiated by VCS) 2014/11/07 14:15:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 65% 2014/11/07 14:18:10 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 64% 2014/11/07 14:21:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 66% 2014/11/07 14:25:08 VCS INFO V-16-1-10299 Resource lsnr_glscyl (Owner: unknown, Group: DB_GLSCENR5) is online on cylibm004 (Not initiated by VCS) 2014/11/07 14:25:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group DB_GLSCENR5 on all nodes 2014/11/07 14:25:08 VCS NOTICE V-16-1-10447 Group DB_GLSCENR5 is online on system cylibm004 2014/11/07 14:27:10 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 64% 2014/11/07 14:30:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 65% 2014/11/07 14:31:10 VCS NOTICE V-16-1-50086 CPU usage on cylibm004 is 70% =========================================================================================== ============================================================================================ Main.cf file - MountPoint = "/DB_GLSCENR5/oracle" BlockDevice = "/dev/GLSCENR5_Oracle" FSType = jfs2 FsckOpt = "-y" ) Mount GLSCENR5_data3 ( Critical = 0 MountPoint = "/DB_GLSCENR5/data3" BlockDevice = "/dev/GLSCENR5_data3" FSType = jfs2 FsckOpt = "-y" ) Netlsnr lsnr_glscyl ( Critical = 0 Owner = oracle Home = "/DB_GLSCENR5/oracle/product/11.2.0.3" TnsAdmin = "/var/opt/oracle" Listener = LISTENER_GLSCENR5 EnvFile = "/oracle/.profile" ) Oracle ora_glscyl ( Critical = 0 Sid = glscenr5 Owner = oracle Home = "/DB_GLSCENR5/oracle/product/11.2.0.3" EnvFile = "/oracle/.profile" ) Proxy DB_GLSCENR5_Proxy ( TargetResName = csgnic ) DB_GLSCENR5_IP requires DB_GLSCENR5_Proxy GLSCENR5_ARCH requires DB_GLS_CENR5_LVMVG GLSCENR5_BACKUP requires DB_GLS_CENR5_LVMVG GLSCENR5_DATA1 requires DB_GLS_CENR5_LVMVG GLSCENR5_DATA2 requires DB_GLS_CENR5_LVMVG GLSCENR5_ORACLE requires DB_GLS_CENR5_LVMVG GLSCENR5_data3 requires DB_GLS_CENR5_LVMVG lsnr_glscyl requires DB_GLSCENR5_IP lsnr_glscyl requires ora_glscyl ora_glscyl requires GLSCENR5_ARCH ora_glscyl requires GLSCENR5_BACKUP ora_glscyl requires GLSCENR5_DATA1 ora_glscyl requires GLSCENR5_DATA2 ora_glscyl requires GLSCENR5_ORACLE ora_glscyl requires GLSCENR5_data3 // resource dependency tree // // group DB_GLSCENR5 // { // Netlsnr lsnr_glscyl // { // Oracle ora_glscyl // { // Mount GLSCENR5_ORACLE // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_DATA2 // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_DATA1 // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_BACKUP // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_ARCH // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_data3 // { // LVMVG DB_GLS_CENR5_LVMVG // } // } // IP DB_GLSCENR5_IP // { // Proxy DB_GLSCENR5_Proxy // } // } // } group DB_GLSCYL ( SystemList = { cylibm003 = 0, cylibm004 = 1 } ) IP DB_GLSCYL_IP ( Critical = 0 Device = en2 Address = "132.189.249.119" NetMask = "255.255.255.128" ) LVMVG DB_GLSCYL_LVMVG ( VolumeGroup = DB_GLS_PRD MajorNumber @cylibm003 = 40 MajorNumber @cylibm004 = 40 ) Mount GLS_ARCH ( Critical = 0 MountPoint = "/DB_GLSCYL/arch" BlockDevice = "/dev/GLSCYL_Arch" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_BACKUP ( Critical = 0 MountPoint = "/DB_GLSCYL/backup" BlockDevice = "/dev/GLSCYL_Backup" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_DATA1 ( Critical = 0 MountPoint = "/DB_GLSCYL/data1" BlockDevice = "/dev/GLSCYL_Data1" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_DATA2 ( Critical = 0 MountPoint = "/DB_GLSCYL/data2" BlockDevice = "/dev/GLSCYL_Data2" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_ORACLE ( Critical = 0 MountPoint = "/DB_GLSCYL/oracle" BlockDevice = "/dev/GLSCYL_Oracle" FSType = jfs2 FsckOpt = "-y" ) Netlsnr lsnr_dbglscyl ( Critical = 0 Owner = oracle Home = "/DB_GLSCYL/oracle/product/11.2.0.3" TnsAdmin = "/var/opt/oracle" Listener = LISTENER_GLSCYL EnvFile = "/oracle/.profile" ) Oracle ora_dbglscyl ( Critical = 0 Sid = glscyl Owner = oracle Home = "/DB_GLSCYL/oracle/product/11.2.0.3" EnvFile = "/oracle/.profile" ) Proxy GLSCYL_PROXY ( TargetResName = csgnic ) DB_GLSCYL_IP requires GLSCYL_PROXY GLS_ARCH requires DB_GLSCYL_LVMVG GLS_BACKUP requires DB_GLSCYL_LVMVG GLS_DATA1 requires DB_GLSCYL_LVMVG GLS_DATA2 requires DB_GLSCYL_LVMVG GLS_ORACLE requires DB_GLSCYL_LVMVG lsnr_dbglscyl requires DB_GLSCYL_IP lsnr_dbglscyl requires ora_dbglscyl ora_dbglscyl requires GLS_ARCH ora_dbglscyl requires GLS_BACKUP ora_dbglscyl requires GLS_DATA1 ora_dbglscyl requires GLS_DATA2 ora_dbglscyl requires GLS_ORACLE // resource dependency tree // // group DB_GLSCYL // { // Netlsnr lsnr_dbglscyl // { // Oracle ora_dbglscyl // { // Mount GLS_BACKUP // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_ARCH // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_ORACLE // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_DATA2 // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_DATA1 // { // LVMVG DB_GLSCYL_LVMVG // } // } // IP DB_GLSCYL_IP // { // Proxy GLSCYL_PROXY // } // } // } ========================================================= Thanks, AllaboutunixSolved2.1KViews1like3CommentsPlan for DIMM replacement activity(VCS nodes)
Hi Team, We have to replace DIMM on the passive node in which VCS services is currently not running.The cross over LLTcables is badly hanged wih each other and Symantec engineer told us that he will manage the cable issue and no require to down the active node. Kindly guide step by step procedure for this activity,also please suggest the prerequisities before starting this activity. This is very very crucial activity as Class A application is running on the active node. Currently, SG is running in Sydneyserver.We have to perform activity on Madagascar server. -- SYSTEM STATE -- System State Frozen A Sydney RUNNING 0 A Madagascar RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService Sydney Y N ONLINE B ClusterService Madagascar Y N OFFLINE B ORA_SG_Group Sydney Y N ONLINE B ORA_SG_Group Madagascar Y N OFFLINE Kindly suggest as soon as possible. Thanks in advance.. AllaboutunixSolved2.1KViews1like6Comments