concurrency violation
Hi Team, This alert came in my environment in one of the AIX server 6.1 ofconcurrency violation. Subject: VCS SevereError for Service Group sapgtsprd, Service group concurrency violation Event Time: Wed Dec 3 06:46:54 EST 2014 Entity Name: sapgtsprd Entity Type: Service Group Entity Subtype: Failover Entity State: Service group concurrency violation Traps Origin: Veritas_Cluster_Server System Name: mapibm625 Entities Container Name: GTS_Prod Entities Container Type: VCS ================================================================================== engineA.log are, 2014/12/03 06:46:54 VCS INFO V-16-1-10299 Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) is online on mapibm625 (Not initiated by VCS) 2014/12/03 06:46:54 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group sapgtsprd 2014/12/03 06:46:54 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group sapgtsprd on all nodes 2014/12/03 06:46:55 VCS WARNING V-16-6-15034 (mapibm625) violation:-Offlining group sapgtsprd on system mapibm625 2014/12/03 06:46:55 VCS INFO V-16-1-50135 User root fired command: hagrp -offline sapgtsprd mapibm625 from localhost 2014/12/03 06:46:55 VCS NOTICE V-16-1-10167 Initiating manual offline of group sapgtsprd on system mapibm625 ====================================================================================================== What isConcurrency Violation in VCS? What are the steps we should have to take to resolve this?Kindly explain in detail. Thanks, AllaboutunixSolved2.8KViews2likes4CommentsService group shows online and cluster service offline
Hi TEam, I have an output of the hastatus which is attached, It shows ClusterService online in 35th system and rest of the system shows Clusterservice Offline however, Service group POWERCENTERSERVICEMANAGER shows online in all systems 35,36,37,38,39,40. I am unable to get it why the Service groupshows online in all the other systems when only 35th system ClusterService is online and rest others are offline. Do that the scenario of Active-Active Cluster? Please help me to understand this scenario.Solved2.2KViews1like2Commentsmnt_app resource failover
Hi, Below is the resource dependecies in my enviroment, what happened is somebody unmount the /app filesystem.So, resource went on faulted state, when i checked that resource criticality it shows mnt_app as non-critical resource 0 and vol_app and app_dg is set as critical 1 The depediencies below shows parent child relationship (mnt_app) as parent and (vol_app)as child. If parent is set as critical and child is non-critical 0, do it failover to another node. Or If child is set as critical or parent as non-critical 0, then it failover? Please assist as soon as possible. root@lyle# hares -dep |grep app PDM_PRD_MG APP_aphelion mnt_app PDM_PRD_MG APP_tibjmsd mnt_app PDM_PRD_MG Blind_check_stopDB mnt_app PDM_PRD_MG IPMultB_pdmprdappdb MNicB_DB PDM_PRD_MG appdg SRDF_app PDM_PRD_MG mnt_app vol_app PDM_PRD_MG vol_activelogs appdg PDM_PRD_MG vol_app appdg PDM_PRD_MG vol_archivelogs appdg PDM_PRD_MG vol_index appdgSolved1.7KViews1like5Commentsnode freeze v/s service group freeze
Hi Team, I came across a DIMM replacement activity in one of our solars servers which are in cluster. Please assist before proceeding for teh activity shall i have to freeze teh service group OR DO i need to freeze the node? ALso, kindly guide the scenario, when we should have toproceed for freezing the SG or freezing the node? Thanks..Solved6.8KViews1like1Comment
Oracle Data Base Replication with VVR under SFCFSHA/DR
Hi All; we are looking for whether VVR can use for the oracle database replication instead of oracle data guard solution. If it is used, do you know veritas gives support for any problem faced. even VVR keeps the write order fidelity, it is not certain the database integrity will ve preserved at the disaster site. do you have any best practices and white papers, experience, anything you suggest for this deploymeny?1.1KViews1like4CommentsPlan for DIMM replacement activity(VCS nodes)
Hi Team, We have to replace DIMM on the passive node in which VCS services is currently not running.The cross over LLTcables is badly hanged wih each other and Symantec engineer told us that he will manage the cable issue and no require to down the active node. Kindly guide step by step procedure for this activity,also please suggest the prerequisities before starting this activity. This is very very crucial activity as Class A application is running on the active node. Currently, SG is running in Sydneyserver.We have to perform activity on Madagascar server. -- SYSTEM STATE -- System State Frozen A Sydney RUNNING 0 A Madagascar RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService Sydney Y N ONLINE B ClusterService Madagascar Y N OFFLINE B ORA_SG_Group Sydney Y N ONLINE B ORA_SG_Group Madagascar Y N OFFLINE Kindly suggest as soon as possible. Thanks in advance.. AllaboutunixSolved2.1KViews1like6CommentsService group concurrency violation
Hi Team, We have alerts ofconcurrency violation, we have two servers in cluster mapibm625, mapibm626 Logs are, 2014/12/26 19:37:03 VCS INFO V-16-1-10299 Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) is online on mapibm625 (Not initiated by VCS) 2014/12/26 19:37:03 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group sapgtsprd 2014/12/26 19:37:03 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group sapgtsprd on all nodes 2014/12/26 19:37:04 VCS WARNING V-16-6-15034 (mapibm625) violation:Offlining group sapgtsprd on system mapibm625 2014/12/26 19:37:04 VCS INFO V-16-1-50135 User root fired command: hagrp -offline sapgtsprd mapibm625 from localhost 2014/12/26 19:37:04 VCS NOTICE V-16-1-10167 Initiating manual offline of group sapgtsprd on system mapibm625 2014/12/26 19:37:04 VCS NOTICE V-16-1-10300 Initiating Offline of Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) on System mapibm625 2014/12/26 19:37:04 VCS INFO V-16-6-15002 (mapibm625) hatrigger:hatrigger executed /opt/VRTSvcs/bin/internal_triggers/violation mapibm625 sapgtsprd successfully 2014/12/26 19:37:04 VCS INFO V-16-10011-306 (mapibm625) Application:App_saposcol:offline:Execution of Stop Program (/opt/VRTSvcs/bin/Saposcol/offline) returned (0). 2014/12/26 19:37:05 VCS INFO V-16-2-13716 (mapibm625) Resource(App_saposcol): Output of the completed operation (offline) ============================================== 2014/12/26 19:37:06 VCS INFO V-16-1-10305 Resource App_saposcol (Owner: Unspecified, Group: sapgtsprd) is offline on mapibm625 (VCS initiated) 2014/12/26 19:37:06 VCS NOTICE V-16-1-10446 Group sapgtsprd is offline on system mapibm625 ======================================================================================== I have asked the application team to look out as whether they are working on the servers because the resource is of SAP(Resource App_saposcol) However, application team has replied that they are not working on it and might theApp_saposcol is online on both of servers which causes the issue. Then, I have checked the status of resources in both the servers and it says, [root@mapibm626]: # hares -state #Resource Attribute System Value App_saposcol State mapibm625 OFFLINE App_saposcol State mapibm626 ONLINE [root@mapibm625]: # hares -state #Resource Attribute System Value App_saposcol State mapibm625 OFFLINE App_saposcol State mapibm626 ONLINE and also checked the current logs of the server however found only, 2014/12/27 13:03:42 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/27 17:03:43 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/27 21:03:44 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 01:03:45 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 05:03:46 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 09:03:47 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 10:56:14 VCS INFO V-16-1-50086 CPU usage on mapibm625 is 61% 2014/12/28 11:26:14 VCS INFO V-16-1-50086 CPU usage on mapibm625 is 61% 2014/12/28 13:03:48 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 14:26:14 VCS INFO V-16-1-50086 CPU usage on mapibm625 is 60% 2014/12/28 17:03:49 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/28 21:03:50 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 01:03:51 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 05:03:52 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 09:03:53 VCS INFO V-16-1-53504 VCS Engine Alive message!! 2014/12/29 13:03:55 VCS INFO V-16-1-53504 VCS Engine Alive message!! ========================================================================== Please assist what could be the possible reasons for this and in future how to avoid this? Thanks, AllaboutunixSolved2.6KViews1like7CommentsRoot cause for resource fault in vcs 6.0
Hi Team, We have a AIX 6.0 server in which vcs 6.0 is runningresource got faulted and now came online automatically. Submitting some details, We only know that there is something wrong with database listener configuration, however unable to find the root cause of it why resource faulted and came online automatically Can you please help to understand about this? Before: [root@cylibm004 /]# hastatus -sum -- SYSTEM STATE -- System State Frozen A cylibm003 RUNNING 0 A cylibm004 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService cylibm003 Y N ONLINE B ClusterService cylibm004 Y N OFFLINE B DB_GLSCENR5 cylibm003 Y N OFFLINE B DB_GLSCENR5 cylibm004 Y N PARTIAL B DB_GLSCYL cylibm003 Y N ONLINE B DB_GLSCYL cylibm004 Y N OFFLINE -- RESOURCES FAILED -- Group Type Resource System C DB_GLSCENR5 Netlsnr lsnr_glscyl cylibm004 After: [root@cylibm004 /]# hastatus -sum -- SYSTEM STATE -- System State Frozen A cylibm003 RUNNING 0 A cylibm004 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService cylibm003 Y N ONLINE B ClusterService cylibm004 Y N OFFLINE B DB_GLSCENR5 cylibm003 Y N OFFLINE B DB_GLSCENR5 cylibm004 Y N ONLINE B DB_GLSCYL cylibm003 Y N ONLINE B DB_GLSCYL cylibm004 Y N OFFLINE -----Original Message----- From: Notifier Sent: Friday, November 07, 2014 2:15 PM Subject: VCS Error for Resource lsnr_glscyl, Resource has faulted Event Time: Fri Nov 7 14:15:06 CST 2014 Entity Name: lsnr_glscyl Entity Type: Resource Entity Subtype: Netlsnr Entity State: Resource has faulted Traps Origin: Veritas_Cluster_Server System Name: cylibm004 Entities Container Name: DB_GLSCENR5 Entities Container Type: Service Group Entities Owner: unknown ============================================================================================================================== EngineA.log - 2014/11/07 14:14:54 VCS WARNING V-16-10011-8 (cylibm004) Netlsnr:lsnr_glscyl:LsnrTest.pl: File /oracle/.profile is not a valid text file 2014/11/07 14:14:55 VCS INFO V-16-20002-211 (cylibm004) Netlsnr:lsnr_glscyl:monitor:Monitor procedure /opt/VRTSagents/ha/bin/Netlsnr/LsnrTest.pl returned the output: Cannot get "LOGNAME" variable. 2014/11/07 14:14:55 VCS ERROR V-16-2-13067 (cylibm004) Agent is calling clean for resource(lsnr_glscyl) because the resource became OFFLINE unexpectedly, on its own. 2014/11/07 14:14:55 VCS NOTICE V-16-20002-42 (cylibm004) Netlsnr:lsnr_glscyl:clean:Listener(LISTENER_GLSCENR5) kill TERM 12845176 2014/11/07 14:15:06 VCS INFO V-16-2-13068 (cylibm004) Resource(lsnr_glscyl) - clean completed successfully. 2014/11/07 14:15:06 VCS WARNING V-16-20002-226 (cylibm004) Netlsnr:lsnr_glscyl:monitor:getargs for process tnslsnr failed with return code 0 2014/11/07 14:15:06 VCS INFO V-16-1-10307 Resource lsnr_glscyl (Owner: unknown, Group: DB_GLSCENR5) is offline on cylibm004 (Not initiated by VCS) 2014/11/07 14:15:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 65% 2014/11/07 14:18:10 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 64% 2014/11/07 14:21:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 66% 2014/11/07 14:25:08 VCS INFO V-16-1-10299 Resource lsnr_glscyl (Owner: unknown, Group: DB_GLSCENR5) is online on cylibm004 (Not initiated by VCS) 2014/11/07 14:25:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group DB_GLSCENR5 on all nodes 2014/11/07 14:25:08 VCS NOTICE V-16-1-10447 Group DB_GLSCENR5 is online on system cylibm004 2014/11/07 14:27:10 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 64% 2014/11/07 14:30:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 65% 2014/11/07 14:31:10 VCS NOTICE V-16-1-50086 CPU usage on cylibm004 is 70% =========================================================================================== ============================================================================================ Main.cf file - MountPoint = "/DB_GLSCENR5/oracle" BlockDevice = "/dev/GLSCENR5_Oracle" FSType = jfs2 FsckOpt = "-y" ) Mount GLSCENR5_data3 ( Critical = 0 MountPoint = "/DB_GLSCENR5/data3" BlockDevice = "/dev/GLSCENR5_data3" FSType = jfs2 FsckOpt = "-y" ) Netlsnr lsnr_glscyl ( Critical = 0 Owner = oracle Home = "/DB_GLSCENR5/oracle/product/11.2.0.3" TnsAdmin = "/var/opt/oracle" Listener = LISTENER_GLSCENR5 EnvFile = "/oracle/.profile" ) Oracle ora_glscyl ( Critical = 0 Sid = glscenr5 Owner = oracle Home = "/DB_GLSCENR5/oracle/product/11.2.0.3" EnvFile = "/oracle/.profile" ) Proxy DB_GLSCENR5_Proxy ( TargetResName = csgnic ) DB_GLSCENR5_IP requires DB_GLSCENR5_Proxy GLSCENR5_ARCH requires DB_GLS_CENR5_LVMVG GLSCENR5_BACKUP requires DB_GLS_CENR5_LVMVG GLSCENR5_DATA1 requires DB_GLS_CENR5_LVMVG GLSCENR5_DATA2 requires DB_GLS_CENR5_LVMVG GLSCENR5_ORACLE requires DB_GLS_CENR5_LVMVG GLSCENR5_data3 requires DB_GLS_CENR5_LVMVG lsnr_glscyl requires DB_GLSCENR5_IP lsnr_glscyl requires ora_glscyl ora_glscyl requires GLSCENR5_ARCH ora_glscyl requires GLSCENR5_BACKUP ora_glscyl requires GLSCENR5_DATA1 ora_glscyl requires GLSCENR5_DATA2 ora_glscyl requires GLSCENR5_ORACLE ora_glscyl requires GLSCENR5_data3 // resource dependency tree // // group DB_GLSCENR5 // { // Netlsnr lsnr_glscyl // { // Oracle ora_glscyl // { // Mount GLSCENR5_ORACLE // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_DATA2 // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_DATA1 // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_BACKUP // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_ARCH // { // LVMVG DB_GLS_CENR5_LVMVG // } // Mount GLSCENR5_data3 // { // LVMVG DB_GLS_CENR5_LVMVG // } // } // IP DB_GLSCENR5_IP // { // Proxy DB_GLSCENR5_Proxy // } // } // } group DB_GLSCYL ( SystemList = { cylibm003 = 0, cylibm004 = 1 } ) IP DB_GLSCYL_IP ( Critical = 0 Device = en2 Address = "132.189.249.119" NetMask = "255.255.255.128" ) LVMVG DB_GLSCYL_LVMVG ( VolumeGroup = DB_GLS_PRD MajorNumber @cylibm003 = 40 MajorNumber @cylibm004 = 40 ) Mount GLS_ARCH ( Critical = 0 MountPoint = "/DB_GLSCYL/arch" BlockDevice = "/dev/GLSCYL_Arch" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_BACKUP ( Critical = 0 MountPoint = "/DB_GLSCYL/backup" BlockDevice = "/dev/GLSCYL_Backup" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_DATA1 ( Critical = 0 MountPoint = "/DB_GLSCYL/data1" BlockDevice = "/dev/GLSCYL_Data1" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_DATA2 ( Critical = 0 MountPoint = "/DB_GLSCYL/data2" BlockDevice = "/dev/GLSCYL_Data2" FSType = jfs2 FsckOpt = "-y" ) Mount GLS_ORACLE ( Critical = 0 MountPoint = "/DB_GLSCYL/oracle" BlockDevice = "/dev/GLSCYL_Oracle" FSType = jfs2 FsckOpt = "-y" ) Netlsnr lsnr_dbglscyl ( Critical = 0 Owner = oracle Home = "/DB_GLSCYL/oracle/product/11.2.0.3" TnsAdmin = "/var/opt/oracle" Listener = LISTENER_GLSCYL EnvFile = "/oracle/.profile" ) Oracle ora_dbglscyl ( Critical = 0 Sid = glscyl Owner = oracle Home = "/DB_GLSCYL/oracle/product/11.2.0.3" EnvFile = "/oracle/.profile" ) Proxy GLSCYL_PROXY ( TargetResName = csgnic ) DB_GLSCYL_IP requires GLSCYL_PROXY GLS_ARCH requires DB_GLSCYL_LVMVG GLS_BACKUP requires DB_GLSCYL_LVMVG GLS_DATA1 requires DB_GLSCYL_LVMVG GLS_DATA2 requires DB_GLSCYL_LVMVG GLS_ORACLE requires DB_GLSCYL_LVMVG lsnr_dbglscyl requires DB_GLSCYL_IP lsnr_dbglscyl requires ora_dbglscyl ora_dbglscyl requires GLS_ARCH ora_dbglscyl requires GLS_BACKUP ora_dbglscyl requires GLS_DATA1 ora_dbglscyl requires GLS_DATA2 ora_dbglscyl requires GLS_ORACLE // resource dependency tree // // group DB_GLSCYL // { // Netlsnr lsnr_dbglscyl // { // Oracle ora_dbglscyl // { // Mount GLS_BACKUP // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_ARCH // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_ORACLE // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_DATA2 // { // LVMVG DB_GLSCYL_LVMVG // } // Mount GLS_DATA1 // { // LVMVG DB_GLSCYL_LVMVG // } // } // IP DB_GLSCYL_IP // { // Proxy GLSCYL_PROXY // } // } // } ========================================================= Thanks, AllaboutunixSolved2.1KViews1like3Comments
about the VCS behaviour on faulted resources
Hi Team, Because ManageFaults is ALL by default, when the resource faults, VCS calls Clean entry point. What is the task for Clean entry point for this resource? As far as I know, when VCS declares the resource as faulted, depending on CRITICAL attribute for resource and AutoFailOver, VCS fail sover the resource group. This resource group stays as faulted on the first node. So, I need to clean fault manually to be able to make the service group online on this node again. So,what the effect of "clean entry point" if the reosurce faults? If service group is faulted on both primary and secondary node, as far as I know, it will not be failed-over. It stays as faulted on both node until clearing it manually? Is there any way to automate the fail-over while the service group are faulted on both node? In fact I would like to understand the role of "clean entry point" for resource when it faults bcause I am always clearing the resource's fault manually? Is it there before declaring the resource is faulted? Please explain me.1.6KViews1like3Comments
Oracle Data Base Replication with VVR under SFCFSHA/DR
Hi All; we are looking for whether VVR can use for the oracle database replication instead of oracle data guard solution. If it is used, do you know veritas gives support for any problem faced. even VVR keeps the write order fidelity, it is not certain the database integrity will ve preserved at the disaster site. do you have any best practices and white papers, experience, anything you suggest for this deploymeny?about the VCS behaviour on faulted resources
Hi Team, Because ManageFaults is ALL by default, when the resource faults, VCS calls Clean entry point. What is the task for Clean entry point for this resource? As far as I know, when VCS declares the resource as faulted, depending on CRITICAL attribute for resource and AutoFailOver, VCS fail sover the resource group. This resource group stays as faulted on the first node. So, I need to clean fault manually to be able to make the service group online on this node again. So,what the effect of "clean entry point" if the reosurce faults? If service group is faulted on both primary and secondary node, as far as I know, it will not be failed-over. It stays as faulted on both node until clearing it manually? Is there any way to automate the fail-over while the service group are faulted on both node? In fact I would like to understand the role of "clean entry point" for resource when it faults bcause I am always clearing the resource's fault manually? Is it there before declaring the resource is faulted? Please explain me.