11-09-2014 10:54 AM
Hi Team,
We have a AIX 6.0 server in which vcs 6.0 is running resource got faulted and now came online automatically.
Submitting some details,
We only know that there is something wrong with database listener configuration, however unable to find the root cause of it why resource faulted and came online automatically
Can you please help to understand about this?
Before:
[root@cylibm004 /]# hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A cylibm003 RUNNING 0
A cylibm004 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B ClusterService cylibm003 Y N ONLINE
B ClusterService cylibm004 Y N OFFLINE
B DB_GLSCENR5 cylibm003 Y N OFFLINE
B DB_GLSCENR5 cylibm004 Y N PARTIAL
B DB_GLSCYL cylibm003 Y N ONLINE
B DB_GLSCYL cylibm004 Y N OFFLINE
-- RESOURCES FAILED
-- Group Type Resource System
C DB_GLSCENR5 Netlsnr lsnr_glscyl cylibm004
After:
[root@cylibm004 /]# hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A cylibm003 RUNNING 0
A cylibm004 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B ClusterService cylibm003 Y N ONLINE
B ClusterService cylibm004 Y N OFFLINE
B DB_GLSCENR5 cylibm003 Y N OFFLINE
B DB_GLSCENR5 cylibm004 Y N ONLINE
B DB_GLSCYL cylibm003 Y N ONLINE
B DB_GLSCYL cylibm004 Y N OFFLINE
-----Original Message-----
From: Notifier
Sent: Friday, November 07, 2014 2:15 PM
Subject: VCS Error for Resource lsnr_glscyl, Resource has faulted
Event Time: Fri Nov 7 14:15:06 CST 2014 Entity Name: lsnr_glscyl Entity Type: Resource Entity Subtype: Netlsnr Entity State: Resource has faulted Traps Origin: Veritas_Cluster_Server System Name: cylibm004 Entities Container Name: DB_GLSCENR5 Entities Container Type: Service Group Entities Owner: unknown
==============================================================================================================================
EngineA.log -
2014/11/07 14:14:54 VCS WARNING V-16-10011-8 (cylibm004) Netlsnr:lsnr_glscyl:LsnrTest.pl: File /oracle/.profile is not a valid text file
2014/11/07 14:14:55 VCS INFO V-16-20002-211 (cylibm004) Netlsnr:lsnr_glscyl:monitor:Monitor procedure /opt/VRTSagents/ha/bin/Netlsnr/LsnrTest.pl returned the output: Cannot get "LOGNAME" variable.
2014/11/07 14:14:55 VCS ERROR V-16-2-13067 (cylibm004) Agent is calling clean for resource(lsnr_glscyl) because the resource became OFFLINE unexpectedly, on its own.
2014/11/07 14:14:55 VCS NOTICE V-16-20002-42 (cylibm004) Netlsnr:lsnr_glscyl:clean:Listener(LISTENER_GLSCENR5) kill TERM 12845176
2014/11/07 14:15:06 VCS INFO V-16-2-13068 (cylibm004) Resource(lsnr_glscyl) - clean completed successfully.
2014/11/07 14:15:06 VCS WARNING V-16-20002-226 (cylibm004) Netlsnr:lsnr_glscyl:monitor:getargs for process tnslsnr failed with return code 0
2014/11/07 14:15:06 VCS INFO V-16-1-10307 Resource lsnr_glscyl (Owner: unknown, Group: DB_GLSCENR5) is offline on cylibm004 (Not initiated by VCS)
2014/11/07 14:15:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 65%
2014/11/07 14:18:10 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 64%
2014/11/07 14:21:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 66%
2014/11/07 14:25:08 VCS INFO V-16-1-10299 Resource lsnr_glscyl (Owner: unknown, Group: DB_GLSCENR5) is online on cylibm004 (Not initiated by VCS)
2014/11/07 14:25:08 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group DB_GLSCENR5 on all nodes
2014/11/07 14:25:08 VCS NOTICE V-16-1-10447 Group DB_GLSCENR5 is online on system cylibm004
2014/11/07 14:27:10 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 64%
2014/11/07 14:30:40 VCS INFO V-16-1-50086 CPU usage on cylibm004 is 65%
2014/11/07 14:31:10 VCS NOTICE V-16-1-50086 CPU usage on cylibm004 is 70%
===========================================================================================
============================================================================================
Main.cf file -
MountPoint = "/DB_GLSCENR5/oracle"
BlockDevice = "/dev/GLSCENR5_Oracle"
FSType = jfs2
FsckOpt = "-y"
)
Mount GLSCENR5_data3 (
Critical = 0
MountPoint = "/DB_GLSCENR5/data3"
BlockDevice = "/dev/GLSCENR5_data3"
FSType = jfs2
FsckOpt = "-y"
)
Netlsnr lsnr_glscyl (
Critical = 0
Owner = oracle
Home = "/DB_GLSCENR5/oracle/product/11.2.0.3"
TnsAdmin = "/var/opt/oracle"
Listener = LISTENER_GLSCENR5
EnvFile = "/oracle/.profile"
)
Oracle ora_glscyl (
Critical = 0
Sid = glscenr5
Owner = oracle
Home = "/DB_GLSCENR5/oracle/product/11.2.0.3"
EnvFile = "/oracle/.profile"
)
Proxy DB_GLSCENR5_Proxy (
TargetResName = csgnic
)
DB_GLSCENR5_IP requires DB_GLSCENR5_Proxy
GLSCENR5_ARCH requires DB_GLS_CENR5_LVMVG
GLSCENR5_BACKUP requires DB_GLS_CENR5_LVMVG
GLSCENR5_DATA1 requires DB_GLS_CENR5_LVMVG
GLSCENR5_DATA2 requires DB_GLS_CENR5_LVMVG
GLSCENR5_ORACLE requires DB_GLS_CENR5_LVMVG
GLSCENR5_data3 requires DB_GLS_CENR5_LVMVG
lsnr_glscyl requires DB_GLSCENR5_IP
lsnr_glscyl requires ora_glscyl
ora_glscyl requires GLSCENR5_ARCH
ora_glscyl requires GLSCENR5_BACKUP
ora_glscyl requires GLSCENR5_DATA1
ora_glscyl requires GLSCENR5_DATA2
ora_glscyl requires GLSCENR5_ORACLE
ora_glscyl requires GLSCENR5_data3
// resource dependency tree
//
// group DB_GLSCENR5
// {
// Netlsnr lsnr_glscyl
// {
// Oracle ora_glscyl
// {
// Mount GLSCENR5_ORACLE
// {
// LVMVG DB_GLS_CENR5_LVMVG
// }
// Mount GLSCENR5_DATA2
// {
// LVMVG DB_GLS_CENR5_LVMVG
// }
// Mount GLSCENR5_DATA1
// {
// LVMVG DB_GLS_CENR5_LVMVG
// }
// Mount GLSCENR5_BACKUP
// {
// LVMVG DB_GLS_CENR5_LVMVG
// }
// Mount GLSCENR5_ARCH
// {
// LVMVG DB_GLS_CENR5_LVMVG
// }
// Mount GLSCENR5_data3
// {
// LVMVG DB_GLS_CENR5_LVMVG
// }
// }
// IP DB_GLSCENR5_IP
// {
// Proxy DB_GLSCENR5_Proxy
// }
// }
// }
group DB_GLSCYL (
SystemList = { cylibm003 = 0, cylibm004 = 1 }
)
IP DB_GLSCYL_IP (
Critical = 0
Device = en2
Address = "132.189.249.119"
NetMask = "255.255.255.128"
)
LVMVG DB_GLSCYL_LVMVG (
VolumeGroup = DB_GLS_PRD
MajorNumber @cylibm003 = 40
MajorNumber @cylibm004 = 40
)
Mount GLS_ARCH (
Critical = 0
MountPoint = "/DB_GLSCYL/arch"
BlockDevice = "/dev/GLSCYL_Arch"
FSType = jfs2
FsckOpt = "-y"
)
Mount GLS_BACKUP (
Critical = 0
MountPoint = "/DB_GLSCYL/backup"
BlockDevice = "/dev/GLSCYL_Backup"
FSType = jfs2
FsckOpt = "-y"
)
Mount GLS_DATA1 (
Critical = 0
MountPoint = "/DB_GLSCYL/data1"
BlockDevice = "/dev/GLSCYL_Data1"
FSType = jfs2
FsckOpt = "-y"
)
Mount GLS_DATA2 (
Critical = 0
MountPoint = "/DB_GLSCYL/data2"
BlockDevice = "/dev/GLSCYL_Data2"
FSType = jfs2
FsckOpt = "-y"
)
Mount GLS_ORACLE (
Critical = 0
MountPoint = "/DB_GLSCYL/oracle"
BlockDevice = "/dev/GLSCYL_Oracle"
FSType = jfs2
FsckOpt = "-y"
)
Netlsnr lsnr_dbglscyl (
Critical = 0
Owner = oracle
Home = "/DB_GLSCYL/oracle/product/11.2.0.3"
TnsAdmin = "/var/opt/oracle"
Listener = LISTENER_GLSCYL
EnvFile = "/oracle/.profile"
)
Oracle ora_dbglscyl (
Critical = 0
Sid = glscyl
Owner = oracle
Home = "/DB_GLSCYL/oracle/product/11.2.0.3"
EnvFile = "/oracle/.profile"
)
Proxy GLSCYL_PROXY (
TargetResName = csgnic
)
DB_GLSCYL_IP requires GLSCYL_PROXY
GLS_ARCH requires DB_GLSCYL_LVMVG
GLS_BACKUP requires DB_GLSCYL_LVMVG
GLS_DATA1 requires DB_GLSCYL_LVMVG
GLS_DATA2 requires DB_GLSCYL_LVMVG
GLS_ORACLE requires DB_GLSCYL_LVMVG
lsnr_dbglscyl requires DB_GLSCYL_IP
lsnr_dbglscyl requires ora_dbglscyl
ora_dbglscyl requires GLS_ARCH
ora_dbglscyl requires GLS_BACKUP
ora_dbglscyl requires GLS_DATA1
ora_dbglscyl requires GLS_DATA2
ora_dbglscyl requires GLS_ORACLE
// resource dependency tree
//
// group DB_GLSCYL
// {
// Netlsnr lsnr_dbglscyl
// {
// Oracle ora_dbglscyl
// {
// Mount GLS_BACKUP
// {
// LVMVG DB_GLSCYL_LVMVG
// }
// Mount GLS_ARCH
// {
// LVMVG DB_GLSCYL_LVMVG
// }
// Mount GLS_ORACLE
// {
// LVMVG DB_GLSCYL_LVMVG
// }
// Mount GLS_DATA2
// {
// LVMVG DB_GLSCYL_LVMVG
// }
// Mount GLS_DATA1
// {
// LVMVG DB_GLSCYL_LVMVG
// }
// }
// IP DB_GLSCYL_IP
// {
// Proxy GLSCYL_PROXY
// }
// }
// }
=========================================================
Thanks,
Allaboutunix
Solved! Go to Solution.
11-14-2014 11:41 AM
If you have non-standard Orcale environments variables then put them in a separate environment file (from profile) owned by root - example create file /opt/VRTSvcs/bin/oraenv and put variables in there.
Mike
11-10-2014 12:19 PM
My guess would be the oracle user edited their profile as you see message:
VCS WARNING V-16-10011-8 (cylibm004) Netlsnr:lsnr_glscyl:LsnrTest.pl: File /oracle/.profile is not a valid text file
It is not a good idea to set the enviroment file to oracles profile. First off you should check if you need to specify env file in VCS as VCS sets all the necessary environments variables so you you only need to use if you are using some non-standard Oracle environments variables and in this case I would create a separate environment file owned by root.
Mike
11-11-2014 01:55 AM
Did you meant to say, if i would have non-standard Orcale environments variables, I should have to declare outside vcs? If yes, then how it should be declared?
11-14-2014 11:41 AM
If you have non-standard Orcale environments variables then put them in a separate environment file (from profile) owned by root - example create file /opt/VRTSvcs/bin/oraenv and put variables in there.
Mike