cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Veritas.com
Support

Cluster cannot detect my Oracle Apps resource is up

Sauskekan
Level 3

‎01-13-2015 04:56 AM

Hi,

i am trying to configure the Oracle Apps to be under cluster when i put all the required vairables required by the resource the cluster couldn't be able detect that the processes is online ,here under the variables:

Oracle Apps Version

12

User Name

applprod

Oracle Home

/binaries/apps/tech_st/10.1.2

Script Home

$INST_TOP/admin/scripts

 

SCRIPT NAME: adstrtal.sh

Server Type

Webserver

 Monitor

/export/home/applprod/monitor/orapps.prob 

Env

$INST_TOP/ora/10.1.2/ERPPROD_erpapp-lh.env

 

Any Idea about what shoud i do.

 

Regards,

10 REPLIES 10

Marianne
Level 6
Partner    VIP    Accredited Certified

‎01-13-2015 05:47 AM

Provide actual path instead of variables used in these attributes:

Script Home      $INST_TOP/admin/scripts

Env                  $INST_TOP/ora/10.1.2/ERPPROD_erpapp-lh.env

 

PS: What is VCS version in this environment?

Please also post main.cf entries for this Service Group.


Handy NetBackup Links
0 Kudos

mikebounds
Level 6
Partner Accredited

‎01-13-2015 05:50 AM

Have you used enviroment variables in VCS as you have listed above as I have never used environment variables in VCS attributes, so try using entering full path without using  enviroment variables.

EDIT: I see Marianne beat me to it !

If this stil doesn't wotk then could you provide extract from main.cf (in /etc/VRTSvcs/conf/config) for your resource.

 

Mike

0 Kudos

Sauskekan
Level 3

‎01-13-2015 07:03 AM

Sorry guys i forgot to save the configuration that's why it didn't appeared here it is :

 

group ERPAPP_SG (
        SystemList = { erp-app2-1 = 0, erp-app2-2 = 1 }
        )

        DiskGroup erpapp_bin_dg (
                DiskGroup = erpapp_dg
                )

        IP erpapp_lh (
                Device = ipmp0
                Address = "172.21.17.148 "
                NetMask = "255.255.252.0"
                )

        Mount erpapp_bin (
                MountPoint = "/binaries"
                BlockDevice = "/dev/vx/dsk/erpapp_dg/erpappbinvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        OracleApps ERP_Ora_Apps (
                User = applprod
                OracleAppsVersion = 12
                OracleHome = "/binaries/apps/tech_st/10.1.2 "
                ScriptHome = "/binaries/inst/apps/ERPPROD_erpapp-lh/admin/scripts "
                ServerType = WebServer
                VirtualHostname = "erpapp-lh.bh.zain.com"
                )

        OracleAppsCM ERP_Ora_CM (
                User = "applprod  "
                OracleHome = "/binaries/apps/tech_st/10.1.2 "
                ScriptHome = "/binaries/inst/apps/ERPPROD_erpapp-lh/admin/scripts "
                SQLDIR = "/apps/apps_st/appl/fnd/12.0.0/sql"
                EnvFile = "/binaries/apps/apps_st/appl/APPSERPPROD_erpapp-lh.env"
                DB_USER = apps
                DB_Password = apps
                VirtualHostname = "erpapp-lh.bh.zain.com"
                ProcMon = { FNDLIBR, FNDSM }
                ProcClean = { "FNDCPMBR " }
                )

        Proxy erpapp_lh-proxy (
                TargetResName = erpappnic
                )

        Volume erpapp_bin_vol (
                Volume = erpappbinvol
                DiskGroup = erpapp_dg
                )

        erpapp_bin requires erpapp_bin_vol
        erpapp_bin_vol requires erpapp_bin_dg
        erpapp_lh requires erpapp_lh-proxy


        // resource dependency tree
        //
        //      group ERPAPP_SG
        //      {
        //      OracleApps ERP_Ora_Apps
        //      OracleAppsCM ERP_Ora_CM
        //      Mount erpapp_bin
        //          {
        //          Volume erpapp_bin_vol
        //              {
        //              DiskGroup erpapp_bin_dg
        //              }
        //          }
        //      IP erpapp_lh
        //          {
        //          Proxy erpapp_lh-proxy
        //          }
        //      }

 

0 Kudos

mikebounds
Level 6
Partner Accredited

‎01-13-2015 11:41 AM

Have a look at the following logs in /var/VRTSvcs/log to see if it shows any errors:

engine_A.log

OracleApps_A.log

Mike

0 Kudos

Sauskekan
Level 3

‎01-14-2015 01:39 AM

Hi Mike,

 

This is what i see in the log file of the OracleApps :

015/01/13 16:22:46 VCS NOTICE V-16-55000-10282 Resource(ERP_Ora_Apps) - (erp-app2-1:monitor) VCSagentFW:logOutputFromCommand:Command line [/bin/env] produced exit code [0] and generated output [HZ=
VCSAG_RES_CONF_LEVEL=0
LC_MONETARY=
VCS_LOG_RESOURCE_NAME=ERP_Ora_Apps
TERM=xterm
SHELL=/usr/bin/bash
CLUSTER_LOGDBG=
VCSONE_LOG_RESOURCE_NAME=ERP_Ora_Apps
VCSONE_HOSTS=
LC_NUMERIC=
VAD_HOME=/opt/VRTSvad
VCS_LOG_AGENT_NAME=OracleApps
A__z=\"*SHLVL
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin/
MAIL=/var/mail/root
LC_MESSAGES=
_=/bin/env
LC_COLLATE=
PWD=/opt/VRTSagents/ha/bin/OracleApps
VCS_HOME=/opt/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCSAG_REG_VERSION=2
LANG=en_US.UTF-8
CLUSTER_HOME=/opt/VRTSvcs
CLUSTER_DIAG=/var/VRTSvcs/diag//agents/OracleApps
VCS_CONF=/etc/VRTSvcs
TZ=localtime
VCSONE_LOG_CATEGORY=55011
SHLVL=6
HOME=/root
CLUSTER_PRODUCT=VCS
VCS_LOG_SCRIPT_NAME=monitor
VCS_LOG=/var/VRTSvcs
LOGNAME=root
LC_CTYPE=
VCSAG_MONITOR_LEVELS=1
VCS_AGFW=1
VCSAG_PIPE_FD=
LC_TIME=
VCS_LOG_CATEGORY=55011]
-bash-4.1# tail -100f  OracleApps_A.log
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21630] PPID [21531] COMMAND [python2.6 /usr/lib/system-config-printer/applet.py]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21632] PPID [21531] COMMAND [/usr/bin/python2.6 /usr/lib/updatemanagernotifier]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21635] PPID [21531] COMMAND [java -Djava.security.policy=/usr/share/vpanels/java.policy com.oracle.solaris.vp.client.swing.App sysmon]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21636] PPID [21531] COMMAND [gnome-power-manager]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21638] PPID [21531] COMMAND [/usr/bin/xscreensaver -nosplash]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21639] PPID [21635] COMMAND [ <defunct>]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21641] PPID [21531] COMMAND [python2.6 /usr/lib/time-slider-notify]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21642] PPID [21531] COMMAND [gnome-volume-control-applet]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21647] PPID [1] COMMAND [/usr/lib/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oaf-ior-]]
2015/01/13 16:22:46 VCS NOTICE V-16-55000-10287 Resource(ERP_Ora_Apps) - (erp-app2-1:monitor) VCSagentFW:messageEngineLog:[ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21649] PPID [1] COMMAND [/usr/lib/trashapplet --oaf-activate-iid=OAFIID:GNOME_Panel_TrashApplet_Factory ]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21653] PPID [1] COMMAND [/usr/lib/gvfsd-trash --spawner :1.9 /org/gtk/gvfs/exec_spaw/0]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21666] PPID [1] COMMAND [/usr/lib/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21668] PPID [1] COMMAND [/usr/lib/notification-area-applet --oaf-activate-iid=OAFIID:GNOME_NotificationA]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21671] PPID [1] COMMAND [/usr/lib/gvfsd-metadata]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21673] PPID [210] COMMAND [/usr/lib/rad/rad -m /usr/lib/rad/transport -m /usr/lib/rad/protocol -m /usr/lib/rad/module -m /usr/lib/rad/site-modules -t pipe:fd=3,exit -e 180]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21675] PPID [1] COMMAND [/usr/lib/notification-daemon]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21677] PPID [1] COMMAND [/usr/bin/gnome-terminal -x /bin/sh -c cd '/root' && exec $SHELL]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21678] PPID [21677] COMMAND [gnome-pty-helper]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21787] PPID [1] COMMAND [/opt/VRTSvcs/bin/Script50Agent -type OracleApps -agdir /opt/VRTSagents/ha/bin/OracleApps]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [25878] PPID [21787] COMMAND [/opt/VRTSperl/bin/perl -Sw /opt/VRTSagents/ha/bin/OracleApps/monitor ERP_Ora_Apps ResLogLevel 1 INFO State 1 0 IState 1 1 User 1 applprod OracleAppsVersion 1 12 OracleHome 1 /binaries/apps/tech_st/10.1.2  ScriptHome 1 /binaries/inst/apps/ERPPROD_erpapp-lh/admin/scripts  EnvFile 1  ServerType 1 WebServer Port 1  DBConnectString 1  SecondLevelMonitor 1 0 MonitorProgram 1  VirtualHostname 1 erpapp-lh.bh.zain.com AEPTimeout 1 60]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [25881] PPID [25878] COMMAND [/usr/ucb/ps -axwwl]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1643] PPID [11] COMMAND [/usr/sbin/ttymon -g -d /dev/console -l console -T vt100 -m ldterm,ttcompat -h -p erp-app2-1 console login:]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [11781] PPID [11780] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [11786] PPID [11781] COMMAND [su -]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [11787] PPID [11786] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21346] PPID [11787] COMMAND [su - applprod]
ffdc:10044:Proc:GetLongProcessListHash:UID [100] PID [21347] PPID [21346] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [17033] PPID [17032] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [25651] PPID [17033] COMMAND [tail -f engine_A.log]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [21679] PPID [21677] COMMAND [/usr/bin/bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [100] PID [12254] PPID [12253] COMMAND [-bash]]
2015/01/13 16:22:46 VCS NOTICE V-16-55000-10288 Resource(ERP_Ora_Apps) - (erp-app2-1:monitor) VCSagentFW:messageEngineLog:[
ffdc:10044:Proc:GetLongProcessListHash:UID [100] PID [12259] PPID [12254] COMMAND [bash]
ffdc:10046:Proc:GetLongProcessListHash:Got [134] processes
ffdc:10050:Proc:FindUidPids:Subroutine <Proc::FindUidPids> called with:
  UID           [100]
ffdc:10052:Proc:FindUidPids:Matched UID [100]
ffdc:10052:Proc:FindUidPids:Matched UID [100]
ffdc:10052:Proc:FindUidPids:Matched UID [100]
ffdc:10052:Proc:FindUidPids:Matched UID [100]
ffdc:10053:Proc:FindUidPids:Got [4] matches
ffdc:20057:<OracleApps::GetProcState> : Number of processes for User applprod = 4
ffdc:20058:<OracleApps::GetProcState> : Filter [^.*/binaries/apps/tech_st/10.1.2 /Apache/Apache/bin/httpd*\b] for Server [webserver]
ffdc:10047:Proc:FilterProcs:Subroutine <Proc::FilterProcs> called with:
  Filter          [^.*/binaries/apps/tech_st/10.1.2 /Apache/Apache/bin/httpd*\b]
ffdc:10049:Proc:FilterProcs:Got [0] matches
ffdc:20059:<OracleApps::GetProcState> : Did not find process [httpd]
ffdc:20058:<OracleApps::GetProcState> : Filter [^.*/binaries/apps/tech_st/10.1.2 /Apache/Apache/bin/rotatelogs*\b] for Server [webserver]
ffdc:10047:Proc:FilterProcs:Subroutine <Proc::FilterProcs> called with:
  Filter          [^.*/binaries/apps/tech_st/10.1.2 /Apache/Apache/bin/rotatelogs*\b]
ffdc:10049:Proc:FilterProcs:Got [0] matches
ffdc:20059:<OracleApps::GetProcState> : Did not find process [rotatelogs]
ffdc:20061:<OracleApps::GetProcState> : Did not find any processes for [webserver]
ffdc:20053:<OracleApps::GetResourceState> did not find all required processes, returning [100]
ffdc:20054:<OracleApps::GetResourceState> returning [100]
ffdc:20103:<main::MonitorEntryPoint> Resource [ERP_Ora_Apps] NOT online with state [100] after first level check.
ffdc:10271:VCSagentFW:entryPointExit:Subroutine VCSagentFW::entryPointExit called with: Exit code [100]
ffdc:10272:VCSagentFW:entryPointExit:Exiting entry point [monitor] with exit code [100]
ffdc:10274:VCSagentFW:entryPointExit:Generating FFDC messages for resource [ERP_Ora_Apps]
]

your help is really apprciated

0 Kudos

Marianne
Level 6
Partner    VIP    Accredited Certified

‎01-14-2015 01:57 AM

Can you check/verify these processes (as per the log)? 

ffdc:20059:<OracleApps::GetProcState> : Did not find process [rotatelogs]
ffdc:20061:<OracleApps::GetProcState> : Did not find any processes for [webserver]
ffdc:20053:<OracleApps::GetResourceState> did not find all required processes, returning [100]


Handy NetBackup Links
0 Kudos

Sauskekan
Level 3

‎01-14-2015 02:28 AM

Hi Marine those processes are already exist and the scripts to start them also there but i am suspecting the agent itself because i imported manually , can you suggest a way to check if the agent import was successfully.

0 Kudos

mikebounds
Level 6
Partner Accredited

‎01-14-2015 05:31 AM

Can you provide an extract from ps -ef showing the rotatelogs and webserver processes.

 

Mike

0 Kudos

Sauskekan
Level 3

‎01-14-2015 08:31 AM

Gus this is the last update when i tried to start the Oracle concurrent manager with the cluster ,


2015/01/14 19:29:53 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ERPAPP_SG on all nodes
2015/01/14 19:29:53 VCS NOTICE V-16-1-10301 Initiating Online of Resource ERP_Ora_CM (Owner: Unspecified, Group: ERPAPP_SG) on System erp-app2-1
2015/01/14 19:29:53 VCS ERROR V-16-55010-20008 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:online:<OracleAppsCM::ArgsValid> MonitorProgram [/export/home/applprod/monitor/oracm.prob] not executable.
                                Check the full pathname and the permissions
2015/01/14 19:29:53 VCS WARNING V-16-55010-20009 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:online:<OracleAppsCM::ArgsValid> Some of the attributes could not be validated. Returning [0]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10274 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:entryPointExit:Generating FFDC messages for resource [ERP_Ora_CM]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10287 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:messageEngineLog:[ffdc:10138:Arch:GetOSType:Subroutine Arch::GetOSType is called
ffdc:10262:VCSagentFW:SetupEntry:Subroutine <VCSagentFW::SetupEntry> called with:
  Entry Point Name         [monitor]
  Category ID              [55010]
ffdc:10263:VCSagentFW:SetupLogging:Subroutine <VCSagentFW::SetupLogging> called with:
  Clean Reason      [FFDC]
  Log Level         [TRACE]
ffdc:10346:LogInt:SetDebugLevel:Information, Set Logging Level To [TRACE]
ffdc:10267:VCSagentFW:SetupLogging:[monitor] Entered by resource instance [ERP_Ora_CM]
ffdc:20092:<monitor> VCSagentFW::SetupEntry returned successfully.
ffdc:20093:<monitor> called with
                         State               [0]
                         Istate              [1]
                         User                [applprod]
                         OracleHome          [/binaries/apps/tech_st/10.1.2]
                         ScriptHome          [/binaries/inst/apps/ERPPROD_erpapp-lh/admin/scripts]
                         SQLDIR              [/binaries/apps/apps_st/appl/fnd/12.0.0/sql]
                         Envfile             [/binaries/apps/apps_st/appl/APPSERPPROD_erpapp-lh.env]
                         DB_USER             [apps]
                         DB_Password         [apps]
                         VirtualHostname     [erpapp-lh]
                         SecondLevelMonitor  [5]
                         MonitorProgram      [/export/home/applprod/monitor/oracm.prob]
ffdc:20094:ProcMon processes : FNDLIBR FNDSM
ffdc:20095:ProcClean processes : FNDLIBR FNDLIBR FNDSM
ffdc:20001:Subroutine <OracleAppsCM::ArgsValid> called with:
  EntryPointName           [monitor]
  SecondLevelMonitor       [5]
  User                     [applprod]
  OracleHome               [/binaries/apps/tech_st/10.1.2]
  ScriptHome               [/binaries/inst/apps/ERPPROD_erpapp-lh/admin/scripts]
  SQLDIR                   [/binaries/apps/apps_st/appl/fnd/12.0.0/sql]
  EnvFile                  [/binaries/apps/apps_st/appl/APPSERPPROD_erpapp-lh.env]
  DB_USER                  [apps]
  DB_PASSWD                [apps]
  MonitorProgram           [/export/home/applprod/monitor/oracm.prob]
  VirtualHostname          [erpapp-lh]
ffdc:20096:<monitor> All arguments validated successfully!
ffdc:20099:Subroutine <main::MonitorEntryPoint> called
ffdc:20125:Subroutine <OracleAppsCM::GetEPTimeout> called with:
  EntryPointName [monitor]
  ResourceType   [OracleAppsCM]
ffdc:20126:<OracleAppsCM::GetEPTimeout> Using AEPTimeout [60] as EP timeout value
ffdc:20030:Subroutine <OracleAppsCM::GetResourceState> called:
  User [applprod] VirtualHost [erpapp-lh]
ffdc:10018:Comms:IsIPLocal:Subroutine <Comms::IsIPLocal> called with:
  Host           [erpapp-lh]
ffdc:10019:Comms:IsIPLocal:<Comms::IsIPLocal> Resolved [erpapp-lh] to Host [erpapp-lh] IPv4 Address [172.21.17.148]
ffdc:10022:Comms:IsIPLocal:<Comms::IsIPLocal> IP Address [172.21.17.148] is plumbed locally
ffdc:20031:<OracleAppsCM::GetResourceState> plumbed locally [erpapp-lh]
ffdc:20036:Subroutine <OracleAppsCM::GetProcState> called:]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10287 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:messageEngineLog:[  User                     [applprod]
ffdc:10040:Proc:GetLongProcessListHash:Subroutine <Proc::GetLongProcessListHash> called with:
   rlhrProcs        [ARRAY(0x78f400)]
   EnvFlag          [0]
ffdc:10138:Arch:GetOSType:Subroutine Arch::GetOSType is called
ffdc:10042:Proc:GetLongProcessListHash:Calling ps command [/usr/ucb/ps] with options [axwwl]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [0] PPID [0] COMMAND [sched]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1] PPID [0] COMMAND [/usr/sbin/init]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [2] PPID [0] COMMAND [pageout]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [3] PPID [0] COMMAND [fsflush]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [5] PPID [0] COMMAND [zpool-rpool]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [6] PPID [0] COMMAND [kmem_task]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [7] PPID [0] COMMAND [intrd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [8] PPID [0] COMMAND [vmtasks]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [11] PPID [1] COMMAND [/lib/svc/bin/svc.startd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [13] PPID [1] COMMAND [/lib/svc/bin/svc.configd]
ffdc:10044:Proc:GetLongProcessListHash:UID [17] PID [46] PPID [1] COMMAND [/lib/inet/netcfgd]
ffdc:10044:Proc:GetLongProcessListHash:UID [15] PID [67] PPID [1] COMMAND [/usr/sbin/dlmgmtd]
ffdc:10044:Proc:GetLongProcessListHash:UID [1] PID [87] PPID [1] COMMAND [/lib/crypto/kcfd]
ffdc:10044:Proc:GetLongProcessListHash:UID [16] PID [106] PPID [1] COMMAND [/lib/inet/ipmgmtd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [115] PPID [1] COMMAND [/lib/inet/in.mpathd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [127] PPID [1] COMMAND [/usr/lib/pfexecd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [153] PPID [1] COMMAND [/usr/lib/sysevent/syseventd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [183] PPID [1] COMMAND [/usr/lib/utmpd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [184] PPID [1] COMMAND [/usr/lib/rad/rad -sp]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [187] PPID [1] COMMAND [/usr/lib/ldoms/drd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [193] PPID [1] COMMAND [/usr/lib/ldoms/ldmad]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [213] PPID [1] COMMAND [/usr/lib/zones/zonestatd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [273] PPID [1] COMMAND [/usr/lib/devfsadm/devfsadmd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [326] PPID [1] COMMAND [/usr/lib/dbus-daemon --system]
ffdc:10044:Proc:GetLongProcessListHash:UID [16] PID [635] PPID [1] COMMAND [/lib/inet/nwamd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [690] PPID [1] COMMAND [/usr/lib/picl/picld]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [945] PPID [1] COMMAND [/lib/svc/method/iscsid]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [954] PPID [1] COMMAND [/usr/sbin/nscd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [958] PPID [1] COMMAND [/usr/lib/inet/ntpd -p /var/run/ntp.pid -g]]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10287 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:messageEngineLog:[ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [985] PPID [1] COMMAND [/usr/lib/efcode/sparcv9/efdaemon]
ffdc:10044:Proc:GetLongProcessListHash:UID [1] PID [1004] PPID [1] COMMAND [/usr/sbin/rpcbind]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1012] PPID [1] COMMAND [/usr/lib/inet/in.ndpd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1535] PPID [1] COMMAND [/usr/lib/hotplugd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1551] PPID [1] COMMAND [/usr/lib/inet/inetd start]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1554] PPID [1] COMMAND [/usr/sbin/cron]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1557] PPID [1] COMMAND [/usr/lib/autofs/automountd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1559] PPID [1557] COMMAND [/usr/lib/autofs/automountd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1593] PPID [1] COMMAND [/usr/lib/hal/hald --daemon=yes]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1595] PPID [1] COMMAND [/usr/sbin/console-kit-daemon]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1596] PPID [1593] COMMAND [hald-runner]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1598] PPID [1] COMMAND [/usr/lib/ssh/sshd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1601] PPID [1] COMMAND [/usr/lib/fm/fmd/fmd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1615] PPID [1] COMMAND [/usr/sbin/syslogd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1631] PPID [1] COMMAND [/usr/sbin/auditd]
ffdc:10044:Proc:GetLongProcessListHash:UID [25] PID [1634] PPID [1] COMMAND [sendmail: Queue runner@00:15:00 for /var/spool/clientmqueue]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1637] PPID [1] COMMAND [sendmail: accepting connections]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1673] PPID [1596] COMMAND [/usr/lib/hal/hald-addon-cpufreq]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1682] PPID [1] COMMAND [/usr/lib/rmvolmgr -s]
ffdc:10044:Proc:GetLongProcessListHash:UID [21] PID [1754] PPID [1] COMMAND [proftpd: (accepting connections)]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1799] PPID [1] COMMAND [/usr/lib/devchassis/devchassisd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [1995] PPID [1] COMMAND [/usr/sbin/gdm-binary]
ffdc:10044:Proc:GetLongProcessListHash:UID [60002] PID [1998] PPID [1] COMMAND [/usr/lib/fm/notify/smtp-notify]
ffdc:10044:Proc:GetLongProcessListHash:UID [60002] PID [2118] PPID [1] COMMAND [/usr/lib/fm/notify/asr-notify]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [2180] PPID [1] COMMAND [/opt/VRTSdbed/bin/vxdbd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [2215] PPID [1] COMMAND [/opt/VRTSgab/gablogd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [2709] PPID [1] COMMAND [/usr/sbin/cupsd -C /etc/cups/cupsd.conf]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [2734] PPID [1] COMMAND [/opt/VRTSsfmh/bin/xprtld -X 1 /etc/opt/VRTSsfmh/xprtld.conf]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [2767] PPID [1] COMMAND [/opt/VRTSvcs/bin/CmdServer]]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10287 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:messageEngineLog:[ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [3166] PPID [1] COMMAND [/usr/lib/ocm/ccr/bin/nmz]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [3728] PPID [1] COMMAND [vxconfigd -k]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [4757] PPID [1598] COMMAND [/usr/lib/ssh/sshd]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [4758] PPID [4757] COMMAND [/usr/lib/ssh/sshd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [5770] PPID [1598] COMMAND [/usr/lib/ssh/sshd]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [5771] PPID [5770] COMMAND [/usr/lib/ssh/sshd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [6046] PPID [1] COMMAND [/opt/VRTSamf/bin/imfd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [6059] PPID [6046] COMMAND [/usr/sbin/vxnotify -ifc]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [10509] PPID [1598] COMMAND [/usr/lib/ssh/sshd]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [10510] PPID [10509] COMMAND [/usr/lib/ssh/sshd]
ffdc:10044:Proc:GetLongProcessListHash:UID [1] PID [17881] PPID [1] COMMAND [/usr/lib/nfs/statd]
ffdc:10044:Proc:GetLongProcessListHash:UID [1] PID [17884] PPID [1] COMMAND [/usr/lib/nfs/nfs4cbd]
ffdc:10044:Proc:GetLongProcessListHash:UID [1] PID [17885] PPID [1] COMMAND [/usr/lib/nfs/nfsmapid]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [17887] PPID [0] COMMAND [nfs4cbd_kproc]
ffdc:10044:Proc:GetLongProcessListHash:UID [1] PID [17888] PPID [1] COMMAND [/usr/lib/nfs/lockd]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [17889] PPID [0] COMMAND [lockd_kproc]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18283] PPID [1] COMMAND [/opt/VRTSvcs/bin/had]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18285] PPID [1] COMMAND [/opt/VRTSvcs/bin/hashadow]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18304] PPID [1] COMMAND [/opt/VRTSvcs/bin/CoordPoint/CoordPointAgent -type CoordPoint]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18307] PPID [1] COMMAND [/opt/VRTSvcs/bin/DiskGroup/DiskGroupAgent -type DiskGroup]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18309] PPID [1] COMMAND [/opt/VRTSvcs/bin/IP/IPAgent -type IP]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18310] PPID [1] COMMAND [/opt/VRTSvcs/bin/Mount/MountAgent -type Mount]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18312] PPID [1] COMMAND [/opt/VRTSvcs/bin/NIC/NICAgent -type NIC]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18314] PPID [1] COMMAND [/opt/VRTSvcs/bin/Phantom/PhantomAgent -type Phantom]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18315] PPID [1] COMMAND [/opt/VRTSvcs/bin/Proxy/ProxyAgent -type Proxy]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18316] PPID [1] COMMAND [/opt/VRTSvcs/bin/Volume/VolumeAgent -type Volume]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [18318] PPID [1] COMMAND [/opt/VRTSvcs/bin/HostMonitor -type HostMonitor -agdir /]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [22917] PPID [1] COMMAND [/opt/VRTSvcs/bin/Script50Agent -type OracleAppsCM -agdir /opt/VRTSagents/ha/bin/OracleAppsCM]]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10287 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:messageEngineLog:[ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [26851] PPID [18307] COMMAND [/opt/VRTSvcs/bin/DiskGroup/imf_getnotification DiskGroup]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [27695] PPID [22917] COMMAND [/opt/VRTSperl/bin/perl -Sw /opt/VRTSagents/ha/bin/OracleAppsCM/monitor ERP_Ora_CM ResLogLevel 1 TRACE State 1 0 IState 1 1 User 1 applprod OracleHome 1 /binaries/apps/tech_st/10.1.2 ScriptHome 1 /binaries/inst/apps/ERPPROD_erpapp-lh/admin/scripts SQLDIR 1 /binaries/apps/apps_st/appl/fnd/12.0.0/sql EnvFile 1 /binaries/apps/apps_st/appl/APPSERPPROD_erpapp-lh.env DB_USER 1 apps DB_Password 1 apps VirtualHostname 1 erpapp-lh SecondLevelMonitor 1 5 MonitorProgram 1 /export/home/applprod/monitor/oracm.prob ProcMon 2 FNDLIBR FNDSM ProcClean 1 FNDLIBR AEPTimeout 1 60]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [27698] PPID [27695] COMMAND [/usr/ucb/ps -axwwl]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [5709] PPID [11] COMMAND [/usr/sbin/ttymon -g -d /dev/console -l console -T vt100 -m ldterm,ttcompat -h -p erp-app2-1 console login:]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [4759] PPID [4758] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [4764] PPID [4759] COMMAND [su -]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [4765] PPID [4764] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [27012] PPID [4765] COMMAND [tail -100f OracleAppsCM_A.log]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [5772] PPID [5771] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [5786] PPID [5772] COMMAND [su -]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [5787] PPID [5786] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [0] PID [23358] PPID [5787] COMMAND [su - applprod]
ffdc:10044:Proc:GetLongProcessListHash:UID [100] PID [23359] PPID [23358] COMMAND [-bash]
ffdc:10044:Proc:GetLongProcessListHash:UID [100] PID [27189] PPID [23359] COMMAND [tail -10f engine_A.log]
ffdc:10044:Proc:GetLongProcessListHash:UID [3000] PID [10511] PPID [10510] COMMAND [-bash]
ffdc:10046:Proc:GetLongProcessListHash:Got [101] processes
ffdc:10050:Proc:FindUidPids:Subroutine <Proc::FindUidPids> called with:
  UID           [100]
ffdc:10052:Proc:FindUidPids:Matched UID [100]
ffdc:10052:Proc:FindUidPids:Matched UID [100]
ffdc:10053:Proc:FindUidPids:Got [2] matches
ffdc:20038:<OracleAppsCM::GetProcState> : Number of processes for User applprod = 2
ffdc:20040:<OracleAppsCM::GetProcState> : Process Filter set to [^FNDLIBR]
ffdc:10047:Proc:FilterProcs:Subroutine <Proc::FilterProcs> called with:
  Filter          [^FNDLIBR]
ffdc:10049:Proc:FilterProcs:Got [0] matches
ffdc:20041:<OracleAppsCM::GetProcState> : Process [FNDLIBR] is not running
ffdc:20040:<OracleAppsCM::GetProcState> : Process Filter set to [^FNDSM]
ffdc:10047:Proc:FilterProcs:Subroutine <Proc::FilterProcs> called with:
  Filter          [^FNDSM]
ffdc:10049:Proc:FilterProcs:Got [0] matches
ffdc:20041:<OracleAppsCM::GetProcState> : Process [FNDSM] is not running]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10288 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:messageEngineLog:[
ffdc:20034:<OracleAppsCM::GetResourceState> did not find all required processes, returning [100]
ffdc:20035:<OracleAppsCM::GetResourceState> returning [100]
ffdc:20102:<main::MonitorEntryPoint> Resource [ERP_Ora_CM] NOT online with state [100] after first level check.
ffdc:10271:VCSagentFW:entryPointExit:Subroutine VCSagentFW::entryPointExit called with: Exit code [100]
ffdc:10272:VCSagentFW:entryPointExit:Exiting entry point [monitor] with exit code [100]
ffdc:10274:VCSagentFW:entryPointExit:Generating FFDC messages for resource [ERP_Ora_CM]
]
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10280 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:logOutputFromCommand:Command line [/bin/env] exited with code [0] , truncating output to approximately [4096] characters [NLS_SORT=BINARY
APPLREP=reports
CE_TOP=/binaries/apps/apps_st/appl/ce/12.0.0
RW=/binaries/apps/tech_st/10.1.2/reports
XLA_TOP=/binaries/apps/apps_st/appl/xla/12.0.0
LNS_TOP=/binaries/apps/apps_st/appl/lns/12.0.0
IGI_TOP=/binaries/apps/apps_st/appl/igi/12.0.0
CUI_TOP=/binaries/apps/apps_st/appl/cui/12.0.0
AHL_TOP=/binaries/apps/apps_st/appl/ahl/12.0.0
ADJREOPTS=-ms128m -mx256m
CA_FILEIO_64=YES
PCCINC=. include=/include include=/precomp/public
PRP_TOP=/binaries/apps/apps_st/appl/prp/12.0.0
PFT_TOP=/binaries/apps/apps_st/appl/pft/12.0.0
HZ=
OA_DOC=/binaries/apps/apps_st/comn/doc
PJI_TOP=/binaries/apps/apps_st/appl/pji/12.0.0
IGC_TOP=/binaries/apps/apps_st/appl/igc/12.0.0
FPA_TOP=/binaries/apps/apps_st/appl/fpa/12.0.0
EGO_TOP=/binaries/apps/apps_st/appl/ego/12.0.0
ASG_TOP=/binaries/apps/apps_st/appl/asg/12.0.0
APPL_TOP=/binaries/apps/apps_st/appl
AFJSMARG=-server -Xmx384m -XX:NewRatio=2 -XX:+UseSerialGC -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc
ORACLE_CONFIG_HOME=/binaries/inst/apps/ERPPROD_erpapp-lh/ora/10.1.2
VCSAG_RES_CONF_LEVEL=0
LC_MONETARY=
APPLLIB=lib
MST_TOP=/binaries/apps/apps_st/appl/mst/12.0.0
IRC_TOP=/binaries/apps/apps_st/appl/irc/12.0.0
VCS_LOG_RESOURCE_NAME=ERP_Ora_CM
APPLSQL=sql
INV_TOP=/binaries/apps/apps_st/appl/inv/12.0.0
IEC_TOP=/binaries/apps/apps_st/appl/iec/12.0.0
AMS_TOP=/binaries/apps/apps_st/appl/ams/12.0.0
APPLORB=/binaries/inst/apps/ERPPROD_erpapp-lh/ora/10.1.2/bin/appsrwrun.sh
XNP_TOP=/binaries/apps/apps_st/appl/xnp/12.0.0
TERM=xterm
SHELL=/usr/bin/bash
CLUSTER_LOGDBG=
FORMS_DISABLE_UNPAD_LOV=TRUE
APPLORC=/binaries/inst/apps/ERPPROD_erpapp-lh/ora/10.1.2/bin/appsrwrun.sh
QP_TOP=/binaries/apps/apps_st/appl/qp/12.0.0
JTM_TOP=/binaries/apps/apps_st/appl/jtm/12.0.0
IGW_TOP=/binaries/apps/apps_st/appl/igw/12.0.0
EDR_TOP=/binaries/apps/apps_st/appl/edr/12.0.0
CSM_TOP=/binaries/apps/apps_st/appl/csm/12.0.0
GWYUID=APPLSYSPUB/PUB
NLS_LANG=American_America.AL32UTF8
ENG_TOP=/binaries/apps/apps_st/appl/eng/12.0.0
DPP_TOP=/binaries/apps/apps_st/appl/dpp/12.0.0
VCSONE_LOG_RESOURCE_NAME=ERP_Ora_CM
VCSONE_HOSTS=
PY_LIB_PATH=/binaries/apps/tech_st/10.1.3/appsutil/jdk/jre/lib/sparc/client:/binaries/apps/tech_st/10.1.3/appsutil/jdk/jre/lib/sparc:/binaries/apps/apps_st/appl/pay/12.0.0/vendor/quantum/lib:/binaries/apps/tech_st/10.1.2/jdk/jre/lib/sparc:/binaries/apps/tech_st/10.1.2/jdk/jre/lib/sparc/native_threads:/binaries/apps/apps_st/appl/cz/12.0.0/bin:/binaries/apps/tech_st/10.1.2/jdk/jre/lib/sparc:/binaries/apps/tech_st/10.1.2/jdk/jre/lib/sparc/native_threads:/binaries/apps/apps_st/appl/cz/12.0.0/bin:/binaries/apps/tech_st/10.1.2/lib32:/binaries/apps/tech_st/10.1.2/lib:/usr/dt/lib:/usr/openwin/lib:/binaries/apps/tech_st/10.1.2/jdk/jre/lib/sparc:/binaries/apps/tech_st/10.1.2/jdk/jre/lib/sparc/native_threads:/binaries/apps/apps_st/appl/sht/12.0.0/lib
PERL5LIB=/binaries/apps/tech_st/10.1.3/perl/lib/5.8.3:/binaries/apps/tech_st/10.1.3/perl/lib/site_perl/5.8.3:/binaries/apps/apps_st/appl/au/12.0.0/perl:/binaries/apps/tech_st/10.1.3/Apache/Apache/mod_perl/lib/site_perl/5.8.3/sun4-solaris-thread-multi
PCC=/bin/proc
SSP_TOP=/binaries/apps/apps_st/appl/ssp/12.0.0
IBU_TOP=/binaries/apps/apps_st/appl/ibu/12.0.0
GMF_TOP=/binaries/apps/apps_st/appl/gmf/12.0.0
FORMS_MESSAGE_ENCRYPTION=TRUE
APPLUSR=usrxit
MSC_TOP=/binaries/apps/apps_st/appl/msc/12.0.0
MRP_TOP=/binaries/apps/apps_st/appl/mrp/12.0.0
LC_NUMERIC=
OA_SECURE=/binaries/apps/apps_st/comn/secure
JAVA_TOP=/binaries/apps/apps_st/comn/java/classes
VNDARSL=
APPLIMG=images
POM_TOP=/binaries/apps/apps_st/appl/pom/12.0.0
IPA_TOP=/binaries/apps/apps_st/appl/ipa/12.0.0
ICX_TOP=/binaries/apps/apps_st/appl/icx/12.0.0
AF_JRE_TOP=/binaries/apps/tech_st/10.1.3/appsutil/jdk/jre
FORMS_PATH=/binaries/apps/apps_st/appl/au/12.0.0/resource:/binaries/apps/apps_st/appl/au/12.0.0/resource/stub
REPORTS_NO_DUMMY_PRINTER=TRUE
RG_TOP=/binaries/apps/apps_st/appl/rg/12.0.0
MTH_TOP=/binaries/apps/apps_st
2015/01/14 19:29:54 VCS NOTICE V-16-55010-10281 (erp-app2-1) OracleAppsCM:ERP_Ora_CM:monitor:VCSagentFW:logOutputFromCommand:The full output is logged to the file [/tmp/ERP_Ora_CM.monitor.aaaWDa4f2.27695]
2015/01/14 19:29:55 VCS INFO V-16-2-13716 (erp-app2-1) Resource(ERP_Ora_CM): Output of the completed operation (monitor)
==============================================
sh[1]: FND_TOP: not found [No such file or directory]
sh[1]: ORACLE_HOME: not found [No such file or directory]
sh[1]: ORACLE_HOME: not found [No such file or directory]

 

Any idea about what's going on

 

 

0 Kudos

mikebounds
Level 6
Partner Accredited

‎01-14-2015 10:42 AM

So is ERP_Ora_Apps resource now working as you are now mentioning ERP_Ora_CM for which the above mentions one issue:

[/export/home/applprod/monitor/oracm.prob] not executable.
                                Check the full pathname and the permissions

 

So have you checked this?

 

Mike

 

0 Kudos