Database recycled but not giving alert

[raunaz]

Hi Experts, 

Our DB admin recycled the oracle database on Oracle level. DBA team did disable the oracle monitoring script on oracle level. So, on VCS the oracle monitoring resource was found offline. But, supprisingly the VCS didn't  captured any error saying the oracle DB was recycled and the service group was not faulted. Was that the correct VCS behaviour ? 

Another thing is, other than engine_A.log where else the log was stored, because when i login to java console i saw some info which is not found in engine_A.log. 

Please help on this, 

Thanks

[Gaurav_S]

Hello,

Little confused with statement ... when you say "on VCS Oracle monitoring resource was found offline"  what type of this resource was ?  was it an oracle resource ?

secondly, was the cluster or service groups frozen at any stage ?

regarding logs, under /var/VRTSvcs/log  you can find all agent logs as well .... you might want to have a look in Oracle_A.log for oracle related messages...

G

[raunaz]

Sorry for the confusion occured, It is aaplication type resource. It is actually a script to monitor the database status. And it is not critical.

Nothing was found inside Oracle_A.log. So i'm thinking of the real log file read by the java console. In our envoronment, all logs was was moved to another places, so basically all the logs inside /var/VRTSvcs/log was emptied. So, i'm thinking of where exactly the VCS java fetch the info. 

[Marianne]

Please post your main.cf?

About the logs - how did you go about to move them? Did you create symbolic links in /var/VRTSvcs/log to the new location?

[raunaz]

Here is the main.cf for oracle and the script to monitor

        Oracle DB2ORA (

                Critical = 0

                Sid = DB2

                Owner = oracle

                Home = "/opt/oracle/product/10.2.0.3.0_64"

                Pfile = "/opt/oracle/product/10.2.0.3.0_64/dbs/initDB2.ora"

                MonScript = "./bin/Oracle/SqlTest.pl"

                )

Application DB2AppOramon (

                Critical = 0

                User = oracle

                StartProgram = "/opt/oracle/scripts/oracle/bin/oracle_mon.pl -s DB2 -k -d 0 1>/dev/null &"

                StopProgram = "/opt/oracle/scripts/oracle/bin/oracle_mon.pl -s DB2 -K"

                MonitorProcesses = {

                         "/opt/oracle/perl/bin/perl -S ./.oracle_mon.pl -s DB2 -k -d 0" }

                )

About the logs, we have a script run by cron No, it will trim the logs which is more than 7 days. the logs was not a link to the others location.

-rw-r--r--   1 root     root           0 Jun  8 23:58 IP_A.log

-rw-r--r--   1 root     root           0 Jun  8 23:58 IP_ipm_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 NIC_A.log

-rw-r--r--   1 root     root           0 Jun  8 23:58 NetAppExport_A.log

-rw-r--r--   1 root     root           0 Jun  8 23:58 NetAppFiler_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 Netlsnr_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 NotifierMngr_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 ORA-09925:

-rw-r--r--   1 root     other          0 Jun  8 23:58 Oracle_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 Phantom_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 Phantom_ipm_A.log

-rw-r--r--   1 root     root           0 Jun  8 23:58 Proxy_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 VRTSWebApp_A.log

-rw-r--r--   1 root     root     32614176 Jun  9 09:56 engine_A.log

-rw-r--r--   1 root     root           0 Jun  8 23:58 engine_B.log

-rw-r--r--   1 root     root           0 Jun  8 23:58 engine_C.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 hashadow-err_A.log

-rw-r--r--   1 root     root           0 Jun  8 23:58 hashadow_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 notifier-err_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 notifier-out_A.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 sg_create.log

-rw-r--r--   1 root     other          0 Jun  8 23:58 sg_create.scr

-rw-r--r--   1 root     other          0 Jun  8 23:58 sg_create2.scr

 # wc -l engine_A.log

       1 engine_A.log

[raunaz]

Could anyone assist me on this, apreciate for the help.

[Marianne · Accepted solution]

Extract from VCS Admin Guide:

About critical and non-critical resources
The Critical attribute for a resource defines whether a service group fails over
when the resource faults. If a resource is configured as non-critical (by setting
the Critical attribute to 0) and no resources depending on the failed resource are
critical, the service group will not fail over. VCS takes the failed resource offline
and updates the group status to ONLINE|PARTIAL. The attribute also determines
whether a service group tries to come online on another node if, during the
group’s online process, a resource fails to come online.

So, the non-critical resources is the reason why service group was not faulted. Was that the correct VCS behaviour ?  YES.

Notification/Alert depends on whether Notifier resource is configured and what level of notification is set.

To understand the difference of what is seen in Java Console and engine_A.log, you need to show us what you are seeing.

I am not comfortable with the way that you 'trim the logs' without copying it first. How often does this script run? How do you know that the info that you're looking for was not discarded by the 'trimming' process?

Rather use the VCS LogSize attribute so that VCS will recycle engine_A log when it gets to the maximum size:

# haclus -modify LogSize <new_size>

Minimum value is = 65536 (equal to 64KB)
Maximum value = 134217728 (equal to 128MB)
Default value = 33554432 (equal to 32MB) 

Agent log files sizes can be specified with LogFileSize attribute:

# hatype -modify <Agent> LogFileSize <new_size>

Please also let us know which OS and VCS version.

[raunaz]

Thanks for pointing the answer right infront of me. yes, the DB was not faulted because it was set to false. 

Regarding the logfile. The file size was set to maximum value by default. And we did backup the files before we trim it.