Forum Discussion

actitud's avatar
actitud
Level 3
11 years ago

Oracle Data Guard over VCS 6.0

Hello experts.

It's being almost a month testing and digging on the VCS documentation regarding "Data Guard and Oracle agents supported by Veritas".

The first I can say, documentation is incomplete, confusing and it assumes many things which is a REAL PAIN for a newbie. 

Anyhow, after weeks of issues and spending hours trying to understand who the whole thing works together, I still have many questions.

Scenario:

  • 2 Linux RH6 physical nodes
  • Oracle 11.2.0.4 and Data Guard configured. This is all SINGLE INSTANCE.
  • NO SHARED storage!

I read the full documentation regarding Oracle Data Guard Agents for Veritas which was pretty much a real challenge.

Questions I have:

  1. Why Netlsnr agent shuts down my listener? Switchover fails because of that. I had to remove the Netlsnr resource and monitor.
  2. How can I perform a smooth switchover from VCS using OraDG and OraDGBroker agents? It's pretty amazing why this information is not provided on the document.
  3. Is it there any other complete documentation regarding OraDG and OraDGBroker operation? It says "agent supports this or that" but it wont tells you the details "how" or "how to make sure this or that happens", or exactly "when". It's really vague.

There is an example of what "vague" means. This is a portion of the document:

Note: For activateStandby, deactivateStandby, flashbackRecover, SnapshotOn, and SnapshotOff action functions, you must increase the values of the ActionTimeout and MonitorInterval attributes for OraDG and OraDGSnap types before running.

To "increase" them. Right. How much??? I found some values on another document but not sure why I have to do this or how to confirm the values are good. There are many other examples like this which makes VCS and Oracle implementation a pain.

 

Thanks for any help you can give me. Ask me for any information and I'll provide it.

Alex.

Agents
business continuity
Cluster Server
Linux
  • mikebounds's avatar
    mikebounds
    11 years ago

    I configured the Oracle DG agent about 3 years ago and it worked fine, but I agree the documentation is not great.  I don't know that much about Oracle, but my understanding from the "About the agent for Oracle Data Guard" is that:

    • Data Guard agent is used for single instance Oracle for all versions and this uses SQL *Plus to control Oracle DG.
    • Data Guard Broker agent is used for RAC but can also be used for single instance Oracle since 11gR2
     
    So if this understanding is correct, as you are using single instance 11gR2, then you have choice of using EITHER Data Guard agent or Data Guard Broker agent, but you should not use both, as you have done.
    As to which agent you should use, I guess this is an Oracle question, but it would be useful if the agent guide gave some direction on this.
     
    When you had your listener resource configured, did you make Oracle resource depends on the Listener resource as in the agent guide.
     
    Have you tested Oracle DG outside of VCS control - it it doesn't work outside of VCS control, then VCS won't be able to get it to work either.  If it works outside of VCS control, but not in VCS, then look at engine_A.log and OraDG_A.log and post any errors.
     
    As an aside, I would not set EnvFile = "/home/oracle/.profile" - this is bad HA practice as if someone changed the .profile this could break VCS (so for instance I have seen a DBA add a menu to .profile to choose what dabase they wanted to log into and this change would break VCS if they had used .profile in VCS).
     
    Mike
     

16 Replies

Sort By
  • actitud's avatar
    actitud
    Level 3
    11 years ago

    Here you have.

    Keep in mind that I have remove the listener resource from here because of the "issue" I had during switchover.

    include "OracleASMTypes.cf"
    include "types.cf"
    include "Db2udbTypes.cf"
    include "OracleTypes.cf"
    include "OraDGTypes.cf"
    include "SybaseTypes.cf"
     
    cluster FWT_CLUSTER (
            UserNames = { admin = lkjfgoijtrhsDsfDffg, mov = asFAdsfjaseSADas }
            ClusterAddress = "10.1.1.10"
            Administrators = { admin, mov }
            HacliUserLevel = COMMANDROOT
            )
     
    system node1 (
            )
     
    system node2 (
            )
     
    group ClusterService (
            SystemList = { node1 = 0, node2 = 1 }
            AutoStartList = { node1, node2 }
            OnlineRetryLimit = 3
            OnlineRetryInterval = 120
            )
     
            IP webip (
                    Device = a1b1
                    Address = "10.1.1.10"
                    NetMask = "255.255.255.192"
                    )
     
            NIC csgnic (
                    Device = a1b1
                    )
     
            webip requires csgnic
     
     
            // IP webip
            //   {
            // NIC csgnic
            //      }
     
     
    group mov_db_grp (
            SystemList = { node1 = 0, node2 = 1 }
            Parallel = 2
            SystemZones = { node1 = 0, node2 = 1 }
            )
     
            Oracle mov_db (
                    Sid @node1 = movp
                    Sid @node2 = movp
                    Owner = oracle
                    Home = "/u01/oracle/product/11.2.0.4"
                    StartUpOpt = CUSTOM
                    EnvFile = "/home/oracle/.profile"
                    )
     
            // resource dependency tree
            //
            //      group mov_db_grp
            //      {
            //      Oracle mov_db
            //      }
     
     
    group oradg_broker_grp (
            SystemList = { node1 = 0, node2 = 1 }
            )
     
            OraDGBroker oradg_broker (
                    Sid = movp
                    Owner = oracle
                    Home = "/u01/oracle/product/11.2.0.4"
                    AgentDebug = 1
                    LinkRes = mov_db
                    )
     
            requires group mov_db_grp online local soft
     
     
            // resource dependency tree
            //
            //      group oradg_broker_grp
            //      {
            //      OraDGBroker oradg_broker
            //      }
     
     
    group oradg_grp (
            SystemList = { node1 = 0, node2 = 1 }
            )
     
            OraDG oradguard (
                    LinkRes = mov_db
                    Flashback = 1
                    )
     
            requires group mov_db_grp online local soft
     
     
            // resource dependency tree
            //
            //      group oradg_grp
            //      {
            //      OraDG oradguard
            //      }
  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    I configured the Oracle DG agent about 3 years ago and it worked fine, but I agree the documentation is not great.  I don't know that much about Oracle, but my understanding from the "About the agent for Oracle Data Guard" is that:

    • Data Guard agent is used for single instance Oracle for all versions and this uses SQL *Plus to control Oracle DG.
    • Data Guard Broker agent is used for RAC but can also be used for single instance Oracle since 11gR2
     
    So if this understanding is correct, as you are using single instance 11gR2, then you have choice of using EITHER Data Guard agent or Data Guard Broker agent, but you should not use both, as you have done.
    As to which agent you should use, I guess this is an Oracle question, but it would be useful if the agent guide gave some direction on this.
     
    When you had your listener resource configured, did you make Oracle resource depends on the Listener resource as in the agent guide.
     
    Have you tested Oracle DG outside of VCS control - it it doesn't work outside of VCS control, then VCS won't be able to get it to work either.  If it works outside of VCS control, but not in VCS, then look at engine_A.log and OraDG_A.log and post any errors.
     
    As an aside, I would not set EnvFile = "/home/oracle/.profile" - this is bad HA practice as if someone changed the .profile this could break VCS (so for instance I have seen a DBA add a menu to .profile to choose what dabase they wanted to log into and this change would break VCS if they had used .profile in VCS).
     
    Mike
     
  • actitud's avatar
    actitud
    Level 3
    11 years ago

    Hello Mike. First of all, thanks a lot for your time on this. Believe me, I spent nights on all these issues and I'm not newbie on Oracle's technologies.

    Anyhow, if what you say is correct, then it makes perfect sense and it would be another disappointed surprise on the documentation as it does't mention the option of one OR another agent. So, it's a FACT that I can not use them both? Just checking.

    Yes,

    • Data Guard Agent uses Sqlplus to perform the data guard operations (such as failovers, switchovers, etc)
    • DG Broker uses the database broker to do the same

    I'll remove DataGuard and see if by using hagrp -switch on the Broker, I can make it work.

    YES. When I had it on the main, I have configured the database resource to depend on the listener, and that's the main reason on why I can't understand why the Netlsnr shut it down when the database comes down :(. Maybe, If I only have the broker and the listener, everything will start to work magically.

    There is nothing like Oracle DG outside VCS. What VCS does (with its agent) is to perform the same tasks you would perform from SqlPlus to do a switchover for example.

    Broker (dgmgrl) works fine from outside VCS. What I'm doing right now is to freeze the agents and then run the switchover but it's clearly a dumb practice just for testing.

    I'll check the profile (which I don't believe is causing any issues).

    Again, thanks for your inputs. I'll test it and let you know the results.

    BTW, the good side on all this is I've faced MANY (if not all) the possible issues so I've learned a lot.

    Thanks,

    Alex.

     

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    Note, if you use BDG roker agent then you the need theStartUpOpt = STARTUP for Oracle resource , not CUSTOM as it needs to be for DG agent.

    Mike

  • actitud's avatar
    actitud
    Level 3
    11 years ago

    Ok. That makes all this more understandable as one of the agents uses

    "StartUpOpt = CUSTOM"

    and the other one

    "StartUpOpt = STARTUP"

    Now, how can I perform the switchover from SVC using the OraDGBroker instead of the dgmgrl?

    Should I perform the switchover manually from dgmgrl?

  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    In VCS just switch oradg_broker_grp to the other node - this will cause VCS to call the online function for the Oracle Broker DG resource which will run dgmgrl failover or switchover.

    Mike

  • actitud's avatar
    actitud
    Level 3
    11 years ago

    I've tried that but nothing happens. Maybe I have to wait a little longer? Trying again right now.

    This is what I've got last time from Engine log:

    2013/11/25 14:54:54 VCS NOTICE V-16-1-10301 Initiating Online of Resource oradg_broker (Owner: Unspecified, Group: oradg_broker_grp) on System node2

    2013/11/25 14:54:56 VCS INFO V-16-1-10298 Resource oradg_broker (Owner: Unspecified, Group: oradg_broker_grp) is online on node2 (VCS initiated)
    2013/11/25 14:54:56 VCS NOTICE V-16-1-10447 Group oradg_broker_grp is online on system node2
     
    Nothing came up from the OraDGBroker_A.log log.
     
    This is what I have on hastatus after the switch
     
    ...
    ...
                    mov_db           node1                ONLINE              
                    mov_db           node2                ONLINE              
                    oradg_broker         node1                OFFLINE             
                    oradg_broker         node2                ONLINE
    ...
    ...
     
    5 minutes and still waiting. Anything to do with the Listener resource? (not currently present on the main.cf)
     
  • mikebounds's avatar
    mikebounds
    Level 6
    11 years ago

    I've read the agent guide again, and as you say it is confusing as it is not clear which statements relate to both agents and which refer to just the "Oracle Data Guard agent"  as oppose to the "Oracle Data Guard broker agent"  - so I think the statement in the introdution:

    The Oracle Data Guard agent supports replicated data clusters on Linux and Solaris operating systems
     
    means ONLY the Oracle Data Guard agent supports replicated data clusters and the "Oracle Data Guard broker agent"  does not and if it doesn't, then by implication the "Oracle Data Guard broker agent"  must support Global clusters with the global cluster option (GCO).  It also appears the the Oracle Data Guard agent supports GCO as well as RDS, but this is not stated in the introduction.
    Just to be clear, as you are a newbie - a replicated data cluster (RDC) is when you are replicating within a single cluster and a global cluster is where you are replicating between 2 clusters.
     
    The Oracle Data Guard agent gives example for RDC and GCO, but the Oracle Data Guard broker agent ony gives an example for GCO.
     
    Also in the "Configuring the agent manually in a global cluster" section it says:
     
    Add a new group with at least one resource of type OraDG or OraDGBroker for VCS
    and for "Configuring the agent manually in a replicated data cluster"
     
    Create an OraDG service group and configure the attributes of the OraDG resource that you added.
     
    So as you have an RDC, then I think you must use the "Oracle Data Guard agent", not the "Oracle Data Guard broker agent" 
     
    Mike

     

  • actitud's avatar
    actitud
    Level 3
    11 years ago

    In all this mess,  your help is obviously priceless. Thanks for that again Mike.

    The main problem when using Data Guard Agent to do the switch (hagrp -switch oradg_grp -to nodeX) is that the actual BROKER configuration (managed by Oracle's dgmgrl) never gets aware of what happened in the backend and it gets "corrupted" showing wrong information. The only solution is to recreate the configuration as it will keep believing the primary database is the one from before the switch happened which is clearly wrong.

    So i guess the only two options are:

    1. Perform the switch(over) from Data Guard Agent using VCS and THEN, recreate dgmgrl configuration
    2. Freeze the Data Guard Agent Group, perform the switchover from dgmgrl, switch the group and unfreeze.

    I guess in case of number 2, I rather should not use dgmgrl at all.

     

    What do you think?

    Alex.