cancel
Showing results for 
Search instead for 
Did you mean: 

adding new volumes to a DG that has a RVG under VCS cluster

grigori
Level 5

hi,

i am having a VCS cluster with GCO and VVR.
on each node of the cluster i have a DG with an associated RVG, this RVG contains 11 data volume for Oracle database, these volumes are getting full so
i am going to add new disks to the DG and create new volumes and mount points to be used by the Oracle Database.

my question: can i add the disks to the DG and volumes to RVG while the database is UP and the replication is ON?
if the answer is no, please let me know what should be performed on the RVG and rlink to add these volumes also what to perform on the database resource group to not failover.

thanks in advance. 

1 ACCEPTED SOLUTION

Accepted Solutions

mikebounds
Level 6
Partner Accredited

Guarav's post is not quite correct so you must not do the steps in the order above - it is VERY important you do them in the order below:

  1. Add disks to diskgroup (vxdisksetup -i and vxdg adddisk)
  2. Create Volume on both Primary and secondary VVR nodes (vxassist make)
  3. Add to RDS (vradmin addvol)
  4. Create fileysystem (mkfs)
  5. Create mount points on all nodes in both clusters
  6. Add resource to VCS (hares -add/modify) on both clusters
  7. I would recommend making resource non-critical and not linked to other resources yet and then online resource - in case there are any typos in resource name or mountpoints you have created
  8. Once resource is online, then link resource (hares -link) and make critical (hares -modify)

Of course you can use VCS Java GUI or VOM to do VCS steps if you prefer.

When you do step 3, it does not copy any data between sites, but it then replicates from that point onwards, so when you then do step 4 and create filesystem, the creation of the filesystem is replicated, so if you had created filesystem first, it would not have been replicated and so the volume would be unusable on the secondary.

Note the primary and secondary VVR volumes will not be the same, but the filesystem metadata and the blocks of files created will be the same and so it will only be unused space in the filesystem that is not the same.  If you want the volumes to be block for block the same, then create volumes on primary and secondary using "init=zero" with the vxassist command which willl create volume and write zero to all blocks (you must still create filesystem AFTER adding to RDS)

Mike

View solution in original post

14 REPLIES 14

Gaurav_S
Moderator
Moderator
   VIP    Certified

Hi,

You should be able to add volumes online .. First step is to get volume created on both primary & secondary sites. Both the volumes on each site must have same size. You can use vxassist to create the volume. Once volume is created you need to create filesystem on it. You can use mkfs to create to filesystem.

Next step would be to add volume to RDS. Refer steps in below link to understand how to add volume to RDS

https://sort.symantec.com/public/documents/sfha/6.1/solaris/productguides/html/sfhas_replication_admin/ch12s01s01.htm

By using vradmin addvol command, it would automatically add DCM volume & mirror it.

Once the volume is setup under RDS & is getting replicated, you need to bring the volume under VCS control. You can use cluster commands to do this task. Refer how to add a resource to service group section on below link

https://sort.symantec.com/public/documents/sfha/6.1/solaris/productguides/html/vcs_admin/ch06s14s02.htm

You resource type would be volume & Mount. Volume resource would add volume to VCS control. Mount resource will add Filesystem under VCS control.

Once resources are created & bought under VCS control, you need to link them appropriately. Mount will depend on volume typically, volume will depend on diskgroup.

Refer this section on how to link resource

https://sort.symantec.com/public/documents/sfha/6.1/solaris/productguides/html/vcs_admin/ch06s14s09.htm

Hope this helps.

 

G

mikebounds
Level 6
Partner Accredited

Guarav's post is not quite correct so you must not do the steps in the order above - it is VERY important you do them in the order below:

  1. Add disks to diskgroup (vxdisksetup -i and vxdg adddisk)
  2. Create Volume on both Primary and secondary VVR nodes (vxassist make)
  3. Add to RDS (vradmin addvol)
  4. Create fileysystem (mkfs)
  5. Create mount points on all nodes in both clusters
  6. Add resource to VCS (hares -add/modify) on both clusters
  7. I would recommend making resource non-critical and not linked to other resources yet and then online resource - in case there are any typos in resource name or mountpoints you have created
  8. Once resource is online, then link resource (hares -link) and make critical (hares -modify)

Of course you can use VCS Java GUI or VOM to do VCS steps if you prefer.

When you do step 3, it does not copy any data between sites, but it then replicates from that point onwards, so when you then do step 4 and create filesystem, the creation of the filesystem is replicated, so if you had created filesystem first, it would not have been replicated and so the volume would be unusable on the secondary.

Note the primary and secondary VVR volumes will not be the same, but the filesystem metadata and the blocks of files created will be the same and so it will only be unused space in the filesystem that is not the same.  If you want the volumes to be block for block the same, then create volumes on primary and secondary using "init=zero" with the vxassist command which willl create volume and write zero to all blocks (you must still create filesystem AFTER adding to RDS)

Mike

Gaurav_S
Moderator
Moderator
   VIP    Certified

Hi Mike,

Not quite convinced with the change in order you are suggesting, happy to discuss this further ..

You suggest to add volume to RDS first & then create the filesystem. This will replicate the filesystem also to secondary site. May be a good thought.

What I suggested was to create filesystem first & then add to RDS. If the volume size is same (which I mentioned in my comment above), FS should be same. I am not quite clear on why volume on Primary & Secondary can not be same ? If the exact block sizes are provided, volume would be same & so as the filesystem on top of it.

 

G

 

mikebounds
Level 6
Partner Accredited

Hi Guarav, are you suggesting create filesytem on both primary and secondary, before adding to RDS or just creating filesystem on primary before adding to RDS?

Mike

Gaurav_S
Moderator
Moderator
   VIP    Certified

Hi Mike,

Creating FS only on primary will give us convenience of not creating FS on secondary if we put the volume into RDS before creating filesystem. This will ensure that even blank FS structure is replicated between primary & secondary.

If we create FS on both primary & secondary before putting volume in RDS, then also I believe it shouldn't cause any issue as it is a fresh volume & filesystem. As we are adding the volume to RVG using vradmin addvol command, the DCM maps will be associated with volume automatically that means data change maps are created before volume is added to RVG. So in this case, if there is any changed data (even though its a fresh vol & filesystem), DCM would know about it & changes will be replicated to other node.

One of the question which can be more helpful here is, what is the plan to copy data on volumes ? I believe if the data is going to get copied to Primary after volume getting added to RVG, then we may have issues & recommendation should be to create FS only on primary, copy data & add to RDS but run a vradmin syncvol. However if data is huge & getting copied to both Primary & secondary before getting volume in RDS, then we need to trust that volume is in sync & a syncvol can be avoided.

Here is a reference from vradmin addvol man page

"If replication is in progress, make sure that the Secondary volume contains the same data as the Primary volume before adding the volume to the RDS. To transfer the data from the Primary volume to the Secondary volume over the network, use the vradmin syncvol command. "

 

G

mikebounds
Level 6
Partner Accredited

Hi Guarav,

See comments below:

Creating FS only on primary will give us convenience of not creating FS on secondary if we put the volume into RDS before creating filesystem. This will ensure that even blank FS structure is replicated between primary & secondary.

Agreed, this is what I suggested in my first post, but what you must not do is create FS only on primary before adding to RVG as this will not work and you will have useless data at the secondary.  I thought this is what you were suggesting in your first post or maybe you were suggesting creating FS on both sites before adding to RVG which I will address next.

If we create FS on both primary & secondary before putting volume in RDS, then also I believe it shouldn't cause any issue as it is a fresh volume & filesystem

When you create a filesystem it may contain GUID, timestamps and potentially hostname or architecture of storage and therefore your 2 filesystems created will not be byte for byte the same.  If you are lucky this may not matter, depending on how this is written and used by the filesystem, but remember VVR supports any filesystem, so you can never be sure this method will work for every filesystem and you may come across issues further down the line (for exampe if a filesystem, later on, re-writes GUID to somewhere else in the meta filesystem, then on the system it does this the GUIDs will be the same, but on the other system, the GUIDs will be different and how this is handled by the other system is unknown).  Therefore this is risky and there are no benefits of this method (as you have mentioned, my method will give convenience of not creating FS on secondary)

As we are adding the volume to RVG using vradmin addvol command, the DCM maps will be associated with volume automatically that means data change maps are created before volume is added to RVG. So in this case, if there is any changed data (even though its a fresh vol & filesystem), DCM would know about it & changes will be replicated to other node.

When you say "if there is any changed data" , what do you mean by this - are you saying DCM is aware of the blocks written to create filesystem (where filesystem is created before adding to RVG) or are you saying DCM is aware of what is different between primary and secondary (example different GUIDs if you create both filesystems first), or do you mean something else.

One of the question which can be more helpful here is, what is the plan to copy data on volumes ? I believe if the data is going to get copied to Primary after volume getting added to RVG, then we may have issues & recommendation should be to create FS only on primary, copy data & add to RDS but run a vradmin syncvol. However if data is huge & getting copied to both Primary & secondary before getting volume in RDS, then we need to trust that volume is in sync & a syncvol can be avoided.

The manual is not great on how syncvol can be used.
grigori wants to add a new blank filesystem and then add an oracle data file to it.  A common mistake made here is that this is done before adding to RVG and then you have an issue as you have to run syncvol on a static volume and if you have already created Oracel data file on it, then you cannot do this and therefore in this situation you choice is:

  1. Bring down DB, umount new volume, run vradmin syncvol and then start up your DB - not ideal as you have now created an outage
  2. Stop replication and resync from scratch - not ideal as now you have no DR until resync has finished, which can be a very long time for large DBs

Therefore the correct method as in my first post is to add empty volume to RDS, then create filesytem and then it is ok to add to create Oracle data files on it.

Mike

 

 

Gaurav_S
Moderator
Moderator
   VIP    Certified

Hi Mike,

let me run some tests to check out on GUID & will let you know how did it go ...

I still believe that mkfs should produce consistent results while creating filesystem on any given size of volume. Regardless of anything underneath, vxfs metadata should be same on any volume created with exact same size.

If I get an opportunity, will try to test syncvol once again.

Hopefully Grigori has the answers post this discussion ..

 

G

 

 

mikebounds
Level 6
Partner Accredited

Hi Gaurav,

It may work in a few tests, but I believe this is an unnecessary risk to create filesystem twice and assume it is the same.

If definately will not be the same for Linux ext and I am not sure about vxfs as it is hard to get lowlevel information about the filesystem.  I don't know what GUID is used for in ext, so you would have to check the source code to find out every instance it is used to check that it may not cause an issue (in particular that ANY fs command, like fsck does not re-write the GUID to another location) - why would you want to do this -much easier to create filesystem ONCE, after adding to RVG so you know both sites are the same.

Still unsure what you think DCM is used for.  As far as I am aware, the DCM is not used when you run "vradmin addvol" - it is only used in initial sync, SRL overflow and after vradmin takeover - if you know different, then please let me know.

Mike

grigori
Level 5

hi Mark and Gaurav.

thanks for the help provided so far, yet i didnt try the steps becuase i got stuck into and RVG issue when adding the new disks to the disk group in the secondary site.

i have mistankly deleted 2 disks that were used as cachevol for secondary RVG and data in secondary become unaccessible, for that i went through deleting the DG and succeeded.

now i am trying to create RVG group again, for that i should delete both RVG in main and DR which seems impossible, even dissociate SRL volume from primary RVG seems impossible

any hint please?

BR, 

grigori
Level 5

hi Mike,

if i am going to useVeritas Cluster Manager GUI to do the above, how can i apply the changes on all cluster nodes specially if i am using the GCO option.

regards

mikebounds
Level 6
Partner Accredited

Add resource to one cluster and then copy resource and paste into other cluster.

Mike

grigori
Level 5

hi Mike,

i just wanted to confirm, should i copy the resources into the second cluster node in order to succeed in failover of the volumes and mount points (when we do the failover test), or it is sufficient to perform the commands/GUI steps on one node and cluster's main.cf will be reloaded automatically after we save by "haconf -dump -makero"

thanks

 

grigori
Level 5

hi,

i found the needed answer in the VCS admin guide:

In a VCS cluster, the first system to be brought online reads the configuration file and creates an internal (in-memory) representation of the configuration. Systems that are brought online after the first system derive their information from systems that are in the cluster.

You must stop the cluster if you need to modify the files manually. Changes made by editing the configuration files take effect when the cluster is restarted. The node where you made the changes should be the first node to be brought back online.

 

regards

grigori
Level 5

Hi Gaurav,

i marked Mike's note as solution because i tried the steps he supplied and there is only one solution in every thread :)

regards