03-28-2011 02:05 PM
Hi guys,
I'm in a nice position to upgrade a VCS cluster before it makes its way into production. The cluster is installed on rhel5.3 5.0.3 MP3 and I'd like to upgrade it to 5.1_SP1.
Glancing through the upgrade guide, it doesn't look like I'd be able to do an online upgrade with 2 nodes? Other posts (my own) advise that veritas doesn't work when the versions differ, and as soon as one node is upgraded... my versions would differ. Is there any way to do an online upgrade? If not - that's cool, just looking for confirmation :)
Second.. the upgrade guide says stop the application agents and resources (my second indication that an online upgrade can't be done).
For syntax.. I've never used these commands before. For hares is it looking for the service group?
hares -offline Service_Group_Name -sys hostnameofmachinerunninggroup (I'd imagine this wouldn't work on the standby host).
For haagent is it looking for the below 4 agents? and then repeat the same on the 2nd host?
haagent -stop NICAgent -sys hostname1
haagent -stop StorageAgent -sys hostname1
haagent -stop IPAgent -sys hostname1
haagent -stop ApplicationAgent -sys hostname1
[root@linopstfg01 ~]# ps -ef | grep Agent
I'm pulling these instructions from
https://sort.symantec.com/public/documents/sf/5.1/linux/pdf/vcs_install.pdf page 234
Is the above correct? Do I have anything wrong?
Solved! Go to Solution.
03-29-2011 10:31 PM
Hello,
GAB & fencing will not be shutdown by running hastop command. The reason I suggested you to shutdown Fencing & GAB on node B was following:
-- lets say you ran an installer on node B, (it should anyways shutdown vxfen & Gab) but still safer to do with our own hands. installer will need to upgrade these components as well, so Gab & Vxfen will be upgraded to 5.1 now once node B is ready to take up services after upgrade, it will not join even Gab membership with node A since there is a version difference now moreover you have active node A accessing disks at this point. I hope you get what I am trying to say here..
so to overcome, on node B, shutdown Fencing & Gab (/etc/init.d/vxfen stop & /etc/init.d/gab stop). This will stop the vxfen & Gab. now you can go ahead & run the upgrade so that both these components are upgraded to 5.1.
Later on node A, you should repeat the same procedure as there also you need to upgrade both of these components in order to join memberships with node B.
Hope its clear..
Gaurav
03-28-2011 02:17 PM
Hello,
In nutshell, yes, cluster will NOT be formed when versions differ, so you are right , when one node is upgraded it won't join the cluster with other (un-upgraded) node. So.. upgrade will be done in an offline method (one node at a time). This procedure will need some outage .. here are steps in short:
Lets say your two nodes are A & B & it runs multiple service groups..
1. Switch/online all your service groups on node A, make sure no service group is online on node B
2. shutdown VCS on node B
3. Upgrade VCS on node B & make sure nothing starts on node B (you can do so by renaming cluster scripts)
4. Shutdown all service groups from node A (here your outage starts)
5. start all cluster services on node B i.e upgraded node, start all service groups here
6. Repeat the upgrade procedure on node A, once upgrade done, it should join cluster with node B automatically.
Now regarding your other questions:
-- Agents need not to be stopped individually, once you stop cluster (had process), it will stop all the agents.
-- Regarding resources, once you offline a service group, it will offline all the resources inside it, so no need to run hares individually.
Regarding the last part, it is applicable to you only if you are using any such agent which is obsolete from 5.1 onwards... so do you use any such agent ? From ps -ef output, I see all agents are still there in 5.1 (forget storageagent as it belongs to vxvm VEA GUI)
Hope its clear
Gaurav
03-29-2011 08:29 AM
Note " /opt/VRTSobc/pal33/bin/vxpal -aStorageAgent-x" is not a VCS agent is is an agent used by Volume Manager, so you can't stop using haagent - this process is normally stopped as part of the SF upgrade.
You can actually upgrade VCS without an outage, by running "hastop -all -force" (which leaves applications running), unload VCS modules, upgrading VCS and starting new version of VCS, but to upgrade SF (VM + vxfs) you need an outage, but you can minimise outage using Gaurav's procedure above.
Mike
03-29-2011 02:29 PM
Hi,
I just tried to upgrade my 2nd node and got the following errors/warnings.
03-29-2011 02:47 PM
actually, I figured that out... should've tried before posting.
However I notice now that I'm trying a "phased" upgrade according to VCS.. the documentation tells me to disable fencing and all sorts of stuff from the upgrade guide. Is this necessary (seems to imply services still running) as opposed to what we're doing with shutting off the services on the 2nd node. It would seem to me that changing config files if services are shut off wouldn't really do much good.
03-29-2011 02:53 PM
well since its a phased upgrade, that means your one node would be still accessing the data ... so changing fencing config won't be good.. even deleting keys won't be necessary. When you have upgraded your node B successfully, shutdown all apps on node A, shutdown cluster, shutdown fencing , GAB on node A & start all of these components on node B .... That should start Gab / fencing / HAD for 5.1 module
Hope its clear
Gaurav
03-29-2011 05:38 PM
should GAB/fencing be shut down on node B before doing the upgrade? I guess what I'm unsure of is what services need to be offline on node B compared to what would need to be shut down on node A?
So far what I've done is..
move everything to node 1
hastop -sys hostname
[root@linopstfg02 ~]# lltconfig
LLT is running
[root@linopstfg02 ~]# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen dbff03 membership 01
Port b gen dbff06 membership 01
then I was going to start the installer and do the upgrade. fencing is still running as well as anything else not stopped by the hastop.
Below I've pasted a ps -ef. Forgive me if I've missed something, I'm not that familiar with VCS in terms of proper services to shut down. I'm not sure if there would be issues with some components still running during the upgrade and components being upgraded while being accessed by the other node. I want to make sure I do this right, and only once so that way I can replicate the exact same behavior on our current production nodes.
03-29-2011 10:31 PM
Hello,
GAB & fencing will not be shutdown by running hastop command. The reason I suggested you to shutdown Fencing & GAB on node B was following:
-- lets say you ran an installer on node B, (it should anyways shutdown vxfen & Gab) but still safer to do with our own hands. installer will need to upgrade these components as well, so Gab & Vxfen will be upgraded to 5.1 now once node B is ready to take up services after upgrade, it will not join even Gab membership with node A since there is a version difference now moreover you have active node A accessing disks at this point. I hope you get what I am trying to say here..
so to overcome, on node B, shutdown Fencing & Gab (/etc/init.d/vxfen stop & /etc/init.d/gab stop). This will stop the vxfen & Gab. now you can go ahead & run the upgrade so that both these components are upgraded to 5.1.
Later on node A, you should repeat the same procedure as there also you need to upgrade both of these components in order to join memberships with node B.
Hope its clear..
Gaurav
03-30-2011 11:01 AM
is there a way to override the admin account password that was originally set with the cluster? after upgrading the clusters it doesn't look like the account works anymore.
03-30-2011 11:04 AM
sorry.. little more data might be useful.
I installed the Java cluster manager (hagui) to try accessing. Everything seems to be online on both boxes.
03-30-2011 11:22 AM
I found that the upgrade seemed to move configs around.. my main.cf was very small and included none of my old service groups (yay for backups).
Moved a backup but that didn't seem to help.. I found a post mentioning..
Login as root, and run the hauser command to add the admin user again:
/opt/VRTSvcs/bin/hauser -add admin -priv Administrator
So I might try that.
03-30-2011 11:31 AM
If you still have a question left that I can move this to a new discussion since this seems to be a different issue than the one for which this thread was started.... let me know if this works else I will move this to different thread.
G
03-30-2011 11:35 AM
yep, different question - looks like my main.cf is being overwritten with a blank vcs cluster config. This can be moved to a different thread - or I can start it myself. Let me know which you'd prefer.