VCS 6.0.1 Solairs 11 Oracle 11Rg2 everything works fine now, except for....

[dariuszz]

getting this error message while offlining the oracle SG.

2013/05/15 10:43:31 VCS ERROR V-16-10001-14058 (solaris11-chi-2) Zone:z3_zone_res:offline:Command [/usr/sbin/zoneadm -z "z3" detach 2>&1] failed with output [umount: warning: /z3root/root not in mnttab

 umount: /z3root/root not mounted

 ERROR: unable to unmount /z3root/root.

 ERROR: Unable to unmount boot environment.

]

I am able to bring up the oracle resource on one of the nodes in a HA zone....

Please see main.cf attached....

[Paresh_Bafna]

Hi,

Could you also share zone configuration file for zone 'z3'? It would be @ /etc/zones/z3.xml

Also, did you online the z3grp through VCS comepletely or manually outside VCS?

[dariuszz]

Okay, Thanks for responding. I am able to get zone/database/listener up on all nodes in the VCS cluster. I am testing hagrp -switch z5grp -to solaris11-chi-1/solaris11-chi-2

I reinstalled VCS, so 

However, the -switch, does not leave the zone in a configured state(when the SG is going offline), I have to manually set configured state  for zone on both nodes.

I am attaching the main.cf as main.txt. 

root@solaris11-chi-1:/# cat /etc/zones/index

# ident "%Z%%M% %I%     %E% SMI"

# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.

# Use is subject to license terms.

#

# DO NOT EDIT: this file is automatically generated by zoneadm(1M)

# and zonecfg(1M).  Any manual changes will be lost.

#

global:installed:/

z5:installed:/z4root:71e7f013-acc1-eaa6-e0ab-8821d024bd57

root@solaris11-chi-1:/#

root@solaris11-chi-2:/# cat /etc/zones/index

# ident "%Z%%M% %I%     %E% SMI"

# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.

# Use is subject to license terms.

#

# DO NOT EDIT: this file is automatically generated by zoneadm(1M)

# and zonecfg(1M).  Any manual changes will be lost.

#

global:installed:/

z5:installed:/z4root:79f123fc-f4fb-6c2c-98d5-b0732d3bd248

root@solaris11-chi-2:/#

root@solaris11-chi-2:/# hastatus -sum

-- SYSTEM STATE

-- System               State                Frozen

A  solaris11-chi-1      RUNNING              0

A  solaris11-chi-2      RUNNING              0

-- GROUP STATE

-- Group           System               Probed     AutoDisabled    State

B  z5grp           solaris11-chi-1      Y          N               OFFLINE

B  z5grp           solaris11-chi-2      Y          N               ONLINE

root@solaris11-chi-2:/#

root@solaris11-chi-2:/# cat /etc/zones/z5.xml

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN" "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1">

<!--

    DO NOT EDIT THIS FILE.  Use zonecfg(1M) instead.

-->

<zone name="z5" zonepath="/z4root" autoboot="true" brand="solaris" ip-type="exclusive" bootargs="-m verbose">

  <dataset name="orabin/orabin" alias="orabin"/>

  <dataset name="oradata/oradata" alias="oradata"/>

  <automatic-network lower-link="auto" linkname="net0" configure-allowed-address="true" link-protection="mac-nospoof" auto-mac-address="2:8:20:5b:1e:d6" mac-address="random"/>

</zone>

root@solaris11-chi-2:/#

[dariuszz]

 umount: /z4root/root not mounted

 ERROR: unable to unmount /z4root/root.

 ERROR: Unable to unmount boot environment.

]

[dariuszz]

My issue is this:

failover is not working. zone remains in "installed" state from the original node it fails FROM, it get's into "running" mode just fine on the node it fails TO, all services come up, database, listener...I have to manually edit the index file to change the zone from installed to configured state for failovber to work....

[Paresh_Bafna]

Hi,

Zone agent does have an attibute to control whether to detach zone root on offline/clean - DetachZonePath.

Though your main.cf shows that DetachZonePath has its default value which is 1, I would request you to check the value of DetachZonePath on running clsuter just to capture any unsaved configuration changes. You could check the value of this attribute by following command:

# /opt/VRTSvcs/bin/hares -value z5_zone DetachZonePath

Value should be 1. If it is not 1, please run following command to set it to 1.

# /opt/VRTSvcs/bin/hares -modify z5_zone DetachZonePath 1

If you see the issue that even after offline zone is in installed state, please enable debug logs for Zone agent and reproduce the issue. Debug log would help capture the exact error.

Debug logs can be enalbed using following command:

# /opt/VRTSvcs/bin/hatype -modify Zone LogDbg 1 2 3 4 5

[Paresh_Bafna · Accepted solution]

Also note that never manually edit any of the zone file like /etc/zones/index or /etc/zones/z5.xml.

If you want to manually change state of the zone from 'installed' to 'configured' run following command.

# /usr/sbin/zoneadm -z z5 detach

Manually editing these files may leave zone in inconsistent state.

[dariuszz]

I wish you continued comfort and peace.Thank you for responding. All values check out, debug levels are set as you suggested.

I did a 

hagrp -switch z5grp -to solaris11-chi-2

and saw this in the engine log:

==============================================

Got notification for group: z5_zone

==============================================

2013/05/20 15:24:42 VCS ERROR V-16-10001-14058 (solaris11-chi-1) Zone:z5_zone:offline:Command [/usr/sbin/zoneadm -z "z5" detach 2>&1] failed with output [umount: warning: /z4root/root not in mnttab

 umount: /z4root/root not mounted

 ERROR: unable to unmount /z4root/root.

 ERROR: Unable to unmount boot environment.

]

Also, saw this:

2013/05/20 15:25:03 VCS INFO V-16-2-13716 (solaris11-chi-2) Resource(z5_pool): Output of the completed operation (online)

==============================================

cannot mount 'z4root' on '/z4root': directory is not empty

cannot import 'z4root': a pool with that name is already created/imported,

and no additional pools with that name were found

==============================================

I can delete the subdirectory called "root" under /z4root....

The good news is that it failed back to node 1, so this means that the "switch" operation left node 1  zone in "configured" state........

will try switching to node 2 again.....hope you read this....

[dariuszz]

2013/05/20 15:36:13 VCS WARNING V-16-10001-20004 (solaris11-chi-2) Zpool:z5_pool:monitor:Warning: The filesystem z4root/rpool with mountpoint /rpool is not mounted. Administrative action may be required

[dariuszz]

root@solaris11-chi-2:/# zfs list

NAME                              USED  AVAIL  REFER  MOUNTPOINT

orabin                           13.1G  20.4G    31K  /orabin

orabin/orabin                    13.1G  20.4G  13.1G  /orabin/orabin

oradata                           368K  33.0G    31K  /oradata

oradata/oradata                    32K  33.0G    32K  /oradata/oradata

rpool                            26.7G  39.8G  73.5K  /rpool

rpool/ROOT                       6.11G  39.8G    31K  legacy

rpool/ROOT/solaris               6.11G  39.8G  2.76G  /

rpool/ROOT/solaris/var           3.27G  39.8G  3.23G  /var

rpool/VARSHARE                    320K  39.8G   320K  /var/share

rpool/dump                       4.06G  39.9G  3.94G  -

rpool/export                       63K  39.8G    32K  /export

rpool/export/home                  31K  39.8G    31K  /export/home

rpool/swap                       16.5G  40.3G  16.0G  -

z4root                           8.00G  25.5G    33K  /z4root

z4root/rpool                     8.00G  25.5G    31K  /rpool

z4root/rpool/ROOT                8.00G  25.5G    31K  legacy

z4root/rpool/ROOT/solaris        8.00G  25.5G  2.95G  /

z4root/rpool/ROOT/solaris/var    5.04G  25.5G  5.04G  /var

z4root/rpool/VARSHARE              39K  25.5G    39K  /var/share

z4root/rpool/export               102K  25.5G    32K  /export

z4root/rpool/export/home           70K  25.5G    32K  /export/home

z4root/rpool/export/home/oracle    38K  25.5G    38K  /export/home/oracle

root@solaris11-chi-2:/#

[dariuszz]

2013/05/20 15:47:49 VCS ERROR V-16-1-10205 Group z5grp is faulted on system solaris11-chi-2

2013/05/20 15:47:49 VCS NOTICE V-16-1-10446 Group z5grp is offline on system solaris11-chi-2

2013/05/20 15:47:49 VCS INFO V-16-1-10493 Evaluating solaris11-chi-1 as potential target node for group z5grp

2013/05/20 15:47:49 VCS INFO V-16-1-10493 Evaluating solaris11-chi-2 as potential target node for group z5grp

2013/05/20 15:47:49 VCS INFO V-16-1-50010 Group z5grp is online or faulted on system solaris11-chi-2

then it faults over to node 1 where it is in "installed" state and gets booted up there - under VCS control

[dariuszz]

This is where things are now:

I reset ZoneResName to null ONLY on the zone root pool z5_pool.

Now, i can switch to node 2 to node2 but cannot switch back to node 1.

I get the following message:

2013/05/21 10:31:35 VCS INFO V-16-1-10298 Resource oradata_pool (Owner: Unspecified, Group: z5grp) is online on solaris11-chi-1 (VCS initiated)

2013/05/21 10:31:35 VCS NOTICE V-16-1-10301 Initiating Online of Resource z5_zone (Owner: Unspecified, Group: z5grp) on System solaris11-chi-1

2013/05/21 10:31:36 VCS DBG_1 V-16-10001-0 (solaris11-chi-1) Zone:z5_zone:online:None of the keys have a valid value. Configuration is not enabled for DR

2013/05/21 10:31:36 VCS DBG_1 V-16-10001-0 (solaris11-chi-1) Zone:z5_zone:online:Attribute WorkLoad not found in the argument list.

2013/05/21 10:31:37 VCS DBG_3 V-16-10001-0 (solaris11-chi-1) Zone:z5_zone:online:Zone [z5] is in [installed] state. Skipping zoneadm attach operation..

2013/05/21 10:31:37 VCS WARNING V-16-10001-14053 (solaris11-chi-1) Zone:z5_zone:online:The value of the BootState attribute is not set. Defaulting BootState to multi-user.

2013/05/21 10:31:44 VCS DBG_3 V-16-10001-0 (solaris11-chi-1) Zone:z5_zone:online:Command [/usr/sbin/zoneadm -z "z5" boot 2>&1] exited with output [zone 'z5': ERROR: no active dataset.

zone 'z5': ERROR: Unable to mount zone root dataset.

zoneadm: zone 'z5': call to zoneadmd failed

] and exitcode [1]