Forum Discussion

Level 6

11 years ago

CVM won't start on remote node with an FSS diskgroup

I am testing FSS (Flexible Shared Storage) on SF 6.1 on RH 5.5 in a Virtual Box VM and when I try to start CVM on the remote node I get:

VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster reason: Disk for disk group not found: retry to add a node failed

Here is my setup:

Node A is master with a local (sdd) and remote disk (B_sdd)

[root@r55v61a ~]# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: r55v61a
[root@r55v61a ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
B_sdd        auto:cdsdisk    -            -            online remote
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    -            -            online exported

Node B is the slave, and sees local (sdd) and remote disk (A_sdd)

[root@r55v61b ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
A_sdd        auto:cdsdisk    -            -            online remote
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    -            -            online exported

On node A, I add an FSS diskgroup, so on node A the disk is local

[root@r55v61a ~]# vxdg -s -o fss=on init fss-dg fd1_La=sdd
[root@r55v61a ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
B_sdd        auto:cdsdisk    -            -            online remote
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    fd1_La       fss-dg       online exported shared

And on node B the disk in fss-dg is remote

[root@r55v61b ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
A_sdd        auto:cdsdisk    fd1_La       fss-dg       online shared remote
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    -            -            online exported

I then stop and start VCS on node B which is when I see the issue:

2014/05/13 12:05:23 VCS INFO V-16-2-13716 (r55v61b) Resource(cvm_clus): Output of the completed operation (online) 
==============================================
ERROR: 
==============================================

2014/05/13 12:05:24 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed

If I destroy fss-dg diskgroup on node A, then CVM will start on node B, so issue is the FSS diskgroup where it seems CVM cannot find the remote disk in the diskgroup

I can also get round issue by stopping VCS on node A and then CVM will start on node B:

[root@r55v61b ~]# hagrp -online cvm -sys r55v61b
[root@r55v61b ~]# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    -            -            online exported

If I then start VCS on node A, then B is able to see the FSS diskgroup:

[root@r55v61b ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
A_sdd        auto:cdsdisk    fd1_La       fss-dg       online shared remote
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    -            -            online exported

I can stop and start VCS on each node when disks are just exported and VCS is able to see disk from other node, but when I create the FSS diskgroup, CVM won't start on the system that has the remote disk - does anybody have any ideas as to why?

Mike

mikebounds
11 years ago
The issue of this post which was CVM would not start if an FSS diskgroup was present, giving error message:

CVMCluster:cvm_clus:monitor:node - state: out of cluster reason: Disk for disk group not found: retry to add a node failed

was resolved by recreating separate diskgroup which was purely CVM (no exported disks). The likely issues was UDID mismatches or conflicts as it would appear with non-FSS failover and CVM diskgroups, all that is required is that VxVM read the private region, but with FSS diskgroups, my theory is the UDID is required to be used to ensure that if you export a disk then it only shows as a remote disk on other systems if the same disk can NOT be seen on the SAN of the remote system and it needs to use the UDID to determine this.

Hence in Virtual box, the same disk will normally show as having different UDID when viewed from different systems, and if this disk is shared then I did indeed see a single disk presented on one server BOTH via the SAN and as a remote disk, but when I made the UDID the same by changing the hostname of one of the nodes so both nodes had the same hostname and hence the same constucted UDID, then VxVM correctly identied the remote disk was available via the SAN and hence ONLY showed the disk as a SAN attached disk and not also a remote disk.

Although in my opening post I was not exporting any shared SAN disks (only local disks), I believe the UDID checking when autoimporting the diskgroups caused the issue.

Mike

21 Replies

novonil_choudhu
Level 4
11 years ago
Can you please paste the o/p of the following commands.

From Node A :

# vxdisk list sdd

# lltstat -nvv|more

# vxprint

From Node B :

# vxdisk list A_sdd

Can you try exporting and then importing the dgs with VCS not in picture :

# hastop -all -force

# vxdg export fss-dg

# vxdg -s -o fss=on import fss-dg

mikebounds

Level 6

11 years ago

Node A:

[root@r55v61a ~]# vxdctl -c mode
mode: enabled: cluster active - SLAVE
master: r55v61b
[root@r55v61a ~]#  vxdisk list sdd
Device:    sdd
devicetag: sdd
type:      auto
clusterid: r55v61c1
disk:      name=fd1_La id=1399977098.91.r55v61a
group:     name=fss-dg id=1399978977.116.r55v61a
info:      format=cdsdisk,privoffset=256,pubslice=3,privslice=3
flags:     online ready private autoconfig exported shared autoimport imported
pubpaths:  block=/dev/vx/dmp/sdd3 char=/dev/vx/rdmp/sdd3
guid:      {c4025770-da89-11e3-a9f1-d5ec014b3593}
udid:      ATA%5FVBOX%20HARDDISK%5FOTHER%5FDISKS%5Fr55v61a.localdomain%5F%2Fdev%2Fsdd
site:      -
version:   3.1
iosize:    min=512 (bytes) max=1024 (blocks)
public:    slice=3 offset=2304 len=517888 disk_offset=0
private:   slice=3 offset=256 len=2048 disk_offset=0
update:    time=1399982408 seqno=0.32
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=1280
logs:      count=1 len=192
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-001343[001088]: copy=01 offset=000192 enabled
 log      priv 001344-001535[000192]: copy=01 offset=000000 enabled
 lockrgn  priv 001536-001679[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
sdd          state=enabled
connectivity: r55v61a 
[root@r55v61a ~]# lltstat -nvv
LLT node information:
    Node                 State    Link  Status  Address
   * 0 r55v61a           OPEN    
                                  eth1   UP      08:00:27:C9:3A:39
                                  eth2   UP      08:00:27:1A:E8:66
     1 r55v61b           OPEN    
                                  eth1   UP      08:00:27:2F:39:51
                                  eth2   UP      08:00:27:15:CD:E8
     2                   CONNWAIT
                                  eth1   DOWN    
                                  eth2   DOWN    
     3                   CONNWAIT
                                  eth1   DOWN    
                                  eth2   DOWN    
[root@r55v61a ~]# vxprint -g fss-dg
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg fss-dg       fss-dg       -        -        -        -        -       -

dm fd1_La       sdd          -        517888   -        -        -       -
[root@r55v61a ~]#

Node B:

[root@r55v61b config]# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: r55v61b
[root@r55v61b config]# vxdisk list A_sdd
Device:    A_sdd
type:      auto
clusterid: r55v61c1
disk:      name=fd1_La id=1399977098.91.r55v61a
group:     name=fss-dg id=1399978977.116.r55v61a
info:      format=cdsdisk,privoffset=256,pubslice=3,privslice=3
flags:     online ready private autoconfig remote exported shared autoimport imported
guid:      {c4025770-da89-11e3-a9f1-d5ec014b3593}
udid:      ATA%5FVBOX%20HARDDISK%5FOTHER%5FDISKS%5Fr55v61a.localdomain%5F%2Fdev%2Fsdd
site:      -
version:   3.1
iosize:    min=512 (bytes) max=1024 (blocks)
public:    slice=3 offset=2304 len=517888 disk_offset=0
private:   slice=3 offset=256 len=2048 disk_offset=0
update:    time=1399982408 seqno=0.32
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=1280
logs:      count=1 len=192
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-001343[001088]: copy=01 offset=000192 enabled
 log      priv 001344-001535[000192]: copy=01 offset=000000 enabled
 lockrgn  priv 001536-001679[000144]: part=00 offset=000000
connectivity: r55v61a 
[root@r55v61b config]# hastop -all -force
[root@r55v61b config]# vxdg deport fss-dg
[root@r55v61b config]# vxdg -s -o fss=on import fss-dg

The import hangs, so after minute I used <ctrl><c> and then I see:

[root@r55v61b config]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
A_sdd        auto:cdsdisk    fd1_La       fss-dg       online shared remote
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    -            -            online exported

[root@r55v61b config]# vxprint -g fss-dg
VxVM vxprint ERROR V-5-1-582 Disk group fss-dg: No such disk group
[root@r55v61b config]# vxdg deport fss-dg
VxVM vxdg ERROR V-5-1-2275 vxdg: Disk group fss-dg: No such disk group
[root@r55v61b config]#

So vxdisk shows fss-dg is imported, but vxprint and vxdg think is is deported.
Node A shows:

[root@r55v61a ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
B_sdd        auto:cdsdisk    -            -            online remote
sda          auto:none       -            -            online invalid
sdb          auto:none       -            -            online invalid
sdc          auto:cdsdisk    -            -            online
sdd          auto:cdsdisk    -            -            online exported

Mike

novonil_choudhu
Level 4
11 years ago
Mike ,

Did you observe any errors / messages in syslog while importing the dg.

What is the CVM protocol and DG version. Please paste the below commands o/ps.

# vxdctl protocolversion

# vxdg list fss-dg
mikebounds
Level 6
11 years ago
In /var/log/messages on node B (there are no logs on node A at time of deport/import) I see:

May 13 16:39:53 r55v61b vxvm:vxconfigd: V-5-1-16252 Disk group deport of fss-dg succeeded.
May 13 16:40:07 r55v61b vxvm:vxconfigd: V-5-1-16765 Selecting configuration database copy from A_sdd from disks: A_sdd
May 13 16:40:07 r55v61b vxvm:vxconfigd: V-5-1-16766 Trying to import the disk group fss-dg using configuration database copy from A_sdd

This is a fresh install of 6.1 with newly create diskgroups so versions are the latest:

[root@r55v61b log]# vxdctl protocolversion Cluster running at protocol 130 [root@r55v61b log]# vxdg list fss-dg Group: fss-dg dgid: 1399978977.116.r55v61a import-id: 33792.135 flags: shared cds version: 190 alignment: 8192 (bytes) local-activation: shared-write cluster-actv-modes: r55v61b=sw r55v61a=sw ssb: on autotagging: on detach-policy: local dg-fail-policy: obsolete ioship: on fss: on storage-sources: r55v61a copies: nconfig=default nlog=default config: seqno=0.1039 permlen=1280 free=1277 templen=2 loglen=192 config disk A_sdd copy 1 len=1280 state=clean online log disk A_sdd copy 1 len=192 [root@r55v61b log]#

I am using fencing in disabled mode and I have a normal cvm diskgroup (I truncated output of vxdisk list) and and before creating FSS diskgroup, CVM would start normally and mount CFS mount on both nodes.

Mike
mikebounds
Level 6
11 years ago
I noticed that after I <ctrl><c> the import, after a while the diskgroup imported so I check the messages log which showed:

May 13 16:39:53 r55v61b vxvm:vxconfigd: V-5-1-16252 Disk group deport of fss-dg succeeded.
May 13 16:40:07 r55v61b vxvm:vxconfigd: V-5-1-16765 Selecting configuration database copy from A_sdd from disks: A_sdd
May 13 16:40:07 r55v61b vxvm:vxconfigd: V-5-1-16766 Trying to import the disk group fss-dg using configuration database copy from A_sdd
May 13 16:41:02 r55v61b Had[624]: VCS CRITICAL V-16-1-50086 Mem usage on r55v61b is 91%
May 13 16:43:02 r55v61b Had[624]: VCS CRITICAL V-16-1-50086 CPU usage on r55v61b is 100%
May 13 16:43:38 r55v61b vxvm:vxconfigd: V-5-1-16254 Disk group import of fss-dg succeeded.

So it is eventually importing and it may be taking so long due to not enough CPU power, but having said that, I have installed a very light-weight O/S without X-windows which runs at 99% idle without VCS running.
With VCS running (just CVM/CFS stuff) the CPU runs at 90% idle
If I online the cvm service group on node B first then it takes 20 seconds for cvm_clus resource to online (this would be importing the other normal diskgroup on just node B) and it also takes 20 seconds for cvm_clus resource to then online on Node A (this would be importing the other normal diskgroup on node A and the fss diskgroup on both systems)

I did an import again on node B and watched the CPU and it was maxed out for nearly 5 mins with vxconfigd taking 90%, so to me this indicates that vxconfigd is doing something wrong as normal operations complete in a reasonable time.

If this does just take excessive CPU to import a diskgroup containing a remote disk, then are there any timeouts I can set for the cvm_clus resource. The only timeouts I can see see are the type OnlineTimeout which is by default 400 seconds and the resource CVMTimeout which is 200 seconds, but the resource seems to be timing out a lot earlier than this:

2014/05/13 19:04:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvm_clus (Owner: Unspecified, Group: cvm) on System r55v61b
2014/05/13 19:04:46 VCS WARNING V-16-20006-1002 (r55v61b) CVMCluster:cvm_clus:online:CVMCluster start failed on this node.
2014/05/13 19:04:47 VCS INFO V-16-2-13716 (r55v61b) Resource(cvm_clus): Output of the completed operation (online)
==============================================
ERROR:
==============================================

2014/05/13 19:04:48 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:05:48 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:06:48 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:06:48 VCS ERROR V-16-2-13066 (r55v61b) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
2014/05/13 19:06:49 VCS INFO V-16-2-13068 (r55v61b) Resource(cvm_clus) - clean completed successfully.
2014/05/13 19:06:50 VCS INFO V-16-2-13072 (r55v61b) Resource(cvm_clus): Agent is retrying online (attempt number 1 of 2).
2014/05/13 19:07:13 VCS WARNING V-16-20006-1002 (r55v61b) CVMCluster:cvm_clus:online:CVMCluster start failed on this node.
2014/05/13 19:07:13 VCS INFO V-16-2-13716 (r55v61b) Resource(cvm_clus): Output of the completed operation (online)
==============================================
ERROR:
==============================================

2014/05/13 19:07:14 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:08:13 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:09:10 VCS INFO V-16-10031-20903 (r55v61b) CFSfsckd:vxfsckd:imf_register:/opt/VRTSamf/bin/amfregister -ipf -ouid=0,euid=0,gid=0,egid=0 -r CFSfsckd -g vxfsckd "/usr/lib/fs/vxfs/vxfsckd" -- "-p /var/adm/cfs/vxfsckd-pid"
2014/05/13 19:09:13 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:09:14 VCS ERROR V-16-2-13066 (r55v61b) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
2014/05/13 19:09:15 VCS INFO V-16-2-13068 (r55v61b) Resource(cvm_clus) - clean completed successfully.
2014/05/13 19:09:15 VCS INFO V-16-2-13072 (r55v61b) Resource(cvm_clus): Agent is retrying online (attempt number 2 of 2).
2014/05/13 19:09:38 VCS WARNING V-16-20006-1002 (r55v61b) CVMCluster:cvm_clus:online:CVMCluster start failed on this node.
2014/05/13 19:09:39 VCS INFO V-16-2-13716 (r55v61b) Resource(cvm_clus): Output of the completed operation (online)
==============================================
ERROR:
==============================================

2014/05/13 19:09:39 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:10:39 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:11:39 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:11:40 VCS ERROR V-16-2-13066 (r55v61b) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
2014/05/13 19:11:41 VCS INFO V-16-2-13068 (r55v61b) Resource(cvm_clus) - clean completed successfully.
2014/05/13 19:11:41 VCS INFO V-16-2-13071 (r55v61b) Resource(cvm_clus): reached OnlineRetryLimit(2).
2014/05/13 19:11:42 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:11:42 VCS ERROR V-16-1-54031 Resource cvm_clus (Owner: Unspecified, Group: cvm) is FAULTED on sys r55v61b

So you can see here that I get a "CVMCluster start failed on this node" error after 23 seconds and then get "node - state: out of cluster
reason: Disk for disk group not found"

repeated at 60 second intervals.

Mike

mikebounds

Level 6

11 years ago

Done some more investigation:

If I try to start CVM manually on node B with FSS diskgroup imported on node A, then I get:

# ./vxclustadm nodestate; ./vxclustadm -m vcs -t gab startnode; while true
> do
> date
> ./vxclustadm nodestate
> sleep 1
> done
state: out of cluster
reason: Disk for disk group not found: user initiated stop
VxVM vxclustadm INFO V-5-2-9687 vxclustadm: Fencing driver is in disabled mode
Tue May 13 20:40:50 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:40:51 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:40:52 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:40:54 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:40:56 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:40:58 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:40:59 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:41:00 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:41:01 BST 2014
state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
Tue May 13 20:41:02 BST 2014
state: out of cluster
reason: Disk for disk group not found: retry to add a node failed

So it is in joining state for just over 10 seconds and then complains "Disk for disk group not found"

If I then destroy FSS diskgroup on node A and rerun manual CVM start again as above I get:

[root@r55v61b bin]# ./vxclustadm nodestate; ./vxclustadm -m vcs -t gab startnode; while true; do date; ./vxclustadm nodestate; sleep 1; done
state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
VxVM vxclustadm INFO V-5-2-9687 vxclustadm: Fencing driver is in disabled mode
Tue May 13 20:46:21 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:46:22 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:46:23 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:46:24 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:46:25 BST 2014
state: joining
reconfig: initialized
Tue May 13 20:46:27 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:46:28 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:46:29 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:46:30 BST 2014
state: joining
reconfig: vxconfigd in join
Tue May 13 20:46:31 BST 2014
state: cluster member
Tue May 13 20:46:33 BST 2014
state: cluster member

So this again is in joining state for about 10 seconds and then successfully joins CVM.

So it seems CVM is quite quickly determining that with an FSS diskgroup it can't see a disk (presumbly the remote disk).

Do you know why this might be happening?

Thanks

Mike

novonil_choudhu
Level 4
11 years ago
Mark,

Please refer to SFHA 6.1 solutions guide http://www.symantec.com/docs/DOC6982 for the FSS limitations. Check whether your configuration comply with FSS requirements. Are your storage scsi3 complaint ?

Here are below limitations :

■ FSS is only supported on clusters of up to 8 nodes.
■ Disk initialization operations should be performed only on nodes with local
connectivity to the disk.
■ FSS does not support the use of boot disks, opaque disks, and non-VxVM disks
for network sharing.
■ Hot-relocation is disabled on FSS disk groups.
■ The vxresize operation is not supported on volumes and file systems from the
slave node.
■ FSS does not support non-SCSI3 disks connected to multiple hosts.
■ Dynamic Lun Expansion (DLE) is not supported.
■ FSS only supports instant data change object (DCO), created using the vxsnap
operationorby specifying"logtype=dcodcoversion=20"attributesduringvolume
creation.
■ By default creating a mirror between SSD and HDD is not supported through
vxassist, as the underlying mediatypes are different. To workaround this issue,
you can create a volume with one mediatype, for instance the HDD, which is
the default mediatype, and then later add a mirror on the SSD.
For example:
# vxassist -g diskgroup make volume size init=none
# vxassist -g diskgroup mirror volume mediatype:ssd
Optimizing storage with Flexible Storage Sharing 135
About Flexible Storage Sharing# vxvol -g diskgroup init active volume
See the "Administering mirrored volumes using vxassist" section in the Symantec
Storage Foundation Cluster File System High Availability Administrator's Guide or
the Symantec Storage Foundation for Oracle RAC Administrator's Guide.

Also check your private interconnects as all the metadata exchanges happen over there in a FSS configuration. I recommend you to check if your llt links and their settings like speed, autoneg , mtu , switch settings etc etc..are desirable.

Please paste the below o/p :

# ifconfig -a

mikebounds

Level 6

11 years ago

Hi Novonil,

My disks are not SCSI3, but they are not "connected to multiple hosts" - I guess this is the point of FSS - you don't need disks in an array - you can use local disks (which probably are not going to be SCSI3).

Just to clarify, as in opening post, I am testing FSS in Virtual box, so the 2 VMs are running on my laptop using virtual networks and virtual disks. So I am not expecting this to be supported as this is not supported from a redundancy poiint of view (my laptop is a SPOF) and a data protection point of view (I am using fencing in disabled mode which gives me no protection against split brain, so in a real environment I should be using I/O fencing or CPS).

FSS was demonstated to me at Vision and this was running on virtual hardware, which I believe was VMWare ESX which I presume was using vmdks which are not SCSI3.

I installed SFCFS HA on Node A and then cloned it to create Node B, so they should be identically configured - below is output of ifconfig -a:
Node A:

[root@r55v61a log]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 08:00:27:7A:FF:1E  
          inet addr:192.168.56.51  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe7a:ff1e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9744 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7611 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:806345 (787.4 KiB)  TX bytes:1783862 (1.7 MiB)

eth0:0    Link encap:Ethernet  HWaddr 08:00:27:7A:FF:1E  
          inet addr:192.168.56.55  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth1      Link encap:Ethernet  HWaddr 08:00:27:C9:3A:39  
          inet6 addr: fe80::a00:27ff:fec9:3a39/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:113984 errors:0 dropped:0 overruns:0 frame:0
          TX packets:121992 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:15288510 (14.5 MiB)  TX bytes:18173014 (17.3 MiB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:1A:E8:66  
          inet6 addr: fe80::a00:27ff:fe1a:e866/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:114602 errors:0 dropped:0 overruns:0 frame:0
          TX packets:121423 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:15335104 (14.6 MiB)  TX bytes:17804072 (16.9 MiB)

eth3      Link encap:Ethernet  HWaddr 08:00:27:B8:54:AE  
          inet addr:192.168.57.51  Bcast:192.168.57.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feb8:54ae/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:720 (720.0 b)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:17 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1796 (1.7 KiB)  TX bytes:1796 (1.7 KiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Node B:

[root@r55v61b bin]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 08:00:27:68:CC:26  
          inet addr:192.168.56.52  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe68:cc26/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:23233 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21826 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1899658 (1.8 MiB)  TX bytes:10908440 (10.4 MiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:2F:39:51  
          inet6 addr: fe80::a00:27ff:fe2f:3951/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:140787 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132047 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:21324780 (20.3 MiB)  TX bytes:19628051 (18.7 MiB)

eth2      Link encap:Ethernet  HWaddr 08:00:27:15:CD:E8  
          inet6 addr: fe80::a00:27ff:fe15:cde8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:140200 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132810 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:21113691 (20.1 MiB)  TX bytes:19811643 (18.8 MiB)

eth3      Link encap:Ethernet  HWaddr 08:00:27:00:F7:01  
          inet addr:192.168.57.52  Bcast:192.168.57.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe00:f701/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:360 (360.0 b)  TX bytes:720 (720.0 b)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:38 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4040 (3.9 KiB)  TX bytes:4040 (3.9 KiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

FSS and CVM are working in terms of I can mount a filesystem on both nodes and write to filesystems from both nodes, - it is just the starting of CVM that is not working for FSS. See below:

Node A:

[root@r55v61a ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              8022104   1531008   6077024  21% /
/dev/sda1               101086     11792     84075  13% /boot
tmpfs                   206120         0    206120   0% /dev/shm
/dev/vx/dsk/fss-dg/fssvol1
                         16384      5488     10478  35% /fss1
/dev/vx/dsk/cvm-dg/volshare1
                          5120      3820      1300  75% /share1
[root@r55v61a ~]# 
[root@r55v61a ~]# echo test > /fss1/created_from_a
[root@r55v61a ~]# echo test > /share1/created_from_a

Node B:

[root@r55v61b /]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              8022104   1534184   6073848  21% /
/dev/sda1               101086     11792     84075  13% /boot
tmpfs                   190632         0    190632   0% /dev/shm
/dev/vx/dsk/fss-dg/fssvol1
                         16384      5488     10478  35% /fss1
/dev/vx/dsk/cvm-dg/volshare1
                          5120      3820      1300  75% /share1
[root@r55v61b /]# echo test > /fss1/created_from_b
[root@r55v61b /]# echo test > /share1/created_from_b
[root@r55v61b /]# ls -l /fss1 /share1
/fss1:
total 2
-rw-r--r-- 1 root root  5 May 13 23:25 created_from_a
-rw-r--r-- 1 root root  5 May 13 23:26 created_from_b
drwxr-xr-x 2 root root 96 May 13 22:55 lost+found

/share1:
total 2
-rw-r--r-- 1 root root  5 May 13 23:25 created_from_a
-rw-r--r-- 1 root root  5 May 13 23:26 created_from_b
drwxr-xr-x 2 root root 96 May 10 14:13 lost+found

Node A:

[root@r55v61a ~]# ls -l /fss1 /share1
/fss1:
total 2
-rw-r--r-- 1 root root  5 May 13 23:25 created_from_a
-rw-r--r-- 1 root root  5 May 13  2014 created_from_b
drwxr-xr-x 2 root root 96 May 13 22:55 lost+found

/share1:
total 2
-rw-r--r-- 1 root root  5 May 13 23:25 created_from_a
-rw-r--r-- 1 root root  5 May 13  2014 created_from_b
drwxr-xr-x 2 root root 96 May 10 14:13 lost+found

Mike

novonil_choudhu
Level 4
11 years ago
Mike,

Please follow this guide for configuring SFHA deployment guide on VMware vmdks. Just to ensure you are not missing anything.

https://www-secure.symantec.com/connect/articles/storage-foundation-cluster-file-system-ha-vmware-vmdk-deployment-guide
mikebounds
Level 6
11 years ago
I have had a look at this, but it is most not applicable because:

I am using Oracle VirtualBox with vmdk, not VMWare ESX and I am not trying to do things like vmotion, so really the Symantec software does not need to be aware it is in a virtual server (as was the case prior 6.0 where SFHA was VMWare unaware)

I do not have an issue with the more complicated writing data from multiple hosts, as FSS is only writing to local disks and actually writing to volumes from multiple hosts is working in regular CVM.

Actually FSS, in terms of wrting to the disks is working too, what is not working is CVM membership - how is this meant to work with FSS. Here is my understanding of what happens when a system tries to join the membership:

In regular CVM, the joining system asks CVM master what disks it has in its shared disk groups and membership is only successful if the joining system see all these disks.

For FSS, with exported disks on the Master, but NO shared diskgroups, when joining systems asks CVM master what disks it has in its shared disk groups, it replies it has no shared diskgroups, so then it joins and ONLY after it joins does it see the exported disks.

For FSS, with exported disks on the Master in a shared diskgroup, when joining systems asks CVM master what disks it has in its shared disk groups, it replies it has a disk (which is an exported disk), but the joining system won't see this disk as it is not a member yet so what appears to be happening is joining system reports it can't see this disk and won't join.

Clearly this can't be how FSS works, but this appears to be what is happening as the joining system IS reporting "Disk for disk group not found", so can you elaborate how the joining process works?

Mike

Forum Discussion

CVM won't start on remote node with an FSS diskgroup

21 Replies

Related Content

Re: How to check VVR replication status

Storage Migration of vvr diskgroup

Re: how to add master in clustered nbu 7.5

Cannot detect remote tape library

Reccuring unmount of Diskgroups

Recent Discussions

DataDomain To StoreOnce Migration

VCS ERROR V-16-1-10205 Group Mediation_NFS is faulted on system com1em03

vxprivutil

URGENT disk oublic region recover

super block corrupt