I noticed that after I <ctrl><c> the import, after a while the diskgroup imported so I check the messages log which showed:
May 13 16:39:53 r55v61b vxvm:vxconfigd: V-5-1-16252 Disk group deport of fss-dg succeeded.
May 13 16:40:07 r55v61b vxvm:vxconfigd: V-5-1-16765 Selecting configuration database copy from A_sdd from disks: A_sdd
May 13 16:40:07 r55v61b vxvm:vxconfigd: V-5-1-16766 Trying to import the disk group fss-dg using configuration database copy from A_sdd
May 13 16:41:02 r55v61b Had[624]: VCS CRITICAL V-16-1-50086 Mem usage on r55v61b is 91%
May 13 16:43:02 r55v61b Had[624]: VCS CRITICAL V-16-1-50086 CPU usage on r55v61b is 100%
May 13 16:43:38 r55v61b vxvm:vxconfigd: V-5-1-16254 Disk group import of fss-dg succeeded.
So it is eventually importing and it may be taking so long due to not enough CPU power, but having said that, I have installed a very light-weight O/S without X-windows which runs at 99% idle without VCS running.
With VCS running (just CVM/CFS stuff) the CPU runs at 90% idle
If I online the cvm service group on node B first then it takes 20 seconds for cvm_clus resource to online (this would be importing the other normal diskgroup on just node B) and it also takes 20 seconds for cvm_clus resource to then online on Node A (this would be importing the other normal diskgroup on node A and the fss diskgroup on both systems)
I did an import again on node B and watched the CPU and it was maxed out for nearly 5 mins with vxconfigd taking 90%, so to me this indicates that vxconfigd is doing something wrong as normal operations complete in a reasonable time.
If this does just take excessive CPU to import a diskgroup containing a remote disk, then are there any timeouts I can set for the cvm_clus resource. The only timeouts I can see see are the type OnlineTimeout which is by default 400 seconds and the resource CVMTimeout which is 200 seconds, but the resource seems to be timing out a lot earlier than this:
2014/05/13 19:04:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvm_clus (Owner: Unspecified, Group: cvm) on System r55v61b
2014/05/13 19:04:46 VCS WARNING V-16-20006-1002 (r55v61b) CVMCluster:cvm_clus:online:CVMCluster start failed on this node.
2014/05/13 19:04:47 VCS INFO V-16-2-13716 (r55v61b) Resource(cvm_clus): Output of the completed operation (online)
==============================================
ERROR:
==============================================
2014/05/13 19:04:48 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:05:48 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:06:48 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:06:48 VCS ERROR V-16-2-13066 (r55v61b) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
2014/05/13 19:06:49 VCS INFO V-16-2-13068 (r55v61b) Resource(cvm_clus) - clean completed successfully.
2014/05/13 19:06:50 VCS INFO V-16-2-13072 (r55v61b) Resource(cvm_clus): Agent is retrying online (attempt number 1 of 2).
2014/05/13 19:07:13 VCS WARNING V-16-20006-1002 (r55v61b) CVMCluster:cvm_clus:online:CVMCluster start failed on this node.
2014/05/13 19:07:13 VCS INFO V-16-2-13716 (r55v61b) Resource(cvm_clus): Output of the completed operation (online)
==============================================
ERROR:
==============================================
2014/05/13 19:07:14 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:08:13 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:09:10 VCS INFO V-16-10031-20903 (r55v61b) CFSfsckd:vxfsckd:imf_register:/opt/VRTSamf/bin/amfregister -ipf -ouid=0,euid=0,gid=0,egid=0 -r CFSfsckd -g vxfsckd "/usr/lib/fs/vxfs/vxfsckd" -- "-p /var/adm/cfs/vxfsckd-pid"
2014/05/13 19:09:13 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:09:14 VCS ERROR V-16-2-13066 (r55v61b) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
2014/05/13 19:09:15 VCS INFO V-16-2-13068 (r55v61b) Resource(cvm_clus) - clean completed successfully.
2014/05/13 19:09:15 VCS INFO V-16-2-13072 (r55v61b) Resource(cvm_clus): Agent is retrying online (attempt number 2 of 2).
2014/05/13 19:09:38 VCS WARNING V-16-20006-1002 (r55v61b) CVMCluster:cvm_clus:online:CVMCluster start failed on this node.
2014/05/13 19:09:39 VCS INFO V-16-2-13716 (r55v61b) Resource(cvm_clus): Output of the completed operation (online)
==============================================
ERROR:
==============================================
2014/05/13 19:09:39 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:10:39 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:11:39 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:11:40 VCS ERROR V-16-2-13066 (r55v61b) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
2014/05/13 19:11:41 VCS INFO V-16-2-13068 (r55v61b) Resource(cvm_clus) - clean completed successfully.
2014/05/13 19:11:41 VCS INFO V-16-2-13071 (r55v61b) Resource(cvm_clus): reached OnlineRetryLimit(2).
2014/05/13 19:11:42 VCS ERROR V-16-20006-1005 (r55v61b) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: Disk for disk group not found: retry to add a node failed
2014/05/13 19:11:42 VCS ERROR V-16-1-54031 Resource cvm_clus (Owner: Unspecified, Group: cvm) is FAULTED on sys r55v61b
So you can see here that I get a "CVMCluster start failed on this node" error after 23 seconds and then get "node - state: out of cluster
reason: Disk for disk group not found"
repeated at 60 second intervals.
Mike