01-27-2010 02:29 AM
Hi,
I installed and configured sfcfsha (SF5.1) on two SLES11(x86_64) with IO-fence disable mode. Share group and share disks have been configured by vxvm. But after configuration everything, NFS and share resources can't be online.(Disks can be mounted on both nodes).
Can anyone help me? Thanks in advance.
1. Before configuring NFS:
arcx3455gbxv5:/opt/VRTS/bin # /opt/VRTS/bin/cfscluster status
Node : arcx3455frpv8
Cluster Manager : running
CVM state : running
No mount point registered with cluster configuration
Node : arcx3455gbxv5
Cluster Manager : running
CVM state : running
No mount point registered with cluster configuration
*************************************************************
arcx3455gbxv5:/opt/VRTS/bin # vxdg list
NAME STATE ID
rootdg enabled 1263890629.7.arcx3455gbxv5
cfs enabled,shared,cds 1264561164.23.arcx3455gbxv5
qlogic enabled,shared,cds 1264492611.18.arcx3455gbxv5
**************************************************************
arcx3455gbxv5:/opt/VRTS/bin # vxdisk list
DEVICE TYPE DISK GROUP STATUS
san_vc0_1 auto:cdsdisk - - online
san_vc0_2 auto:cdsdisk - - online
san_vc0_3 auto:cdsdisk - - online
san_vc0_4 auto:cdsdisk qlogic01 qlogic online shared
san_vc0_5 auto:cdsdisk qlogic02 qlogic online shared
san_vc0_6 auto:cdsdisk qlogic03 qlogic online shared
san_vc0_7 auto:cdsdisk qlogic04 qlogic online shared
san_vc0_8 auto:cdsdisk cfs01 cfs online shared
sda auto:sliced rootdisk rootdg online
(cfs group and cfs01 disk are for CNFS components)
2. Configure a VCS configuration for CFS/CVM
arcx3455gbxv5:/opt/VRTS/bin # cfscluster config
Warning: V-35-420: CVM Configuration already exists
Warning: V-35-419: Cluster already configured
cfscluster: CFS Cluster Configured Successfully
3.Configure the CNFS components
arcx3455gbxv5:/opt/VRTS/bin # cfsshare config cfs c01 /locks
Mount Point is being added...
/locks added to the cluster-configuration
Mounting...
[/dev/vx/dsk/cfs/c01] mounted successfully at /locks on arcx3455frpv8
[/dev/vx/dsk/cfs/c01] mounted successfully at /locks on arcx3455gbxv5
But warning and errormessages appear in /opt/VRTvcs/log/engineA.log
2010/01/27 03:05:25 VCS NOTICE V-16-1-10301 Initiating Online of Resource nfs (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:05:26 VCS WARNING V-16-10031-7002 (arcx3455frpv8) NFS:nfs:online:Sending SIGKILL signal to nfsd (11174 11175 11176 11177 11178 11179 11180 11181).
2010/01/27 03:05:26 VCS INFO V-16-2-13716 (arcx3455frpv8) Resource(nfs): Output of the completed operation (online)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:07:06 VCS INFO V-16-1-50135 User root fired command: hares -online nfs arcx3455gbxv5 from localhost
2010/01/27 03:07:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455frpv8
2010/01/27 03:07:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455gbxv5
2010/01/27 03:07:06 VCS NOTICE V-16-1-10301 Initiating Online of Resource nfs (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:07:06 VCS WARNING V-16-10031-7002 (arcx3455gbxv5) NFS:nfs:online:Sending SIGKILL signal to nfsd (11626 11627 11628 11629 11630 11631 11632 11633).
2010/01/27 03:07:07 VCS INFO V-16-2-13716 (arcx3455gbxv5) Resource(nfs): Output of the completed operation (online)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:07:28 VCS ERROR V-16-2-13066 (arcx3455frpv8) Agent is calling clean for resource(nfs) because the resource is not up even after online completed.
2010/01/27 03:07:28 VCS WARNING V-16-10031-7002 (arcx3455frpv8) NFS:nfs:clean:Sending SIGKILL signal to nfsd (13067 13068 13069 13070 13071 13072 13073 13074).
2010/01/27 03:07:29 VCS INFO V-16-2-13716 (arcx3455frpv8) Resource(nfs): Output of the completed operation (clean)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:07:29 VCS INFO V-16-2-13068 (arcx3455frpv8) Resource(nfs) - clean completed successfully.
2010/01/27 03:07:29 VCS INFO V-16-2-13071 (arcx3455frpv8) Resource(nfs): reached OnlineRetryLimit(0).
2010/01/27 03:07:29 VCS ERROR V-16-1-10303 Resource nfs (Owner: unknown, Group: cfsnfssg) is FAULTED (timed out) on sys arcx3455frpv8
2010/01/27 03:07:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cfsnfs_locks (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:07:30 VCS INFO V-16-1-10305 Resource cfsnfs_locks (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (VCS initiated)
2010/01/27 03:07:30 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg1 (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:07:32 VCS INFO V-16-1-10305 Resource cvmvoldg1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (VCS initiated)
2010/01/27 03:07:32 VCS ERROR V-16-1-10205 Group cfsnfssg is faulted on system arcx3455frpv8
2010/01/27 03:07:32 VCS NOTICE V-16-1-10446 Group cfsnfssg is offline on system arcx3455frpv8
2010/01/27 03:07:32 VCS INFO V-16-6-0 (arcx3455frpv8) postoffline:Invoked with arg0=arcx3455frpv8, arg1=cfsnfssg
2010/01/27 03:07:32 VCS INFO V-16-6-15002 (arcx3455frpv8) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline arcx3455frpv8 cfsnfssg successfully
2010/01/27 03:09:09 VCS ERROR V-16-2-13066 (arcx3455gbxv5) Agent is calling clean for resource(nfs) because the resource is not up even after online completed.
2010/01/27 03:09:09 VCS WARNING V-16-10031-7002 (arcx3455gbxv5) NFS:nfs:clean:Sending SIGKILL signal to nfsd (15200 15201 15202 15203 15204 15205 15206 15207).
2010/01/27 03:09:10 VCS INFO V-16-2-13716 (arcx3455gbxv5) Resource(nfs): Output of the completed operation (clean)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:09:10 VCS INFO V-16-2-13068 (arcx3455gbxv5) Resource(nfs) - clean completed successfully.
2010/01/27 03:09:10 VCS INFO V-16-2-13071 (arcx3455gbxv5) Resource(nfs): reached OnlineRetryLimit(0).
2010/01/27 03:09:10 VCS ERROR V-16-1-10303 Resource nfs (Owner: unknown, Group: cfsnfssg) is FAULTED (timed out) on sys arcx3455gbxv5
2010/01/27 03:09:10 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cfsnfs_locks (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:09:11 VCS INFO V-16-1-10305 Resource cfsnfs_locks (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (VCS initiated)
2010/01/27 03:09:11 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg1 (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:09:14 VCS INFO V-16-1-10305 Resource cvmvoldg1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (VCS initiated)
2010/01/27 03:09:14 VCS ERROR V-16-1-10205 Group cfsnfssg is faulted on system arcx3455gbxv5
2010/01/27 03:09:14 VCS NOTICE V-16-1-10446 Group cfsnfssg is offline on system arcx3455gbxv5
2010/01/27 03:09:14 VCS INFO V-16-6-0 (arcx3455gbxv5) postoffline:Invoked with arg0=arcx3455gbxv5, arg1=cfsnfssg
2010/01/27 03:09:14 VCS INFO V-16-6-15002 (arcx3455gbxv5) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline arcx3455gbxv5 cfsnfssg successfully
********************************************
arcx3455gbxv5:/opt/VRTS/bin # /opt/VRTS/bin/cfscluster status
Node : arcx3455frpv8
Cluster Manager : running
CVM state : running
MOUNT POINT SHARED VOLUME DISK GROUP STATUS
/locks c01 cfs NOT MOUNTED
Node : arcx3455gbxv5
Cluster Manager : running
CVM state : running
MOUNT POINT SHARED VOLUME DISK GROUP STATUS
/locks c01 cfs NOT MOUNTED
************************************************************************
4. Add and mount the NFS shared CFS file system to the VCS configuration
arcx3455gbxv5:/opt/VRTS/bin # cfsshare add qlogic q01 /mnt/q01 rw,no_root_squash all=rw
Mount Point is being added...
/mnt/q01 added to the cluster-configuration
Mounting...
[/dev/vx/dsk/qlogic/q01] mounted successfully at /mnt/q01 on arcx3455frpv8
[/dev/vx/dsk/qlogic/q01] mounted successfully at /mnt/q01 on arcx3455gbxv5
Warning: V-35-465: Resource [nfs] is not online on system [arcx3455frpv8].
Warning: V-35-465: Resource [share1] is not online on system [arcx3455frpv8].
Warning: V-35-465: Resource [nfs] is not online on system [arcx3455gbxv5].
Warning: V-35-465: Resource [share1] is not online on system [arcx3455gbxv5].
EngineA.log will be added later.
5.arcx3455gbxv5:/opt/VRTS/bin # /opt/VRTS/bin/cfscluster status
Node : arcx3455frpv8
Cluster Manager : running
CVM state : running
MOUNT POINT SHARED VOLUME DISK GROUP STATUS
/mnt/q01 q01 qlogic NOT MOUNTED
/locks c01 cfs NOT MOUNTED
Node : arcx3455gbxv5
Cluster Manager : running
CVM state : running
MOUNT POINT SHARED VOLUME DISK GROUP STATUS
/mnt/q01 q01 qlogic NOT MOUNTED
/locks c01 cfs NOT MOUNTED
arcx3455gbxv5:/opt/VRTS/bin # cfsshare display
SHARE RESOURCE MOUNTPOINT SHARE OPTIONS
share1 /mnt/q01 rw,no_root_squash
**************************************************************************
6. arcx3455gbxv5:/opt/VRTS/bin # cat /etc/VRTSvcs/conf/config/main.cf
include "OracleASMTypes.cf"
include "types.cf"
include "ApplicationNone.cf"
include "CFSTypes.cf"
include "CVMTypes.cf"
include "Db2udbTypes.cf"
include "OracleTypes.cf"
include "SybaseTypes.cf"
cluster dqq (
UserNames = { admin = aHHpIBgGHoEChEIjIE }
ClusterAddress = "9.11.111.66"
Administrators = { admin }
HacliUserLevel = COMMANDROOT
)
system arcx3455frpv8 (
)
system arcx3455gbxv5 (
)
group ClusterService (
SystemList = { arcx3455gbxv5 = 0, arcx3455frpv8 = 1 }
AutoStartList = { arcx3455gbxv5, arcx3455frpv8 }
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)
IP webip (
Device = eth0
Address = "9.11.111.66"
NetMask = "255.255.254.0"
)
NIC csgnic (
Device = eth0
)
webip requires csgnic
// resource dependency tree
//
// group ClusterService
// {
// IP webip
// {
// NIC csgnic
// }
// }
group cfsnfssg (
SystemList = { arcx3455frpv8 = 0, arcx3455gbxv5 = 1 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { arcx3455frpv8, arcx3455gbxv5 }
)
ApplicationNone app (
MonitorProgram = "/opt/VRTSvcs/bin/ApplicationNone/lockdstatdmon"
)
CFSMount cfsmount1 (
Critical = 0
MountPoint = "/mnt/q01"
BlockDevice = "/dev/vx/dsk/qlogic/q01"
MountOpt @arcx3455frpv8 = rw
MountOpt @arcx3455gbxv5 = rw
NodeList = { arcx3455frpv8, arcx3455gbxv5 }
)
CFSMount cfsnfs_locks (
Critical = 0
MountPoint = "/locks"
BlockDevice = "/dev/vx/dsk/cfs/c01"
NodeList = { arcx3455frpv8, arcx3455gbxv5 }
)
CVMVolDg cvmvoldg1 (
Critical = 0
CVMDiskGroup = cfs
CVMVolume = { c01 }
CVMActivation @arcx3455frpv8 = sw
CVMActivation @arcx3455gbxv5 = sw
)
CVMVolDg cvmvoldg2 (
Critical = 0
CVMDiskGroup = qlogic
CVMVolume = { q01 }
CVMActivation @arcx3455frpv8 = sw
CVMActivation @arcx3455gbxv5 = sw
)
NFS nfs (
)
Share share1 (
PathName = "/mnt/q01"
Options = "rw,no_root_squash"
NFSRes = nfs
)
requires group cvm online local firm
cfsmount1 requires cvmvoldg2
cfsnfs_locks requires cvmvoldg1
share1 requires cfsmount1
share1 requires nfs
// resource dependency tree
//
// group cfsnfssg
// {
// ApplicationNone app
// CFSMount cfsnfs_locks
// {
// CVMVolDg cvmvoldg1
// }
// Share share1
// {
// NFS nfs
// CFSMount cfsmount1
// {
// CVMVolDg cvmvoldg2
// }
// }
// }
group cfsnfssg_dummy (
SystemList = { arcx3455frpv8 = 0, arcx3455gbxv5 = 1 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { arcx3455frpv8, arcx3455gbxv5 }
)
requires group cvm online local firm
// resource dependency tree
//
// group cfsnfssg_dummy
// {
// }
group cvm (
SystemList = { arcx3455frpv8 = 0, arcx3455gbxv5 = 1 }
AutoFailOver = 0
Parallel = 1
AutoStartList = { arcx3455frpv8, arcx3455gbxv5 }
)
CFSfsckd vxfsckd (
ActivationMode @arcx3455frpv8 = { cfs = sw, qlogic = sw }
ActivationMode @arcx3455gbxv5 = { cfs = sw, qlogic = sw }
)
CVMCluster cvm_clus (
CVMClustName = dqq
CVMNodeId = { arcx3455frpv8 = 1, arcx3455gbxv5 = 0 }
CVMTransport = gab
CVMTimeout = 200
)
CVMVxconfigd cvm_vxconfigd (
Critical = 0
CVMVxconfigdArgs = { syslog }
)
cvm_clus requires cvm_vxconfigd
vxfsckd requires cvm_clus
// resource dependency tree
//
// group cvm
// {
// CFSfsckd vxfsckd
// {
// CVMCluster cvm_clus
// {
// CVMVxconfigd cvm_vxconfigd
// }
// }
// }
01-27-2010 02:31 AM
engine_A.log when add and mount the NFS shared CFS file system to the VCS configuration
arcx3455gbxv5:/var/VRTSvcs/log # tail -f engine_A.log
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -add cvmvoldg2 CVMVolDg cfsnfssg from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -modify cvmvoldg2 CVMDiskGroup qlogic from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -modify ... -add cvmvoldg2 CVMVolume q01 from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -local vxfsckd ActivationMode from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -local cvmvoldg2 CVMActivation from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -modify cvmvoldg2 CVMActivation sw arcx3455frpv8 from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -modify cvmvoldg2 CVMActivation sw arcx3455gbxv5 from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -modify cvmvoldg2 Critical 0 from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: hares -modify cvmvoldg2 Enabled 1 from localhost
2010/01/27 03:14:26 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -add cfsmount1 CFSMount cfsnfssg from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 MountPoint /mnt/q01 from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 BlockDevice /dev/vx/dsk/qlogic/q01 from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -local cfsmount1 MountOpt from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -local cfsmount1 RemountRes from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -local cfsmount1 ForceOff from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -local cfsmount1 SetPrimary from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 MountOpt rw arcx3455frpv8 from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 NodeList arcx3455frpv8 from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 Critical 0 from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -link cfsmount1 cvmvoldg2 0 0 from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 Enabled 1 from localhost
2010/01/27 03:14:27 VCS WARNING V-16-10031-1087 (arcx3455frpv8) CVMVolDg:cvmvoldg2:attr_changed:Resource cvmvoldg2 is not ONLINE on this node
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2010/01/27 03:14:27 VCS INFO V-16-1-50135 User root fired command: hares -local cfsmount1 RemountRes from localhost
2010/01/27 03:14:28 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 RemountRes DONE arcx3455gbxv5 from localhost
2010/01/27 03:14:28 VCS WARNING V-16-10031-1087 (arcx3455gbxv5) CVMVolDg:cvmvoldg2:attr_changed:Resource cvmvoldg2 is not ONLINE on this node
2010/01/27 03:14:28 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2010/01/27 03:14:28 VCS INFO V-16-1-50135 User root fired command: hares -local cfsmount1 MountOpt from localhost
2010/01/27 03:14:28 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 MountOpt rw arcx3455gbxv5 from localhost
2010/01/27 03:14:28 VCS INFO V-16-1-50135 User root fired command: hares -modify ... -add cfsmount1 NodeList arcx3455gbxv5 from localhost
2010/01/27 03:14:28 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2010/01/27 03:14:29 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:14:29 VCS INFO V-16-1-50135 User root fired command: MSG_RES_PROBE cfsmount1 arcx3455gbxv5 from localhost
2010/01/27 03:14:29 VCS WARNING V-16-10031-1087 (arcx3455gbxv5) CVMVolDg:cvmvoldg2:attr_changed:Resource cvmvoldg2 is not ONLINE on this node
2010/01/27 03:14:29 VCS WARNING V-16-10031-1087 (arcx3455frpv8) CVMVolDg:cvmvoldg2:attr_changed:Resource cvmvoldg2 is not ONLINE on this node
2010/01/27 03:14:31 VCS INFO V-16-1-10304 Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (First probe)
2010/01/27 03:14:31 VCS INFO V-16-1-10304 Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (First probe)
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -clear cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -online cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -clear cfsmount1 arcx3455gbxv5 from localhost
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -online cfsmount1 arcx3455gbxv5 from localhost
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -online cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg2 arcx3455frpv8 from localhost
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -local cfsmount1 RemountRes from localhost
2010/01/27 03:14:33 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 RemountRes DONE arcx3455gbxv5 from localhost
2010/01/27 03:14:35 VCS INFO V-16-1-50135 User root fired command: hares -online cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:14:35 VCS INFO V-16-1-10304 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (First probe)
2010/01/27 03:14:35 VCS INFO V-16-1-10306 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (Previous State = OFFLINE)
2010/01/27 03:14:35 VCS INFO V-16-1-10304 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (First probe)
2010/01/27 03:14:35 VCS INFO V-16-1-10306 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (Previous State = OFFLINE)
2010/01/27 03:14:37 VCS INFO V-16-1-50135 User root fired command: hares -online cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:14:37 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455frpv8
2010/01/27 03:14:37 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455gbxv5
2010/01/27 03:14:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:14:39 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg2 arcx3455frpv8 from localhost
2010/01/27 03:14:40 VCS INFO V-16-10031-1046 (arcx3455frpv8) CVMVolDg:cvmvoldg2:online:resource cvmvoldg2 is online
2010/01/27 03:15:11 VCS INFO V-16-1-10298 Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) is online on arcx3455frpv8 (VCS initiated)
2010/01/27 03:15:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:15:12 VCS INFO V-16-20011-5506 (arcx3455frpv8) CFSMount:cfsmount1:online:CFSMOUNT Onlined :MountOptions : -t vxfs -o cluster,rw,mntlock=VCS Block Device /dev/vx/dsk/qlogic/q01 : MountPoint /mnt/q01
2010/01/27 03:15:13 VCS INFO V-16-1-10298 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is online on arcx3455frpv8 (VCS initiated)
2010/01/27 03:15:16 VCS INFO V-16-1-50135 User root fired command: hares -online cfsmount1 arcx3455gbxv5 from localhost
2010/01/27 03:15:16 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455frpv8
2010/01/27 03:15:16 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455gbxv5
2010/01/27 03:15:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:15:19 VCS INFO V-16-10031-1046 (arcx3455gbxv5) CVMVolDg:cvmvoldg2:online:resource cvmvoldg2 is online
2010/01/27 03:15:21 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg2 arcx3455gbxv5 from localhost
2010/01/27 03:15:51 VCS INFO V-16-1-10298 Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) is online on arcx3455gbxv5 (VCS initiated)
2010/01/27 03:15:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:15:51 VCS INFO V-16-20011-5506 (arcx3455gbxv5) CFSMount:cfsmount1:online:CFSMOUNT Onlined :MountOptions : -t vxfs -o cluster,rw,mntlock=VCS Block Device /dev/vx/dsk/qlogic/q01 : MountPoint /mnt/q01
2010/01/27 03:15:52 VCS INFO V-16-1-10298 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is online on arcx3455gbxv5 (VCS initiated)
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: haconf -makerw from localhost
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: hares -add share1 Share cfsnfssg from localhost
2010/01/27 03:15:53 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Share/ShareAgent for resource type Share successfully started at Wed Jan 27 03:15:53 2010
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: hares -modify share1 PathName /mnt/q01 from localhost
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: hares -modify share1 Options rw,no_root_squash from localhost
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: hares -modify share1 NFSRes nfs from localhost
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: hares -modify share1 Enabled 1 from localhost
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: hares -link share1 nfs 0 0 from localhost
2010/01/27 03:15:53 VCS INFO V-16-1-10304 Resource share1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (First probe)
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: hares -link share1 cfsmount1 0 0 from localhost
2010/01/27 03:15:53 VCS INFO V-16-1-10304 Resource share1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (First probe)
2010/01/27 03:15:53 VCS INFO V-16-1-50135 User root fired command: haconf -dump -makero from localhost
2010/01/27 03:15:54 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455frpv8 from localhost
2010/01/27 03:15:54 VCS INFO V-16-1-50135 User root fired command: hares -clear cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:15:54 VCS INFO V-16-1-50135 User root fired command: hares -clear nfs arcx3455frpv8 from localhost
2010/01/27 03:15:54 VCS INFO V-16-1-10307 Resource nfs (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (Not initiated by VCS)
2010/01/27 03:15:54 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg2 arcx3455frpv8 from localhost
2010/01/27 03:15:56 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455frpv8 from localhost
2010/01/27 03:15:56 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455frpv8
2010/01/27 03:15:56 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455gbxv5
2010/01/27 03:15:56 VCS NOTICE V-16-1-10301 Initiating Online of Resource nfs (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:15:56 VCS WARNING V-16-10031-7002 (arcx3455frpv8) NFS:nfs:online:Sending SIGKILL signal to nfsd (13419 13420 13421 13422 13423 13424 13425 13426).
2010/01/27 03:15:57 VCS INFO V-16-2-13716 (arcx3455frpv8) Resource(nfs): Output of the completed operation (online)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:15:58 VCS INFO V-16-1-50135 User root fired command: hares -clear cfsmount1 arcx3455frpv8 from localhost
2010/01/27 03:15:58 VCS INFO V-16-1-50135 User root fired command: hares -clear nfs arcx3455frpv8 from localhost
2010/01/27 03:18:00 VCS ERROR V-16-2-13066 (arcx3455frpv8) Agent is calling clean for resource(nfs) because the resource is not up even after online completed.
2010/01/27 03:18:00 VCS WARNING V-16-10031-7002 (arcx3455frpv8) NFS:nfs:clean:Sending SIGKILL signal to nfsd (14906 14907 14908 14909 14910 14911 14912 14913).
2010/01/27 03:18:01 VCS INFO V-16-2-13716 (arcx3455frpv8) Resource(nfs): Output of the completed operation (clean)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:18:01 VCS INFO V-16-2-13068 (arcx3455frpv8) Resource(nfs) - clean completed successfully.
2010/01/27 03:18:01 VCS INFO V-16-2-13071 (arcx3455frpv8) Resource(nfs): reached OnlineRetryLimit(0).
2010/01/27 03:18:01 VCS ERROR V-16-1-10303 Resource nfs (Owner: unknown, Group: cfsnfssg) is FAULTED (timed out) on sys arcx3455frpv8
2010/01/27 03:18:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:18:02 VCS INFO V-16-1-10305 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (VCS initiated)
2010/01/27 03:18:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) on System arcx3455frpv8
2010/01/27 03:18:04 VCS INFO V-16-1-10305 Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455frpv8 (VCS initiated)
2010/01/27 03:18:04 VCS ERROR V-16-1-10205 Group cfsnfssg is faulted on system arcx3455frpv8
2010/01/27 03:18:04 VCS NOTICE V-16-1-10446 Group cfsnfssg is offline on system arcx3455frpv8
2010/01/27 03:18:04 VCS INFO V-16-6-0 (arcx3455frpv8) postoffline:Invoked with arg0=arcx3455frpv8, arg1=cfsnfssg
2010/01/27 03:18:04 VCS INFO V-16-6-15002 (arcx3455frpv8) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline arcx3455frpv8 cfsnfssg successfully
2010/01/27 03:18:05 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455frpv8 from localhost
2010/01/27 03:18:07 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455frpv8 from localhost
.....
2010/01/27 03:18:52 VCS INFO V-16-1-50135 User root fired command: hares -modify cfsmount1 Primary arcx3455gbxv5 from localhost
.....
2010/01/27 03:18:52 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455frpv8 from localhost
2010/01/27 03:19:40 VCS INFO V-16-1-50135 User root fired command: hares -clear cfsmount1 arcx3455gbxv5 from localhost
2010/01/27 03:19:40 VCS INFO V-16-1-50135 User root fired command: hares -clear nfs arcx3455gbxv5 from localhost
2010/01/27 03:19:40 VCS INFO V-16-1-10307 Resource nfs (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (Not initiated by VCS)
2010/01/27 03:19:40 VCS INFO V-16-1-50135 User root fired command: hares -clear cvmvoldg2 arcx3455gbxv5 from localhost
2010/01/27 03:19:42 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455gbxv5 from localhost
2010/01/27 03:19:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455frpv8
2010/01/27 03:19:42 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cfsnfssg on node arcx3455gbxv5
2010/01/27 03:19:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource nfs (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:19:43 VCS WARNING V-16-10031-7002 (arcx3455gbxv5) NFS:nfs:online:Sending SIGKILL signal to nfsd (15669 15670 15671 15672 15673 15674 15675 15676).
2010/01/27 03:19:43 VCS INFO V-16-2-13716 (arcx3455gbxv5) Resource(nfs): Output of the completed operation (online)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:19:44 VCS INFO V-16-1-50135 User root fired command: hares -clear cfsmount1 arcx3455gbxv5 from localhost
2010/01/27 03:19:44 VCS INFO V-16-1-50135 User root fired command: hares -clear nfs arcx3455gbxv5 from localhost
2010/01/27 03:21:45 VCS ERROR V-16-2-13066 (arcx3455gbxv5) Agent is calling clean for resource(nfs) because the resource is not up even after online completed.
2010/01/27 03:21:45 VCS WARNING V-16-10031-7002 (arcx3455gbxv5) NFS:nfs:clean:Sending SIGKILL signal to nfsd (18915 18916 18917 18918 18919 18920 18921 18922).
2010/01/27 03:21:46 VCS INFO V-16-2-13716 (arcx3455gbxv5) Resource(nfs): Output of the completed operation (clean)
==============================================
mount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
fs.nfs.nlm_grace_period = 90
==============================================
2010/01/27 03:21:46 VCS INFO V-16-2-13068 (arcx3455gbxv5) Resource(nfs) - clean completed successfully.
2010/01/27 03:21:46 VCS INFO V-16-2-13071 (arcx3455gbxv5) Resource(nfs): reached OnlineRetryLimit(0).
2010/01/27 03:21:46 VCS ERROR V-16-1-10303 Resource nfs (Owner: unknown, Group: cfsnfssg) is FAULTED (timed out) on sys arcx3455gbxv5
2010/01/27 03:21:46 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:21:47 VCS INFO V-16-1-10305 Resource cfsmount1 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (VCS initiated)
2010/01/27 03:21:47 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) on System arcx3455gbxv5
2010/01/27 03:21:50 VCS INFO V-16-1-10305 Resource cvmvoldg2 (Owner: unknown, Group: cfsnfssg) is offline on arcx3455gbxv5 (VCS initiated)
2010/01/27 03:21:50 VCS ERROR V-16-1-10205 Group cfsnfssg is faulted on system arcx3455gbxv5
2010/01/27 03:21:50 VCS NOTICE V-16-1-10446 Group cfsnfssg is offline on system arcx3455gbxv5
2010/01/27 03:21:50 VCS INFO V-16-6-0 (arcx3455gbxv5) postoffline:Invoked with arg0=arcx3455gbxv5, arg1=cfsnfssg
2010/01/27 03:21:50 VCS INFO V-16-6-15002 (arcx3455gbxv5) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline arcx3455gbxv5 cfsnfssg successfully
2010/01/27 03:21:52 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455gbxv5 from localhost
2010/01/27 03:21:54 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455gbxv5 from localhost
.....
2010/01/27 03:23:27 VCS INFO V-16-1-50135 User root fired command: hares -online share1 arcx3455gbxv5 from localhost
01-27-2010 11:25 PM
01-28-2010 02:23 AM
01-28-2010 11:38 PM
02-03-2010 11:34 PM