01-16-2016 03:03 PM
(Please excuse the kludged service group names ) I think I'm doing something wrong with NFSRestart Trying to run two NFS shares, one on each node. NFS agent fails to start nfsd or is very delayed
# showmount -e rpc mount export: RPC: Unable to receive; errno = Connection refused 2016/01/16 14:46:17 VCS INFO V-16-1-10298 Resource dCVS_NFSRestart
(Owner: unknown, Group: dCVS_Group) is online on cm-cvs-22 (VCS initiated) 2016/01/16 14:49:02 VCS ERROR V-16-2-13067 (cm-cvs-21) Agent is calling clean for resource(dCVS_NFS) because the resource became OFFLINE unexpectedly, on its own. 2016/01/16 14:49:03 VCS INFO V-16-2-13716 (cm-cvs-21) Resource(dCVS_NFS): Output of the completed operation (clean) ============================================== mount: nfsd already mounted or /proc/fs/nfsd busy mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd fs.nfs.nlm_grace_period = 90 ============================================== 2016/01/16 14:49:03 VCS INFO V-16-2-13068 (cm-cvs-21) Resource(dCVS_NFS) - clean completed successfully. 2016/01/16 14:49:03 VCS ERROR V-16-2-13073 (cm-cvs-21) Resource(dCVS_NFS) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 1) the resource. 2016/01/16 14:49:04 VCS WARNING V-16-10031-7002 (cm-cvs-21) NFS:dCVS_NFS:online:Sending SIGKILL signal to nfsd (9674 9675 9676 9677 9678 9679 9680 9681 9682 9683 9684 9685 9686 9687 9688 9689). 2016/01/16 14:49:05 VCS INFO V-16-2-13716 (cm-cvs-21) Resource(dCVS_NFS): Output of the completed operation (online) #############main.cf################ include "types.cf" cluster cvsnew ( UserNames = { admin = hello } Administrators = { admin } ) system cm-cvs-21 ( ) system cm-cvs-22 ( ) group ClusterService ( SystemList = { cm-cvs-21 = 0, cm-cvs-22 = 1 } AutoStartList = { cm-cvs-21, cm-cvs-22 } OnlineRetryLimit = 3 OnlineRetryInterval = 120 ) IP webip ( Device = eth0 Address = "10.x.243.35" NetMask = "255.255.252.0" ) NIC csgnic ( Device = eth0 ) NotifierMngr ntfr ( SmtpServer = "mailhost.blah.blah.blah.com" SmtpRecipients = { root = Warning } ) ntfr requires csgnic webip requires csgnic // resource dependency tree // // group ClusterService // { // NotifierMngr ntfr // { // NIC csgnic // } // IP webip // { // NIC csgnic // } // } group dCVS_Group ( SystemList = { cm-cvs-22 = 0, cm-cvs-21 = 1 } AutoStartList = { cm-cvs-22, cm-cvs-21 } PreOnline @cm-cvs-22 = 1 PreOnline @cm-cvs-21 = 1 ) Apache dCVS_Apache ( Enabled = 0 httpdDir = "/usr/sbin" HostName = cvs ConfigFile = "/etc/httpd/conf/httpd.conf" ) DiskGroup dCVS_cvsdg ( DiskGroup = cvsdg01- ) IP dCVS_IP ( Device = eth0 Address = "10.x.243.79" NetMask = "255.255.252.0" ) Mount dCVS_Mount01 ( MountPoint = "/u/cvsdg01" BlockDevice = "/dev/vx/dsk/cvsdg01-/vol01" FSType = vxfs MountOpt = "ioerror=disable" FsckOpt = "-y" ) NFSRestart dCVS_NFSRestart ( NFSRes = dCVS_NFS ) NIC dCVS_NIC ( Device = eth0 ) Proxy sProxy_NFS ( TargetResName = dCVS_NFS ) Share dCVS_Share01 ( PathName = "/u/cvsdg01" Client = "*" Options = "rw,sync,insecure,no_root_squash" NFSRes = dCVS_NFS ) Volume dCVS_vol01 ( DiskGroup = cvsdg01- Volume = vol01 ) dCVS_IP requires dCVS_NIC dCVS_IP requires dCVS_Share01 dCVS_Mount01 requires dCVS_vol01 dCVS_NFSRestart requires dCVS_IP dCVS_NFSRestart requires dCVS_Mount01 dCVS_Share01 requires dCVS_Mount01 dCVS_Share01 requires sProxy_NFS dCVS_vol01 requires dCVS_cvsdg // resource dependency tree // // group dCVS_Group // { // Apache dCVS_Apache // NFSRestart dCVS_NFSRestart // { // Mount dCVS_Mount01 // { // Volume dCVS_vol01 // { // DiskGroup dCVS_cvsdg // } // } // IP dCVS_IP // { // Share dCVS_Share01 // { // Proxy sProxy_NFS // Mount dCVS_Mount01 // { // Volume dCVS_vol01 // { // DiskGroup dCVS_cvsdg // } // } // } // NIC dCVS_NIC // } // } // } group dCVS_Group-02 ( SystemList = { cm-cvs-22 = 0, cm-cvs-21 = 1 } AutoStartList = { cm-cvs-22, cm-cvs-21 } PreOnline @cm-cvs-22 = 1 PreOnline @cm-cvs-21 = 1 ) DiskGroup dCVS_cvsdg02 ( DiskGroup = cvsdg02- ) IP dCVS_IP02 ( Device = eth0 Address = "10.x.243.221" NetMask = "255.255.252.0" ) Mount dCVS_Mount0102 ( MountPoint = "/u/cvsdg02" BlockDevice = "/dev/vx/dsk/cvsdg02-/vol01" FSType = vxfs MountOpt = "ioerror=disable" FsckOpt = "-y" ) NFSRestart dCVS_NFSRestart02 ( NFSRes = dCVS_NFS ) NIC dCVS_NIC02 ( Device = eth0 ) Proxy sProxy_NFS02 ( TargetResName = dCVS_NFS ) Share dCVS_Share0102 ( PathName = "/u/cvsdg02" Client = "*" Options = "rw,sync,insecure,no_root_squash" NFSRes = dCVS_NFS ) Volume dCVS_vol0102 ( DiskGroup = cvsdg02- Volume = vol01 ) dCVS_IP02 requires dCVS_NIC02 dCVS_IP02 requires dCVS_Share0102 dCVS_Mount0102 requires dCVS_vol0102 dCVS_NFSRestart02 requires dCVS_IP02 dCVS_NFSRestart02 requires dCVS_Mount0102 dCVS_Share0102 requires dCVS_Mount0102 dCVS_Share0102 requires sProxy_NFS02 dCVS_vol0102 requires dCVS_cvsdg02 // resource dependency tree // // group dCVS_Group-02 // { // NFSRestart dCVS_NFSRestart02 // { // IP dCVS_IP02 // { // NIC dCVS_NIC02 // Share dCVS_Share0102 // { // Proxy sProxy_NFS02 // Mount dCVS_Mount0102 // { // Volume dCVS_vol0102 // { // DiskGroup dCVS_cvsdg02 // } // } // } // } // Mount dCVS_Mount0102 // { // Volume dCVS_vol0102 // { // DiskGroup dCVS_cvsdg02 // } // } // } // } group dNFS_Group ( SystemList = { cm-cvs-22 = 0, cm-cvs-21 = 1 } Parallel = 1 AutoStartList = { cm-cvs-22, cm-cvs-21 } ) NFS dCVS_NFS ( Nproc = 16 ) Phantom NFS_phantom ( Critical = 0 ) // resource dependency tree // // group dNFS_Group // { // Phantom NFS_phantom // NFS dCVS_NFS // }
01-16-2016 03:09 PM
Eventually nfsd will start so the remaining problem is that when both groups are on node1 and one group is switched to the node2, the agent stops nfsd on node1, and then it has the same long delay as when the service first starts.
01-18-2016 06:23 PM
Let's see if this formats better
include "types.cf"
cluster cvsnew (
UserNames = { admin = ghiAhcHeiDiiGqiChf }
Administrators = { admin }
)
system cm-cvs-21 (
)
system cm-cvs-22 (
)
group ClusterService (
SystemList = { cm-cvs-21 = 0, cm-cvs-22 = 1 }
AutoStartList = { cm-cvs-21, cm-cvs-22 }
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)
IP webip (
Device = eth0
Address = "10.x.243.35"
NetMask = "255.255.252.0"
)
NIC csgnic (
Device = eth0
)
NotifierMngr ntfr (
SmtpServer = "mailhost.blah.blah.blah.com"
SmtpRecipients = { root = Warning }
)
ntfr requires csgnic
webip requires csgnic
// resource dependency tree
//
// group ClusterService
// {
// NotifierMngr ntfr
// {
// NIC csgnic
// }
// IP webip
// {
// NIC csgnic
// }
// }
group dCVS_Group (
SystemList = { cm-cvs-22 = 0, cm-cvs-21 = 1 }
AutoStartList = { cm-cvs-22, cm-cvs-21 }
PreOnline @cm-cvs-22 = 1
PreOnline @cm-cvs-21 = 1
)
Apache dCVS_Apache (
Enabled = 0
httpdDir = "/usr/sbin"
HostName = cvs
ConfigFile = "/etc/httpd/conf/httpd.conf"
)
DiskGroup dCVS_cvsdg (
DiskGroup = cvsdg01-
)
IP dCVS_IP (
Device = eth0
Address = "10.x.243.79"
NetMask = "255.255.252.0"
)
Mount dCVS_Mount01 (
MountPoint = "/u/cvsdg01"
BlockDevice = "/dev/vx/dsk/cvsdg01-/vol01"
FSType = vxfs
MountOpt = "ioerror=disable"
FsckOpt = "-y"
)
NFSRestart dCVS_NFSRestart (
NFSRes = dCVS_NFS
)
NIC dCVS_NIC (
Device = eth0
)
Proxy sProxy_NFS (
TargetResName = dCVS_NFS
)
Share dCVS_Share01 (
PathName = "/u/cvsdg01"
Client = "*"
Options = "rw,sync,insecure,no_root_squash"
NFSRes = dCVS_NFS
)
Volume dCVS_vol01 (
DiskGroup = cvsdg01-
Volume = vol01
)
dCVS_IP requires dCVS_NIC
dCVS_IP requires dCVS_Share01
dCVS_Mount01 requires dCVS_vol01
dCVS_NFSRestart requires dCVS_IP
dCVS_NFSRestart requires dCVS_Mount01
dCVS_Share01 requires dCVS_Mount01
dCVS_Share01 requires sProxy_NFS
dCVS_vol01 requires dCVS_cvsdg
// resource dependency tree
//
// group dCVS_Group
// {
// Apache dCVS_Apache
// NFSRestart dCVS_NFSRestart
// {
// Mount dCVS_Mount01
// {
// Volume dCVS_vol01
// {
// DiskGroup dCVS_cvsdg
// }
// }
// IP dCVS_IP
// {
// Share dCVS_Share01
// {
// Proxy sProxy_NFS
// Mount dCVS_Mount01
// {
// Volume dCVS_vol01
// {
// DiskGroup dCVS_cvsdg
// }
// }
// }
// NIC dCVS_NIC
// }
// }
// }
group dCVS_Group-02 (
SystemList = { cm-cvs-22 = 0, cm-cvs-21 = 1 }
AutoStartList = { cm-cvs-22, cm-cvs-21 }
PreOnline @cm-cvs-22 = 1
PreOnline @cm-cvs-21 = 1
)
DiskGroup dCVS_cvsdg02 (
DiskGroup = cvsdg02-
)
IP dCVS_IP02 (
Device = eth0
Address = "10.x.243.221"
NetMask = "255.255.252.0"
)
Mount dCVS_Mount0102 (
MountPoint = "/u/cvsdg02"
BlockDevice = "/dev/vx/dsk/cvsdg02-/vol01"
FSType = vxfs
MountOpt = "ioerror=disable"
FsckOpt = "-y"
)
NFSRestart dCVS_NFSRestart02 (
NFSRes = dCVS_NFS
)
NIC dCVS_NIC02 (
Device = eth0
)
Proxy sProxy_NFS02 (
TargetResName = dCVS_NFS
)
Share dCVS_Share0102 (
PathName = "/u/cvsdg02"
Client = "*"
Options = "rw,sync,insecure,no_root_squash"
NFSRes = dCVS_NFS
)
Volume dCVS_vol0102 (
DiskGroup = cvsdg02-
Volume = vol01
)
dCVS_IP02 requires dCVS_NIC02
dCVS_IP02 requires dCVS_Share0102
dCVS_Mount0102 requires dCVS_vol0102
dCVS_NFSRestart02 requires dCVS_IP02
dCVS_NFSRestart02 requires dCVS_Mount0102
dCVS_Share0102 requires dCVS_Mount0102
dCVS_Share0102 requires sProxy_NFS02
dCVS_vol0102 requires dCVS_cvsdg02
// resource dependency tree
//
// group dCVS_Group-02
// {
// NFSRestart dCVS_NFSRestart02
// {
// IP dCVS_IP02
// {
// NIC dCVS_NIC02
// Share dCVS_Share0102
// {
// Proxy sProxy_NFS02
// Mount dCVS_Mount0102
// {
// Volume dCVS_vol0102
// {
// DiskGroup dCVS_cvsdg02
// }
// }
// }
// }
// Mount dCVS_Mount0102
// {
// Volume dCVS_vol0102
// {
// DiskGroup dCVS_cvsdg02
// }
// }
// }
// }
group dNFS_Group (
SystemList = { cm-cvs-22 = 0, cm-cvs-21 = 1 }
Parallel = 1
AutoStartList = { cm-cvs-22, cm-cvs-21 }
)
NFS dCVS_NFS (
Nproc = 16
)
Phantom NFS_phantom (
Critical = 0
)
// resource dependency tree
//
// group dNFS_Group
// {
// Phantom NFS_phantom
// NFS dCVS_NFS
// }
01-19-2016 10:37 PM
Hi.
Can you kindly provide which VCS and OS version are you using?
Also the VCS configuration may need modification based on the the VCS version. In the recent VCS releases we configure two NFSRestart resources in each service group. On one of the NFSRestart resource we set the Lower attribute to 1.
https://sort.symantec.com/public/documents/sfha/6.2/linux/productguides/html/vcs_bundled_agents/ch04s03s04.htm
Also you gan go through https://www.veritas.com/community/articles/how-make-nfs-shares-highly-available-using-atleast-resource-dependency post to give you more info.
When you switch the service group the nfsd daemon is stopped by NFSRestart agent and then re-started again.
Regards,
Sudhir