SFRAC 4.1 error on solaris ..urgent help needed

[symsonu]

I am trying to stop cluster on node rmgdbs02 by hastop -local

it got stuck as few mount resources not going offline as below

-- SYSTEM STATE
-- System               State                Frozen

A  rmgdbs02             LEAVING              0
A  rmgdbs03             RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  SID_SAVS        rmgdbs02             Y          N               STOPPING|PARTIAL
B  SID_SAVS        rmgdbs03             Y          N               ONLINE
B  SID_SBPP        rmgdbs02             Y          N               STOPPING|PARTIAL
B  SID_SBPP        rmgdbs03             Y          N               ONLINE
B  SID_SPTL        rmgdbs02             Y          N               STOPPING|PARTIAL
B  SID_SPTL        rmgdbs03             Y          N               ONLINE
B  cvm             rmgdbs02             Y          N               STOPPING|PARTIAL|FAULTED
B  cvm             rmgdbs03             Y          N               ONLINE

-- RESOURCES FAILED
-- Group           Type                 Resource             System

C  cvm             CFSMount             orabin_mnt           rmgdbs02

-- RESOURCES OFFLINING
-- Group           Type            Resource             System               IState

F  SID_SAVS        CFSMount        SAVS_MOUNT_arch      rmgdbs02             W_OFFLINE_PROPAGATE
F  SID_SBPP        CFSMount        SBPP_MOUNT_arch      rmgdbs02             W_OFFLINE_PROPAGATE
F  SID_SPTL        CFSMount        SPTL_MOUNT_admin     rmgdbs02             W_OFFLINE_PROPAGATE
F  SID_SPTL        CFSMount        SPTL_MOUNT_data01    rmgdbs02             W_OFFLINE_PROPAGATE
F  SID_SPTL        CFSMount        SPTL_MOUNT_data02    rmgdbs02             W_OFFLINE_PROPAGATE
F  cvm             CVMCluster      cvm_clus             rmgdbs02             W_OFFLINE_PROPAGATE

There are also continuous  messages in log files since yesterday as below..

2012/04/16 06:27:03 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data02
2012/04/16 06:27:03 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/admin
2012/04/16 06:27:03 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/admin
2012/04/16 06:27:04 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data01
2012/04/16 06:27:04 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data01
2012/04/16 06:27:04 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data02
2012/04/16 06:29:17 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/16 06:31:33 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                     

Please help me to resolve the situation to bring all online....

Quick response will be apppreciated

Regards

Sumit

[scottmyers]

Sumit,

Please open a support case with Symantec Technical Support.

Phone number is 800-342-0652

Thanks.

[Marianne]

Version 4.1 has reached EOSL just over a year ago (March 31, 2011)... http://www.symantec.com/docs/TECH48244

[symsonu]

Yes , it has reached EOSL, can someone suggest what below  error mssg  suugest?

2012/04/16 06:29:17 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/16 06:31:33 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:  

I could not find this error code V-16-10001-5507. googled a lot but in vein.

Regards

Sumit                    

[Gaurav_S]

Sumit,

error message is self descriptive ... filesystems are disabled .... that means any changes to metadeta is disabled ...

system would have had an IO error resulting in disabling of filesystem. Does this filesystem lies on SAN disks ? If yes, check out the messages file or logs to find if there were IOerrors on the disks.

you may also notice some vxfs errors in the messages like

WARNING: msgcnt x: vxfs: mesg 003: vx_mapbad...
WARNING: msgcnt x: vxfs: mesg 004: vx_mapbad...
WARNING: msgcnt x: vxfs: mesg 005: vx_mapbad...

easiest way would be to force unmount the FS, do a full fsck & remount again ... You should either freeze the cluster or stop the cluster while doing this fsck & manual umount process..

G

[symsonu]

Hello Gaurav,

Thanks a lot for the positive response and hope for me to get this resolved.

I have stopped the cluster using hastop -local -force

root@rmgdbs02 # hastatus -summ
VCS ERROR V-16-1-10600 Cannot connect to VCS engine
VCS WARNING V-16-1-11046 Local system not available
root@rmgdbs02 #

yes, there are  vxfs error messages and I/o error on SAN disk. yes all these filesystems are on SAN

below are the logs to analyse

Apr 14 19:09:28 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 1 mesg 075: V-2-75: replay fsck failed for /dev/vx/dsk/orasrv_dg/srvm_vol file system
Apr 14 19:09:28 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 2 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/orasrv_dg/srvm_vol file system disabled
Apr 14 19:09:28 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 3 mesg 075: V-2-75: replay fsck failed for /dev/vx/dsk/orabinvol_dg/orabinvol file system
Apr 14 19:09:28 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 4 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/orabinvol_dg/orabinvol file system disabled
Apr 14 19:10:00 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 5 mesg 075: V-2-75: replay fsck failed for /dev/vx/dsk/dbsavsdg/data01 file system
Apr 14 19:10:00 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 6 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/dbsavsdg/data01 file system disabled
Apr 14 19:10:01 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 7 mesg 075: V-2-75: replay fsck failed for /dev/vx/dsk/dbsavsdg/data02 file system
Apr 14 19:10:01 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 8 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/dbsavsdg/data02 file system disabled
Apr 14 19:10:01 rmgdbs02 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 9 mesg 075: V-2-75: replay fsck failed for /dev/vx/dsk/dbsavsdg/admin file system

root@rmgdbs02 # df -h | grep -i stat
df: cannot statvfs /opt/oracle_srvm: I/O error
df: cannot statvfs /opt/oracle: I/O error
df: cannot statvfs /apps/oracle/SBPP/admin: I/O error
df: cannot statvfs /apps/oracle/SBPP/data02: I/O error
df: cannot statvfs /apps/oracle/SBPP/data01: I/O error
df: cannot statvfs /apps/oracle/SAVS/data02: I/O error
df: cannot statvfs /apps/oracle/SAVS/data01: I/O error
df: cannot statvfs /apps/oracle/SAVS/admin: I/O error

root@rmgdbs02 # cat /etc/mnttab | grep -i disable
/dev/vx/dsk/orasrv_dg/srvm_vol  /opt/oracle_srvm        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4489c40      1288538848
/dev/vx/dsk/orabinvol_dg/orabinvol      /opt/oracle     vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483a98      1288538848
/dev/vx/dsk/archsbppdg/arch     /apps/oracle/SBPP/arch  vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448bf68      1288538890
/dev/vx/dsk/archsavsdg/arch     /apps/oracle/SAVS/arch  vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448dea8      1288538890
/dev/vx/dsk/archsptldg/arch     /apps/oracle/SPTL/arch  vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4481f40      1288538890
/dev/vx/dsk/dbsbppdg/admin      /apps/oracle/SBPP/admin vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448232a      1288538927
/dev/vx/dsk/dbsbppdg/data02     /apps/oracle/SBPP/data02        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4482329      1288538927
/dev/vx/dsk/dbsbppdg/data01     /apps/oracle/SBPP/data01        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4482328      1288538927
/dev/vx/dsk/dbsavsdg/data02     /apps/oracle/SAVS/data02        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4489859      1288538931
/dev/vx/dsk/dbsptldg/admin      /apps/oracle/SPTL/admin vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483e82      1288538931
/dev/vx/dsk/dbsavsdg/data01     /apps/oracle/SAVS/data01        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4489858      1288538931
/dev/vx/dsk/dbsptldg/data02     /apps/oracle/SPTL/data02        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483e81      1288538931
/dev/vx/dsk/dbsptldg/data01     /apps/oracle/SPTL/data01        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483e80      1288538931
/dev/vx/dsk/dbsavsdg/admin      /apps/oracle/SAVS/admin vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448985a      1288538931

root@rmgdbs02 # umount -f /opt/oracle
/usr/lib/fs/vxfs/umount: syntax error at line 1: `)' unexpected
root@rmgdbs02 #
root@rmgdbs02 # fsck /dev/vx/rdsk/orabinvol_dg/orabinvol
** /dev/vx/rdsk/orabinvol_dg/orabinvol
BAD SUPERBLOCK AT BLOCK 16: MAGIC NUMBER WRONG

LOOK FOR ALTERNATE SUPERBLOCKS WITH MKFS? y

LOOK FOR ALTERNATE SUPERBLOCKS WITH NEWFS? y

SEARCH FOR ALTERNATE SUPERBLOCKS FAILED.

USE GENERIC SUPERBLOCK FROM MKFS? y

CALCULATED GENERIC SUPERBLOCK WITH MKFS
If filesystem was created with manually-specified geometry, using
auto-discovered superblock may result in irrecoverable damage to
filesystem and user data.

CANCEL FILESYSTEM CHECK? ^Croot@rmgdbs02 # fsck -F vxfs /dev/vx/rdsk/orabinvol_dg/orabinvol
fsck: execve("/usr/lib/fs/vxfs/sparcv9/fsck") failed
fsck: execve("/usr/lib/fs/vxfs/sparcv7/fsck") failed
fsck: cannot find/execute "fsck" in ISA subdirectories

Can you please analyse the fsck of these filesystems, the same filessystems are mounted on other node as this is Active/Active cluster using CFS.
I canceled the fsck and then on running it again , it gave me above error.
I canceled it because I thought might be it corrupts the filesystem on other node as the same volume is mounted on other node too.

please suggest.

[Har-D]

Hi Sumit,

1. NEVER USE INCORRECT FSCK PARAMETERS (e.g. Filesystem argument in your case), as it corrupts the FS. Reason being, the when running FSCK w/o Filesystem argument, will check /etc/vfstab for the correct FSType, if it doesn't find, it will check /etc/default/fs, which is generally UFS. So the above output shows that FSCK considered the FSType as UFS and hence, errored. Not sure, upto what extent it made changes to the FS, hence corrupting the FS.

2. It seems they are CFS mounts and the volumes are mounted on other node. Why you want to FSCK?

3. vx_disable, states that FS is disabled. What is the state of this FS on the other node? Are those FS still mounted with applications or I/Os running on those FS?

Try the following :

1. Unmount all the CFS volumes from all the nodes (via VCS) and remount them one-by-one on one node (where VCS is running) first and then do it on second node.

2. If it doesn't help, try to manage some downtime and reboot all the nodes.

[symsonu]

Hello Hari,

yes, I am aware of the fact that fsck without proper fstype can corrupt filesystem thats why I Ctrl C the command . on other nodes  service groups are online and all filesystems are mounted

-- SYSTEM STATE
-- System               State                Frozen

A  rmgdbs02             EXITED               0
A  rmgdbs03             RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  SID_SAVS        rmgdbs02             Y          Y               OFFLINE
B  SID_SAVS        rmgdbs03             Y          N               ONLINE
B  SID_SBPP        rmgdbs02             Y          Y               OFFLINE
B  SID_SBPP        rmgdbs03             Y          N               ONLINE
B  SID_SPTL        rmgdbs02             Y          Y               OFFLINE
B  SID_SPTL        rmgdbs03             Y          N               ONLINE
B  cvm             rmgdbs02             Y          Y               OFFLINE
B  cvm             rmgdbs03             Y          N               ONLINE

No error messages for vxfs in other node.

root@rmgdbs03 # cat /var/adm/messages | grep -i vxfs
root@rmgdbs03 # df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/bootdg/rootvol
                       3.9G   3.1G   826M    80%    /
/proc                    0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
fd                       0K     0K     0K     0%    /dev/fd
/dev/vx/dsk/bootdg/var
                       3.9G   1.4G   2.5G    36%    /var
swap                    11G    64K    11G     1%    /var/run
dmpfs                   11G     0K    11G     0%    /dev/vx/dmp
dmpfs                   11G     0K    11G     0%    /dev/vx/rdmp
swap                   500M   1.7M   498M     1%    /tmp
/dev/vx/dsk/bootdg/export
                        44G    26G    17G    60%    /export
rmgnasas02:/vol/rmgprod/postbus
                       3.0T   1.8T   1.2T    62%    /shared/postbus
rmgnasas02:/vol/rmgprod/dbdumps
                       3.0T   1.8T   1.2T    62%    /shared/dbdumps
rmgnasas02:/vol/rmgprod/foreignexchange
                       3.0T   1.8T   1.2T    62%    /shared/foreignexchange
/dev/odm                 0K     0K     0K     0%    /dev/odm
/dev/vx/dsk/orabinvol_dg/orabinvol
                       5.9G   4.4G   1.4G    76%    /opt/oracle
/dev/vx/dsk/orasrv_dg/srvm_vol
                       6.5G   5.3G   1.1G    83%    /opt/oracle_srvm
/dev/vx/dsk/archsptldg/arch
                        27G   426M    25G     2%    /apps/oracle/SPTL/arch
/dev/vx/dsk/archsbppdg/arch
                        27G   224M    25G     1%    /apps/oracle/SBPP/arch
/dev/vx/dsk/archsavsdg/arch
                        27G   224M    25G     1%    /apps/oracle/SAVS/arch
/dev/vx/dsk/dbsbppdg/admin
                       1.0G   234M   740M    25%    /apps/oracle/SBPP/admin
/dev/vx/dsk/dbsbppdg/data01
                        10G   5.8G   3.9G    60%    /apps/oracle/SBPP/data01
/dev/vx/dsk/dbsavsdg/data02
                       6.0G    18M   5.6G     1%    /apps/oracle/SAVS/data02
/dev/vx/dsk/dbsavsdg/admin
                       1.0G   218M   756M    23%    /apps/oracle/SAVS/admin
/dev/vx/dsk/dbsptldg/data01
                       116G    99G    15G    87%    /apps/oracle/SPTL/data01
/dev/vx/dsk/dbsavsdg/data01
                        10G   7.6G   2.2G    78%    /apps/oracle/SAVS/data01
/dev/vx/dsk/dbsptldg/data02
                       105G    87G    17G    84%    /apps/oracle/SPTL/data02
/dev/vx/dsk/dbsptldg/admin
                       9.4G   8.0G   1.3G    86%    /apps/oracle/SPTL/admin
/dev/vx/dsk/dbsbppdg/data02
                       6.0G    18M   5.6G     1%    /apps/oracle/SBPP/data02
rmgnasas02:/vol/rmgstage/logs
                       982G    70G   913G     8%    /shared/logs
root@rmgdbs03 # cat /etc/mnttab
/dev/vx/dsk/bootdg/rootvol      /       ufs     rw,intr,largefiles,logging,xattr,onerror=panic,suid,dev=4480000 1288537790
/proc   /proc   proc    dev=5180000     1288537789
mnttab  /etc/mnttab     mntfs   dev=5240000     1288537789
fd      /dev/fd fd      rw,suid,dev=5280000     1288537790
/dev/vx/dsk/bootdg/var  /var    ufs     rw,intr,largefiles,logging,xattr,onerror=panic,suid,dev=4497700 1288537792
swap    /var/run        tmpfs   xattr,dev=1     1288537792
dmpfs   /dev/vx/dmp     tmpfs   xattr,dev=2     1288537793
dmpfs   /dev/vx/rdmp    tmpfs   xattr,dev=3     1288537793
swap    /tmp    tmpfs   xattr,size=500m,dev=4   1288537800
/dev/vx/dsk/bootdg/export       /export ufs     rw,intr,largefiles,logging,xattr,onerror=panic,suid,dev=4497701 1288537800
rmgnasas02:/vol/rmgprod/postbus /shared/postbus nfs     bg,hard,intr,ro,xattr,dev=53c0001       1288537823
rmgnasas02:/vol/rmgprod/dbdumps /shared/dbdumps nfs     bg,hard,intr,xattr,dev=53c0002  1288537823
rmgnasas02:/vol/rmgprod/foreignexchange /shared/foreignexchange nfs     bg,hard,intr,ro,xattr,dev=53c0003       1288537823
/dev/odm        /dev/odm        odm     dev=54c0000     1288537826
/dev/vx/dsk/orabinvol_dg/orabinvol      /opt/oracle     vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483a98      1288538054
/dev/vx/dsk/orasrv_dg/srvm_vol  /opt/oracle_srvm        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4489c40      1288538054
/dev/vx/dsk/archsptldg/arch     /apps/oracle/SPTL/arch  vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4481f40      1288538098
/dev/vx/dsk/archsbppdg/arch     /apps/oracle/SBPP/arch  vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448bf68      1288538099
/dev/vx/dsk/archsavsdg/arch     /apps/oracle/SAVS/arch  vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448dea8      1288538099
/dev/vx/dsk/dbsbppdg/admin      /apps/oracle/SBPP/admin vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448232a      1288538140
/dev/vx/dsk/dbsbppdg/data01     /apps/oracle/SBPP/data01        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4482328      1288538140
/dev/vx/dsk/dbsavsdg/data02     /apps/oracle/SAVS/data02        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4489859      1288538140
/dev/vx/dsk/dbsavsdg/admin      /apps/oracle/SAVS/admin vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=448985a      1288538141
/dev/vx/dsk/dbsptldg/data01     /apps/oracle/SPTL/data01        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483e80      1288538141
/dev/vx/dsk/dbsavsdg/data01     /apps/oracle/SAVS/data01        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4489858      1288538142
/dev/vx/dsk/dbsptldg/data02     /apps/oracle/SPTL/data02        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483e81      1288538143
/dev/vx/dsk/dbsptldg/admin      /apps/oracle/SPTL/admin vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4483e82      1288538143
/dev/vx/dsk/dbsbppdg/data02     /apps/oracle/SBPP/data02        vxfs    rw,suid,log,largefiles,qio,cluster,ioerror=disable,crw,dev=4482329      1288538143
rmgnasas02:/vol/rmgstage/logs   /shared/logs    nfs     bg,soft,rw,xattr,dev=53c0004    1288767294
 

[Har-D]

Ok.

What happens if you try mounting them again (starting cluster)? Did they error out? Are all the required ports started (gabconfig -a) ?

[symsonu]

root@rmgdbs02 # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen  11c5903 membership 01
Port b gen  11c5905 membership 01
Port d gen  11c5902 membership 01
Port f gen  11c5913 membership 01
Port o gen  11c5907 membership 01
Port q gen  11c5911 membership 01
Port v gen  11c590d membership 01
Port w gen  11c591c membership 01

I have started the cluster and then we can see that  two CFSmount resources didnot come online as below

root@rmgdbs02 # hastatus -summ

-- SYSTEM STATE
-- System               State                Frozen

A  rmgdbs02             RUNNING              0
A  rmgdbs03             RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  SID_SAVS        rmgdbs02             Y          N               PARTIAL
B  SID_SAVS        rmgdbs03             Y          N               ONLINE
B  SID_SBPP        rmgdbs02             Y          N               PARTIAL
B  SID_SBPP        rmgdbs03             Y          N               ONLINE
B  SID_SPTL        rmgdbs02             Y          N               PARTIAL
B  SID_SPTL        rmgdbs03             Y          N               ONLINE
B  cvm             rmgdbs02             Y          N               STARTING|PARTIAL
B  cvm             rmgdbs03             Y          N               ONLINE

-- RESOURCES ONLINING
-- Group           Type            Resource             System               IState

E  cvm             CFSMount        orabin_mnt           rmgdbs02             W_ONLINE
E  cvm             CFSMount        srvm_mnt             rmgdbs02             W_ONLINE

[Har-D]

Looks good that it joined cluster fine (where is port-h, by the way :))

Couple of things to look further :

- What does engine logs tell?

- What about other CFS mounts?

Also try the following :

> wait till it faults and immediately FLUSH and freeze the node to avoid VCS to offline the the whole SG, after finding the FAULT with the CFSMount resource.

> Now try mounting the CFSmounts manually and check, what error they give.

[symsonu]

H port is also there

root@rmgdbs03 # gabconfig -a
GAB Port Memberships
===============================================================
Port a gen  11c5903 membership 01
Port b gen  11c5905 membership 01
Port d gen  11c5902 membership 01
Port f gen  11c5913 membership 01
Port h gen  11c5915 membership 01
Port o gen  11c5907 membership 01
Port q gen  11c5911 membership 01
Port v gen  11c590d membership 01
Port w gen  11c591c membership 0

2012/04/18 06:46:50 VCS INFO V-16-1-50135 User root fired command: MSG_CLUSTER_STOP_SYS_FORCE from localhost
2012/04/18 06:46:50 VCS NOTICE V-16-1-10322 System rmgdbs02 (Node '0') changed state from RUNNING to EXITING_FORCIBLY
2012/04/18 06:46:50 VCS NOTICE V-16-1-10446 Group SID_SAVS is offline on system rmgdbs02
2012/04/18 06:46:50 VCS NOTICE V-16-1-10446 Group SID_SBPP is offline on system rmgdbs02
2012/04/18 06:46:50 VCS NOTICE V-16-1-10446 Group SID_SPTL is offline on system rmgdbs02
2012/04/18 06:46:50 VCS NOTICE V-16-1-10446 Group cvm is offline on system rmgdbs02
2012/04/18 06:46:50 VCS NOTICE V-16-1-10322 System rmgdbs02 (Node '0') changed state from EXITING_FORCIBLY to EXITED
2012/04/18 06:46:50 VCS ERROR V-16-1-50016 Setting SYSSTATE to RSM_EXITED for node : rmgdbs02.
2012/04/18 10:15:53 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/04/18 10:15:53 VCS NOTICE V-16-1-11050 VCS engine version=4.1
2012/04/18 10:15:53 VCS NOTICE V-16-1-11051 VCS engine join version=4.1001
2012/04/18 10:15:53 VCS NOTICE V-16-1-11052 VCS engine pstamp=4.1 11/08/05-16:44:01
2012/04/18 10:15:53 VCS NOTICE V-16-1-10114 Opening GAB library
2012/04/18 10:15:53 VCS NOTICE V-16-1-10619 'HAD' starting on: rmgdbs02
2012/04/18 10:15:53 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms
2012/04/18 10:15:58 VCS INFO V-16-1-10077 Received new cluster membership
2012/04/18 10:15:58 VCS NOTICE V-16-1-10080 System (rmgdbs02) - Membership: 0x3, Jeopardy: 0x0
2012/04/18 10:15:58 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/04/18 10:15:58 VCS NOTICE V-16-1-10086 System rmgdbs02 (Node '0') is in Regular Membership - Membership: 0x3
2012/04/18 10:15:58 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/04/18 10:15:58 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/04/18 10:15:58 VCS WARNING V-16-1-50129 Operation 'haclus -modify' rejected as the node is in CURRENT_DISCOVER_WAIT state
2012/04/18 10:15:58 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'rmgdbs03'
2012/04/18 10:15:58 VCS NOTICE V-16-1-10322 System rmgdbs03 (Node '1') changed state from INITING to RUNNING
2012/04/18 10:15:58 VCS NOTICE V-16-1-10322 System rmgdbs02 (Node '0') changed state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2012/04/18 10:15:58 VCS NOTICE V-16-1-10464 Requesting snapshot from node: 1
2012/04/18 10:15:58 VCS NOTICE V-16-1-10465 Getting snapshot.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/04/18 10:15:58 VCS WARNING V-16-1-40117 Invalid IP address specified.
2012/04/18 10:15:58 VCS NOTICE V-16-1-10181 Group SID_SAVS AutoRestart set to 1
2012/04/18 10:15:58 VCS NOTICE V-16-1-10181 Group SID_SBPP AutoRestart set to 1
2012/04/18 10:15:58 VCS NOTICE V-16-1-10181 Group SID_SPTL AutoRestart set to 1
2012/04/18 10:15:58 VCS NOTICE V-16-1-10181 Group cvm AutoRestart set to 1
2012/04/18 10:16:00 VCS INFO V-16-1-10466 End of snapshot received from node: 1.  snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/04/18 10:16:00 VCS NOTICE V-16-1-10032 VxFEN driver available. Local node id=0
2012/04/18 10:16:00 VCS NOTICE V-16-1-10467 Replaying broadcast queue. snapped_membership: 0x3 current_membership: 0x3 current_jeopardy_membership: 0x0
2012/04/18 10:16:00 VCS NOTICE V-16-1-10322 System rmgdbs02 (Node '0') changed state from REMOTE_BUILD to RUNNING
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSMount/CFSMountAgent for resource type CFSMount successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSQlogckd/CFSQlogckdAgent for resource type CFSQlogckd successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CFSfsckd/CFSfsckdAgent for resource type CFSfsckd successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMCluster/CVMClusterAgent for resource type CVMCluster successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVolDg/CVMVolDgAgent for resource type CVMVolDg successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/CVMVxconfigd/CVMVxconfigdAgent for resource type CVMVxconfigd successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Netlsnr/NetlsnrAgent for resource type Netlsnr successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:00 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Oracle/OracleAgent for resource type Oracle successfully started at Wed Apr 18 10:16:00 2012
2012/04/18 10:16:01 VCS INFO V-16-1-10304 Resource cvm_listener (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:01 VCS INFO V-16-1-10304 Resource IP_Listener (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:02 VCS INFO V-16-1-10304 Resource Oracle_SAVS (Owner: unknown, Group: SID_SAVS) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:02 VCS INFO V-16-1-10304 Resource Oracle_SBPP (Owner: unknown, Group: SID_SBPP) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:02 VCS INFO V-16-1-10304 Resource Oracel_SPTL (Owner: unknown, Group: SID_SPTL) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:04 VCS INFO V-16-1-10304 Resource vxfsckd (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:04 VCS INFO V-16-1-10297 Resource cvm_vxconfigd (Owner: unknown, Group: cvm) is online on rmgdbs02 (First probe)
2012/04/18 10:16:04 VCS INFO V-16-1-10304 Resource qlogckd (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:04 VCS INFO V-16-1-10297 Resource cvm_clus (Owner: unknown, Group: cvm) is online on rmgdbs02 (First probe)
2012/04/18 10:16:04 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node rmgdbs02
2012/04/18 10:16:04 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group cvm on node rmgdbs03
2012/04/18 10:16:05 VCS INFO V-16-1-10304 Resource orabin_voldg (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:06 VCS INFO V-16-1-10297 Resource SPTL_MOUNT_arch (Owner: unknown, Group: SID_SPTL) is online on rmgdbs02 (First probe)
2012/04/18 10:16:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs02
2012/04/18 10:16:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs03
2012/04/18 10:16:06 VCS INFO V-16-1-10297 Resource SBPP_MOUNT_arch (Owner: unknown, Group: SID_SBPP) is online on rmgdbs02 (First probe)
2012/04/18 10:16:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SBPP on node rmgdbs02
2012/04/18 10:16:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SBPP on node rmgdbs03
2012/04/18 10:16:06 VCS INFO V-16-1-10297 Resource SAVS_MOUNT_arch (Owner: unknown, Group: SID_SAVS) is online on rmgdbs02 (First probe)
2012/04/18 10:16:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SAVS on node rmgdbs02
2012/04/18 10:16:06 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SAVS on node rmgdbs03
2012/04/18 10:16:06 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data01
2012/04/18 10:16:06 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/admin
2012/04/18 10:16:06 VCS INFO V-16-1-10304 Resource SBPP_VOL_oracle (Owner: unknown, Group: SID_SBPP) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:06 VCS INFO V-16-1-10304 Resource orasrv_dg (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:07 VCS INFO V-16-1-10297 Resource SPTL_MOUNT_admin (Owner: unknown, Group: SID_SPTL) is online on rmgdbs02 (First probe)
2012/04/18 10:16:07 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs02
2012/04/18 10:16:07 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs03
2012/04/18 10:16:07 VCS INFO V-16-1-10304 Resource SBPP_MOUNT_data01 (Owner: unknown, Group: SID_SBPP) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:07 VCS INFO V-16-1-10304 Resource SBPP_MOUNT_admin (Owner: unknown, Group: SID_SBPP) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:07 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data02
2012/04/18 10:16:07 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data02
2012/04/18 10:16:07 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/admin
2012/04/18 10:16:07 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data01
2012/04/18 10:16:08 VCS INFO V-16-1-10304 Resource SAVS_VOL_oracle (Owner: unknown, Group: SID_SAVS) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:08 VCS INFO V-16-1-10304 Resource SAVS_MOUNT_admin (Owner: unknown, Group: SID_SAVS) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:08 VCS INFO V-16-1-10304 Resource SBPP_MOUNT_data02 (Owner: unknown, Group: SID_SBPP) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:08 VCS INFO V-16-1-10304 Resource SAVS_MOUNT_data02 (Owner: unknown, Group: SID_SAVS) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:08 VCS INFO V-16-1-10304 Resource SAVS_MOUNT_data01 (Owner: unknown, Group: SID_SAVS) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:16:09 VCS INFO V-16-1-10297 Resource SPTL_MOUNT_data02 (Owner: unknown, Group: SID_SPTL) is online on rmgdbs02 (First probe)
2012/04/18 10:16:09 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs02
2012/04/18 10:16:09 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs03
2012/04/18 10:16:09 VCS INFO V-16-1-10297 Resource SPTL_MOUNT_data01 (Owner: unknown, Group: SID_SPTL) is online on rmgdbs02 (First probe)
2012/04/18 10:16:09 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs02
2012/04/18 10:16:09 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs03
2012/04/18 10:16:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:16:10 VCS INFO V-16-1-10304 Resource srvm_mnt (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:10 VCS INFO V-16-1-10304 Resource orabin_mnt (Owner: unknown, Group: cvm) is offline on rmgdbs02 (First probe)
2012/04/18 10:16:10 VCS NOTICE V-16-1-10438 Group cvm has been probed on system rmgdbs02
2012/04/18 10:16:10 VCS NOTICE V-16-1-10442 Initiating auto-start online of group cvm on system rmgdbs02
2012/04/18 10:16:10 VCS NOTICE V-16-1-10301 Initiating Online of Resource qlogckd (Owner: unknown, Group: cvm) on System rmgdbs02
2012/04/18 10:16:10 VCS NOTICE V-16-1-10301 Initiating Online of Resource IP_Listener (Owner: unknown, Group: cvm) on System rmgdbs02
2012/04/18 10:16:10 VCS INFO V-16-1-10297 Resource SPTL_VOL_arch (Owner: unknown, Group: SID_SPTL) is online on rmgdbs02 (First probe)
2012/04/18 10:16:10 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs02
2012/04/18 10:16:10 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs03
2012/04/18 10:16:10 VCS INFO V-16-1-10297 Resource SAVS_VOL_arch (Owner: unknown, Group: SID_SAVS) is online on rmgdbs02 (First probe)
2012/04/18 10:16:10 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SAVS on node rmgdbs02
2012/04/18 10:16:10 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SAVS on node rmgdbs03
2012/04/18 10:16:10 VCS NOTICE V-16-1-10438 Group SID_SAVS has been probed on system rmgdbs02
2012/04/18 10:16:10 VCS INFO V-16-1-10297 Resource SBPP_VOL_arch (Owner: unknown, Group: SID_SBPP) is online on rmgdbs02 (First probe)
2012/04/18 10:16:10 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SBPP on node rmgdbs02
2012/04/18 10:16:10 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SBPP on node rmgdbs03
2012/04/18 10:16:10 VCS NOTICE V-16-1-10438 Group SID_SBPP has been probed on system rmgdbs02
2012/04/18 10:16:13 VCS INFO V-16-1-10298 Resource qlogckd (Owner: unknown, Group: cvm) is online on rmgdbs02 (VCS initiated)
2012/04/18 10:16:13 VCS NOTICE V-16-1-10301 Initiating Online of Resource vxfsckd (Owner: unknown, Group: cvm) on System rmgdbs02
2012/04/18 10:16:13 VCS INFO V-16-1-10298 Resource IP_Listener (Owner: unknown, Group: cvm) is online on rmgdbs02 (VCS initiated)
2012/04/18 10:16:14 VCS INFO V-16-1-10297 Resource SPTL_VOL_oracle (Owner: unknown, Group: SID_SPTL) is online on rmgdbs02 (First probe)
2012/04/18 10:16:14 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs02
2012/04/18 10:16:14 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group SID_SPTL on node rmgdbs03
2012/04/18 10:16:14 VCS NOTICE V-16-1-10438 Group SID_SPTL has been probed on system rmgdbs02
2012/04/18 10:16:16 VCS INFO V-16-1-10298 Resource vxfsckd (Owner: unknown, Group: cvm) is online on rmgdbs02 (VCS initiated)
2012/04/18 10:16:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource orasrv_dg (Owner: unknown, Group: cvm) on System rmgdbs02
2012/04/18 10:16:16 VCS NOTICE V-16-1-10301 Initiating Online of Resource orabin_voldg (Owner: unknown, Group: cvm) on System rmgdbs02
2012/04/18 10:16:23 VCS INFO V-16-10001-1046 (rmgdbs02) CVMVolDg:orabin_voldg:online:resource orabin_voldg is online
2012/04/18 10:16:24 VCS INFO V-16-10001-1046 (rmgdbs02) CVMVolDg:orasrv_dg:online:resource orasrv_dg is online
2012/04/18 10:16:59 VCS INFO V-16-1-10298 Resource orabin_voldg (Owner: unknown, Group: cvm) is online on rmgdbs02 (VCS initiated)
2012/04/18 10:16:59 VCS NOTICE V-16-1-10301 Initiating Online of Resource orabin_mnt (Owner: unknown, Group: cvm) on System rmgdbs02
2012/04/18 10:17:00 VCS INFO V-16-1-10298 Resource orasrv_dg (Owner: unknown, Group: cvm) is online on rmgdbs02 (VCS initiated)
2012/04/18 10:17:00 VCS NOTICE V-16-1-10301 Initiating Online of Resource srvm_mnt (Owner: unknown, Group: cvm) on System rmgdbs02
2012/04/18 10:17:03 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:17:04 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:17:04 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:17:05 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:17:06 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:17:07 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:17:08 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:17:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:18:08 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:18:09 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:18:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:18:10 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:18:10 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:18:10 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 1 of 16).
2012/04/18 10:18:11 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:18:11 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 1 of 16).
2012/04/18 10:18:14 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:18:14 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:18:15 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:18:15 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:18:17 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:18:17 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:18:20 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:18:20 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:19:20 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:19:20 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:19:21 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:19:21 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:19:22 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:19:22 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 2 of 16).
2012/04/18 10:19:22 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:19:22 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 2 of 16).
2012/04/18 10:19:25 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:19:25 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:19:26 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:19:26 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:19:27 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:19:28 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:19:29 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:19:30 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:20:30 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:20:30 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:20:31 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:20:31 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:20:32 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:20:32 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 3 of 16).
2012/04/18 10:20:33 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:20:33 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 3 of 16).
2012/04/18 10:20:35 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:20:35 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:20:36 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:20:36 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:20:38 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:20:38 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:20:40 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:20:40 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:21:08 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/admin
2012/04/18 10:21:08 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data01
2012/04/18 10:21:08 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data01
2012/04/18 10:21:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/admin
2012/04/18 10:21:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data02
2012/04/18 10:21:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data02
2012/04/18 10:21:41 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:21:41 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:21:41 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:21:41 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:21:42 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:21:42 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 4 of 16).
2012/04/18 10:21:42 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:21:42 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 4 of 16).
2012/04/18 10:21:45 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:21:45 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:21:46 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle.
mount: I/O error
2012/04/18 10:21:46 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle.
mount: I/O error
2012/04/18 10:21:49 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:21:49 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:21:51 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:21:51 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:22:51 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:22:51 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:22:52 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:22:52 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:22:53 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:22:53 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 5 of 16).
2012/04/18 10:22:53 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:22:53 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 5 of 16).
2012/04/18 10:22:57 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:22:57 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:22:57 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:22:58 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:22:59 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:22:59 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:23:01 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:23:02 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:24:00 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:24:01 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:24:01 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:24:02 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:24:02 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:24:02 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 6 of 16).
2012/04/18 10:24:03 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:24:03 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 6 of 16).
2012/04/18 10:24:06 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:24:07 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:24:07 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:24:08 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:24:09 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:24:10 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:24:11 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:24:13 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:25:11 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:25:12 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:25:13 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:25:13 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 7 of 16).
2012/04/18 10:25:13 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:25:14 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:25:15 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:25:15 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 7 of 16).
2012/04/18 10:25:16 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:25:17 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:25:18 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:25:19 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:25:19 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:25:21 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:25:22 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:25:23 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:26:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/admin
2012/04/18 10:26:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data01
2012/04/18 10:26:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data01:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data01
2012/04/18 10:26:09 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SBPP_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SBPP/data02
2012/04/18 10:26:10 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_admin:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/admin
2012/04/18 10:26:10 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:SAVS_MOUNT_data02:monitor:File System Disabled:                                 MountPoint : /apps/oracle/SAVS/data02
2012/04/18 10:26:22 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:26:22 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:26:24 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:26:24 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 8 of 16).
2012/04/18 10:26:24 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:26:25 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:26:26 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:26:26 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 8 of 16).
2012/04/18 10:26:27 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:26:28 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:26:29 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:26:30 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:srvm_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle_srvm.
mount: I/O error
2012/04/18 10:26:30 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:orabin_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:26:32 VCS INFO V-16-10001-5506 (rmgdbs02) CFSMount:srvm_mnt:online:CFSMOUNT Onlined :MountOptions : -F vxfs -o cluster,           Block Device /dev/vx/dsk/orasrv_dg/srvm_vol : MountPoint /opt/oracle_srvm
2012/04/18 10:26:33 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:26:34 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:27:32 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:orabin_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle
2012/04/18 10:27:33 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(orabin_mnt) because the resource is not up even after online completed.
2012/04/18 10:27:34 VCS INFO V-16-2-13068 (rmgdbs02) Resource(orabin_mnt) - clean completed successfully.
2012/04/18 10:27:34 VCS INFO V-16-2-13072 (rmgdbs02) Resource(orabin_mnt): Agent is retrying online (attempt number 9 of 16).
2012/04/18 10:27:35 VCS ERROR V-16-10001-5507 (rmgdbs02) CFSMount:srvm_mnt:monitor:File System Disabled:                                 MountPoint : /opt/oracle_srvm
2012/04/18 10:27:35 VCS ERROR V-16-2-13066 (rmgdbs02) Agent is calling clean for resource(srvm_mnt) because the resource is not up even after online completed.
2012/04/18 10:27:36 VCS INFO V-16-2-13068 (rmgdbs02) Resource(srvm_mnt) - clean completed successfully.
2012/04/18 10:27:36 VCS INFO V-16-2-13072 (rmgdbs02) Resource(srvm_mnt): Agent is retrying online (attempt number 9 of 16).
2012/04/18 10:27:38 VCS WARNING V-16-10001-5502 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Failed on this Node MountOptions -F vxfs -o cluster,                 Block Device /dev/vx/dsk/orabinvol_dg/orabinvol : MountPoint /opt/oracle
2012/04/18 10:27:39 VCS WARNING V-16-10001-5508 (rmgdbs02) CFSMount:orabin_mnt:online:Mount Error : mount: cannot stat mount-point /opt/oracle.

other CFS mounts are showing offlineroot@rmgdbs03 # hares -dep SBPP_MOUNT_admin
#Group       Parent           Child
SID_SBPP     Oracle_SBPP      SBPP_MOUNT_admin
SID_SBPP     SBPP_MOUNT_admin SBPP_VOL_oracle
root@rmgdbs03 # hares -dep SBPP_VOL_oracle
#Group       Parent            Child
SID_SBPP     SBPP_MOUNT_admin  SBPP_VOL_oracle
SID_SBPP     SBPP_MOUNT_data01 SBPP_VOL_oracle
SID_SBPP     SBPP_MOUNT_data02 SBPP_VOL_oracle
SID_SBPP     SBPP_VOL_oracle   SBPP_MOUNT_arch
root@rmgdbs03 # hares -dep SBPP_MOUNT_arch
#Group       Parent          Child
SID_SBPP     SBPP_MOUNT_arch SBPP_VOL_arch
SID_SBPP     SBPP_VOL_oracle SBPP_MOUNT_arch
root@rmgdbs03 # hares -dep SBPP_VOL_arch
#Group       Parent          Child
SID_SBPP     SBPP_MOUNT_arch SBPP_VOL_arch
root@rmgdbs03 # hares -state SBPP_MOUNT_data02
#Resource         Attribute        System     Value
SBPP_MOUNT_data02 State            rmgdbs02   OFFLINE
SBPP_MOUNT_data02 State            rmgdbs03   ONLINE

The cluster state remains as it is and will not change/fault.should i still wait?

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  SID_SAVS        rmgdbs02             Y          N               PARTIAL
B  SID_SAVS        rmgdbs03             Y          N               ONLINE
B  SID_SBPP        rmgdbs02             Y          N               PARTIAL
B  SID_SBPP        rmgdbs03             Y          N               ONLINE
B  SID_SPTL        rmgdbs02             Y          N               PARTIAL
B  SID_SPTL        rmgdbs03             Y          N               ONLINE
B  cvm             rmgdbs02             Y          N               STOPPING|PARTIAL|FAULTED
B  cvm             rmgdbs03             Y          N               ONLINE

-- RESOURCES FAILED
-- Group           Type                 Resource             System

C  cvm             CFSMount             orabin_mnt           rmgdbs02
C  cvm             CFSMount             srvm_mnt             rmgdbs02

-- RESOURCES OFFLINING
-- Group           Type            Resource             System               IState

F  cvm             CVMCluster      cvm_clus             rmgdbs02             W_OFFLINE_PROPAGATE

[Har-D]

Hi Sumit,

Run the following command from the other node to check if FSCK flag is set :

# echo "8192B.p S" | fsdb -F vxfs /dev/vx/dsk/<DG>/<vol> | grep clean

e.g. the output would be like the below :

flags 101 mod x clean xx

Flag ends with 1, means full fsck is set.

If it is set, you will need to unmount the affected CFS mounts from both the nodes and will need to run Full fsck on those volumes.

[symsonu]

root@rmgdbs02 # hastatus -summ

-- SYSTEM STATE
-- System               State                Frozen

A  rmgdbs02             RUNNING              0
A  rmgdbs03             RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  SID_SAVS        rmgdbs02             Y          N               PARTIAL
B  SID_SAVS        rmgdbs03             Y          N               ONLINE
B  SID_SBPP        rmgdbs02             Y          N               PARTIAL
B  SID_SBPP        rmgdbs03             Y          N               ONLINE
B  SID_SPTL        rmgdbs02             Y          N               PARTIAL
B  SID_SPTL        rmgdbs03             Y          N               ONLINE
B  cvm             rmgdbs02             Y          N               PARTIAL|FAULTED
B  cvm             rmgdbs03             Y          N               ONLINE

-- RESOURCES FAILED
-- Group           Type                 Resource             System

C  cvm             CFSMount             orabin_mnt           rmgdbs02
C  cvm             CFSMount             srvm_mnt             rmgdbs02

-- RESOURCES OFFLINING
-- Group           Type            Resource             System               IState

F  cvm             CVMCluster      cvm_clus             rmgdbs02             W_OFFLINE
root@rmgdbs02 # hares -state orabin_mnt
#Resource    Attribute        System     Value
orabin_mnt   State            rmgdbs02   FAULTED
orabin_mnt   State            rmgdbs03   ONLINE
root@rmgdbs02 # hares -display orabin_mnt
#Resource    Attribute        System     Value
orabin_mnt   Group            global     cvm
orabin_mnt   Type             global     CFSMount
orabin_mnt   AutoStart        global     1
orabin_mnt   Critical         global     1
orabin_mnt   Enabled          global     1
orabin_mnt   LastOnline       global     rmgdbs03
orabin_mnt   MonitorOnly      global     0
orabin_mnt   ResourceOwner    global     unknown
orabin_mnt   TriggerEvent     global     0
orabin_mnt   ArgListValues    rmgdbs02   /opt/oracle    /dev/vx/dsk/orabinvol_dg/orabinvol      ""
orabin_mnt   ArgListValues    rmgdbs03   /opt/oracle    /dev/vx/dsk/orabinvol_dg/orabinvol      ""
orabin_mnt   ConfidenceLevel  rmgdbs02   0
orabin_mnt   ConfidenceLevel  rmgdbs03   100
orabin_mnt   Flags            rmgdbs02
orabin_mnt   Flags            rmgdbs03
orabin_mnt   IState           rmgdbs02   not waiting
orabin_mnt   IState           rmgdbs03   not waiting
orabin_mnt   Probed           rmgdbs02   1
orabin_mnt   Probed           rmgdbs03   1
orabin_mnt   Start            rmgdbs02   1
orabin_mnt   Start            rmgdbs03   1
orabin_mnt   State            rmgdbs02   FAULTED
orabin_mnt   State            rmgdbs03   ONLINE
orabin_mnt   BlockDevice      global     /dev/vx/dsk/orabinvol_dg/orabinvol
orabin_mnt   ComputeStats     global     0
orabin_mnt   ForceOff         global
orabin_mnt   MountOpt         global
orabin_mnt   MountPoint       global     /opt/oracle
orabin_mnt   MountType        global
orabin_mnt   NodeList         global
orabin_mnt   Policy           global
orabin_mnt   Primary          global     rmgdbs03
orabin_mnt   RemountRes       global
orabin_mnt   SetPrimary       global
orabin_mnt   MonitorTimeStats rmgdbs02   Avg    0       TS
orabin_mnt   MonitorTimeStats rmgdbs03   Avg    0       TS
orabin_mnt   ResourceInfo     rmgdbs02   State  Stale   Msg             TS
orabin_mnt   ResourceInfo     rmgdbs03   State  Valid   Msg             TS

root@rmgdbs02 # hares -state orabin_mnt
#Resource    Attribute        System     Value
orabin_mnt   State            rmgdbs02   FAULTED
orabin_mnt   State            rmgdbs03   ONLINE
root@rmgdbs02 # hares -display orabin_mnt
#Resource    Attribute        System     Value
orabin_mnt   Group            global     cvm
orabin_mnt   Type             global     CFSMount
orabin_mnt   AutoStart        global     1
orabin_mnt   Critical         global     1
orabin_mnt   Enabled          global     1
orabin_mnt   LastOnline       global     rmgdbs03
orabin_mnt   MonitorOnly      global     0
orabin_mnt   ResourceOwner    global     unknown
orabin_mnt   TriggerEvent     global     0
orabin_mnt   ArgListValues    rmgdbs02   /opt/oracle    /dev/vx/dsk/orabinvol_dg/orabinvol      ""
orabin_mnt   ArgListValues    rmgdbs03   /opt/oracle    /dev/vx/dsk/orabinvol_dg/orabinvol      ""
orabin_mnt   ConfidenceLevel  rmgdbs02   0
orabin_mnt   ConfidenceLevel  rmgdbs03   100
orabin_mnt   Flags            rmgdbs02
orabin_mnt   Flags            rmgdbs03
orabin_mnt   IState           rmgdbs02   not waiting
orabin_mnt   IState           rmgdbs03   not waiting
orabin_mnt   Probed           rmgdbs02   1
orabin_mnt   Probed           rmgdbs03   1
orabin_mnt   Start            rmgdbs02   1
orabin_mnt   Start            rmgdbs03   1
orabin_mnt   State            rmgdbs02   FAULTED
orabin_mnt   State            rmgdbs03   ONLINE
orabin_mnt   BlockDevice      global     /dev/vx/dsk/orabinvol_dg/orabinvol
orabin_mnt   ComputeStats     global     0
orabin_mnt   ForceOff         global
orabin_mnt   MountOpt         global
orabin_mnt   MountPoint       global     /opt/oracle
orabin_mnt   MountType        global
orabin_mnt   NodeList         global
orabin_mnt   Policy           global
orabin_mnt   Primary          global     rmgdbs03
orabin_mnt   RemountRes       global
orabin_mnt   SetPrimary       global
orabin_mnt   MonitorTimeStats rmgdbs02   Avg    0       TS
orabin_mnt   MonitorTimeStats rmgdbs03   Avg    0       TS
orabin_mnt   ResourceInfo     rmgdbs02   State  Stale   Msg             TS
orabin_mnt   ResourceInfo     rmgdbs03   State  Valid   Msg             TS
root@rmgdbs02 # hares -display srvm_mnt
#Resource    Attribute        System     Value
srvm_mnt     Group            global     cvm
srvm_mnt     Type             global     CFSMount
srvm_mnt     AutoStart        global     1
srvm_mnt     Critical         global     1
srvm_mnt     Enabled          global     1
srvm_mnt     LastOnline       global     rmgdbs03
srvm_mnt     MonitorOnly      global     0
srvm_mnt     ResourceOwner    global     unknown
srvm_mnt     TriggerEvent     global     0
srvm_mnt     ArgListValues    rmgdbs02   /opt/oracle_srvm       /dev/vx/dsk/orasrv_dg/srvm_vol  ""
srvm_mnt     ArgListValues    rmgdbs03   /opt/oracle_srvm       /dev/vx/dsk/orasrv_dg/srvm_vol  ""
srvm_mnt     ConfidenceLevel  rmgdbs02   0
srvm_mnt     ConfidenceLevel  rmgdbs03   100
srvm_mnt     Flags            rmgdbs02
srvm_mnt     Flags            rmgdbs03
srvm_mnt     IState           rmgdbs02   not waiting
srvm_mnt     IState           rmgdbs03   not waiting
srvm_mnt     Probed           rmgdbs02   1
srvm_mnt     Probed           rmgdbs03   1
srvm_mnt     Start            rmgdbs02   1
srvm_mnt     Start            rmgdbs03   1
srvm_mnt     State            rmgdbs02   FAULTED
srvm_mnt     State            rmgdbs03   ONLINE
srvm_mnt     BlockDevice      global     /dev/vx/dsk/orasrv_dg/srvm_vol
srvm_mnt     ComputeStats     global     0
srvm_mnt     ForceOff         global
srvm_mnt     MountOpt         global
srvm_mnt     MountPoint       global     /opt/oracle_srvm
srvm_mnt     MountType        global
srvm_mnt     NodeList         global
srvm_mnt     Policy           global
srvm_mnt     Primary          global     rmgdbs03
srvm_mnt     RemountRes       global
srvm_mnt     SetPrimary       global
srvm_mnt     MonitorTimeStats rmgdbs02   Avg    0       TS
srvm_mnt     MonitorTimeStats rmgdbs03   Avg    0       TS
srvm_mnt     ResourceInfo     rmgdbs02   State  Stale   Msg             TS
srvm_mnt     ResourceInfo     rmgdbs03   State  Valid   Msg             TS

============================================================================================================

checking the status of fsck flags on these two filesystems from other node which is CFS primary and master node

root@rmgdbs03 # echo "8192B.p S" | fsdb -F vxfs /dev/vx/dsk/orabinvol_dg/orabinvol |grep clean
flags 0 mod 0 clean 3c
root@rmgdbs03 #  echo "8192B.p S" | fsdb -F vxfs /dev/vx/dsk/orasrv_dg/srvm_vol | grep clean
flags 0 mod 0 clean 3c
root@rmgdbs03 #