CFS Health Check ?
Hi All,
We have recently built a 4-node Oracle single-instance CFS cluster, with more to on the way.
The cluster will host many databases, and all nodes will be active. The databases will migrate from non-clustered to CFS clustered servers, and we are concerned about what effect CFS may have on performance.
Concerns:
- Suitability of our LLT links to handle intra-node communication generated by CFS ?
- Location of CFS primary, for each filesytem, and how to determine if it is optimal ?
- When multiple nodes write to the same filesystem simultaeously, would that be detrimental to performance ?
- I am aware of GLM lock/release as one source of CFS intra-node traffic, but wondering if there are other significant sources, such as VxFS cache coherency ?
- How to best monitor CFS overhead, intra-node traffic, and overall health ?
Thank your for reading !
Ken W
---------------------Configuration---------------------
Red Hat RHEL 6.5
SFCFSHA 6.1.1 with P200 patch
2 x 1 Gbit ethernet as LLT links, on dedicated VLANs
Oracle 11.2.0.3 and 12.1.0.2
Oracle code local to each node (outside of cluster control)
Oracle data files on CFS filesystem dedicated to only datases running on that node.
Oracle Redo on a single shared filesytem for all nodes.
-------------------------------------------------------------
- Hi, Ken In a summary, you can use common performance tools like : top, vmstat etc to monitor CFS system performance, cpu usage, memory usage, io busy. Several suggestions: 1. use fsadm -DE
to check fragment report of file system, avoid fragmentation. 2. avoid too many small files under one directory. 3. multinode nodes write to a file system simultaeously, try to put data in different directory. More refer guide https://sort.symantec.com/public/documents/sfha/6.1/linux/productguides/html/sfcfs_admin/pt05.htm About your concern: - Suitability of our LLT links to handle intra-node communication generated by CFS ? < StarFly... Thank you for CFS best practice recommendation.
In addition, I have found...
1. Monitor LLT link utilization at the switch VLAN level using Cisco tools.
2. Very little performance degrade when CFS primary is located on an "off" node (non-writer).
3. CFS primary can be manually set after CFS is onlined (non-persistent).
# fsclustadm setprimary /mnt (sets to this node)
# fsclustadm show primary /mnt (report current primary)
4. CFS "Policy" attribute will set the primary automatically every time CFS is onlined, from a list of primaries. This is a persistent setting.
# hares -modify <CFSMount_resource> Policy NodeB NodeA
5. Found 'glmstat' command but could not locate any use cases.
6. Recommended articles:
How to set primary node for a cluster file system
http://www.symantec.com/business/support/index?page=content&id=TECH39266Moving CFS Primary when using SFCFS for fast failover (mikebounds)
https://www-secure.symantec.com/connect/forums/moving-cfs-primary-when-using-sfcfs-fast-failoverThank You,
Ken W
