CFS Primary operations

[PROTON]

Hi,

Can anybody list the 'Special' operations that are performed only by a CFS primary node for an FS, like creating ckpt, etc.

Need to get more info on this.

Thanks.

[ScottK]

It depends on the version.

If you go back to version 4.0, many operations had to be done through the primary -- most operations involving the allocation of free space, for example.

From then until now, a key focus has been to allow more operations to be performed by all nodes, because that improves the scalability of the overall cluster -- since the master is no longer a bottleneck for an operation that any node can perform. Most of those changes were made in the 5.0 release.

In short, you may want to specify the version of interest.

[joseph_dangelo]

There are two distinct designations within CVM and CFS.  Master/Slave and Primar/Secondary.  The latter of course is associated with CFS.  The most important role a Primary Node plays for a particular FS is the distribution and control of File Locking and interfacing with the GLM (Global Lock Manager).

The locking itself is handled via CFS for both metadata and user data coherency.  The locking mechanism (GLM) is specifically designed to coordinate access to file system metadata i.e. inodes & free lists).   Each CFS mount is assigned a GLM primary.  The primary is responsible for file create and delete operations. This allows for the load balancing of the lock coherency operations across all available cluster nodes. Furthermore, GLM will also ensure that when a particular file is being accessed, not only will that host have an exclusive “lock” on the particular data range being written to, but that all other nodes of the cluster that have the same file cached will have their cached copy invalidated.

All inter-node GLM communication is handled via Port f of GAB while all block changes are over the SAN directly from each host.  GLM is essentially a traffic cop that allows or disallows access to a particular portion of file instead of the entire file itself.  This architecture (range locking) allows for a single file to be access concurrently for reads and writes across all cluster nodes. 

Hope this helps.

Joe D

[PROTON]

I am interested in 5.0 and 5.1 releases.

[ScottK · Accepted solution]

Good points, Joe --

Couple of other notes

GLM hashes the ownership of file locks across the cluster for scalability, so once a file is created, the primary is not a bottleneck for locks. What if one node is constantly accessing a file? GLM will notice that and delegate the ownership for that file.

Joe also hit on CVM Master/Slave. At the volume layer, operations are much more centralized compared with CFS primary/secondary. So the CVM Master will take care of virtually all metadata operations involving volumes -- resize, relayout, create, remove, rename, etc.