I am runnign a concurrent sequenital write test from 8 nodes. The test creates 5 x 40GiB files on each node. What I have observed (vxfs 7.4 Linux).
1. Start the file creates on all nodes at the same time
2. The first node file creates run quickly, then finish.
3. After the first node finishes the file creation on the rest of the nodes starts and runs much slower than the first node.
Whay the imbalance?
Out of 8 nodes in cluster, node where file system is mounted first, becomes primary node for the file system. Other nodes where the file system is mounted afterwards becomes secondary nodes for the file system. It seems that in your case, 'first node' is primary node for the file system. This you can confirm with "fsclustadm [-v] showprimary mount_point". Primary node may perform the operation little faster as it does not need to get extent/inode delegation from delegation master (primary node) over network. Secondary node needs to get inode/extent delegation from primary node. This involves message exchanges over network.
Could you pls mention following (just to get better idea about operation being performed) -
Does each node is creating its files in seperate directory ?
What command is being used for sequential write (dd or something else)
How much time difference is being seen in these operation on nodes ?
Also what is the file system size ?
