Veritas cluster issue

[ARUP_JYOTI_THAK]

Hi

We have a VCS of two node. Two file systems are under VCS configured from VXVM. One file system got 100% full.Now we have rebooted the cluster node & the mount points started to show,but after some time that file system got disappear,Checked with concerned DG that also got disabed,we tried to manually import & got succeded.But after starting the volumes we tried to mount the filesystem under the specified mount point. But the below error giving---

# mount  /dev/vx/dsk/bgw1dg/vol01 /var/opt/BGw/Server1

mount: /dev/vx/dsk/bgw1dg/vol01 is not this fstype

#

Kindly suggest what could be the cause.

regards...Arup

[Marianne · Accepted solution]

If this is VxFS filesystem, add '-F' vxfs to the mount command.

Maybe best to fsck filesystem first: 

fsck -F vxfs /dev/vx/rdsk/bgw1dg/vol01 

Then mount it:

 mount -F vxfs /dev/vx/dsk/bgw1dg/vol01 /var/opt/BGw/Server1

The reason why diskgroup 'disappeared' after reboot is because it is under cluster control and imported with '-t'. 

PS: Not sure how you hoped for a reboot to fix the 100% full issue? 
Seems your Service Group is not configured to automatically failover to 2nd node when system is rebooted?

[Daniel_Matheus]

Hi Arup,

As Marianne suggested a reboot won't fix a full file system.

You either need to remove data or increase the volume and the file system (For that you wll also need to make some space otherwise the resize will bail out).

For the mount not appearing.

The diskgroup needs to be imported, the volume started and the file system mounted.

So please make sure that you have a diskgroup resource, a volume resource and a mount resource in your service group with correct dependencies set (i.e. mount-->volume-->diskgroup)

In case you are using Linux you need to use mount -t

regards,
Dan

[ARUP_JYOTI_THAK]

Thanks Marianne,the reboot was done from the CU end.Now the issue got resolved. My query is that after reboot mount point are showing,but after certain period of time it again got disappeared & what I have found that the DG also get deported automatically.As this is 100% full..that is why????

@ Daniel--As I mentioned mount point appeared but got disappear again,as because of the space.

As after manually mounting the filesystem, & deleting files & making some spaces,that issue got resolved.

regards...Arup

[g_lee · Accepted solution]

As this is 100% full..that is why????

As everyone above has pointed out, the filesystem being 100% full is not going to cause the fs to unmount or the dg to deport in and of itself.

If the dg and filesystem are clustered, and an application depends on them couldn't write to the filesystem due to it being full, if this would be enough to fault the application, this could have triggered a failover if the application was configured as Critical - so VCS would have taken the group (including the filesystem and dg) offline and brought them up elsewhere if it was able.

However, this is only one possible scenario based on many many assumptions - you need to look at your configuration and logs to determine exactly what happened in your case.

[Daniel_Matheus]

As g_lee pointed out, if the file system got mounted after the reboot and then unmounted the probable reason would be that another resource which sits on top of the mount resource is critical and faulted (probably application due to space issue). You should check the engine_A log for details (/var/VRTSvcs/log/engine_A.log)

[g_lee: edited to amend username]