CFS mount problem with VCS 5.1

[Mukund_Bhatia]

Hi,

We have a three node cluster running on VCS 5.1. The cluster consists of three BL685cG6 servers running on OEL 5u3. In the VCS logs (/var/VRTSvcs/log), the below info messages are continuously logged.

2011/04/14 05:10:26 VCS INFO V-16-2-13716 (dyl06611app07) Resource(dr_b2borr01_as): Output of the completed operation (monitor)
==============================================
Can't locate Sys.pm in @INC (@INC contains: /opt/VRTSvcs/lib /opt/VRTS/.ACCLib/vcs /opt/VRTS/.ACCLib/system /opt/VRTS/.ACCLib/network /opt/VRTSagents/ha/bin/
WebLogic9 /opt/VRTSperl/lib/5.10.0/x86_64-linux-thread-multi /opt/VRTSperl/lib/5.10.0 /opt/VRTSperl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /opt/VRTSp
erl/lib/site_perl/5.10.0 /opt/VRTSperl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /opt/VRTSperl/lib/site_perl/5.10.0 /opt/VRTSperl/lib/site_perl .) at /o
pt/VRTSagents/ha/bin/WebLogic9/monitor line 67.
BEGIN failed--compilation aborted at /opt/VRTSagents/ha/bin/WebLogic9/monitor line 67.
 

And subsequently the CFS mounts goes offline by giving the following error

==============================================

2011/04/14 05:10:33 VCS INFO V-16-2-13716 (dyl06611app07) Resource(dyorrapp01_cfs_dyorrapp01_shared_soaord02-dg): Output of the completed operation (monitor)

==============================================
basename: write error: No space left on device
dirname: write error: No space left on device
basename: missing operand
Try `basename --help' for more information.
uname: write error: No space left on device
uname: write error: No space left on device
grep: write error: No space left on device
grep: write error: No space left on device
expr: write error: No space left on device
/opt/VRTSvcs/bin/CVMVolDg/monitor[632]: [: 0: unknown operator
expr: write error: No space left on device
/opt/VRTSvcs/bin/CVMVolDg/monitor[767]: [: argument expected

2011/04/14 05:10:36 VCS INFO V-16-1-10307 Resource dyorrapp01_cfs_dyorrapp01_shared_soaord01-dg (Owner: Unspecified, Group: REF_B2BSOAORD01_CFS) is offline o
n dyl06611app07 (Not initiated by VCS)
 

Would anyone the cause of this info messages, errors and the relevant fix for this issue ?

Thanks,

Mukund

[Gaurav_S]

Hi Mukund,

very first thing I would check is what is the usage of your filesystems  ? Any common system filesystem like /  or  /var or swap is getting filled up ?

monitor script is facing issues to update the file it seems...  btw what is VCS version you are using ?

also, can u paste the resource defination from main.cf ??

Gaurav

[mikebounds · Accepted solution]

Looks like you have 2 issues:

First issue I think is that you have not installed VRTSacclib, or the version you have installed is too old.  Download latest ACC library from https://sort.symantec.com/agents

Second issue, as Gaurav says is probably /var or /tmp is full.

Mike