VCS ERROR V-16-2-13067 SERVER01 Agent is calling clean for resource because the resource became OFFLINE unexpectedly, on its own.

[sachinsharma10]

Hi,,

I have 8 nodes Veritas cluster running on RHEL 5.5 in which we are using Firedrill resources.

For some time now FireDrill resources faulted everyday with below error.

LOGS:

2011/01/21 20:38:06 VCS INFO V-16-20054-101 SERVER01 MirrorViewSnap:mirrorviewsnap_ora:monitor:Ping output: PING XX.XX.XX.XX (XX.XX.XX.XX) 56(84) bytes of data.
64 bytes from XX.XX.XX.XX: icmp_seq=1 ttl=125 time=0.289 ms
--- XX.XX.XX.XX ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.289/0.289/0.289/0.000 ms
2011/01/21 20:38:22 VCS ERROR V-16-2-13067 SERVER01  Agent is calling clean for resource(fd_mnt_oradata2) because the resource became OFFLINE unexpectedly, on its own.
2011/01/21 20:38:23 VCS NOTICE V-16-10031-5512 SERVER01  Mount:fd_mnt_oradata2:clean:Trying force umount with signal 9...
2011/01/21 20:38:23 VCS INFO V-16-2-13716 SERVER01  Resource(fd_mnt_oradata2): Output of the completed operation (clean)
==============================================
Cannot stat /oradata2: Input/output error
Cannot stat /oradata2: Input/output error
Cannot stat /oradata2: Input/output error
==============================================
2011/01/21 20:38:23 VCS INFO V-16-2-13068 SERVER01 Resource(fd_mnt_oradata2) - clean completed successfully.
2011/01/21 20:38:24 VCS INFO V-16-1-10307 Resource fd_mnt_oradata2 (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (Not initiated by VCS)
2011/01/21 20:38:24 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fd_LISTENER (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:24 VCS INFO V-16-20002-40 SERVER01 Netlsnr:fd_LISTENER:offline:lsnrctl returned the following output
+--------------------------------------------------------------------+
LD_LIBRARY_PATH - /usr/lib:
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 24-APR-2012 20:38:24
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=xx.xx.xx.xx)(PORT=1530)))
The command completed successfully
+====================================================================+
2011/01/21 20:38:26 VCS INFO V-16-1-10305 Resource fd_LISTENER (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (VCS initiated)
2011/01/21 20:38:26 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fd_PDDCOTC (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:26 VCS WARNING V-16-20002-23 SERVER01 Oracle:fd_PDDCOTC:offline:Oracle database PDDCOTCF not running
2011/01/21 20:38:27 VCS ERROR V-16-2-13067 SERVER01 Agent is calling clean for resource(fd_mnt_oradata1) because the resource became OFFLINE unexpectedly, on its own.
2011/01/21 20:38:28 VCS NOTICE V-16-10031-5512 SERVER01 Mount:fd_mnt_oradata1:clean:Trying force umount with signal 9...
2011/01/21 20:38:28 VCS INFO V-16-2-13716 SERVER01 Resource(fd_mnt_oradata1): Output of the completed operation (clean)

==============================================
Cannot stat /oradata1: Input/output error
Cannot stat /oradata1: Input/output error
Cannot stat /oradata1: Input/output error
==============================================
2011/01/21 20:38:28 VCS INFO V-16-1-10305 Resource fd_PDDCOTC (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (VCS initiated)
2011/01/21 20:38:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fd_ip_listener (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fd_mnt_oradata1 (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:28 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fd_oradata2 (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:28 VCS INFO V-16-2-13068 SERVER01 Resource(fd_mnt_oradata1) - clean completed successfully.
2011/01/21 20:38:29 VCS INFO V-16-1-10305 Resource fd_mnt_oradata1 (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (VCS initiated)
2011/01/21 20:38:29 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fd_oradata1 (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:29 VCS INFO V-16-1-10306 Resource fd_mnt_oradata1 (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (Previous State = OFFLINE)
2011/01/21 20:38:30 VCS INFO V-16-2-13716 SERVER01 Resource(fd_oradata2): Output of the completed operation (offline)
==============================================
VxVM vxprint ERROR V-5-1-582 Disk group ora_dg_fd: No such disk group
==============================================
2011/01/21 20:38:30 VCS INFO V-16-1-10305 Resource fd_ip_listener (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (VCS initiated)
2011/01/21 20:38:31 VCS INFO V-16-2-13716 SERVER01 Resource(fd_oradata1): Output of the completed operation (offline)
==============================================
VxVM vxprint ERROR V-5-1-582 Disk group ora_dg_fd: No such disk group
==============================================
2011/01/21 20:38:31 VCS INFO V-16-1-10305 Resource fd_oradata2 (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (VCS initiated)
2011/01/21 20:38:31 VCS INFO V-16-1-10305 Resource fd_oradata1 (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (VCS initiated)
2011/01/21 20:38:31 VCS NOTICE V-16-1-10300 Initiating Offline of Resource fd_ora_dg (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:31 VCS WARNING V-16-10031-1521 SERVER01 DiskGroup:fd_ora_dg:offline:The command *vxvol -g ora_dg_fd stopall* failed. Doing a forced stop.
2011/01/21 20:38:32 VCS INFO V-16-2-13716 SERVER01 Resource(fd_ora_dg): Output of the completed operation (offline)
==============================================
VxVM vxvol ERROR V-5-1-607 Diskgroup ora_dg_fd not found
VxVM vxvol ERROR V-5-1-607 Diskgroup ora_dg_fd not found
VxVM vxdg ERROR V-5-1-580 Disk group ora_dg_fd: Flush failed: Disk group is disabled
==============================================

2011/01/21 20:38:33 VCS INFO V-16-1-10305 Resource fd_ora_dg (Owner: unknown, Group: fd_oracle) is offline on SERVER01 (VCS initiated)
2011/01/21 20:38:33 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mirrorviewsnap_ora (Owner: unknown, Group: fd_oracle) on System SERVER01
2011/01/21 20:38:41 VCS INFO V-16-20054-101 SERVER01 MirrorViewSnap:mirrorviewsnap_ora:offline:Ping output: PING XX.XX.XX.XX (XX.XX.XX.XX) 56(84) bytes of data.
64 bytes from XX.XX.XX.XX: icmp_seq=1 ttl=125 time=0.271 ms

[jstucki]

Did the environment work well for some time, and then one day just start experiencing this problem daily?  Does the problem occur at approximately the same time each day?

Has somebody upgraded the NaviCLI software at approximately the same time that the errors started happening?

What is the version of VCS you're using?  And what is the version of the NaviCLI software?

How often do you bring the fire drill service group online?  How long do you leave the fire drill service group online (each time)?

Have you checked the cron jobs scheduled on the system, to see if a cron job was added which might interfere with the VCS configuration?  Are there cron jobs which run NaviCLI commands?

-John

[mikebounds]

There are 2 ways a resource is determined offline in VCS:

1. Monitor routine returns offline
2. Monitor routine timesout 4 times (4 by default) in a row.

Which of these are you seeing - if you are seeing the latter then there will messages in log saying monitor timed out.

I THOUGHT the last message before calling clean if situation was 2 was something like "monitor timed out 4 times in a row so calling clean", not " resource became OFFLINE unexpectedly, on its own", so I think situation is 1, but you then see error "Cannot stat /oradata2: Input/output error" which suggests there is a problem detemining the state of the mount, siggesting monitor maybe timing out.

Does this happen at the same time every day and is it always once a day, or sometimes more than once or not at all?

How often and at what time do you run your firedrill and long do you leave the group online

Mike