Had[14065]: [ID 702911 daemon.notice] VCS ERROR V-16-20010-302

[forever_hwang]

# hastatus -sum

-- SYSTEM STATE

-- System               State                Frozen

A  UMTS-Aart-EMS        RUNNING              0

-- GROUP STATE

-- Group           System               Probed     AutoDisabled    State

B  ClusterService  UMTS-Aart-EMS        Y          N               ONLINE

B  netnumen        UMTS-Aart-EMS        Y          N               ONLINE

B  replication     UMTS-Aart-EMS        Y          N               ONLINE

-- RESOURCES OFFLINING

-- Group           Type            Resource             System               IState

G  replication     RVG             rep-rvg              UMTS-Aart-EMS        W_OFFLINE_PATH

-- WAN HEARTBEAT STATE

-- Heartbeat       To                   State

M  Icmp            cluster2             UNKNOWN

-- REMOTE CLUSTER STATE

-- Cluster         State

N  cluster2        LOST_CONN

-- REMOTE SYSTEM STATE

-- cluster:system       State                Frozen

O  cluster2:UMTS-Gent-EMS RUNNING              0

-- REMOTE GROUP STATE

-- Group           cluster:system       Probed     AutoDisabled    State

P  netnumen        cluster2:UMTS-Gent-EMS Y          N               OFFLINE

# dmesg | tail -8

Jan  3 10:57:51 UMTS-Aart-EMS Had[14065]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13079 (UMTS-Aart-EMS) Resource(rep-rvg): The last 13800 invocations of the clean procedure have failed.

Jan  3 10:58:54 UMTS-Aart-EMS Had[14065]: [ID 702911 daemon.notice] VCS ERROR V-16-20010-302 (UMTS-Aart-EMS) RVG:rep-rvg:clean:The forced stop of RVG netnumenrvg using the command 'vxrvg -g netnumendg -f stop netnumenrvg' failed

Jan  3 10:59:56 UMTS-Aart-EMS last message repeated 1 time

Jan  3 11:00:58 UMTS-Aart-EMS Had[14065]: [ID 702911 daemon.notice] VCS ERROR V-16-20010-302 (UMTS-Aart-EMS) RVG:rep-rvg:clean:The forced stop of RVG netnumenrvg using the command 'vxrvg -g netnumendg -f stop netnumenrvg' failed

Jan  3 11:07:10 UMTS-Aart-EMS last message repeated 6 times

Jan  3 11:08:12 UMTS-Aart-EMS Had[14065]: [ID 702911 daemon.notice] VCS ERROR V-16-20010-302 (UMTS-Aart-EMS) RVG:rep-rvg:clean:The forced stop of RVG netnumenrvg using the command 'vxrvg -g netnumendg -f stop netnumenrvg' failed

Jan  3 11:08:13 UMTS-Aart-EMS Had[14065]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13079 (UMTS-Aart-EMS) Resource(rep-rvg): The last 13810 invocations of the clean procedure have failed.

Jan  3 11:09:16 UMTS-Aart-EMS Had[14065]: [ID 702911 daemon.notice] VCS ERROR V-16-20010-302 (UMTS-Aart-EMS) RVG:rep-rvg:clean:The forced stop of RVG netnumenrvg using the command 'vxrvg -g netnumendg -f stop netnumenrvg' failed

# date

Fri Jan  3 11:10:09 CET 2014

[CraigV]

...what application is this for? You've tagged Solaris as the OS but that's about it...

Thanks!

**** Moved to correct Forum by MvdB ****

[Marianne]

There seems to be something wrong at VVR level. 
Do you know if any changes were made to replication? 

There is not enough info in the information provided to tell us what exactly is wrong.

Please locate the /var/VRTSvcs/log/engine_A.log file and copy all entries for Jan 3.
We need to see where first fault was logged.

Please also post  /etc/VRTSvcs/conf/config/main.cf
(Copy log and main.cf to txt files and upload as File attachments).

Depending on what we see in logs and cluster config, we will ask you to check VVR config and status.

[mikebounds]

I assume the netnumen service group contains your application which is dependent on the "replication" service group and therefore the RVG resource will not be able to offline as the netnumen service group is online which presumbly has mounts in it for volumes in the RVG.

The RVG resource should not attempt to offline without first offlining the netnumen service group, so you should check you have a service group dependency between netnumen and "replication" and that the dependency is "online local hard" (NOT "online local firm") - you can check this by running "hagrp -dep"

To get rid of the W_OFFLINE state, if you are using Java GUI, then you can copy RVG resource, check what resources it is linked to, then delete resource, then paste it back in and link it back to the resources.

Mike