VCS ERROR V-16-10031-9512 (inactive_nodename) Process:<myprocess> Process does not exist

VCS monitors, by default, every 5 mins on inactive nodes (determined by OfflineMonitorInterval type attribute) and every minute (determined by MonitorInterval type attribute).

But if a Monitor returns offline it should not normally log a message - do you have LogDbg attribute set on Process Type?  I would guess you don't as it looks as though severity of message is ERROR which means the message would be sent even if LogDbg was not set.

As State is unknown this suggests more than VCS just can't find process, but there is some error determining this.  Could you post extract from main.cf showing process resource and any other log messages in the engine_A.log AND Process_A.log relating to this resource at around the same time.  If there are no other message it maybe worth setting DBG_AGDEBUG and DBG_AGTRACE on LogDbg for Process type:

"Why would VCS even be looking for the process on the inactive node?"

Because it needs to failover to the inactive node should the need arise.

VCS needs to know the state of all resources before a node can be considered as failover target. It needs to know at all times that a failover target is available.

Please verify resource attributes on the inactive node.

Mike - here is the section from main.cf:

        Process CDproc (
                Critical = 0
                PathName = "/connect_direct/ndm/bin/cdpmgr"
                Arguments = "/connect_direct/ndm/bin/cdpmgr -i /connect_direct/ndm/cfg/aucdirect/initparm.cfg"
                )
 

The only output in engine_A.log is this every five minutes:

2011/12/14 08:40:25 VCS ERROR V-16-10031-9512 (pvpr1o12) Process:CDproc:monitor:Process(/connect_direct/ndm/bin/cdpmgr) does not exist.
2011/12/14 08:45:24 VCS ERROR V-16-10031-9512 (pvpr1o12) Process:CDproc:monitor:Process(/connect_direct/ndm/bin/cdpmgr) does not exist.
2011/12/14 08:50:25 VCS ERROR V-16-10031-9512 (pvpr1o12) Process:CDproc:monitor:Process(/connect_direct/ndm/bin/cdpmgr) does not exist.
2011/12/14 08:55:24 VCS ERROR V-16-10031-9512 (pvpr1o12) Process:CDproc:monitor:Process(/connect_direct/ndm/bin/cdpmgr) does not exist.

There are no recent messages in Process_A.log (none since Dec 2nd).

Marianne - it certainly makes sense that VCS knows the status of the inactive node.  But why issue messages that the process resource was not found if VCS is not going to do the same for the DG, NIC, mounts and volumes? 

I am pretty new to this; what would you like to see regarding this:

"Please verify resource attributes on the inactive node."

Hi sclind,

The error message is teilling you that it cannot find the path "/connect_direct/ndm/bin/cdpmgr".  It is trying to detemine if the Process agent is running but it cannot find the path to the scripts/process that you provided.  As a result it cannot validate the state of the resource.

If I had to guess, the path "/connect_direct/ndm/bin/cdpmgr" is being mounted in your service gorup and does not exist on the passive node.  VCS is expecting that the binary path is available on all nodes and the data files are being moved during failover.

The NIC, DG, mount and volume resources have all of their binaries needed for the resource to perform online, offline and monitor entry points installed on all nodes.

The process agent has its binaries on all nodes too but you have configured it to run additional binaries from the "/connect_direct/ndm/bin/cdpmgr" path.

Thanks,

Wally

Wally - "/connect_direct/ndm/bin/cdpmgr" is the complete path and program name.   But why would VCS even be looking for the process on the passive/inactive node?  By definition it's not running there. 

The base problem we have here is we monitor for any line that has "ERROR" in it (as opposed to INFO, CRITICAL, WARNING, etc).  With the exception of this message all other lines I've ever seen that say "ERROR" are *real* errors.  It just seems odd to me that VCS writes a "ERROR" level message to the log for what does not seem to really be an error.

Hi sclind,

VCS is checking if the process is running on the passive node or not.  As Marianne mentioined, if it does not check the resource state  periodically then VCS cannot be sure of the state on the passive node and the resource would be marked as being in an "UNKNOWN" state and would not allow failover to that node.

I'm a Windows guy, so I can just give you basica guidence as to what the resource is doing on non-Windows platforms.  There might be a way to disable the logging of this message but the real fix would be to determine why the agent cannot find the path that it needs.

I would recommend opening a case with Symantec Technical Support so that you can get to an expert that deals with this agent on the OS platform that you are using.  They would be able to provide more assistance than I can on your configuration.

Thanks,

Wally

Wally is probably right in that binaries does not exist on inactive node, but from your initial post it looks as though there are 3 nodes where you resource is:

1. Online on node 1
2. Offline on node 2
3. Unknown on node 3

So if Process binary is on shared storage then I don't know why node 2 is not giving error messages also.

I would consider this to be a bug.  I had the same issue with the Informix Agent just over a year ago where I was getting errors on the inactive node as the binaries did not exist as there were on shared storage.  I raised a call and the issue was resolved in the next release of the agent.  So as Wally says, I would log a Support call.

Alternatively you could use Application resource type - this would require you to write an offline script, but if there is a graceful way to shutdown cdpmgr, like "cdpmgrt stop" or equiv, then this would be better as using Process resource type means VCS just send SIGTERM to process.

Mike

Mike - it is true the binaries do not exist on the inactive node; they are in the mounted file system.

There are only two nodes in the cluster; node 1 being onlinee and node 2 being offline.  The error messages are issued for the offline node.

I see what I wrote can be misleading:

                myprocess               inactive_node  OFFLINE
                myprocess               inactive_node        |STATE UNKNOWN|

"inactive_node" is the one offline node.  It's odd that it is both OFFLINE and STATE UNKNOWN.