Agent exiting in vcs 6.1 on rhel6.6

Hi All,

 

i have 10 nodes vcs cluster setup on rhel 6.6  ,here 28 service groups are running . some are parallel and mostly are failover service groups

i am getting below error around work

 

 

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10121]: VCS ERROR V-16-2-13120 Thread(4151479088) Error receiving from the engine. Agent(LVMVolumeGroup) is exiting

Jun 19 05:36:50 HTNDPUEDSVC01 t of memory [5967]

Jun 19 05:36:50 HTNDPUEDSVC01 lloc() FAILED at file Memory.C, line 376, memory could not be allocated. Process ID 5967 dumping core deliberately!

Jun 19 05:37:12 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 22 sec

Jun 19 05:37:13 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 23 sec

Jun 19 05:37:14 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 24 sec

Jun 19 05:37:15 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 25 sec

Jun 19 05:37:16 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 26 sec

Jun 19 05:37:17 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 27 sec

Jun 19 05:37:18 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 28 sec

Jun 19 05:37:19 HTNDPUEDSVC01 abrt[14811]: Write error: No space left on device

Jun 19 05:37:19 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20057 Port h[GAB_USER_CLIENT (refcount 0)] process 5967 inactive 29 sec

Jun 19 05:37:19 HTNDPUEDSVC01 abrt[14811]: Error writing '/var/spool/abrt/ccpp-2016-06-19-05:36:50-5967.new/coredump'

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20058 Port h[GAB_USER_CLIENT (refcount 0)] process 5967: heartbeat failed, killing process

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20059 Port h[GAB_USER_CLIENT (refcount 0)] heartbeat interval 30000 msec. Statistics:

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20129 Port h: heartbeats in 0 ~ 6000 msec: 75483871

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20129 Port h: heartbeats in 6000 ~ 12000 msec: 0

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20129 Port h: heartbeats in 12000 ~ 18000 msec: 0

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20129 Port h: heartbeats in 18000 ~ 24000 msec: 0

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20129 Port h: heartbeats in 24000 ~ 30000 msec: 0

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20088 System information:

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20089     number of cpu:                 16

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20090     physical memory:               49426504 K

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20091     free memory:                   5399360 K

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20041 Port h: client process failure: killing process

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB WARNING V-15-1-20161 Port h client process killed, GAB will initiate regmon action syslog after 200 sec

Jun 19 05:37:20 HTNDPUEDSVC01 kernel: GAB INFO V-15-1-20032 Port h closed

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10130]: VCS ERROR V-16-2-13120 Thread(4151511776) Error receiving from the engine. Agent(HostMonitor) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10121]: VCS ERROR V-16-2-13120 Thread(4151479088) Error receiving from the engine. Agent(LVMVolumeGroup) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10133]: VCS ERROR V-16-2-13120 Thread(4151663328) Error receiving from the engine. Agent(Proxy) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10116]: VCS ERROR V-16-2-13120 Thread(4151875280) Error receiving from the engine. Agent(Application) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10128]: VCS ERROR V-16-2-13120 Thread(4151543504) Error receiving from the engine. Agent(VMwareDisks) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10120]: VCS ERROR V-16-2-13120 Thread(4151569120) Error receiving from the engine. Agent(IP) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10122]: VCS ERROR V-16-2-13120 Thread(4151507680) Error receiving from the engine. Agent(LVMLogicalVolume) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10124]: VCS ERROR V-16-2-13120 Thread(4151524144) Error receiving from the engine. Agent(Phantom) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 AgentFramework[10126]: VCS ERROR V-16-2-13120 Thread(4152023856) Error receiving from the engine. Agent(NIC) is exiting.

Jun 19 05:37:20 HTNDPUEDSVC01 hashadow[4005]: VCS ERROR V-16-1-11103 VCS exited. It will restart

 

Please suggest what action i should take at this place of error

 

 

 

 

1 Solution

Accepted Solutions
Highlighted
Accepted Solution!

Re: Agent exiting in vcs 6.1 on rhel6.6

Hi,

How is the load on the system ? These errors are quite common if system is over loaded. I would recommend to have a look at system performance to begin with. What VCS is displaying above might be a result of poor performance of server.

Also, I see the error that no space left on the device, is system low on space where logs are getting written ?

If you google around "port h client process failure", you will find many articles .. for e.g.

https://www.veritas.com/support/en_US/article.TECH1794.html

 

G

 

2 Replies
Highlighted
Accepted Solution!

Re: Agent exiting in vcs 6.1 on rhel6.6

Hi,

How is the load on the system ? These errors are quite common if system is over loaded. I would recommend to have a look at system performance to begin with. What VCS is displaying above might be a result of poor performance of server.

Also, I see the error that no space left on the device, is system low on space where logs are getting written ?

If you google around "port h client process failure", you will find many articles .. for e.g.

https://www.veritas.com/support/en_US/article.TECH1794.html

 

G

 

Re: Agent exiting in vcs 6.1 on rhel6.6

Thanks Gaurav,

 

I think it would be helpfull