cancel
Showing results for 
Search instead for 
Did you mean: 

need to know the meaning of logs

symsonu
Level 6

 

 

Can sommeone please tell me what has happened?

 

2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 1 (priveth1) node 1 in trouble
2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 0 (priveth0) node 1 in trouble
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 8 sec (1698350566)
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 8 sec (1698356198)
2013 Jun 19 17:42:27 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 9 sec (1698350566)
2013 Jun 19 17:42:27 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 9 sec (1698356198)
2013 Jun 19 17:42:28 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 10 sec (1698350566)
2013 Jun 19 17:42:28 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 10 sec (1698356198)
2013 Jun 19 17:42:29 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 11 sec (1698350566)
2013 Jun 19 17:42:29 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 11 sec (1698356198)
2013 Jun 19 17:42:30 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 12 sec (1698350566)
2013 Jun 19 17:42:30 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 12 sec (1698356198)
2013 Jun 19 17:42:31 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 13 sec (1698350566)
2013 Jun 19 17:42:31 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 13 sec (1698356198)
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 4 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 14 sec (1698350566)
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 3 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 4 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 2 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 14 sec (1698356198)
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 3 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 1 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 2 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 0 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 15 sec (1698350566)
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 1 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 1 (priveth1) node 1 expired
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 0 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port h gen  1132317 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port v gen  113231a membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port w gen  113231c membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port a gen  1132305 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port b gen  1132314 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port f gen  113231e membership 0
2013 Jun 19 17:42:38 kyornas051_01 Had[30829]: VCS INFO V-16-1-10077 Received new cluster membership
2013 Jun 19 17:42:38 kyornas051_01 kernel: VXFEN INFO V-11-1-68 Completed ejection of leaving node(s) from data disks.
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-7899 CVM_VOLD_CHANGE command received
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-13170 Preempting CM NID 1
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-0 Calling join complete
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-8062 master: not a cluster startup
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-10994 join completed for node 0
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-4123 cluster established successfully
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED
2013 Jun 19 17:42:39 kyornas051_01 sfsfs_event.network.alert: Node kyornas051_02 went offline.
2013 Jun 19 17:42:39 kyornas051_01 sshd[17509]: Accepted publickey for root from 172.16.0.3 port 42449 ssh2
2013 Jun 19 17:42:39 kyornas051_01 sshd[17515]: Accepted publickey for root from 172.16.0.3 port 42450 ssh2
2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 617 Phase 0 - /dev/vx/dsk/sfsdg/_nlm_ - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0
2013 Jun 19 17:42:41 kyornas051_01 kernel:

1 ACCEPTED SOLUTION

Accepted Solutions

g_lee
Level 6

Which part is unclear - the logs appear to be fairly self explanatory?

Node 0 kyornas051_01 did not get a response from Node 1 kyornas051_02 over LLT links priveth0 and priveth1 for 16 consecutive seconds:

2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired

so node 1 was paniced/taken down, hence:

2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED

this is expected behaviour - see https://sort.symantec.com/public/documents/sfha/6.0.1/solaris/productguides/html/vcs_admin/ch09s02s02.htm

View solution in original post

3 REPLIES 3

g_lee
Level 6

Which part is unclear - the logs appear to be fairly self explanatory?

Node 0 kyornas051_01 did not get a response from Node 1 kyornas051_02 over LLT links priveth0 and priveth1 for 16 consecutive seconds:

2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired

so node 1 was paniced/taken down, hence:

2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED

this is expected behaviour - see https://sort.symantec.com/public/documents/sfha/6.0.1/solaris/productguides/html/vcs_admin/ch09s02s02.htm

Daniel_Matheus
Level 4
Employee Accredited Certified

As per above, either the 2nd node became unresponsive or the network was disrupted and so the node got fenced out. This is indeed expected behaviour. Node will be marked as faulted until it rejoins the cluster

avsrini
Level 4
Employee Accredited Certified

Hi Sonu,

what happened to node kyornas051_02? Was it rebooted?