Forum Discussion

symsonu's avatar
symsonu
Level 6
12 years ago

need to know the meaning of logs

 

 

Can sommeone please tell me what has happened?

 

2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 1 (priveth1) node 1 in trouble
2013 Jun 19 17:42:20 kyornas051_01 kernel: LLT INFO V-14-1-10205 link 0 (priveth0) node 1 in trouble
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 8 sec (1698350566)
2013 Jun 19 17:42:26 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 8 sec (1698356198)
2013 Jun 19 17:42:27 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 9 sec (1698350566)
2013 Jun 19 17:42:27 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 9 sec (1698356198)
2013 Jun 19 17:42:28 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 10 sec (1698350566)
2013 Jun 19 17:42:28 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 10 sec (1698356198)
2013 Jun 19 17:42:29 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 11 sec (1698350566)
2013 Jun 19 17:42:29 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 11 sec (1698356198)
2013 Jun 19 17:42:30 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 12 sec (1698350566)
2013 Jun 19 17:42:30 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 12 sec (1698356198)
2013 Jun 19 17:42:31 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 13 sec (1698350566)
2013 Jun 19 17:42:31 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 13 sec (1698356198)
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 4 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 14 sec (1698350566)
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 3 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 4 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 2 more to go.
2013 Jun 19 17:42:32 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 14 sec (1698356198)
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 3 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 1 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 2 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (priveth1) node 1. 0 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 1 (priveth1) node 1 inactive 15 sec (1698350566)
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 1 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 1 (priveth1) node 1 expired
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (priveth0) node 1. 0 more to go.
2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port h gen  1132317 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port v gen  113231a membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port w gen  113231c membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port a gen  1132305 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port b gen  1132314 membership 0
2013 Jun 19 17:42:38 kyornas051_01 kernel: GAB INFO V-15-1-20036 Port f gen  113231e membership 0
2013 Jun 19 17:42:38 kyornas051_01 Had[30829]: VCS INFO V-16-1-10077 Received new cluster membership
2013 Jun 19 17:42:38 kyornas051_01 kernel: VXFEN INFO V-11-1-68 Completed ejection of leaving node(s) from data disks.
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-7899 CVM_VOLD_CHANGE command received
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-13170 Preempting CM NID 1
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-0 Calling join complete
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-8062 master: not a cluster startup
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-10994 join completed for node 0
2013 Jun 19 17:42:38 kyornas051_01 vxvm:vxconfigd: V-5-1-4123 cluster established successfully
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED
2013 Jun 19 17:42:39 kyornas051_01 sfsfs_event.network.alert: Node kyornas051_02 went offline.
2013 Jun 19 17:42:39 kyornas051_01 sshd[17509]: Accepted publickey for root from 172.16.0.3 port 42449 ssh2
2013 Jun 19 17:42:39 kyornas051_01 sshd[17515]: Accepted publickey for root from 172.16.0.3 port 42450 ssh2
2013 Jun 19 17:42:41 kyornas051_01 kernel: vxfs: msgcnt 617 Phase 0 - /dev/vx/dsk/sfsdg/_nlm_ - Blocking buffer reads for recovery. gencnt 1 primary 0 leavers: 0x2 0x0 0x0 0x0
2013 Jun 19 17:42:41 kyornas051_01 kernel:

business continuity
Cluster Server
Monitoring
Roadmap
Solaris
Troubleshooting
Upgrade
  • g_lee's avatar
    g_lee
    12 years ago

    Which part is unclear - the logs appear to be fairly self explanatory?

    Node 0 kyornas051_01 did not get a response from Node 1 kyornas051_02 over LLT links priveth0 and priveth1 for 16 consecutive seconds:

    2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired

    so node 1 was paniced/taken down, hence:

    2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED

    this is expected behaviour - see https://sort.symantec.com/public/documents/sfha/6.0.1/solaris/productguides/html/vcs_admin/ch09s02s02.htm

  • g_lee's avatar
    g_lee
    Level 6
    12 years ago

    Which part is unclear - the logs appear to be fairly self explanatory?

    Node 0 kyornas051_01 did not get a response from Node 1 kyornas051_02 over LLT links priveth0 and priveth1 for 16 consecutive seconds:

    2013 Jun 19 17:42:33 kyornas051_01 kernel: LLT INFO V-14-1-10032 link 0 (priveth0) node 1 inactive 15 sec (1698356198)
2013 Jun 19 17:42:34 kyornas051_01 kernel: LLT INFO V-14-1-10509 link 0 (priveth0) node 1 expired

    so node 1 was paniced/taken down, hence:

    2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10079 System kyornas051_02 (Node \'1\') is in Down State - Membership: 0x1
2013 Jun 19 17:42:39 kyornas051_01 Had[30829]: VCS ERROR V-16-1-10322 System kyornas051_02 (Node \'1\') changed state from RUNNING to FAULTED

    this is expected behaviour - see https://sort.symantec.com/public/documents/sfha/6.0.1/solaris/productguides/html/vcs_admin/ch09s02s02.htm

  • Daniel_Matheus's avatar
    Daniel_Matheus
    Level 4
    12 years ago

    As per above, either the 2nd node became unresponsive or the network was disrupted and so the node got fenced out. This is indeed expected behaviour. Node will be marked as faulted until it rejoins the cluster

  • avsrini's avatar
    avsrini
    Level 4
    12 years ago

    Hi Sonu,

    what happened to node kyornas051_02? Was it rebooted?