Cannot unload GAB and LLT on RHEL 6.0

[sirin_zarin]

Hi all, i have next problem

# lltstat -n
LLT node information:
    Node                 State    Links
     0 srv-n1       OPEN        2
   * 1 srv-n2       OPEN        2

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   7b4d01 membership 01
Port b gen   7b4d05 membership 01
Port d gen   7b4d04 membership 01
Port h gen   7b4d11 membership 01

# /opt/VRTSvcs/bin/haconf -dump -makero
VCS WARNING V-16-1-10369 Cluster not writable.

# /opt/VRTSvcs/bin/hastop -all -force

# /etc/init.d/vxfen stop
Stopping vxfen..
Stopping vxfen.. Done

# /etc/init.d/gab stop
Stopping GAB:
ERROR! Cannot unload GAB module. Clients still exist
Kill/Stop clients corresponding to following ports.
GAB Port Memberships
===============================================================
Port d gen   7b4d04 membership 01

# /etc/init.d/llt stop
Stopping LLT:
LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 3 port(s)
LLT:Warning: lltconfig failed. Retrying [1]
LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 3 port(s)
LLT:Warning: lltconfig failed. Retrying [2]
LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 3 port(s)
LLT:Warning: lltconfig failed. Retrying [3]
LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 3 port(s)
LLT:Warning: lltconfig failed. Retrying [4]
LLT lltconfig ERROR V-14-2-15121 LLT unconfigure aborted, unregister 3 port(s)
LLT:Warning: lltconfig failed. Retrying [5]
LLT:Error: lltconfig failed

OK, i see lsmod and modinfo for details...

# lsmod
Module                  Size  Used by
vxodm                 206291  1
vxgms                 284352  0
vxglm                 289848  0
gab                   283317  4
llt                   180985  5 gab
autofs4                27683  3
sunrpc                241630  1
dmpCLARiiON            11771  1
dmpap                   9390  1
vxspec                  3174  6
vxio                 3261814  1 vxspec
vxdmp                 377776  20 vxspec,vxio
cpufreq_ondemand       10382  1
acpi_cpufreq            8593  3
freq_table              4847  2 cpufreq_ondemand,acpi_cpufreq
ipv6                  321209  60
vxportal                5940  0
fdd                    53457  1 vxodm
vxfs                 2957815  2 vxportal,fdd
exportfs                4202  0
serio_raw               4816  0
i2c_i801               11190  0
iTCO_wdt               11708  0
iTCO_vendor_support     3022  1 iTCO_wdt
ioatdma                57872  9
dca                     7099  1 ioatdma
i5k_amb                 5039  0
hwmon                   2464  1 i5k_amb
i5000_edac              8833  0
edac_core              46055  3 i5000_edac
sg                     30186  0
shpchp                 33448  0
e1000e                140051  0
ext4                  353979  3
mbcache                 7918  1 ext4
jbd2                   89033  1 ext4
dm_mirror              14003  1
dm_region_hash         12200  1 dm_mirror
dm_log                 10088  3 dm_mirror,dm_region_hash
sr_mod                 16162  0
cdrom                  39769  1 sr_mod
sd_mod                 37221  18
crc_t10dif              1507  1 sd_mod
pata_acpi               3667  0
ata_generic             3611  0
ata_piix               22588  0
ahci                   39105  4
qla2xxx               280129  24
scsi_transport_fc      50893  1 qla2xxx
scsi_tgt               12107  1 scsi_transport_fc
radeon                797054  1
ttm                    46942  1 radeon
drm_kms_helper         32113  1 radeon
drm                   200778  3 radeon,ttm,drm_kms_helper
i2c_algo_bit            5664  1 radeon
i2c_core               31274  5 i2c_i801,radeon,drm_kms_helper,drm,i2c_algo_bit
dm_mod                 76856  20 dm_mirror,dm_log

# rmmod gab
ERROR: Module gab is in use
[root@srv-vrts-n2 ~]# modinfo gab
filename:       /lib/modules/2.6.32-71.el6.x86_64/veritas/vcs/gab.ko
license:        Proprietary. Send bug reports to support@veritas.com
description:    Group Membership and Atomic Broadcast 5.1.120.000-SP1PR2
author:         VERITAS Software Corp.
srcversion:     F43C75576C05662FB0ED8C8
depends:        llt
vermagic:       2.6.32-71.el6.x86_64 SMP mod_unload modversions
parm:           gab_logflag:int
parm:           gab_numnids:maximum nodes in the cluster (1-128) (int)
parm:           gab_numports:maximum gab ports allowed (1-32) (int)
parm:           gab_flowctrl:queue depth that causes flow-control (1-128) (int)
parm:           gab_logbufsize:internal log buffer size in bytes (8100-65400) (int)
parm:           gab_msglogsize:maximum messages in internal message log (128-4096) (int)
parm:           gab_isolate_time:maximum time to wait for isolated client (16000-240000) (int)
parm:           gab_kill_ntries:number of times to attempt to kill client (3-10) (int)
parm:           gab_kstat_size:Number of system statistics to maintain in GAB 60-240 (int)
parm:           gab_conn_wait:maximum number of wait for CONNECTS message (1-256) (int)
parm:           gab_ibuf_count:maximum number of intermediate buffers (0-32) (int)

# modinfo llt
filename:       /lib/modules/2.6.32-71.el6.x86_64/veritas/vcs/llt.ko
license:        Proprietary. Send bug reports to support@veritas.com
author:         VERITAS Software Corp.
description:    Low Latency Transport 5.1.120.000-SP1PR2
srcversion:     AF11D9C04A71073E1ADCFC8
depends:
vermagic:       2.6.32-71.el6.x86_64 SMP mod_unload modversions
parm:           llt_maxnids:maximum nodes in the cluster (1-128) (int)
parm:           llt_maxports:maximum llt ports allowed (1-32) (int)
parm:           llt_nqthread:number of kernel threads to use (2-5) (int)
parm:           llt_basetimer:frequency of base timer ((10 * 1000)-(500 * 1000)) (int)

Hm.... ok, i run /etc/init.d/gab stop with debug

...

+ echo 'Stopping GAB: '
Stopping GAB:
+ mod_isloaded
++ lsmod
++ grep '^gab\ '
+ return
+ mod_isconfigured
++ LANG=C
++ LC_ALL=C
++ /sbin/gabconfig -l
++ grep 'Driver state'
++ grep -q Configured
+ return
+ /sbin/gabconfig -U
+ ret=1
+ '[' '!' 1 -eq 0 ']'
+ echo 'ERROR! Cannot unload GAB module. Clients still exist'
ERROR! Cannot unload GAB module. Clients still exist
+ echo 'Kill/Stop clients corresponding to following ports.'
Kill/Stop clients corresponding to following ports.
+ LANG=C
+ LC_ALL=C
+ /sbin/gabconfig -a
+ grep -v 'Port a gen'
GAB Port Memberships

...

ok, i use /sbin/gabconfig -l and -U

# /sbin/gabconfig -l
GAB Driver Configuration
Driver state         : Configured
Partition arbitration: Disabled
Control port seed    : Enabled
Halt on process death: Disabled
Missed heartbeat halt: Disabled
Halt on rejoin       : Disabled
Keep on killing      : Disabled
Quorum flag          : Disabled
Restart              : Enabled
Node count           : 2
Send queue limit     : 128
Recv queue limit     : 128
IOFENCE timeout (ms) : 15000
Stable timeout (ms)  : 5000

# /sbin/gabconfig -U
GAB /sbin/gabconfig ERROR V-15-2-25014 clients still registered

but it did not help... i find this topic https://www-secure.symantec.com/connect/forums/unable-stop-gab-llt-vcs51solaris-10 and stop ODM

# /etc/init.d/vxodm stop
Stopping ODM

and run /etc/init.d/gab stop , but

# /etc/init.d/gab stop
Stopping GAB:
GAB has usage count greater than zero. Cannot unload

I again see /sbin/gabconfig -l 

# /sbin/gabconfig -l
GAB Driver Configuration
Driver state         : Unconfigured
Partition arbitration: Disabled
Control port seed    : Disabled
Halt on process death: Disabled
Missed heartbeat halt: Disabled
Halt on rejoin       : Disabled
Keep on killing      : Disabled
Quorum flag          : Disabled
Restart              : Disabled
Node count           : 0
Send queue limit     : 128
Recv queue limit     : 128
IOFENCE timeout (ms) : 15000
Stable timeout (ms)  : 5000
[root@srv-vrts-n2 ~]# /sbin/gabconfig -a
GAB Port Memberships
===============================================================

and again run with debug /etc/init.d/gab stop

...

+ echo 'Stopping GAB: '
Stopping GAB:
+ mod_isloaded
++ lsmod
++ grep '^gab\ '
+ return
+ mod_isconfigured
++ LANG=C
++ LC_ALL=C
++ /sbin/gabconfig -l
++ grep 'Driver state'
++ grep -q Configured
+ return
+ mod_unload
++ lsmod
++ grep '^gab '
++ awk '{print $3}'
+ USECNT=1
+ '[' -z 1 ']'
+ '[' 5 '!=' 0 ']'
+ ps -e
+ grep gablogd
+ '[' 1 -ne 0 ']'
+ GAB_UNLOAD_RETRIES=0
+ '[' 0 '!=' 0 ']'
++ lsmod
++ grep '^gab '
++ awk '{print $3}'
+ USECNT=1
+ '[' 1 -gt 0 ']'
+ echo 'GAB has usage count greater than zero. Cannot unload'
GAB has usage count greater than zero. Cannot unload
+ return 1

...

and agian run lsmod

lsmod | grep gab
gab                   283317  1
llt                   180985  1 gab
[root@srv-vrts-n2 ~]# rmmod gab
ERROR: Module gab is in use

what can be done in such a situation?

[mikebounds]

Your initial problem was that you did not unload ODM (port d) before trying to unload GAB first.  When you ran  /etc/init.d/gab stop this tries to configure gab (gabconfig -U) and then unload gab from kernel, but as ODM was configured, then gabconfig -U should have failed.  Quite often if you do things in the wrong order, GAB gets screwed up and then you can't unload things and you have to reboot.

Please show what output of gabconfig -a is now - if it show port a and port d, then you need to see if you can stop odm.  If "gabconfg -a" JUST shows "port a" and gabconfig -U does not work, then I have never found away round this, so I think you will have to reboot.

Mike 

[sirin_zarin]

Ok, i reboot systems , ODM running

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   6c5601 membership 01
Port b gen   6c5605 membership 01
Port d gen   6c5604 membership 01
Port f gen   6c560e membership 01
Port h gen   6c5603 membership 01
Port u gen   6c560c membership 01
Port v gen   6c5607 membership 01
Port w gen   6c5609 membership 01

i stop ODM

# /etc/init.d/vxodm stop
Stopping ODM

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   6c5601 membership 01
Port b gen   6c5605 membership 01
Port f gen   6c560e membership 01
Port h gen   6c5603 membership 01
Port u gen   6c560c membership 01
Port v gen   6c5607 membership 01
Port w gen   6c5609 membership 01

[mikebounds]

Do you have any more queries.  Basically if you want to unload gab, then gabconfig -a, must only show port a.  Stopping VCS without -force will stop ports for CVM and CFS (and port h) and then should just be left with port b (fencing) and port d (ODM) which you need to stop before stopping GAB (port a)

Mike

[sirin_zarin · Accepted solution]

Hi all, this

# /opt/VRTSvcs/bin/hastop -all -force (or -sys <node name>)
# /etc/init.d/vxglm stop
# /etc/init.d/vxodm stop
# /etc/init.d/vxgms stop
# /etc/init.d/vxfen stop
# /etc/init.d/gab stop
# /etc/init.d/llt stop

work ;)