Forum Discussion

S_D1's avatar
S_D1
Level 2
9 years ago

unable to start mysql rhel6.5 - vcs (6.2)

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =
Following are the settings in main.cf

group app_servicegroup (
        SystemList = { node71 = 0, node72 = 1 }
        AutoStartList = { node71, node72 }
        )

        DiskGroup mysql_DG_resource (
                DiskGroup = mysql_dg
                )

        Mount mysql_mount_resource (
                MountPoint = "/var/lib/mysql"
                BlockDevice = "/dev/vx/dsk/mysql_dg/mysqlvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        MySQL mysql_service_resource (
                ResLogLevel = TRACE
                MySQLAdminPasswd = "password"
                BaseDir = "/var/lib/mysql"
                DataDir = "/var/lib/mysql/mysql"
                MyCnf = "/etc/my.cnf"
                )

        Volume mysql_volume_resource (
                DiskGroup = mysql_dg
                Volume = mysqlvol
                )

        mysql_mount_resource requires mysql_volume_resource
        mysql_service_resource requires mysql_mount_resource
        mysql_volume_resource requires mysql_DG_resource

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

- mysql is installed in /var/lib/mysql which is on share storage.

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

Tried to clear the resource and bring the resource on primary node node71.
[root@node71 MySQL]# hares -clear mysql_service_resource -sys node71
[root@node71 MySQL]# hares -online mysql_service_resource -sys node71

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

- Following are the engine_A.log generated during the above steps.

2016/03/17 12:11:29 VCS INFO V-16-1-50135 User root fired command: hares -clear mysql_service_resource  node71  from localhost
2016/03/17 12:11:29 VCS INFO V-16-1-10307 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (Not initiated by VCS)
2016/03/17 12:11:50 VCS INFO V-16-1-50135 User root fired command: hares -online mysql_service_resource  node71  from localhost
2016/03/17 12:11:50 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group app_servicegroup on all nodes
2016/03/17 12:11:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:11:51 VCS INFO V-16-1-10298 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/17 12:11:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:11:51 VCS INFO V-16-1-10298 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/17 12:11:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:11:55 VCS INFO V-16-1-10298 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/17 12:11:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:13:58 VCS ERROR V-16-2-13066 (node71) Agent is calling clean for resource(mysql_service_resource) because the resource is not up even after online completed.
2016/03/17 12:14:00 VCS INFO V-16-2-13068 (node71) Resource(mysql_service_resource) - clean completed successfully.
2016/03/17 12:14:00 VCS INFO V-16-2-13071 (node71) Resource(mysql_service_resource): reached OnlineRetryLimit(0).
2016/03/17 12:14:01 VCS ERROR V-16-1-54031 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is FAULTED on sys node71
2016/03/17 12:14:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:14:01 VCS INFO V-16-6-15015 (node71) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed
2016/03/17 12:14:03 VCS INFO V-16-1-10305 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/17 12:14:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:14:03 VCS INFO V-16-1-10305 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/17 12:14:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:14:04 VCS INFO V-16-1-10305 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/17 12:14:04 VCS ERROR V-16-1-10205 Group app_servicegroup is faulted on system node71
2016/03/17 12:14:04 VCS NOTICE V-16-1-10446 Group app_servicegroup is offline on system node71
2016/03/17 12:14:04 VCS INFO V-16-1-10493 Evaluating node71 as potential target node for group app_servicegroup
2016/03/17 12:14:04 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node71
2016/03/17 12:14:04 VCS INFO V-16-1-10493 Evaluating node72 as potential target node for group app_servicegroup
2016/03/17 12:14:04 VCS ERROR V-16-1-10200 System node72 is not available

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

I have brought the node72 down to avoid failover to node72 since mysql resource is not configured yet.

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

mysql agent and acc_agent are installed on the nodes.

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

I am able to mount share store and then start mysql service manually. Also, shared storage is able to failover to node72 and vice versa.

=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

[root@node71 config]# rpm -qa | grep -i mysql-server
mysql-server-5.1.73-5.el6_6.x86_64
=   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =   =

Please advice.
 

business continuity
Cluster Server
Linux
  • S_D1's avatar
    S_D1
    Level 2
    9 years ago

    I have tried plain text password and vcsencrypt -agent to encrypt the password as well.

    [root@node71 ~]# ps -ef | grep -i mysql
    root     15512     1  0 Mar16 ?        00:01:47 /opt/VRTSvcs/bin/Script50Agent -type MySQL -agdir /opt/VRTSagents/ha/bin/MySQL
    root     19948 15512  0 13:47 ?        00:00:01 /opt/VRTSperl/bin/perl -Sw /opt/VRTSagents/ha/bin/MySQL/online mysql_service_resource ResLogLevel 1 TRACE State 1 0 IState 1 1 MySQLUser 1 root MySQLAdmin 1 root MySQLAdminPasswd 1 DNGxxxxxxxxxxjG EnvFile 1  BaseDir 1 /var/lib/mysql DataDir 1 /var/lib/mysql/mysql MyCnf 1 /etc/my.cnf HostName 1  Port 1 3306 SecondLevelMonitor 1 0 MonitorProgram 1  AEPTimeout 1 300
    root     20666 12581  0 13:52 pts/0    00:00:00 grep -i mysql
     

    df shows shared mounted when I try to bring mysql_service_resource online.

    Filesystem                     Size  Used Avail Use% Mounted on
    /dev/mapper/VolGroup-lv_root    28G  7.5G   19G  30% /
    tmpfs                          495M     0  495M   0% /dev/shm
    /dev/sda1                      485M   35M  426M   8% /boot
    tmpfs                          4.0K     0  4.0K   0% /dev/vx
    /dev/vx/dsk/mysql_dg/mysqlvol   14G   63M   14G   1% /var/lib/mysql

  • S_D1's avatar
    S_D1
    Level 2
    9 years ago

    Latest update is that as per below logs mysql_service_resource "resource is not up even after online completed." but actually mysql server comes up and I am able to access that but vcs is not able to see that the service is up and thus mysql server goes down as resource moves to faulted. Any advice / suggestion?

    2016/03/21 04:58:02 VCS INFO V-16-2-13068 (node71) Resource(mysql_service_resource) - clean completed successfully.
    2016/03/21 04:58:02 VCS INFO V-16-2-13071 (node71) Resource(mysql_service_resource): reached OnlineRetryLimit(0).
    2016/03/21 04:58:03 VCS ERROR V-16-1-54031 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is FAULTED on sys node71
    2016/03/21 04:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 04:58:04 VCS INFO V-16-6-15015 (node71) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed
    2016/03/21 04:58:06 VCS INFO V-16-1-10305 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
    2016/03/21 04:58:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 04:58:06 VCS INFO V-16-1-10305 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
    2016/03/21 04:58:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 04:58:08 VCS INFO V-16-1-10305 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
    2016/03/21 04:58:08 VCS ERROR V-16-1-10205 Group app_servicegroup is faulted on system node71
    2016/03/21 04:58:08 VCS NOTICE V-16-1-10446 Group app_servicegroup is offline on system node71
    2016/03/21 04:58:08 VCS INFO V-16-1-10493 Evaluating node71 as potential target node for group app_servicegroup
    2016/03/21 04:58:08 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node71
    2016/03/21 04:58:08 VCS INFO V-16-1-10493 Evaluating node72 as potential target node for group app_servicegroup
    2016/03/21 04:58:08 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node72
    2016/03/21 05:05:05 VCS INFO V-16-1-50135 User root fired command: hares -clear mysql_service_resource  node71  from localhost
    2016/03/21 05:05:05 VCS INFO V-16-1-10307 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (Not initiated by VCS)
    2016/03/21 05:06:14 VCS INFO V-16-1-50135 User root fired command: hares -modify mysql_service_resource  MySQLAdmin  MySQLAdmin  from localhost
    2016/03/21 05:06:28 VCS INFO V-16-1-50135 User root fired command: hares -online mysql_service_resource  node71  from localhost
    2016/03/21 05:06:28 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group app_servicegroup on all nodes
    2016/03/21 05:06:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 05:06:29 VCS INFO V-16-1-10298 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
    2016/03/21 05:06:29 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 05:06:29 VCS INFO V-16-1-10298 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
    2016/03/21 05:06:29 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 05:06:33 VCS INFO V-16-1-10298 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
    2016/03/21 05:06:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 05:13:17 VCS ERROR V-16-2-13066 (node71) Agent is calling clean for resource(mysql_service_resource) because the resource is not up even after online completed.
    2016/03/21 05:13:22 VCS INFO V-16-2-13068 (node71) Resource(mysql_service_resource) - clean completed successfully.
    2016/03/21 05:13:22 VCS INFO V-16-2-13071 (node71) Resource(mysql_service_resource): reached OnlineRetryLimit(0).
    2016/03/21 05:13:23 VCS ERROR V-16-1-54031 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is FAULTED on sys node71
    2016/03/21 05:13:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 05:13:23 VCS INFO V-16-6-15015 (node71) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed
    2016/03/21 05:13:25 VCS INFO V-16-1-10305 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
    2016/03/21 05:13:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 05:13:25 VCS INFO V-16-1-10305 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
    2016/03/21 05:13:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
    2016/03/21 05:13:26 VCS INFO V-16-1-10305 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
    2016/03/21 05:13:26 VCS ERROR V-16-1-10205 Group app_servicegroup is faulted on system node71
    2016/03/21 05:13:26 VCS NOTICE V-16-1-10446 Group app_servicegroup is offline on system node71
    2016/03/21 05:13:26 VCS INFO V-16-1-10493 Evaluating node71 as potential target node for group app_servicegroup
    2016/03/21 05:13:26 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node71
    2016/03/21 05:13:26 VCS INFO V-16-1-10493 Evaluating node72 as potential target node for group app_servicegroup
    2016/03/21 05:13:26 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node72