03-16-2016 11:59 PM
= = = = = = = = = = = = = = = = = = = = = = = = = = =
Following are the settings in main.cf
group app_servicegroup (
SystemList = { node71 = 0, node72 = 1 }
AutoStartList = { node71, node72 }
)
DiskGroup mysql_DG_resource (
DiskGroup = mysql_dg
)
Mount mysql_mount_resource (
MountPoint = "/var/lib/mysql"
BlockDevice = "/dev/vx/dsk/mysql_dg/mysqlvol"
FSType = vxfs
FsckOpt = "-y"
)
MySQL mysql_service_resource (
ResLogLevel = TRACE
MySQLAdminPasswd = "password"
BaseDir = "/var/lib/mysql"
DataDir = "/var/lib/mysql/mysql"
MyCnf = "/etc/my.cnf"
)
Volume mysql_volume_resource (
DiskGroup = mysql_dg
Volume = mysqlvol
)
mysql_mount_resource requires mysql_volume_resource
mysql_service_resource requires mysql_mount_resource
mysql_volume_resource requires mysql_DG_resource
= = = = = = = = = = = = = = = = = = = = = = = = = = =
- mysql is installed in /var/lib/mysql which is on share storage.
= = = = = = = = = = = = = = = = = = = = = = = = = = =
Tried to clear the resource and bring the resource on primary node node71.
[root@node71 MySQL]# hares -clear mysql_service_resource -sys node71
[root@node71 MySQL]# hares -online mysql_service_resource -sys node71
= = = = = = = = = = = = = = = = = = = = = = = = = = =
- Following are the engine_A.log generated during the above steps.
2016/03/17 12:11:29 VCS INFO V-16-1-50135 User root fired command: hares -clear mysql_service_resource node71 from localhost
2016/03/17 12:11:29 VCS INFO V-16-1-10307 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (Not initiated by VCS)
2016/03/17 12:11:50 VCS INFO V-16-1-50135 User root fired command: hares -online mysql_service_resource node71 from localhost
2016/03/17 12:11:50 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group app_servicegroup on all nodes
2016/03/17 12:11:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:11:51 VCS INFO V-16-1-10298 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/17 12:11:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:11:51 VCS INFO V-16-1-10298 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/17 12:11:51 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:11:55 VCS INFO V-16-1-10298 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/17 12:11:55 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:13:58 VCS ERROR V-16-2-13066 (node71) Agent is calling clean for resource(mysql_service_resource) because the resource is not up even after online completed.
2016/03/17 12:14:00 VCS INFO V-16-2-13068 (node71) Resource(mysql_service_resource) - clean completed successfully.
2016/03/17 12:14:00 VCS INFO V-16-2-13071 (node71) Resource(mysql_service_resource): reached OnlineRetryLimit(0).
2016/03/17 12:14:01 VCS ERROR V-16-1-54031 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is FAULTED on sys node71
2016/03/17 12:14:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:14:01 VCS INFO V-16-6-15015 (node71) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed
2016/03/17 12:14:03 VCS INFO V-16-1-10305 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/17 12:14:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:14:03 VCS INFO V-16-1-10305 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/17 12:14:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/17 12:14:04 VCS INFO V-16-1-10305 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/17 12:14:04 VCS ERROR V-16-1-10205 Group app_servicegroup is faulted on system node71
2016/03/17 12:14:04 VCS NOTICE V-16-1-10446 Group app_servicegroup is offline on system node71
2016/03/17 12:14:04 VCS INFO V-16-1-10493 Evaluating node71 as potential target node for group app_servicegroup
2016/03/17 12:14:04 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node71
2016/03/17 12:14:04 VCS INFO V-16-1-10493 Evaluating node72 as potential target node for group app_servicegroup
2016/03/17 12:14:04 VCS ERROR V-16-1-10200 System node72 is not available
= = = = = = = = = = = = = = = = = = = = = = = = = = =
I have brought the node72 down to avoid failover to node72 since mysql resource is not configured yet.
= = = = = = = = = = = = = = = = = = = = = = = = = = =
mysql agent and acc_agent are installed on the nodes.
= = = = = = = = = = = = = = = = = = = = = = = = = = =
I am able to mount share store and then start mysql service manually. Also, shared storage is able to failover to node72 and vice versa.
= = = = = = = = = = = = = = = = = = = = = = = = = = =
[root@node71 config]# rpm -qa | grep -i mysql-server
mysql-server-5.1.73-5.el6_6.x86_64
= = = = = = = = = = = = = = = = = = = = = = = = = = =
Please advice.
03-17-2016 01:28 AM
I have tried plain text password and vcsencrypt -agent to encrypt the password as well.
[root@node71 ~]# ps -ef | grep -i mysql
root 15512 1 0 Mar16 ? 00:01:47 /opt/VRTSvcs/bin/Script50Agent -type MySQL -agdir /opt/VRTSagents/ha/bin/MySQL
root 19948 15512 0 13:47 ? 00:00:01 /opt/VRTSperl/bin/perl -Sw /opt/VRTSagents/ha/bin/MySQL/online mysql_service_resource ResLogLevel 1 TRACE State 1 0 IState 1 1 MySQLUser 1 root MySQLAdmin 1 root MySQLAdminPasswd 1 DNGxxxxxxxxxxjG EnvFile 1 BaseDir 1 /var/lib/mysql DataDir 1 /var/lib/mysql/mysql MyCnf 1 /etc/my.cnf HostName 1 Port 1 3306 SecondLevelMonitor 1 0 MonitorProgram 1 AEPTimeout 1 300
root 20666 12581 0 13:52 pts/0 00:00:00 grep -i mysql
df shows shared mounted when I try to bring mysql_service_resource online.
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root 28G 7.5G 19G 30% /
tmpfs 495M 0 495M 0% /dev/shm
/dev/sda1 485M 35M 426M 8% /boot
tmpfs 4.0K 0 4.0K 0% /dev/vx
/dev/vx/dsk/mysql_dg/mysqlvol 14G 63M 14G 1% /var/lib/mysql
03-20-2016 04:53 PM
Latest update is that as per below logs mysql_service_resource "resource is not up even after online completed." but actually mysql server comes up and I am able to access that but vcs is not able to see that the service is up and thus mysql server goes down as resource moves to faulted. Any advice / suggestion?
2016/03/21 04:58:02 VCS INFO V-16-2-13068 (node71) Resource(mysql_service_resource) - clean completed successfully.
2016/03/21 04:58:02 VCS INFO V-16-2-13071 (node71) Resource(mysql_service_resource): reached OnlineRetryLimit(0).
2016/03/21 04:58:03 VCS ERROR V-16-1-54031 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is FAULTED on sys node71
2016/03/21 04:58:03 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 04:58:04 VCS INFO V-16-6-15015 (node71) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed
2016/03/21 04:58:06 VCS INFO V-16-1-10305 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/21 04:58:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 04:58:06 VCS INFO V-16-1-10305 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/21 04:58:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 04:58:08 VCS INFO V-16-1-10305 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/21 04:58:08 VCS ERROR V-16-1-10205 Group app_servicegroup is faulted on system node71
2016/03/21 04:58:08 VCS NOTICE V-16-1-10446 Group app_servicegroup is offline on system node71
2016/03/21 04:58:08 VCS INFO V-16-1-10493 Evaluating node71 as potential target node for group app_servicegroup
2016/03/21 04:58:08 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node71
2016/03/21 04:58:08 VCS INFO V-16-1-10493 Evaluating node72 as potential target node for group app_servicegroup
2016/03/21 04:58:08 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node72
2016/03/21 05:05:05 VCS INFO V-16-1-50135 User root fired command: hares -clear mysql_service_resource node71 from localhost
2016/03/21 05:05:05 VCS INFO V-16-1-10307 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (Not initiated by VCS)
2016/03/21 05:06:14 VCS INFO V-16-1-50135 User root fired command: hares -modify mysql_service_resource MySQLAdmin MySQLAdmin from localhost
2016/03/21 05:06:28 VCS INFO V-16-1-50135 User root fired command: hares -online mysql_service_resource node71 from localhost
2016/03/21 05:06:28 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group app_servicegroup on all nodes
2016/03/21 05:06:28 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 05:06:29 VCS INFO V-16-1-10298 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/21 05:06:29 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 05:06:29 VCS INFO V-16-1-10298 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/21 05:06:29 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 05:06:33 VCS INFO V-16-1-10298 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is online on node71 (VCS initiated)
2016/03/21 05:06:33 VCS NOTICE V-16-1-10301 Initiating Online of Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 05:13:17 VCS ERROR V-16-2-13066 (node71) Agent is calling clean for resource(mysql_service_resource) because the resource is not up even after online completed.
2016/03/21 05:13:22 VCS INFO V-16-2-13068 (node71) Resource(mysql_service_resource) - clean completed successfully.
2016/03/21 05:13:22 VCS INFO V-16-2-13071 (node71) Resource(mysql_service_resource): reached OnlineRetryLimit(0).
2016/03/21 05:13:23 VCS ERROR V-16-1-54031 Resource mysql_service_resource (Owner: Unspecified, Group: app_servicegroup) is FAULTED on sys node71
2016/03/21 05:13:23 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 05:13:23 VCS INFO V-16-6-15015 (node71) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed
2016/03/21 05:13:25 VCS INFO V-16-1-10305 Resource mysql_mount_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/21 05:13:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 05:13:25 VCS INFO V-16-1-10305 Resource mysql_volume_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/21 05:13:25 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) on System node71
2016/03/21 05:13:26 VCS INFO V-16-1-10305 Resource mysql_DG_resource (Owner: Unspecified, Group: app_servicegroup) is offline on node71 (VCS initiated)
2016/03/21 05:13:26 VCS ERROR V-16-1-10205 Group app_servicegroup is faulted on system node71
2016/03/21 05:13:26 VCS NOTICE V-16-1-10446 Group app_servicegroup is offline on system node71
2016/03/21 05:13:26 VCS INFO V-16-1-10493 Evaluating node71 as potential target node for group app_servicegroup
2016/03/21 05:13:26 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node71
2016/03/21 05:13:26 VCS INFO V-16-1-10493 Evaluating node72 as potential target node for group app_servicegroup
2016/03/21 05:13:26 VCS INFO V-16-1-50010 Group app_servicegroup is online or faulted on system node72