This post is the fourth of a series that started here.
From the previous of this series, we now have resources configured but instead of starting MySQL, Pacemaker invokes a script to start (or restart) the EC2 instance running MySQL. This blog post describes the instance restart script. Remember, I am more a DBA than a script writer so it might not be written in the most optimal way.
First, let’s recap what’s the script has to perform (the full script is given below).
- Kill the MySQL EC2 instance if running
- Make sure the MySQL EC2 instance is stopped
- Prepare the user-data script for the new MySQL EC2 instance
- Launch the new MySQL instance
- Make sure it is running
- Reconfigure local heartbeat
- Broadcast the new MySQL instance IP to the application servers
Kill the MySQL EC2 instance
In order to kill the existing MySQL EC2 instance, we first have to identify it. This is done by:
CODE:
-
OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $AMI_HA_MYSQL | egrep “running|pending” | tail -n 1 | cut -d‘|’ -f3`
by filtering on the AMI type of the instance. Since an instance can be listed at the “stopped” state, it is mandatory to filter for states “running” or “pending”. Then the instance is terminated with:
CODE:
-
ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
Make sure the MySQL EC2 instance is stopped
Terminating an EC2 instance is not instantaneous, we can confirm an instance is really stopped by monitoring its status and wait until it is actually “terminated”. The code below is how the script performs this task.
CODE:
-
#wait until the old instance is terminated it takes a few seconds to stop
-
done=“false”
-
while [ $done == “false” ]
-
do
-
status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | grep -c terminated`
-
if [ “$status” -eq “1” ]; then
-
done=“true”
-
else
-
ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
-
sleep 5
-
fi
-
done
Prepare the user-data script for the new MySQL EC2 instance
The new MySQL instance will be running heartbeat. Since we cannot use neither Ethernet broadcast or multicast, we need to configure the new instance so that it communicates through unicast with its partner node in the cluster, the node on which the restart script is run. This configuration is achieved by providing a user-data script (see the hamysql.user-data below) which completes the heartbeat configuration of the new instance. The hamysql.user-data script only performs a search and replace operation on the /etc/ha.d/ha.cf file and then restart the heartbeat service. In order for this to work properly, we just have to put the IP of the current instance in the script like here:
CODE:
-
OUR_IP=`/sbin/ifconfig eth0 | grep ‘inet addr’ | cut -d‘:’ -f2 | cut -d‘ ‘ -f1`
-
#Now, modify the user-data script, we need to put our IP address in
-
if [ “$OUR_IP” == “” ]
-
then
-
echo “Error getting Our IP”
-
else
-
perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g” $USER_DATA_SCRIPT
-
fi
Launch the new MySQL instance
Once things are ready, a new MySQL instance can be launched with:
CODE:
-
#Now we are ready to start a new one
-
INSTANCE_INFO=`ec2-run-instances -K $PK -C $CERT $AMI_HA_MYSQL -n 1 -g $HA_SECURITY_GROUP -f $USER_DATA_SCRIPT -t $INSTANCE_TYPE -z $INSTANCE_ZONE -k $INSTANCE_KEY | /usr/local/bin/filtre_instances.pl`
-
-
#wait until the new instance is running it take a few seconds to start
-
NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d‘|’ -f3`
Out of this operation, we retrieve the new instance “instance_id”.
Make sure it is running
Since we know the “instance_id” of the new instance, checking if it is running is easy:
CODE:
-
done=“false”
-
while [ $done == “false” ]
-
do
-
INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl`
-
status=`echo $INSTANCE_INFO | grep -c running`
-
if [ “$status” -eq “1” ]; then
-
done=“true”
-
else
-
sleep 5
-
fi
-
done
Reconfigure local heartbeat
Now, Heartbeat, on the monitoring host, must be informed of the IP address of its new partner. In order to achieve this, a search and replace operation in the local ha.cf file followed of restart of Heartbeat is sufficient.
CODE:
-
#Set the IP in /etc/ha.d/ha.cf and ask heartbeat to reload its config
-
MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | cut -d‘|’ -f2`
-
perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g” /etc/ha.d/ha.cf
-
/etc/init.d/heartbeat reload
Broadcast the new MySQL instance IP to the application servers
The final phase is to inform the application servers that the IP of the MySQL has changed. The best way to list those application servers is through a security group and, provided the appropriate ssh keys have been exchanged, this code will push the IP update.
CODE:
-
TMPFILE=`mktemp`
-
ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $CLIENT_SECURITY_GROUP> $TMPFILE
-
-
while read line
-
do
-
IP=`echo $line | cut -d‘|’ -f2`
-
ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./updated_xinetd.sh $MYSQL_IP
-
done <$TMPFILE
-
-
rm $TMPFILE
The full script:
CODE:
-
#!/bin/bash
-
HA_SECURITY_GROUP=testyves
-
CLIENT_SECURITY_GROUP=hamysql-client
-
CLIENT_SCRIPT=/usr/local/bin/update_client.sh
-
AMI_HA_MYSQL=ami-84a74fed
-
EBS_DATA_VOL=vol-aefawf
-
USER_DATA_SCRIPT=/usr/local/bin/hamysql.user-data
-
PK=/usr/local/bin/pk-FNMBRRABFRKVICBDZ4IOOSF7YROYZRZW.pem
-
CERT=/usr/local/bin/cert-FNMBRRABFRKVICBDZ4IOOSF7YROYZRZW.pem
-
INSTANCE_TYPE=m1.small
-
INSTANCE_ZONE=us-east-1c
-
INSTANCE_KEY=yves-key
-
-
OUR_IP=`/sbin/ifconfig eth0 | grep ‘inet addr’ | cut -d‘:’ -f2 | cut -d‘ ‘ -f1`
-
#Now, modify the user-data script, we need to put our IP address in
-
if [ “$OUR_IP” == “” ]
-
then
-
echo “Error getting Our IP”
-
else
-
perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $OUR_IP/g” $USER_DATA_SCRIPT
-
fi
-
-
while [ 1 ]; do
-
-
#First thing to do, terminate the other instance ID
-
OLD_INSTANCE_ID=`ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $AMI_HA_MYSQL | egrep “running|pending” | tail -n 1 | cut -d‘|’ -f3`
-
-
if [ “$OLD_INSTANCE_ID” == “” ]
-
then
-
#no running instance
-
:
-
else
-
ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
-
-
#wait until the old instance is terminated it takes a few seconds to stop
-
done=“false”
-
while [ $done == “false” ]
-
do
-
status=`ec2-describe-instances -K $PK -C $CERT $OLD_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | grep -c terminated`
-
if [ “$status” -eq “1” ]; then
-
done=“true”
-
else
-
ec2-terminate-instances -K $PK -C $CERT $OLD_INSTANCE_ID> /dev/null
-
sleep 5
-
fi
-
done
-
fi
-
-
#Now we are ready to start a new one
-
INSTANCE_INFO=`ec2-run-instances -K $PK -C $CERT $AMI_HA_MYSQL -n 1 -g $HA_SECURITY_GROUP -f $USER_DATA_SCRIPT -t $INSTANCE_TYPE -z $INSTANCE_ZONE -k $INSTANCE_KEY | /usr/local/bin/filtre_instances.pl`
-
-
#wait until the new instance is running it take a few seconds to start
-
NEW_INSTANCE_ID=`echo $INSTANCE_INFO | cut -d‘|’ -f3`
-
-
if [ “$NEW_INSTANCE_ID” == “” ]
-
then
-
echo “Error creating the new instance”
-
else
-
-
done=“false”
-
while [ $done == “false” ]
-
do
-
INSTANCE_INFO=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl`
-
status=`echo $INSTANCE_INFO | grep -c running`
-
if [ “$status” -eq “1” ]; then
-
done=“true”
-
else
-
sleep 5
-
fi
-
done
-
-
#Set the IP in /etc/ha.d/ha.cf and ask heartbeat to reload its config
-
MYSQL_IP=`ec2-describe-instances -K $PK -C $CERT $NEW_INSTANCE_ID | /usr/local/bin/filtre_instances.pl | cut -d‘|’ -f2`
-
perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 $MYSQL_IP/g” /etc/ha.d/ha.cf
-
-
TMPFILE=`mktemp`
-
ec2-describe-instances -K $PK -C $CERT | /usr/local/bin/filtre_instances.pl | grep $CLIENT_SECURITY_GROUP> $TMPFILE
-
-
while read line
-
do
-
IP=`echo $line | cut -d‘|’ -f2`
-
ssh -i /usr/local/bin/update_mysql ubuntu@$IP sudo ./updated_xinetd.sh $MYSQL_IP
-
done <$TMPFILE
-
-
rm $TMPFILE
-
-
/etc/init.d/heartbeat reload
-
fi
-
-
-
sleep 300 # 5 min before attempting again. Normally heartbeat should kill the script before
-
done
The hamysql.user-data script:
The script sets the IP of the monitor host in the heartbeat ha.cf configuration file and then, finishes up some missing configuration settings of the AMI.
CODE:
-
#!/bin/bash
-
sudo hostname hamysql
-
sudo perl -pi -e “s/ucast eth0 (\d+)(\.\d+){3}/ucast eth0 10.220.230.18/g” /etc/ha.d/ha.cf
-
-
# to eventually be added to the ebs image
-
sudo perl -pi -e ‘s/bind-address/#bind-address/g’ /etc/mysql/my.cnf
-
sudo service mysql restart
-
sleep 5
-
/usr/bin/mysql -u root -proot -e “grant all on *.* to root@’%’ identified by ‘root'”
-
sudo /etc/init.d/heartbeat start
Entry posted by Yves Trudeau |
No comment
Add to: 
| 
| 
| 
| 
