Personal blog of Yzmir Ramirez

Do you need to get to grips with MySQL proxies? Or maybe you could do with discovering the latest developments and plans for Percona’s software?

Well, wait no more because …

on Wednesday November 15, 2017, we bring you a webinar double bill.

Meanwhile, return a little later in the day at 10:00 am PST / 1:00 pm EST (UTC-8) to hear Percona CEO Peter Zaitsev discuss what’s new in Percona open source software. In “Percona Software News and Roadmap Update – Q4 2017”, Peter will talk about new features in Percona software, show some quick demos and share highlights from the Percona open source software roadmap. He will also talk about new developments in Percona commercial services and finish with a Q&A.

 

You are, of course, very welcome to register for either one or both webinars. Please register for your place soon!

---

Peter Zaitsev, Percona CEO and Co-Founder

Peter Zaitsev co-founded Percona and assumed the role of CEO in 2006. As one of the foremost experts on MySQL strategy and optimization, Peter leveraged both his technical vision and entrepreneurial skills to grow Percona from a two-person shop to one of the most respected open source companies in the business. With over 150 professionals in 30+ countries, Peter’s venture now serves over 3000 customers – including the “who’s who” of internet giants, large enterprises and many exciting startups. Percona was named to the Inc. 5000 in 2013, 2014, 2015 and 2016. Peter was an early employee at MySQL AB, eventually leading the company’s High Performance Group. A serial entrepreneur, Peter co-founded his first startup while attending Moscow State University, where he majored in Computer Science. Peter is a co-author of High Performance MySQL: Optimization, Backups, and Replication, one of the most popular books on MySQL performance. Peter frequently speaks as an expert lecturer at MySQL and related conferences, and regularly posts on the Percona Database Performance Blog. Fortune and DZone have both tapped Peter as a contributor, and his recent ebook Practical MySQL Performance Optimization is one of percona.com’s most popular downloads.

Colin Charles, Chief Evangelist

Colin Charles is the Chief Evangelist at Percona. He was previously on the founding team for MariaDB Server in 2009, worked in MySQL since 2005 and been a MySQL user since 2000. Before joining MySQL, he worked actively on the Fedora and OpenOffice.org projects. He’s well known within many open source communities and has spoken on the conference circuit.

 

This post is the third of a series focusing on the MySQL high availability solutions available in 2017.

The first post looked at the elders, the technologies that have been around for more than ten years. The second post talked about the adults, the more recent and mature technologies. In this post, we will look at the emerging MySQL high availability solutions. The “baby” MySQL high availability solutions I chose for the blog are group replication, proxies and distributed storage.

Group replication

Group replication is the Oracle response to Galera. The term “InnoDB cluster” means a cluster using group replication. The goal is offering similar functionalities, especially the almost synchronous feature.

At first glance, the group replication implementation appears to be rather elegant. The basis is the GTID replication mode. The nodes of an InnoDB cluster share a single UUID sequence. To control the replication lag, Oracle added a flow control layer. While Galera requires unanimity, group replication only requires a majority. The majority protocol in use is derived from Paxos. A majority protocol makes the cluster more resilient to a slow node.

Like Galera, when you add flow control you needs queues. Group replication has two queues. There is one queue for the certification process and one queue for the appliers. What is interesting in the Oracle approach is the presence of a throttling mechanism. When flow control is requested by a node, instead of halting the processing of new transactions like Galera, the rate of transactions is throttled. That can help to meet strict timing SLAs.

Because the group replication logic is fairly similar to Galera, they suffer from the same limitations: large transactions, latency and hot rows. Group replication is recent. The first GA version is 5.7.17, from December 2016. It is natural then that it has a number of sharp edges. I won’t extend too much here, but if you are interested read here, here. I am confident over time group replication will get more polished. Some automation, like the Galera SST process, would also be welcome.

Given the fact the technology is recent, I know no Percona customer using group replication in production.

Proxies

Intelligent proxies can be viewed as another type of upcoming MySQL high availability solution. It is not strictly MySQL. In fact, this solution is more of a mix of other solutions.

The principle is simple: you connect to a proxy, and the proxy directs you to a valid MySQL server. The proxy has to monitor the states of the back-end servers, and maybe even perform actions on them. Of course, the proxy layer must not become a single point of failure. There should be more than one proxy host for basic HA. If more that one proxy is used at the same time, they’ll have to agree on the state of the back-end servers. For example, on a cluster using MySQL async replication, if the proxies are not sending the write traffic to the same host, things will quickly become messy.

There are few ways of achieving this. The simplest solution is an active-passive setup where only one proxy is active at a given time. You’ll need some kind of logic to determine if the proxy host is available or not. Typical choices will use tools like keepalived or Pacemaker.

A second option is to have the proxies agree to a deterministic way of identifying a writer node. For example, with a Galera-based cluster, the sane back-end node with the lowest wsrep_local_index could be the writer node.

Finally, the proxies could talk to each other and coordinate. Such an approach is promising. It could allow a single proxy to perform the monitoring and inform its peers of the results. It would allow also coordinated actions on the cluster when a failure is detected.

Currently, there are a few options in terms of proxies:

• ProxySQL: An open-source that understands the MySQL protocol and can do R/W splitting, query caching, sharding, SQL firewalling, etc. A new alpha level feature, mirroring, targets the inter-proxy communication need.
• MaxScale: No longer fully open-source (BSL), but understands the MySQL protocol. Can do R/W splitting, sharding, binlog serving, SQL firewalling, etc.
• MySQL Router: MySQL Router is an open-source proxy developed by Oracle for InnoDB Cluster (Group replication). It understands the MySQL protocol and also supports the new X protocol. It can do R/W splitting.
• HAProxy: HAProxy is a popular open-source TCP level proxy. It doesn’t understand the MySQL protocol. It needs helper scripts, responding to HTTP type requests, to figure the node’s health.

To these open source proxies, there are two well-known commercial proxy-like solutions, Tungsten and ScaleArc. Both of these technologies are mature and are not “babies” in terms of age and traction. On top of these, there are also numerous hardware-based load balancer solutions.

The importance of proxies in MySQL high availability has led Percona to include ProxySQL in the latest releases of Percona XtraDB Cluster. In collaboration with the ProxySQL maintainer, René Cannaò, features have been added to make ProxySQL aware of the Percona XtraDB Cluster state.

Proxies are already often deployed in MySQL high availability solutions. Often proxies are only doing load balancing type work. We start to see deployment using proxies for more advanced things, like read/write splitting and sharding.

Distributed storage

 

This MySQL high availability solution is a project I am interested in. It is fair to say it is more a “fetus” than a real “baby,” since I know nobody using it in production. You can see this solution as a shared storage approach on steroids.

The simplest solution requires a three-node Ceph cluster. The nodes also run MySQL and the datadir is a Ceph RBD block device. Data in Ceph is automatically replicated to multiple hosts. This built-in data replication is an important component of the solution. Also, Ceph RBD supports snapshots and clones. A clone is a copy of the whole data set that consumes only the data that changed (delta) in terms of storage. Our three MySQL servers will thus not use three full copies of the dataset, but only one full copy and two deltas. As time passes, the deltas grow. When they are too large, we can simply generate new snapshots and clones and be back to day one. The generation of a new snapshot and clone takes a few seconds, and doesn’t require stopping MySQL.

The obvious use case for the distributed storage approach is a read-intensive workload on a very large dataset. The setup can handle a lot of writes. The higher the write load, the more frequently there will be a snapshot refresh. Keep in mind that refreshing a snapshot of a 10 TB data set takes barely more time than for a 1 GB data set.

For that purpose, I wrote an SST script for Percona XtraDB Cluster that works with Ceph. I blogged about it here. I also wrote a Ceph snapshot/clone backup script that can provision a slave from a master snapshot. I’ll blog about how to use this Ceph backup script in the near future.

Going further with distributed storage, multiple MySQL instances could use the same data pages. Ceph would be use as a distributed object store for InnoDB pages. This would allow to build an open-source Aurora like database. Coupled with Galera or Group replication, you could have a highly-available MySQL cluster sharing a single copy of the dataset.

I started to modify MySQL, actually Percona Server for MySQL 5.7, to add support for Ceph/Rados. Rados is the object store protocol of Ceph. There is still a lot of effort needed to make it work. My primary job is not development, so progress is slow. My work can be found (here). The source compiles well but MySQL doesn’t fully start. I need to debug where things are going wrong.

Adding a feature to MySQL like that is an awesome way to learn the internals of MySQL. I would really appreciate any help if you are interested in this project.

Conclusion

Over the three articles in this series, we have covered the 2017 landscape of MySQL high availability solutions. The first focused on the old timers, “the elders”, composed of: replication, shared storage and NDB. The second articles dealt with the solutions that are more recent and have a good traction: Galera and RDS Aurora. The conclusion of the series is the current article, which looked at what could be possibly coming in term of MySQL high availability solutions.

The main goal of this series is to help planning the deployment of MySQL in a highly-available way. I hope it can be used for hints and pointers to get better and more efficient solutions.

In this blog post, we’ll discuss ProxySQL versus MaxScale for OLTP RO workloads.

Continuing my series of READ-ONLY benchmarks (you can find the other posts here: https://www.percona.com/blog/2016/04/07/mysql-5-7-sysbench-oltp-read-results-really-faster/ and https://www.percona.com/blog/2016/03/28/mysql-5-7-primary-key-lookup-results-is-it-really-faster), in this post I want to see how much overhead a proxy adds. At this

In my opinion, there are only two solid proxy software options for MySQL at the moment: ProxySQL and MaxScale. In the past, there was also MySQL Proxy, but it is pretty much dead for now. Its replacement, MySQl Router, is still in the very early stages and seriously lacks any features that would compete with ProxySQL and MaxScale. This will most likely change in the future – when MySQL Router adds more features, I will reevaluate them then!

To test the proxies, I will start with a very simple setup to gauge basic performance characteristics. I will use a sysbench client and proxy running on the same box. Sysbench connects to the proxy via local socket (for minimal network and TCP overhead), and the proxy is connected to a remote MySQL via a 10Gb network. This way, the proxy and sysbench share the same server resources.

Other parameters:

• CPU: 56 logical CPU threads servers Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
• sysbench ten tables x 10mln rows, Pareto distribution
• OS: Ubuntu 15.10 (Wily Werewolf)
• MySQL 5.7
• MaxScale version 1.4.1
• ProxySQL version 1.2.0b

You can find more details about benchmarks, scripts and configs here: https://github.com/Percona-Lab/benchmark-results/tree/201603-mysql55-56-57-RO/remote-OLTP-proxy-may.

An important parameter to consider is how much of the CPU resources you allocate for a proxy. Both ProxySQL and MaxScale allow you to configure how many threads they can use to process user requests and to route queries. I’ve found that 16 threads for ProxySQL 8 threads for  MaxScale is optimal (I will also show 16 threads for MaxScale in this). Both proxies also allow you to setup simple load-balancing configurations, or to work in read-write splitting mode. In this case, I will use simple load balancing, since there are no read-write splitting requirements in a read-only workload).

ProxySQL

First result: How does ProxySQL perform compared to vanilla MySQL 5.7?

As we can see, there is a noticeable drop in performance with ProxySQL. This is expected, as ProxySQL does extra work to process queries. What is good though is that ProxySQL scales with increasing user connections.

One of the tricks that ProxySQL has is a “fast-forward” mode, which minimizes overhead from processing (but as a drawback, you can’t use many of the other features). Out of curiosity, let’s see how the “fast-forward” mode performs:

MaxScale

Now let’s see what happens with MaxScale. Before showing the next chart, let me not it contains “error bars,” which are presented as vertical bars. Basically, an “error bar” shows a standard deviation: the longer the bar, the more variation was observed during the experiment. We want to see less variance, as it implies more stable performance.

Here are results for MaxScale versus ProxySQL:

We can see that with lower numbers of threads both proxies are nearly similar, but MaxScale has a harder time scaling over 100 threads. On average, MaxScale’s throughput is worse, and there is a lot of variation. In general, we can see that MaxScale demands more CPU resources and uses more of the CPU per request (compared to ProxySQL). This holds true if we run MaxScale with 16 threads (instead of 8):

MaxScale with 16 threads does not handle the workload well, and there is a lot of variation along with some visible scalability issues.

To summarize, here is a chart with relative performance (vanilla MySQL 5.7 is shown as 1):

While this chart does show that MaxScale has less overhead from 1-6 threads, it doesn’t scale as user load increases.

We wrapped up Thursday morning’s Percona Live 2016 keynotes (April 21)  with The MySQL Community Award Ceremony. The MySQL Community Awards is a community-based initiative, with the goal of publicly recognizing contributors to the MySQL ecosystem. The idea is to publicly recognize contributors to the MySQL ecosystem. The entire process of discussing, voting and awarding is controlled by an independent group of community members, typically based on past winners or their representatives, as well as known contributors. It is a self-appointed, self-declared, self-making-up-the-rules-as-it-goes committee. It is also very aware of the importance of the community; a no-nonsense, non-political, adhering to tradition, self-criticizing committee.

The categories can be found here.

We celebrated the members of the open source community, and some big open source contributors from the past year! The winners this year were:

Community Contributors of the Year:

• Bill Karwin, Senior Database Architect and Software Engineer at SchoolMessenger
• Domas Mituzas, Small Data Engineer at Facebook
• Yoshinori Matsunobu, Data Engineer at Facebook

Application of the Year:

• MaxScale

Corporate Contributor of the Year:

• Booking.com

 

Congratulation to all the winners! You can watch the whole ceremony below (it’s about 15 minutes long).

In this blog post, we’ll discuss read-write split routing in MaxScale.

The two previous posts have shown how to setup high availability (HA) with Maxscale using asynchronous replication and how we monitor replication.

Now let’s focus on the routing module performing read-write splits.

This is our current configuration:

[Splitter Service]
type=service
router=readwritesplit
servers=percona1, percona2
max_slave_replication_lag=30
user=maxscale
passwd=264D375EC77998F13F4D0EC739AABAD4

This router module is designed to spread the read queries across multiple servers (slaves by default), and send the write queries to a single server: the master.

This module is intended to work with Asynchronous Master-Slave replication but also with Galera replication if you plan to write to a single node.

So what is routed to the Master?

• Write statements
• All statements within an open transaction, even if this transaction is read only
• Store procedure and user-defined function call.
• DDL statements
• Execution of prepared statements (EXECUTE)
• All statements using temporary tables

Example:

• percona1: master
• percona2 and percona3: slaves

Let’s connect to MaxScale with the MySQL’s interactive client:

mysql> select @@hostname;
+------------+
| @@hostname |
+------------+
| percona2   |
+------------+
mysql> start transaction;
mysql> select @@hostname;
+------------+
| @@hostname |
+------------+
| percona1   |
+------------+
mysql> rollback;
mysql> select @@hostname;
+------------+
| @@hostname |
+------------+
| percona2   |
+------------+

Now let’s try with a READ ONLY transaction:

mysql> start transaction read only;
mysql> select @@hostname;
+------------+
| @@hostname |
+------------+
| percona1   |
+------------+

As we can see, MaxScale doesn’t support READ ONLY transactions. It considers them the same as any transaction. This means they are routed to the master as a WRITE.

We’ve already seen the 

max_slave_replication_lag

 optional parameter, but there are some others:

• max_slave_connections

  : defines the maximum number of slaves a router session uses, the default is to use all the ones available

• use_sql_variables_in

  : defines where queries’ reading session variables should be routed. Valid values are master and all (the latter being the default)

• weightby

  : defines the name of the value used to calculate the weights of the server

Now let’s play with the

weightby

 . So in this configuration, we will target 10% of the reads to percona2, and 90% to percona3:

[Splitter Service]
type=service
router=readwritesplit
servers=percona1, percona2, percona3
weightby=myweight
...
[percona2]
type=server
address=192.168.90.3
port=3306
protocol=MySQLBackend
myweight=1
[percona3]
type=server
address=192.168.90.4
port=3306
protocol=MySQLBackend
myweight=9

We restart MaxScale, and verify the settings of the service:

# maxadmin -pmariadb show service "Splitter Service"
Service 0x363b460
	Service:                             Splitter Service
	Router:                              readwritesplit (0x7fe7f1e88540)
	State:                                       Started
	Number of router sessions:           	0
	Current no. of router sessions:      	0
	Number of queries forwarded:          	0
	Number of queries forwarded to master:	0
	Number of queries forwarded to slave: 	0
	Number of queries forwarded to all:   	0
	Master/Slave percentage:		0.00%
	Connection distribution based on myweight server parameter.
		Server               Target %    Connections  Operations
		                               Global  Router
		percona3             90.0%     0       0       0
		percona2             10.0%     0       0       0
		percona1             100.0%     0       0       0
	Started:                             Wed Feb 24 22:39:27 2016
	Root user access:                    Disabled
	Backend databases
		192.168.90.4:3306  Protocol: MySQLBackend
		192.168.90.3:3306  Protocol: MySQLBackend
		192.168.90.2:3306  Protocol: MySQLBackend
	Routing weight parameter:            myweight
	Users data:                          0x36397c0
	Total connections:                   2
	Currently connected:                 2
	SSL:  Disabled

The target % seems correct, let’s test it!

for i in `seq 1 10`;
do mysql -h 192.168.90.5 -BN -umanager -ppercona -e "select @@hostname; select sleep(10)" 2>/dev/null &
done
percona2
percona2
percona2
percona2
percona2
percona2
percona3
percona3
percona3
percona3

That doesn’t seem good! Let’s check the service again:

Service 0x363b460
	Service:                             Splitter Service
	Router:                              readwritesplit (0x7fe7f1e88540)
	State:                                       Started
	Number of router sessions:           	10
	Current no. of router sessions:      	10
	Number of queries forwarded:          	30
	Number of queries forwarded to master:	0
	Number of queries forwarded to slave: 	30
	Number of queries forwarded to all:   	0
	Master/Slave percentage:		0.00%
	Connection distribution based on myweight server parameter.
		Server               Target %    Connections  Operations
		                               Global  Router
		percona3             90.0%     10      10      5
		percona2             10.0%     10      10      5
		percona1             100.0%     10      10      0
	Started:                             Wed Feb 24 22:39:27 2016
	Root user access:                    Disabled
	Backend databases
		192.168.90.4:3306  Protocol: MySQLBackend
		192.168.90.3:3306  Protocol: MySQLBackend
		192.168.90.2:3306  Protocol: MySQLBackend
	Routing weight parameter:            myweight
	Users data:                          0x36397c0
	Total connections:                   12
	Currently connected:                 12
	SSL:  Disabled

Five operations for both . . . this looks like a normal load balancer, 50%-50%.

So that doesn’t work as we expected. Let’s have a look at other router options:

• slave_selection_criteria

  . Controls how the router chooses the slaves and how it load balances the sessions. There are some parameter options:

  • LEAST_GLOBAL_CONNECTIONS. Slave with least connections from MaxScale, not on the server itself
  • LEAST_ROUTER_CONNECTIONS. Slave with least connections from this service
  • LEAST_BEHIND_MASTER. Slave with smallest replication lag
  • LEAST_CURRENT_OPRTATIONS. Slave with least active operations (this is the default)

• master_accept_reads

  . Uses the master for reads

The are some others; please check the online manual for:

• max_sescmd_history

• disable_sescmd_history

That explains the behavior we just observed. But what if we want to use the weight setting, and not spread the reads equivalently on the slaves?

I found the answer on IRC in the #maxscale freenode. Markus Makela (markusjm) explained to me that the default configuration in 1.3.0 is to use all the slaves, and load balance the actual statements. So to achieve what we want to do, we need to use these options in the service section:

router_options=slave_selection_criteria=LEAST_GLOBAL_CONNECTIONS
max_slave_connections=1

Let’s test it:

for i in `seq 1 10`;
do mysql -h 192.168.90.5 -BN -umanager -ppercona -e "select @@hostname; select sleep(10)" 2>/dev/null &
done
percona3
percona3
percona3
percona3
percona3
percona2
percona3
percona3
percona3
percona3
Service 0x1d88560
	Service:                             Splitter Service
	Router:                              readwritesplit (0x7f9c018c3540)
	State:                                       Started
	Number of router sessions:           	10
	Current no. of router sessions:      	10
	Number of queries forwarded:          	30
	Number of queries forwarded to master:	0
	Number of queries forwarded to slave: 	30
	Number of queries forwarded to all:   	0
	Master/Slave percentage:		0.00%
	Connection distribution based on myweight server parameter.
		Server               Target %    Connections  Operations
		                               Global  Router
		percona3             90.0%     9       9       9
		percona2             10.0%     1       1       1
		percona1             100.0%     10      10      0
	Started:                             Wed Feb 24 22:58:21 2016
	Root user access:                    Disabled
	Backend databases
		192.168.90.4:3306  Protocol: MySQLBackend
		192.168.90.3:3306  Protocol: MySQLBackend
		192.168.90.2:3306  Protocol: MySQLBackend
	Routing weight parameter:            myweight
	Users data:                          0x1d8a480
	Total connections:                   12
	Currently connected:                 12
	SSL:  Disabled

Yes! It worked as expected!

max_slave_connections

 sets the maximum number of slaves a router session uses at any moment. The default is to use all available slaves. When we set it to 1, we get one master and one slave connection per client, and the connections are balanced according to the server weights. The new mechanism uses statements instead of connections for load balancing (see MXS-588).

Finally, this routing module also support routing hints. I’ll cover them in my next MaxScale post.

More information: https://github.com/mariadb-corporation/MaxScale/blob/develop/Documentation/Routers/ReadWriteSplit.md

In this post, we’ll address how MaxScale monitors servers. We saw in the

We saw in the previous post how we could deal with high availability (HA) and read-write split using MaxScale.

If you remember from the previous post, we used this section to monitor replication:

[Replication Monitor]
type=monitor
module=mysqlmon
servers=percona1, percona2, percona3
user=maxscale
passwd=264D375EC77998F13F4D0EC739AABAD4
monitor_interval=1000
script=/usr/local/bin/failover.sh
events=master_down

But what are we monitoring? We are monitoring the assignment of master and slave roles inside MaxScale according to the actual replication tree in the cluster using the default check from the

mysqlmon

 monitoring modules.

There are other monitoring modules available with MaxScale:

• galera monitor
• multi-master monitor
• ndb cluster monitor

So back to our setup. MaxScale monitors the roles of our servers involved in replication. We can see the status of every server like this:

# maxadmin -pmariadb show server percona2
Server 0x1cace90 (percona2)
	Server:                              192.168.90.3
	Status:                              Slave, Running
	Protocol:                    MySQLBackend
	Port:                                3306
	Server Version:			5.6.28-76.1-log
	Node Id:                     2
	Master Id:                   1
	Slave Ids:
	Repl Depth:                  1
	Number of connections:               0
	Current no. of conns:                0
	Current no. of operations:   0

Now if we stop the slave, we can see:

# maxadmin -pmariadb show server percona2
Server 0x1cace90 (percona2)
	Server:                              192.168.90.3
	Status:                              Running
	Protocol:                    MySQLBackend
	Port:                                3306
	Server Version:			5.6.28-76.1-log
	Node Id:                     2
	Master Id:                   -1
	Slave Ids:
	Repl Depth:                  -1
	Number of connections:               40
	Current no. of conns:                0
	Current no. of operations:   0
# maxadmin -pmariadb list servers
Servers.
-------------------+-----------------+-------+-------------+--------------------
Server             | Address         | Port  | Connections | Status
-------------------+-----------------+-------+-------------+--------------------
percona1           | 192.168.90.2    |  3306 |           0 | Master, Running
percona2           | 192.168.90.3    |  3306 |           0 | Running
percona3           | 192.168.90.4    |  3306 |           0 | Slave, Running
-------------------+-----------------+-------+-------------+--------------------

and in the MaxScale logs:

2016-02-23 14:29:09   notice : Server changed state: percona2[192.168.90.3:3306]: lost_slave

Now if the slave is lagging, nothing happens, and we will then keep sending reads to a slave that is not up to date

To avoid that situation, we can add to the “[Replication Monitor]” section the following parameter:

detect_replication_lag=true

If we do so, MaxScale (if it has enough privileges) will create a schema

maxscale_schema

  with a table

replication_heartbeat

 . This table will be used to verify the replication lag like pt-heartbeat does.

When enabled, after we restart MaxScale, we can see the slave lag:

# maxadmin -pmariadb show server percona2
Server 0x2784f00 (percona2)
	Server:                              192.168.90.3
	Status:                              Slave, Running
	Protocol:                    MySQLBackend
	Port:                                3306
	Server Version:			5.6.28-76.1-log
	Node Id:                     2
	Master Id:                   1
	Slave Ids:
	Repl Depth:                  1
	Slave delay:		670
	Last Repl Heartbeat:	Tue Feb 23 14:25:24 2016
	Number of connections:               0
	Current no. of conns:                0
	Current no. of operations:   0

Does this mean that now the node won’t be reached (no queries will be routed to it)?

Let’s check:

percona3 mysql> select @@hostname;
+------------+
| @@hostname |
+------------+
| percona2   |
+------------+

That doesn’t sound good…

# maxadmin -pmariadb show server percona2
Server 0x2784f00 (percona2)
	Server:                              192.168.90.3
	Status:                              Slave, Running
	Protocol:                    MySQLBackend
	Port:                                3306
	Server Version:			5.6.28-76.1-log
	Node Id:                     2
	Master Id:                   1
	Slave Ids:
	Repl Depth:                  1
	Slave delay:		1099
	Last Repl Heartbeat:	Tue Feb 23 14:25:24 2016
	Number of connections:               1
	Current no. of conns:                1
	Current no. of operations:   0

We can see that there is

1 current connection

 .

How come? The monitoring actually works as expected, but we didn’t configure our

Splitter Service

  to not use that lagging slave.

We need to configure it like this:

[Splitter Service]
type=service
router=readwritesplit
servers=percona1, percona2
max_slave_replication_lag=30
...

And now, if the slave lags for 30 seconds or more, it won’t be used.

But what happen if for any reason we need to stop all the slaves (or if replication breaks)?

To find out, I performed a

STOP SLAVE;

  on percona2 and percona3. This what we see in the logs:

2016-02-23 22:55:16   notice : Server changed state: percona2[192.168.90.3:3306]: lost_slave
2016-02-23 22:55:34   notice : Server changed state: percona1[192.168.90.2:3306]: lost_master
2016-02-23 22:55:34   notice : Server changed state: percona3[192.168.90.4:3306]: lost_slave
2016-02-23 22:55:34   error  : No Master can be determined. Last known was 192.168.90.2:3306
2016-02-23 22:55:45   error  : Couldn't find suitable Master from 2 candidates.
2016-02-23 22:55:45   error  : 140003532506880 [session_alloc] Error : Failed to create Splitter Service session because routercould not establish a new router session, see earlier error.
2016-02-23 22:55:46   error  : Couldn't find suitable Master from 2 candidates.
2016-02-23 22:55:46   error  : 140003542996736 [session_alloc] Error : Failed to create Splitter Service session because routercould not establish a new router session, see earlier error.

If there are no more slaves, the master is not a master anymore, and the routing doesn’t work. The service is unavailable!

As soon as we start a slave, the service is back:

2016-02-23 22:59:17   notice : Server changed state: percona3[192.168.90.4:3306]: new_slave
2016-02-23 22:59:17   notice : A Master Server is now available: 192.168.90.2:3306

Can we avoid this situation when all slaves are stopped?

Yes we can, but we need to add into the monitoring section the following line:

detect_stale_master=true

If we stop  the two slaves again, in MaxScale’s log we can now read:

2016-02-23 23:02:19   notice : Server changed state: percona2[192.168.90.3:3306]: lost_slave
2016-02-23 23:02:46   warning: [mysql_mon]: root server [192.168.90.2:3306] is no longer Master, let's use it again even  if it could be a stale master, you have been warned!
2016-02-23 23:02:46   notice : Server changed state: percona3[192.168.90.4:3306]: lost_slave

And we can still connect to our service and use the single master.

Next time we will see how the read-write split works.

In this post, the first one of a Maxscale series, I describe how to use MariaDB’s MaxScale and MySQL-utilities with MySQL Asynchronous replication.

When we talk about high availability with asynchronous replication, we always think about MHA or PRM. But if we want to transparently use the slave(s) for READs, what can we use ?

Description:

• Three MySQL servers, but one has very limited resources and will never be able to handle the production load. In fact this node is used for backup and some back-office queries.
• We would like to use one of the nodes as a master and the other two as slaves, but only one will be addressed by the application for the READs. If needed, that same node will become the master.
• The application doesn’t handle READ and WRITE connections, and it’s impossible to change it.

To achieve our goals, we will use MaxScale and it’s R/W filter. When using Maxscale and asynchronous replication with MariaDB, it’s possible to use MariaDB’s replication manager, which is a wonderful tool written in Go. Unfortunately, this tool doesn’t support standard MySQL. To replace it, I used then the Oracle’s MySQL-Utilities.

Our three nodes are:

• percona1 (master)
• percona2 (powerful slave)
• percona3 (weak slave)

It’s mandatory in this solution to use GTID, as it’s the only method supported by the mysql-utilities we are using.

This is the MaxScale configuration:

[maxscale]
threads=4
[Splitter Service]
type=service
router=readwritesplit
servers=percona1, percona2
user=maxscale
passwd=264D375EC77998F13F4D0EC739AABAD4
[Splitter Listener]
type=listener
service=Splitter Service
protocol=MySQLClient
port=3306
socket=/tmp/ClusterMaster
[percona1]
type=server
address=192.168.90.2
port=3306
protocol=MySQLBackend
[percona2]
type=server
address=192.168.90.3
port=3306
protocol=MySQLBackend
[percona3]
type=server
address=192.168.90.4
port=3306
protocol=MySQLBackend
[Replication Monitor]
type=monitor
module=mysqlmon
servers=percona1, percona2, percona3
user=maxscale
passwd=264D375EC77998F13F4D0EC739AABAD4
monitor_interval=1000
script=/usr/local/bin/failover.sh
events=master_down
[CLI]
type=service
router=cli
[CLI Listener]
type=listener
service=CLI
protocol=maxscaled
address=localhost
port=6603

As you can notice, the Splitter Service contains only the two nodes able to handle the load.

And to perform the failover, in the Replication Monitor section, we define a script to use when the master is down.

That script calls mysqlrpladmin from the mysql-utilities.

In the script we also define the following line to be sure the weak slave will never become a master.

never_master=192.168.90.4

When everything is setup and running, you should see something like this:

# maxadmin -pmariadb list servers
Servers.
-------------------+-----------------+-------+-------------+--------------------
Server             | Address         | Port  | Connections | Status
-------------------+-----------------+-------+-------------+--------------------
percona1           | 192.168.90.2    |  3306 |          15 | Master, Running
percona2           | 192.168.90.3    |  3306 |        1025 | Slave, Running
percona3           | 192.168.90.4    |  3306 |           0 | Slave, Running
-------------------+-----------------+-------+-------------+--------------------

So as you can see, Maxscale discovers on its own which server is the master; this doesn’t need to be specified in the configuration.

You can also use mysqldrpladmin utility to verify the cluster’s health:

# /usr/bin/mysqlrpladmin --rpl-user=repl:replpercona --master=manager:percona@192.168.90.2:3306 --slaves=manager:percona@192.168.90.3:3306,manager:percona@192.168.90.4:3306  health
# Checking privileges.
#
# Replication Topology Health:
+---------------+-------+---------+--------+------------+---------+
| host          | port  | role    | state  | gtid_mode  | health  |
+---------------+-------+---------+--------+------------+---------+
| 192.168.90.2  | 3306  | MASTER  | UP     | ON         | OK      |
| 192.168.90.3  | 3306  | SLAVE   | UP     | ON         | OK      |
| 192.168.90.4  | 3306  | SLAVE   | UP     | ON         | OK      |
+---------------+-------+---------+--------+------------+---------+

Try it with --verbose

When we test with sysbench, and we stop the master, we can see that there are some errors due to disconnects. Also, during the promotion of the new master, sysbench can’t reconnect:

[  20s] queue length: 0, concurrency: 0
[  21s] threads: 8, tps: 2.00, reads: 28.00, writes: 8.00, response time: 107.61ms (95%), errors: 0.00, reconnects:  0.00
[  21s] queue length: 0, concurrency: 0
[  22s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  22s] queue length: 0, concurrency: 0
[  23s] threads: 8, tps: 1.00, reads: 14.00, writes: 4.00, response time: 100.85ms (95%), errors: 0.00, reconnects:  0.00
[  23s] queue length: 0, concurrency: 0
[  24s] threads: 8, tps: 0.00, reads: 11.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  24s] queue length: 0, concurrency: 1
[  25s] threads: 8, tps: 1.00, reads: 3.00, writes: 4.00, response time: 235.41ms (95%), errors: 0.00, reconnects:  0.00
[  25s] queue length: 0, concurrency: 0
[  26s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  26s] queue length: 0, concurrency: 0
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
[  27s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  27s] queue length: 0, concurrency: 3
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
[  28s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  28s] queue length: 0, concurrency: 4
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
[  29s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  29s] queue length: 0, concurrency: 5
[  30s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  30s] queue length: 0, concurrency: 5
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
[  31s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[  31s] queue length: 0, concurrency: 7
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
WARNING: Both max-requests and max-time are 0, running endless test
sysbench 0.5:  multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 8
Target transaction rate: 1/sec
Report intermediate results every 1 second(s)
Random number generator seed is 0 and will be ignored
Threads started!
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1045: failed to create new session
PANIC: unprotected error in call to Lua API (Failed to connect to the database)
WARNING: Both max-requests and max-time are 0, running endless test
sysbench 0.5:  multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 8
Target transaction rate: 1/sec
Report intermediate results every 1 second(s)
Random number generator seed is 0 and will be ignored
Threads started!
[   1s] threads: 8, tps: 1.99, reads: 27.93, writes: 7.98, response time: 211.49ms (95%), errors: 0.00, reconnects:  0.00
[   1s] queue length: 0, concurrency: 0
[   2s] threads: 8, tps: 1.00, reads: 14.00, writes: 4.00, response time: 51.01ms (95%), errors: 0.00, reconnects:  0.00
[   2s] queue length: 0, concurrency: 0
[   3s] threads: 8, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00
[   3s] queue length: 0, concurrency: 0
[   4s] threads: 8, tps: 1.00, reads: 13.99, writes: 4.00, response time: 80.28ms (95%), errors: 0.00, reconnects:  0.00

It took 8 seconds to automatically failover.

Then we can see the status of the servers:

# maxadmin -pmariadb list serversServers.
-------------------+-----------------+-------+-------------+--------------------
Server             | Address         | Port  | Connections | Status
-------------------+-----------------+-------+-------------+--------------------
percona1           | 192.168.90.2    |  3306 |          17 | Down
percona2           | 192.168.90.3    |  3306 |        1025 | Master, Running
percona3           | 192.168.90.4    |  3306 |           0 | Slave, Running
-------------------+-----------------+-------+-------------+--------------------

If we re-start percona1, we can see now:

# maxadmin -pmariadb list servers
Servers.
-------------------+-----------------+-------+-------------+--------------------
Server             | Address         | Port  | Connections | Status
-------------------+-----------------+-------+-------------+--------------------
percona1           | 192.168.90.2    |  3306 |          17 | Running
percona2           | 192.168.90.3    |  3306 |        1025 | Master, Running
percona3           | 192.168.90.4    |  3306 |           0 | Slave, Running
-------------------+-----------------+-------+-------------+--------------------

To add the node again in the asynchronous replication as a slave, we need to use another MySQL utility, mysqlreplicate:

# mysqlreplicate --master=manager:percona@192.168.90.3 --slave=manager:percona@192.168.90.2 --rpl-user=repl:replpercona
# master on 192.168.90.3: ... connected.
# slave on 192.168.90.2: ... connected.
# Checking for binary logging on master...
# Setting up replication...
# ...done.

This is source of failover.sh:

#!/bin/bash
# failover.sh
# wrapper script to mysqlrpladmin
# user:password pair, must have administrative privileges.
user=manager:percona
# user:password pair, must have REPLICATION SLAVE privileges.
repluser=repl:replpercona
never_master=192.168.90.4
ARGS=$(getopt -o '' --long 'event:,initiator:,nodelist:' -- "$@")
eval set -- "$ARGS"
while true; do
    case "$1" in
        --event)
            shift;
            event=$1
            shift;
        ;;
        --initiator)
            shift;
            initiator=$1
            shift;
        ;;
        --nodelist)
            shift;
            nodelist=$1
            shift;
        ;;
        --)
            shift;
            break;
        ;;
    esac
done
# find the candidates
for i in $(echo $nodelist | sed s/,/n/g)
do
  if [[ "$i" =~ "$never_master" ]]
  then
     # do nothing
     echo nothing >/dev/null
  else
     if [[ "$i" =~ "$initiator" ]]
     then
	# do nothing
        echo nothing >/dev/null
     else
        candidates="$candidates,${user}@${i}"
     fi
  fi
  if [[ "$i" =~ "$initiator" ]]
  then
     # do nothing
     echo nothing >/dev/null
  else
     slaves="$slaves,${user}@${i}"
  fi
done
cmd="/usr/bin/mysqlrpladmin --rpl-user=$repluser --slaves=${slaves#?} --candidates=${candidates#?} failover"
# uncomment following line for debug
#echo $cmd >> /tmp/fred
eval $cmd

In the next post, we will focus on the monitoring module used in this configuration.

Last week I hosted a webinar on using MySQL in the cloud for High Availability (HA) alongside 451 Research analyst Jason Stamper. You can watch the recording and also download the slides (free) here. Just click the “Register” button at the end of that page.

We had several excellent questions and we didn’t have time to get to several of them in the allotted time. I’m posting them here along with the answers. Feel free to ask follow-up questions in the comments below.

---

Q: Can the TokuDB engine be used in a PXC environment?

A: No, TokuDB cannot currently be used in a PXC environment, the only supported engine in Percona XtraDB Cluster 5.6 is InnoDB.

—

Q: With Galera replication (PXC), is balancing the load on each node?

A: No, you need to implement your own load balancing and HA layer between your clients and the Percona XtraDB Cluster server.  Examples mentioned in the webinar include HAProxy and F5 BigIP.

—

Q: What’s the best version of Percona XtraDB Cluster regarding InnoDB performance?

A: In general for best performance you should be using the latest release of Percona XtraDB Cluster 5.6, which is currently 5.6.24, released on June 3rd, 2015.

—

Q: Can I redirect my writes in Percona XtraDB Cluster to multiple nodes using the HAProxy? While trying with SysBench I can see write-only goes to first nodes in PXC while reads does goes to multiple nodes.

A: Yes you can configure HAProxy to distribute both reads and writes across all of your nodes in a Percona XtraDB Cluster environment. Perhaps SysBench created only one database connection for all writes, and so haproxy kept those confined to only one host. You may want to experiment with parallel_prepare.lua.

—

Q: What’s the optimal HA for small datasets (db is less than 10gb)?

A: The optimal HA deployment for small datasets would be dependent on your level of recovery required (tolerance for loss of transactions) and time that you can be in an unavailable state (seconds, minutes, hours?).  Unfortunately there isn’t a single answer to your question, however, if you are interested in further discussion on this point Percona would be happy to coordinate a time to speak.  Please feel free to contact me directly and we can continue the conversation at michael.coburn@percona.com.

—

 Q: Is there a concept of local master vs. remote master with PXC?

A: No there is no concept of local vs remote master.  All nodes in a Percona XtraDB Cluster can now be classified as Master, regardless of their proximity to the clients.

—

Q: Are there any concerns when considering AWS RDS or AURORA DB for MySQL HA in the Cloud?

A: Regarding AWS RDS, yes this a good option for MySQL HA in the Cloud.  I unfortunately haven’t worked with Aurora DB that much yet so I don’t have an opinion on it’s suitability for HA in the Cloud.

—

Q: We tried out PXC awhile back and it used to lock everything whenever any ddl was done. Has that changed?

A: We would have to look at the specifics of your environment, however, there have been numerous improvements in the 1½ years of development since Percona XtraDB Cluster went Generally Available (GA) on January 30th, 2014 in version 5.6.15.

—

Q: Is using the arbitrator a must?

A: No the arbitrator role via the garbd daemon is generally only used when operating in a minimal environment of two nodes that contain the data and you need a third node for quorum but don’t want to store the data a third time.

—

Q: Can we do a cluster across different zones?

A: Yes you can. However be aware that the latency incurred for all cluster certification operations will be impacted by the round trip time between nodes.

—

Q: Does PXC also support the MyISAM database?

A: No, Percona XtraDB Cluster does not support any storage engine other than InnoDB as of PXC 5.6.

—

Q: How do load balancers affect the throughput in a Galera-based setup given that the write would be limited by the slowest node?

A: Load balancers will introduce some measure of additional latency in the form of CPU time in the load balancer layer as it evaluates its own ruleset, and also in network time due to additional hop via load balancer.  Otherwise there should be no perceptible difference in the write throughput of a Percona XtraDB Cluster with and without a load balancer as it relates to the “slowest node” factor.

—

Q: Have you used MaxScale yet? If so, what are your thoughts?

A: Unfortunately I haven’t used MaxScale however Yves Trudeau, Percona Principal Architect, has recently written about MaxScale in this blog post.

—

Q: How do you configure timeout and maintain persistent connection to HAProxy?

A: I would encourage you to refer to the HAProxy Documentation.

The post Q&A: High availability when using MySQL in the cloud appeared first on MySQL Performance Blog.

Ever since MySQL replication has existed, people have dreamed of a good solution to automatically split read from write operations, sending the writes to the MySQL master and load balancing the reads over a set of MySQL slaves. While if at first it seems easy to solve, the reality is far more complex.

First, the tool needs to make sure it parses and analyses correctly all the forms of SQL MySQL supports in order to sort writes from reads, something that is not as easy as it seems. Second, it needs to take into account if a session is in a transaction or not.

While in a transaction, the default transaction isolation level in InnoDB, Repeatable-read, and the MVCC framework insure that you’ll get a consistent view for the duration of the transaction. That means all statements executed inside a transaction must run on the master but, when the transaction commits or rollbacks, the following select statements on the session can be again load balanced to the slaves, if the session is in autocommit mode of course.

Then, what do you do with sessions that set variables? Do you restrict those sessions to the master or you replay them to the slave? If you replay the set variable commands, you need to associate the client connection to a set of MySQL backend connections, made of at least a master and a slave. What about temporary objects like with “create temporary table…”? How do you deal when a slave lags behind or what if worse, replication is broken? Those are just a few of the challenges you face when you want to build a tool to perform read/write splitting.

Over the last few years, a few products have tried to tackle the read/write split challenge. The MySQL_proxy was the first attempt I am aware of at solving this problem but it ended up with many limitations. ScaleARC does a much better job and is very usable but it stills has some limitations. The latest contender is MaxScale from MariaDB and this post is a road story of my first implementation of MaxScale for a customer.

Let me first introduce what is MaxScale exactly. MaxScale is an open source project, developed by MariaDB, that aims to be a modular proxy for MySQL. Most of the functionality in MaxScale is implemented as modules, which includes for example, modules for the MySQL protocol, client side and server side.

Other families of available modules are routers, monitors and filters. Routers are used to determine where to send a query, Read/Write splitting is accomplished by the readwritesplit router. The readwritesplit router uses an embedded MySQL server to parse the queries… quite clever and hard to beat in term of query parsing.

There are other routers available, the readconnrouter is basically a round-robin load balancer with optional weights, the schemarouter is a way to shard your data by schema and the binlog router is useful to manage a large number of slaves (have a look at Booking.com’s Jean-François Gagné’s talk at PLMCE15 to see how it can be used).

Monitors are modules that maintain information about the backend MySQL servers. There are monitors for a replicating setup, for Galera and for NDB cluster. Finally, the filters are modules that can be inserted in the software stack to manipulate the queries and the resultsets. All those modules have well defined APIs and thus, writing a custom module is rather easy, even for a non-developer like me, basic C skills are needed though. All event handling in MaxScale uses epoll and it supports multiple threads.

Over the last few months I worked with a customer having a challenging problem. On a PXC cluster, they have more than 30k queries/s and because of their write pattern and to avoid certification issues, they want to have the possibility to write to a single node and to load balance the reads. The application is not able to do the Read/Write splitting so, without a tool to do the splitting, only one node can be used for all the traffic. Of course, to make things easy, they use a lot of Java code that set tons of sessions variables. Furthermore, for ISO 27001 compliance, they want to be able to log all the queries for security analysis (and also for performance analysis, why not?). So, high query rate, Read/Write splitting and full query logging, like I said a challenging problem.

We experimented with a few solutions. One was a hardware load balancer that failed miserably – the implementation was just too simple, using only regular expressions. Another solution we tried was ScaleArc but it needed many rules to whitelist the set session variables and to repeat them to multiple servers. ScaleArc could have done the job but all the rules increases the CPU load and the cost is per CPU. The queries could have been sent to rsyslog and aggregated for analysis.

Finally, the HA implementation is rather minimalist and we had some issues with it. Then, we tried MaxScale. At the time, it was not GA and was (is still) young. Nevertheless, I wrote a query logging filter module to send all the queries to a Kafka cluster and we gave it a try. Kafka is extremely well suited to record a large flow of queries like that. In fact, at 30k qps, the 3 Kafka nodes are barely moving with cpu under 5% of one core. Although we encountered some issues, remember MaxScale is very young, it appeared to be the solution with the best potential and so we moved forward.

The folks at MariaDB behind MaxScale have been very responsive to the problems we encountered and we finally got to a very usable point and the test in the pilot environment was successful. The solution is now been deployed in the staging environment and if all goes well, it will be in production soon. The following figure is simplified view of the internals of MaxScale as configured for the customer:

The blocks in the figure are nearly all defined in the configuration file. We define a TCP listener using the MySQL protocol (client side) which is linked with a router, either the readwritesplit router or the readconn router.

The first step when routing a query is to assign the backends. This is where the read/write splitting decision is made. Also, as part of the steps required to route a query, 2 filters are called, regexp (optional) and Genlog. The regexp filter may be used to hot patch a query and the Genlog filter is the logging filter I wrote for them. The Genlog filter will send a json string containing about what can be found in the MySQL general query log plus the execution time.

Authentication attempts are also logged but the process is not illustrated in the figure. A key point to note, the authentication information is cached by MaxScale and is refreshed upon authentication failure, the refresh process is throttled to avoid overloading the backend servers. The servers are continuously monitored, the interval is adjustable, and the server status are used when the decision to assign a backend for a query is done.

In term of HA, I wrote a simple Pacemaker resource agent for MaxScale that does a few fancy things like load balancing with IPTables (I’ll talk about that in future post). With Pacemaker, we have a full fledge HA solution with quorum and fencing on which we can rely.

Performance wise, it is very good – a single core in a virtual environment was able to read/write split and log to Kafka about 10k queries per second. Although MaxScale supports multiple threads, we are still using a single thread per process, simply because it yields a slightly higher throughput and the custom Pacemaker agent deals with the use of a clone set of MaxScale instances. Remember we started early using MaxScale and the beta versions were not dealing gracefully with threads so we built around multiple single threaded instances.

So, since a conclusion is needed, MaxScale has proven to be a very useful and flexible tool that allows to elaborate solutions to problems that were very hard to tackle before. In particular, if you need to perform read/write splitting, then, try MaxScale, it is best solution for that purpose I have found so far. Keep in touch, I’ll surely write other posts about MaxScale in the near future.

The post MaxScale: A new tool to solve your MySQL scalability problems appeared first on MySQL Performance Blog.