In this blog post, we’ll look at how Percona XtraDB Cluster maintenance mode uses ProxySQL to take cluster nodes offline without impacting workloads.
Percona XtraDB Cluster Maintenance Mode
Since Percona XtraDB Cluster offers a high availability solution, it must consider a data flow where a cluster node gets taken down for maintenance (through isolation from a cluster or complete shutdown).
Percona XtraDB Cluster facilitated this by introducing a maintenance mode. Percona XtraDB Cluster maintenance mode reduces the number of abrupt workload failures if a node is taken down using ProxySQL (as a load balancer).
The central idea is delaying the core node action and allowing ProxySQL to divert the workload.
How ProxySQL Manages Percona XtraDB Cluster Maintenance Mode
With Percona XtraDB Cluster maintenance mode, ProxySQL marks the node as OFFLINE when a user triggers a shutdown signal (or wants to put a specific node into maintenance mode):
- When a user triggers a shutdown, Percona XtraDB Cluster node sets
pxc_maint_mode
to SHUTDOWN (from the DISABLED default) and sleep for x seconds (dictated by
pxc_maint_transition_period
— 10 secs by default). ProxySQL
auto detects this change and marks the node as OFFLINE. With this change, ProxySQL avoids opening new connections for any DML transactions, but continues to service existing queries untilpxc_maint_transition_period
. Once the sleep period is complete, Percona XtraDB Cluster delivers a real shutdown signal — thereby giving ProxySQL enough time to transition the workload.
- If the user needs to take a node into maintenance mode, the user can simply set
pxc_maint_mode
to MAINTENANCE. With that,
pxc_maint_mode
is updated and the client connection updating it goes into sleep for x seconds (as dictated by
pxc_maint_transition_period
) before giving back control to the user. ProxySQL
auto-detects this change and marks the node as OFFLINE. With this change ProxySQL avoids opening new connections for any DML transactions but continues to service existing queries. - ProxySQL auto-detects this change in maintenance state and then automatically re-routes traffic, thereby reducing abrupt workload failures.
Technical Details:
- The ProxySQL Galera checker script continuously monitors the state of individual nodes by checking the
pxc_maint_mode
variable status (in addition to the existingwsrep_local_state
) using the ProxySQL scheduler feature - Scheduler is a Cron-like implementation integrated inside ProxySQL, with millisecond granularity.
- If
proxysql_galera_checker
detectspxc_maint_mode = SHUTDOWN | MAINTENANCE
, then it marks the node as OFFLINE_SOFT. This avoids the creation of new connections (or workloads) on the node.
Sample
proxysql_galera_checker
log:
Thu Dec 8 11:21:11 GMT 2016 Enabling config Thu Dec 8 11:21:17 GMT 2016 Check server 10:127.0.0.1:25000 , status ONLINE , wsrep_local_state 4 Thu Dec 8 11:21:17 GMT 2016 Check server 10:127.0.0.1:25100 , status ONLINE , wsrep_local_state 4 Thu Dec 8 11:21:17 GMT 2016 Check server 10:127.0.0.1:25200 , status ONLINE , wsrep_local_state 4 Thu Dec 8 11:21:17 GMT 2016 Changing server 10:127.0.0.1:25200 to status OFFLINE_SOFT due to SHUTDOWN Thu Dec 8 11:21:17 GMT 2016 Number of writers online: 2 : hostgroup: 10 Thu Dec 8 11:21:17 GMT 2016 Enabling config Thu Dec 8 11:21:22 GMT 2016 Check server 10:127.0.0.1:25000 , status ONLINE , wsrep_local_state 4 Thu Dec 8 11:21:22 GMT 2016 Check server 10:127.0.0.1:25100 , status ONLINE , wsrep_local_state 4 Thu Dec 8 11:21:22 GMT 2016 Check server 10:127.0.0.1:25200 , status OFFLINE_SOFT , wsrep_local_state 4
Ping us below with any questions or comments.