In this blog post, we’ll discuss trackable flow control in Percona XtraDB Cluster.
Introduction
Percona XtraDB Cluster has a self-regulating mechanism called Flow Control. This mechanism helps to avoid a situation wherein the weakest/slowest member of the cluster falls significantly behind other members of the cluster.
When a member of a cluster is slow at applying write-sets (while simultaneously continuing to receive write-sets from the cluster group channel), then the incoming/receive queue grows in size. If this queue crosses a set threshold (gcs.fc_limit), the node emits a FLOW_CONTROL message asking other members to slow down or halt processing.
While this happens transparently for the end-user, it’s important for the cluster administrator to know whether a node is using flow-control. If it is, it can affect overall cluster productivity.
Finding if a node is in flow control
FLOW_CONTROL is not a persistent state. A node enters FLOW_CONTROL once the queue size exceeds the set threshold. It is released back again once the queue size falls back below the low-end watermark.
So how can one view the higher and lower watermarks?
Up until now, this was exposed only in the log files. However, starting with Percona XtraDB Cluster 5.7.17-29.20, this is now exposed and available through SHOW STATUS:
mysql> show status like 'wsrep_flow_control_interval'; +-----------------------------+------------+ | Variable_name | Value | +-----------------------------+------------+ | wsrep_flow_control_interval | [ 12, 23 ] | +-----------------------------+------------+ 1 row in set (0.00 sec) mysql> set global wsrep_provider_options="gcs.fc_limit=100"; Query OK, 0 rows affected (0.00 sec) mysql> show status like 'wsrep_flow_control_interval'; +-----------------------------+-------------+ | Variable_name | Value | +-----------------------------+-------------+ | wsrep_flow_control_interval | [ 71, 141 ] | +-----------------------------+-------------+ 1 row in set (0.00 sec) mysql> set global wsrep_provider_options="gcs.fc_factor=0.75"; Query OK, 0 rows affected (0.00 sec) mysql> show status like 'wsrep_flow_control_interval'; +-----------------------------+--------------+ | Variable_name | Value | +-----------------------------+--------------+ | wsrep_flow_control_interval | [ 106, 141 ] | +-----------------------------+--------------+ 1 row in set (0.01 sec)
As you can see, the wsrep_flow_control_interval status variable emits a range representing the lower and higher watermark. A value set of (12, 23) means that if the incoming queue size is greater than 23, then FLOW_CONTROL is enabled. If the size falls below 12, FLOW_CONTROL is released.
Still, this does not show us if the node is using FLOW_CONTROL at any given moment.
To address this, we simultaneously introduced a wsrep_flow_control_status status variable into the same release. This boolean status variable tells the user if the node is in FLOW_CONTROL or not. Once the node is out of flow-control, the variable is switched to OFF, and vice versa to ON when the node is employing flow-control:
show status like '%flow%'; ..... | wsrep_flow_control_sent | 28156 | | wsrep_flow_control_recv | 28156 | | wsrep_flow_control_interval | [ 106, 141 ] | | wsrep_flow_control_status | ON |
Finally, the wsrep_flow_control_sent/recv counter can be used to track FLOW_CONTROL status. This shows an aggregated count of how many times flow-control kicked in. You can clear them using FLUSH STATUS.
mysql> show status like '%flow%'; ..... | wsrep_flow_control_sent | 31491 | | wsrep_flow_control_recv | 31491 | | wsrep_flow_control_interval | [ 106, 141 ] | | wsrep_flow_control_status | OFF |
Takeaway thoughts
We hope you enjoy the new features added to Percona XtraDB Cluster, including flow control tracking!
We like feedback from customers (via the usual Customer Support website), and where possible from the community (via Percona JIRA and Launchpad), on which features they most want to see. We recently re-engineered Percona XtraDB Cluster to have much-increased performance (a request originating from the community), and the new IST Progress Meter and Flow Control Monitoring were introduced to aid in solving customer-facing challenges.
If you have a feature you would like to see, let us know!
Note: Special thanks for Kenn Takara and Roel Van de Paar for helping me edit this post.