Percona Toolkit 3.5.5 was released on October 3, 2023. This blog post covers the main changes in this release.
pt-online-schema-change improvements
After nibble hook
This was contributed by Marcelo HP Ferreira at https://github.com/percona/percona-toolkit/pull/645. The feature allows you to define a custom plugin that will be triggered after copying each nibble.
Here are a few examples of what you can print with this hook:
- Copy rate
- Number of rows per nibble
- Run time per nibble
- Chunk size when using dynamic chunks
You can find an example plugin in the sample file
t/pt–online–schema–change/samples/plugins/on_copy_rows_after_nibble.pm
Using option –skip-check-slave-lag multiple times
pt–online–schema–change always supported multiple invocations of option
—skip–check–slave–lag but practically used only the first one; other replicas are still checked for the lag as reported at PT-2241. Contribution by PinoCao (PR-622) fixes this issue. Now, this option works as expected. You can specify multiple replicas for which you do not want to check lag.
No failure on replica disconnect
This is one more feature that was broken before. As described in the user manual for the
—max–lag option:
The tool waits forever for replicas to stop lagging. If any replica is stopped, the tool waits forever until the replica is started. The data copy continues when all replicas are running and not lagging too much.
However, this was not the case. In case of replica disconnect,
pt–online–schema–change just stopped working. This was especially unfortunate if this happened after a few hours of data copying. This behavior was reported at PT-2168 and PT-2096
The reason why this feature did not work lies in a few places. One is a debugging code for bug fix PT-1760 that was not removed. As a result, the tool died in case of replica disconnect instead of handling the error.
But even after removing this debugging code, there exist multiple places in the code where the tool can die. Fix for PT-2168 adds checks for these situations and, depending on the option
—fail–on–stopped–replication , either dies or continues waiting when the replica is available again. Now,
pt–online–schema–change should die less frequently.
pt–online–schema–change was failing at replica loss for many years, and users may rely on this behavior. Therefore, the option
—fail–on–stopped–replication is enabled by default. This means that you need to disable it (
—nofail–on–stopped–replication ) if you want
pt–online–schema–change to continue copying data after one of the replicas is down.
pt–online–schema–change also updates the replica list while waiting. If you use option
—recursion–method , set to processlist or hosts, this happens automatically. If you use
—recursion–method=dsn , you need to update the DSN table manually, so the tool re-reads it and updates the list of replicas it monitors.
K8 support
Percona Toolkit is shipped with the
pt–k8s–debug–collector tool that collects diagnostic data from Percona Kubernetes operators. It collects system information together with the output of
pt–mysql–summary ,
pt–mongodb–summary , or
pg_gather tools, depending on the operator. In version 3.5.5, support for PostgreSQL operator version two has been added. We are planning to add new features to this tool for every release. It will become a Swiss army knife for Percona Kubernetes Operators troubleshooting.
Quality improvements
Perl tools in Percona Toolkit come with a regression test suite that uses its own module that fires up MySQL sandbox (
lib/Sandbox.pm ) and extended Perl module
Test::More (
lib/PerconaTest.pm ). Unfortunately, in the last few years, not all tests were stable, and the suite was not run at each code change. After noticing this, we started working on fixing existing tests so they do not fail due to improvements in new MySQL versions or environmental issues. Fix for PT-2156: Fix tests for lib finishes this effort. We now run tests locally before releasing the product. We plan to make regression tests part of the release process starting from the next version.
Another move in this direction is standardizing the output of the common options, such as format of the option
—version . This is done as a bug fix for PT-2235: pt-mongodb-index-check does not support option –version and PT-2236: pt-secure-collect, pt-pg-summary do not follow PT standard for option –version.
Percona Toolkit also fixed typos and trailing whitespace issues in this release. Viktor Szépe contributed to these changes. He also updated our GitHub workflows, so such issues should not show up in the future.
Quality control is not only about code fixes but also about testing. In the latest releases, we get CVE reports for Go tools from Kushal Haldar. They include more vulnerabilities than Open Source tools that we use, can find, and help us improve the quality of Go-based tools.
Community contributions
Percona Toolkit continuously receives contributions. In the latest releases, we try to accept as many contributions as possible. At the same time, we started requiring minimal quality control, such as adding test cases and running regression tests before accepting the patch. There are still a few pull requests waiting for final resolution, but we hope we will process all of them by the end of the year.
For this release, we want to thank:
- Kushal Haldar for his reports about vulnerabilities in Go-based tools
- Viktor Szépe for pull requests, improving the quality of our code
- Marcelo HP Ferreira for after nibble hook to
pt–online–schema–change - PinoCao for
—skip–check–slave–lag bug fix
Percona Toolkit is a collection of advanced open source command-line tools, developed and used by the Percona technical staff, that are engineered to perform a variety of MySQL, MariaDB, MongoDB, and PostgreSQL server and system tasks that are too difficult or complex to perform manually.
