Jul
23
2026
--

Percona Operator for MongoDB 1.23.0: ClusterSync Migration, Vector Search, and PVC Snapshot Backups


Percona Operator for MongoDB 1.23.0 makes the operator a place you move to, not just a place you start. A new ClusterSync component clones a live source and follows its change streams, so leaving a hosted service is a short cutover rather than a long outage. Alongside it, this release adds semantic vector search and storage-layer snapshot backups, two features that matter most once the data is yours to run.

The three headline features are Percona ClusterSync for MongoDB, vector search, and PVC snapshot backups. ClusterSync clones and continuously replicates a live source into an operator-managed cluster. Vector search brings semantic queries to Percona Server for MongoDB. PVC snapshot backups move backups off the network path and onto the storage layer.

This release also widens where you can run it, adding official Rancher Kubernetes Engine (RKE2) support and full ARM64 images. Much of what shipped here traces back to requests on forums.percona.com and the public issue tracker.

 

In this post, you’ll learn about:

  • ClusterSync migration and replication
  • Vector search for semantic queries
  • PVC snapshot backups
  • Other improvements worth knowing about

 

Zero-Downtime Migration with Percona ClusterSync

Moving a live MongoDB database onto the operator has always been the awkward first step. Dump-and-restore needs a maintenance window sized to your data, and hand-built replication between a source and a target is fragile to set up and easy to get wrong. This release introduces Percona ClusterSync for MongoDB (PCSM) as an operator-managed component, so the migration path is via a Kubernetes object rather than a runbook.

 

Why it matters

The common case is migrating a hosted MongoDB service, for example MongoDB Atlas, for an operator-managed Percona Server for MongoDB cluster you control end to end. A typical trigger in production is a hosted-service bill that climbs with the workload, or a compliance requirement to keep data inside your own VPC and region: a team running a user-profile store on Atlas points PCSM at it, lets the target catch up over a day or two while the application keeps serving from Atlas, then cuts over in a maintenance window measured in seconds. PCSM clones the existing data, then tracks ongoing changes through MongoDB change streams, so the target stays current while you validate it. When you are ready, you cut the application over during a short window rather than a long one. The same mechanism keeps a continuously updated replica for non-production use or a hybrid-cloud copy.

How it works

PCSM runs as its own container, deployed and managed through a new PerconaServerMongoDBClusterSync custom resource. It performs an initial clone from the source connection string, then consumes change stream events to apply subsequent writes to the target. A mode field controls the lifecycle: running starts or resumes replication, paused holds it, and finalized stops replication.

 

Wiring it up

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBClusterSync
metadata:
  name: my-cluster-sync
spec:
  clusterName: my-target-cluster-name
  image: percona/percona-clustersync-mongodb:0.9.0
  # mode controls the PCSM lifecycle intent. Allowed values:
  #   running   - start/resume replication (default)
  #   paused    - pause an active replication
  #   finalized - stop replication
  mode: running
  source:
    uri: mongodb://source-cluster-mongos.source-namespace.svc.cluster.local:27017
    credentialsSecret: my-cluster-sync-source
  # excludeNamespaces lists MongoDB namespaces (db or db.collection) to skip.
  # excludeNamespaces:
  #   - admin
  #   - local

clusterName names the operator-managed target that receives the data. source.uri and source.credentialsSecret point at the database you are migrating from, which can be Atlas, a self-managed replica set, or another operator cluster. mode is the control you drive the cutover with: run to catch up, pause to hold, then finalize once the application points at the new cluster. The optional excludeNamespaces list skips databases or collections you do not want to copy.
 

Cutover and rollback

The cutover is yours to time, not the operator’s. During the running replication, the target trails the source by the change-stream lag, which you watch until it is small and steady. You then stop writes on the source, let the last events drain, and repoint the application at the target cluster. Because the source keeps serving until you move the application, a rollback before cutover is simply leaving the application where it is. After cutover, treat the move as one-way once writes flow to the target, so verify the target thoroughly during the sync window rather than after.

Note: The PCSM component ships at version 0.9.0 with this release. Test the full migration and cutover against a staging copy before you run it on production data, and keep the source available until you have verified the target.

 

Vector search for semantic queries

Vector search retrieves results by meaning rather than exact keyword match, which is the retrieval pattern behind semantic search and retrieval-augmented generation for AI applications. Teams that already store their data in MongoDB have had to copy vectors into a separate engine to do this, which adds a system to run and a pipeline to keep in sync. In production, this is the pattern behind a support tool that surfaces past tickets describing the same problem in different words, a product catalog that returns items by intent rather than exact keywords, and a RAG service that grounds a model on internal documents. This release lets you store and query vector data alongside your regular documents in Percona Server for MongoDB, so those workloads query one system instead of two.
 

How it works

The operator deploys and manages the mongot search process, wires its authentication and TLS to the rest of the cluster, and keeps the search index synchronized for both replica set and sharded deployments. Applications query the index through the same MongoDB connection they already use, so you add semantic search without a second client, a second driver, or a second set of credentials. You do not stand up or secure a separate search tier; the operator treats mongot as another managed component of the cluster.
 

Wiring it up

Enable the search component in the custom resource:

spec:
  search:
    enabled: true
    image: perconalab/percona-server-mongodb-operator:main-mongot
    size: 1
    storage:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 10Gi
    resources:
      requests:
        cpu: "2"
        memory: 2Gi

size sets how many search nodes to run, and storage gives the search index its own PersistentVolumeClaim so it does not compete with the database volume. Size the resources block to your index: vector indexes are memory-sensitive, so give mongot enough headroom for the corpus you intend to query.

Note: Vector search is a tech preview in 1.23.0 and is not recommended for production yet. It requires Percona Server for MongoDB 8.3 or later.

 

PVC snapshot backups

Logical and streamed physical backups both push data across the network to object storage, and for a multi-terabyte cluster, that path is the bottleneck. Backups run long, restores run longer, and both compete with production traffic for CPU and bandwidth. This release adds backups built on PersistentVolumeClaim snapshots, which takes the storage layer directly.

 

Why it matters

A PVC snapshot is a point-in-time copy of your data volumes taken at the storage layer through the Kubernetes VolumeSnapshot API. Because the operator asks the storage provider for a snapshot instead of streaming bytes out, a backup typically completes in seconds or minutes regardless of database size, and a restore is correspondingly fast. Two production situations show the difference: a nightly backup that no longer fits its window as a cluster grows past a few terabytes, and a staging refresh that ties up resources for hours while it restores a streamed copy. A storage-layer snapshot turns both into a near-instant operation. The speed comes from how the storage layer implements snapshots: instead of copying the whole volume, most backends record only the blocks that changed since the previous snapshot and reference the rest, so the cost tracks your change rate rather than the total database size. Snapshots also work with encrypted and TLS-enabled clusters, and they use fewer cluster resources because there is no long-running data-transfer job.

 

Wiring it up

The operator takes snapshot backups in two ways: on demand through a PerconaServerMongoDBBackup object, or on a schedule through a backup task. The scheduled form looks like this, using the external type and a VolumeSnapshotClass:

spec:
  backup:
    tasks:
      - name: daily-snapshot
        enabled: false
        schedule: "0 0 * * *"
        retention:
          count: 1
          type: count
          deleteFromStorage: true
        type: external
        volumeSnapshotClass: YOUR-VOLUME-SNAPSHOT-CLASS

type: external tells the operator to take a storage-layer snapshot rather than stream a backup, and volumeSnapshotClass names the VolumeSnapshotClass your CSI driver provides. The retention block prunes old snapshots on the schedule you set. Your storage provider must support the Kubernetes VolumeSnapshot API for this to work.

Snapshot backups complement the streamed and logical backups the operator already supports; they do not replace them. Snapshots usually live in the same storage account and region as the volumes they copy, so keep a streamed backup to object storage for off-site and cross-region disaster recovery. A practical policy pairs frequent fast snapshots for quick local recovery with a less frequent streamed backup for durability, and the operator runs both from the same backup.tasks list.


Note:
PVC snapshot backups are a tech preview in 1.23.0 and are not recommended for production yet. Snapshot portability and retention semantics depend on your CSI driver, so test restores before you rely on them.

 

Other improvements

Beyond the three headline features, 1.23.0 ships a set of enhancements that smooth day-two operations:

  • Operator-generated connection string Secrets (K8SPSMDB-1537): the operator now publishes a ready-to-use MongoDB connection string (URI) in a Kubernetes Secret for the databaseAdmin user. An application can read that one Secret and connect to it, instead of building the URI itself from Pod names, Services, TLS settings, and credentials.
  • Workload Identity for GCS backups (K8SPSMDB-1602): back up to Google Cloud Storage without storing a service-account JSON key in a Secret.
  • Oracle Cloud Infrastructure Object Storage (K8SPSMDB-1644) and Alibaba Cloud OSS (K8SPSMDB-1519): two more native backup destinations.
  • Restore a collection under a different name (K8SPSMDB-1603): use selective.nsFrom and nsTo to restore one collection alongside the live one for inspection or recovery.
  • External nodes as arbiters (K8SPSMDB-1031): set arbiterOnly: true on an external node to place a tie-breaker vote in a third location without a data-bearing member.
  • cert-manager ClusterIssuer and TLS policy (K8SPSMDB-1413, K8SPSMDB-1458): point the operator at an existing ClusterIssuer, and use certManagementPolicy to keep certificate lifecycle fully under your control.
  • Tunable reconciliation interval (K8SPSMDB-1571): set RECONCILE_INTERVAL to reduce Kubernetes API load on large fleets (default 5s).
  • Query Analytics via mongolog for PMM (K8SPSMDB-1546): choose mongolog as the QAN source in Percona Monitoring and Management.
  • Custom sidecar health probes (K8SPSMDB-1701, K8SPSMDB-1728) and StatefulSet revisionHistoryLimit (K8SPSMDB-1572): finer control over probes and rollout history. 

For the full list, including bug fixes, see the release notes linked below.

 

Conclusion

Percona Operator for MongoDB 1.23.0 covers the arc from getting data in to keeping it safe: ClusterSync brings a live database onto the operator with a short cutover, vector search lets one system serve both documents and semantic queries, and PVC snapshot backups take the network out of the backup path. With RKE2 and full ARM64 support added, more of that runs on the platforms teams actually use. If there is a workflow you still script around the operator, tell us on the forum, since that is where releases like this one come from.


Try Percona Operator for MongoDB 1.23.0

 

The post Percona Operator for MongoDB 1.23.0: ClusterSync Migration, Vector Search, and PVC Snapshot Backups appeared first on Percona.

Jul
15
2026
--

PostgreSQL Meta Commands that save time every day

When most people start working with PostgreSQL, they quickly learn SQL:

SELECT * FROM employees;

But very soon, another world opens up inside psql — a set of commands that don’t look like SQL, don’t end with semicolons.

These are PostgreSQL Meta Commands, and they quietly power the daily workflow of almost every experienced DBA.

Meta commands are not about querying data — they are about navigating, inspecting, and controlling the PostgreSQL session/database efficiently.

What exactly are Meta Commands?

Meta commands are special instructions interpreted by psql, not PostgreSQL itself.

That means:

  • They are not SQL
  • They execute instantly on the client side
  • They are specific to the psql terminal tool
  • They do not end with semicolon like SQL statements
  • The main focus area for meta commands is database interaction and not the interaction with the data in the database.

Cheat Sheet (Quick Reference) 

The most commonly used meta commands are as follows. There are many more apart from these, however, below are the most frequently used ones:

Connect and Manage Sessions

These commands help discover databases, establish connections, and verify the current session.

\c Connect to another database 
\l List all the databases available in the cluster
\l+ List all the databases available in the cluster with more details, like DB Size, etc
\conninfo Displays information about the current database connection

Please find the example of the commands used to connect and manage sessions in the screenshot below:

Inspect Database Objects                

The
\d 
family of commands is one of the most powerful features of
psql 
. These commands can be used to discover database objects, inspect their definitions, and view additional metadata.

\d Describe database objects or list objects visible in the current search path.
\d object_name Describe a specific table, view, sequence, or other database object.
\d+ object_name Display extended information about an object.
\dt List tables. Supports schema names and wildcard patterns.
\di List indexes. Supports wildcard patterns.
\dn List schemas in the current database.
\du List database roles.
\db List tablespaces
\dx List installed extensions
\df List functions and procedures
\sf function name Displays the source code of the specific function/procedure

Using object names and wildcards

Most object-inspection commands accept object names, schema-qualified names, and wildcard patterns.

For example:

\dt

Lists all tables in the current search path.

\dt public.*

Lists all tables in the public schema.

The same pattern matching is supported by several other meta-commands, including \di, \df, and the \d family.

Please find the example of the \d family commands in the screenshot below:

Format Query Results

Several meta-commands are available to improve the readability of query output, particularly when working with wide result sets.

\x [on|off|auto] Toggle expanded (vertical) display
\o filename Redirect query output to a file or pipe.
\o Restore query output to the terminal.

Monitor Query Executions

These commands assist in measuring query performance and repeatedly executing queries for monitoring purposes.

\timing [on|off] Toggle Query execution timing
\watch seconds Re-execute the current query at the specified interval

Execute and Automate tasks

These commands simplify repetitive tasks and enable integration between psql, SQL scripts, and the operating system

\i filename Execute the commands from the file
\gexec Execute each field returned by a query as an SQL statement.
\! command Execute a shell command without leaving a psql prompt

Get Help

Built-in help commands provide quick access to both psql meta-command documentation and PostgreSQL SQL syntax without leaving the terminal.

\? Display all available psql meta-commands.
\h List SQL commands for which syntax help is available.
\h command Display syntax help for a specific SQL command.

What is .psqlrc?

.psqlrc is a startup file in the home directory that psql reads when a session begins. It can hold meta-commands and SQL that run before the first prompt. The main benefit is consistent defaults — timing, formatting, and a custom prompt — without repeating setup each time, which speeds daily work and reduces connection mistakes across databases.

A minimal .psqlrc might look like this:

\timing on 
\x auto

These settings load automatically on every new psql session as highlighted below:

Conclusion

PostgreSQL is powerful because of SQL — but for DBAs, psql meta commands make daily management far easier and more efficient.

Most developers use only a handful like \dt or \d. But experienced DBAs rely on a much broader toolkit to:

  • Investigate production issues faster
  • Navigate systems efficiently
  • Reduce reliance on repetitive SQL
  • Repetitive tasks can be automated
  • Debug complex problems quickly

An easy way to understand the relationship between SQL and PostgreSQL meta commands is to compare them to driving a car.

SQL is like driving the car — it is the primary means of reaching a destination. It is used to retrieve, insert, update, and delete data, enabling applications and users to interact with the information stored in the database.

Meta commands, on the other hand, are like the car’s dashboard. While the dashboard does not move the vehicle, it provides essential information such as speed, fuel level, engine health, navigation status, and warning indicators. Driving without a dashboard is certainly possible, but it would mean operating with limited visibility into the vehicle’s condition and performance.

Similarly, SQL is responsible for manipulating and retrieving data, whereas PostgreSQL meta commands provide valuable insight into the database environment itself. They help administrators inspect database objects, navigate schemas, monitor sessions, examine roles and privileges, review object definitions, and perform numerous administrative tasks efficiently.

In essence, SQL enables interaction with the data, while meta commands enable interaction with the PostgreSQL environment. Together, they form a complementary toolkit that allows database professionals to work more effectively, troubleshoot issues faster, and administer PostgreSQL with greater confidence.

The post PostgreSQL Meta Commands that save time every day appeared first on Percona.

Jul
14
2026
--

Inside MySQL 9.7 LTS Features

MySQL 9.7, a Long-Term Support (LTS) release, incorporates a variety of potential features spanning across multiple technical domains. This article covers some of the primary features introduced and evaluates their practical utility within the MySQL database environment.

Following the End-of-Life (EOL) status of MySQL 8.0, this subsequent LTS release is designed to provide enhanced stability alongside significant architectural innovations.

Let’s discuss each of these features below with some examples and usage.

Flow-control monitoring in Group Replication

Flow control monitoring has been improved and provides more granularity by introducing the additional status variables listed below.

  • Gr_flow_control_throttle_count : It denotes the number of transactions that have been throttled.
  • Gr_flow_control_throttle_time_sum :It denotes the time in microseconds that transactions have been throttled.
  • Gr_flow_control_throttle_active_count :It denotes the number of transactions currently being throttled.
  • Gr_flow_control_throttle_last_throttle_timestamp : It denotes the most recent date and time that a transaction was throttled.

To use these status variables, we must install the “Group Replication Flow Control Statistics”  component.

mysql> Install component 'file://component_group_replication_flow_control_stats';

After the component is installed, the statistics will be visible.

mysql> SELECT * FROM performance_schema.global_status WHERE VARIABLE_NAME LIKE 'Gr_flow_control%';
+--------------------------------------------------+----------------+
| VARIABLE_NAME                                    | VARIABLE_VALUE |
+--------------------------------------------------+----------------+
| Gr_flow_control_throttle_active_count            | 0              |
| Gr_flow_control_throttle_count                   | 0              |
| Gr_flow_control_throttle_last_throttle_timestamp |                |
| Gr_flow_control_throttle_time_sum                | 0              |
+--------------------------------------------------+----------------+

Multi-threaded applier extended statistics

We now have additional verbosity for the Applier threads for both Asynchronous and Group Replication topologies. This means we can get more details of the transactions or potential misbehaviours during the transactions applier stage. This feature is particularly useful for troubleshooting performance bottlenecks in multi-threaded replication environments, where understanding the specific cause of lag can be challenging.

This requires installing the “Replication Applier Metrics” component.

mysql> Install component 'file://component_replication_applier_metrics';

Upon successful installation of the requisite component, the performance schema tables facilitate tracking of transaction details and various performance metrics during the replication applier phase. For instance, monitoring the table “replication_applier_metrics” enables observing channel-specific operations.

mysql> SELECT * FROM performance_schema.replication_applier_metrics where CHANNEL_NAME='group_replication_applier'\G;
*************************** 1. row ***************************
                                CHANNEL_NAME: group_replication_applier
                  TOTAL_ACTIVE_TIME_DURATION: 0
                          LAST_APPLIER_START: 0000-00-00 00:00:00
                TRANSACTIONS_COMMITTED_COUNT: 0
                  TRANSACTIONS_ONGOING_COUNT: 0
                  TRANSACTIONS_PENDING_COUNT: 0
       TRANSACTIONS_COMMITTED_SIZE_BYTES_SUM: 0
    TRANSACTIONS_ONGOING_FULL_SIZE_BYTES_SUM: 0
TRANSACTIONS_ONGOING_PROGRESS_SIZE_BYTES_SUM: 0
         TRANSACTIONS_PENDING_SIZE_BYTES_SUM: NULL
                      EVENTS_COMMITTED_COUNT: 0
            WAITS_FOR_WORK_FROM_SOURCE_COUNT: 0
         WAITS_FOR_WORK_FROM_SOURCE_SUM_TIME: 0
            WAITS_FOR_AVAILABLE_WORKER_COUNT: 0
         WAITS_FOR_AVAILABLE_WORKER_SUM_TIME: 0
      WAITS_COMMIT_SCHEDULE_DEPENDENCY_COUNT: 0
   WAITS_COMMIT_SCHEDULE_DEPENDENCY_SUM_TIME: 0
         WAITS_FOR_WORKER_QUEUE_MEMORY_COUNT: 0
      WAITS_FOR_WORKER_QUEUE_MEMORY_SUM_TIME: 0
              WAITS_WORKER_QUEUES_FULL_COUNT: 0
           WAITS_WORKER_QUEUES_FULL_SUM_TIME: 0
             WAITS_DUE_TO_COMMIT_ORDER_COUNT: 0
          WAITS_DUE_TO_COMMIT_ORDER_SUM_TIME: 0
        TIME_TO_READ_FROM_RELAY_LOG_SUM_TIME: 0

In addition to aggregate metrics, MySQL 9.7 provides a way to inspect the progress of individual worker threads via monitoring stats in the “replication_applier_progress_by_worker” table. This level of detail helps administrators identify if a single transaction is monopolising a specific worker, causing overall replication delay.

mysql> SELECT * FROM performance_schema.replication_applier_progress_by_worker\G;
*************************** 1. row ***************************
                          CHANNEL_NAME: group_replication_applier
                             WORKER_ID: 0
                             THREAD_ID: 62
              ONGOING_TRANSACTION_TYPE: UNASSIGNED
   ONGOING_TRANSACTION_FULL_SIZE_BYTES: 0
ONGOING_TRANSACTION_APPLIED_SIZE_BYTES: 0
*************************** 2. row ***************************
                          CHANNEL_NAME: group_replication_applier
                             WORKER_ID: 1
                             THREAD_ID: 63
              ONGOING_TRANSACTION_TYPE: UNASSIGNED
   ONGOING_TRANSACTION_FULL_SIZE_BYTES: 0
ONGOING_TRANSACTION_APPLIED_SIZE_BYTES: 0
*************************** 3. row ***************************
                          CHANNEL_NAME: group_replication_applier
                             WORKER_ID: 2
                             THREAD_ID: 64
              ONGOING_TRANSACTION_TYPE: UNASSIGNED
   ONGOING_TRANSACTION_FULL_SIZE_BYTES: 0
ONGOING_TRANSACTION_APPLIED_SIZE_BYTES: 0
*************************** 4. row ***************************
                          CHANNEL_NAME: group_replication_applier
                             WORKER_ID: 3
                             THREAD_ID: 65
              ONGOING_TRANSACTION_TYPE: UNASSIGNED
   ONGOING_TRANSACTION_FULL_SIZE_BYTES: 0
ONGOING_TRANSACTION_APPLIED_SIZE_BYTES: 0

Automatic eviction & rejoin

The Group Replication resource manager now provides auto-eviction functionality, which we can configure using the available options. This basically ensures that the unhealthy node is removed from the Group to maintain the cluster’s high availability and overall performance.

This requires installing the “group replication resource manager” component.

mysql> INSTALL COMPONENT 'file://component_group_replication_resource_manager';

Once the component is available,  we can use various options to decide the node expulsion policy.

1) Applier channel

We can set the applier channel replication lag threshold values using the configuration parameter below.

mysql> set global group_replication_resource_manager.applier_channel_lag = <value>;

If lag exceeds  “applier_channel_lag” threshold 10 times or more in a row, this server is expelled from the group. The status variable below is used for tracking the lag exceed rate.

mysql> show global status like 'Gr_resource_manager_applier_channel_lag';
+-----------------------------------------+-------+
| Variable_name                           | Value |
+-----------------------------------------+-------+
| Gr_resource_manager_applier_channel_lag | 0     |
+-----------------------------------------+-------+


2) Recovery Channel

Similarly, we can define a threshold for the group member recovery process to attempt to rejoin the cluster. 

mysql> set global group_replication_resource_manager.recovery_channel_lag = <value>;

If the secondary’s recovery lag exceeds “recovery_channel_lag”, 10 times or more in succession, the server is expelled from the group. 

mysql show global status like 'Gr_resource_manager_recovery_channel_lag';
+------------------------------------------+-------+
| Variable_name                            | Value |
+------------------------------------------+-------+
| Gr_resource_manager_recovery_channel_lag | 0     |
+------------------------------------------+-------+

3) Memory/Resource Usage

We can also define an expelled condition based on the group member’s memory or resource usage %.

mysql> set global group_replication_resource_manager.memory_used_limit = 10;

If the memory usage exceeds memory_used_limit % by 10 or more consecutive times, the node will be expelled from the group.

mysql> show global status like 'Gr_resource_manager_memory_used%';
+---------------------------------+-------+
| Variable_name                   | Value |
+---------------------------------+-------+
| Gr_resource_manager_memory_used | 78    |
+---------------------------------+-------+
1 row in set (0.002 sec)

In addition to the discussed options above, we can also track various server status variables to monitor group replication and the resource manager component.

mysql> select * from performance_schema.global_status where variable_name in ('Gr_resource_manager_applier_channel_threshold_hits','Gr_resource_manager_applier_channel_eviction_timestamp','Gr_resource_manager_recovery_channel_threshold_hits','Gr_resource_manager_recovery_channel_eviction_timestamp','Gr_resource_manager_memory_threshold_hits','Gr_resource_manager_memory_eviction_timestamp');
+---------------------------------------------------------+----------------+
| VARIABLE_NAME                                           | VARIABLE_VALUE |
+---------------------------------------------------------+----------------+
| Gr_resource_manager_applier_channel_eviction_timestamp  |                |
| Gr_resource_manager_applier_channel_threshold_hits      | 0              |
| Gr_resource_manager_memory_eviction_timestamp           |                |
| Gr_resource_manager_memory_threshold_hits               | 6703           |
| Gr_resource_manager_recovery_channel_eviction_timestamp |                |
| Gr_resource_manager_recovery_channel_threshold_hits     | 0              |
+---------------------------------------------------------+----------------+
6 rows in set (0.003 sec)

The expelled node can attempt to automatically rejoin based on the value of the group_replication_autorejoin_tries variable.

mysql> show variables like '%group_replication_autorejoin_tries%';
+------------------------------------+-------+
| Variable_name                      | Value |
+------------------------------------+-------+
| group_replication_autorejoin_tries | 3     |
+------------------------------------+-------+
1 row in set (0.006 sec)

If the node cannot join, it will perform the behaviour specified in the group_replication_exit_state_action variable.

mysql> show variables like '%group_replication_exit_state_action%';
+-------------------------------------+--------------+
| Variable_name                       | Value        |
+-------------------------------------+--------------+
| group_replication_exit_state_action | OFFLINE_MODE |
+-------------------------------------+--------------+
1 row in set (0.005 sec)

After a server is evicted from the group (for whatever reason), it gets a grace period (group_replication_resource_manager) when it rejoins. During this period, the Resource Manager won’t immediately kick it out again, even if it’s still lagging or breaching the defined threshold as discussed above.

mysql> show variables like '%group_replication_resource_manager.quarantine_time%';
+----------------------------------------------------+-------+
| Variable_name                                      | Value |
+----------------------------------------------------+-------+
| group_replication_resource_manager.quarantine_time | 3600  |
+----------------------------------------------------+-------+

Up-to-date aware Primary election

The Primary election process is more mature and cohesive. The Group Replication Manager now uses the most up-to-date status as a criterion for selecting the new primary.

Here is how the Group Replication Manager performs the most up-to-date primary selection prior to MySQL v9.7.

  1. The lowest MySQL version is checked for each member.
  2. If more than one member is running the lowest MySQL Server version, each member’s weight is determined by the “group_replication_member_weight” system variable.
  3. If there is more than one member running the lowest MySQL Server version, and also more than one of those members has the highest member weight, the third factor considered is the lexicographical order of the generated server UUIDs “server_uuid” of each group member. The member with the lowest server UUID is chosen as the new primary.

In MySQL version 9.7, “group_replication_elect_prefers_most_updated” was introduced, so the failover will be determined by how many transactions are in the secondary backlog. Basically the secondary with the least backlog will be selected as Primary.

Now, it will consider the “most up-to-date” node first, then “weight” and then “UUID”

To use “group_replication_elect_prefers_most_updated”, we need to install the “Group Replication Primary Election” component listed below on each Group Member.

mysql> Install component 'file://component_group_replication_elect_prefers_most_updated';

By default, the most up-to-date group member selection is enabled. We need to make sure it’s enabled on all Group Members. 

mysql> select @@group_replication_elect_prefers_most_updated.enabled;
+--------------------------------------------------------+
| @@group_replication_elect_prefers_most_updated.enabled |
+--------------------------------------------------------+
|                                                      1 |
+--------------------------------------------------------+
1 row in set (0.007 sec)

In the event that a new primary is elected via the most up-to-date selection mechanism, this metric represents the transaction processing differential between the newly designated primary and the secondary node with the highest level of synchronisation.

mysql> show status like 'Gr_latest_primary_election_by_most_uptodate_members_trx_delta';
+---------------------------------------------------------------+-------+
| Variable_name                                                 | Value |
+---------------------------------------------------------------+-------+
| Gr_latest_primary_election_by_most_uptodate_members_trx_delta | 0     |
+---------------------------------------------------------------+-------+

Also, we can track the timestamp of the most recent primary election on the most up-to-date node.

mysql> show status like 'Gr_latest_primary_election_by_most_uptodate_member_timestamp';
+--------------------------------------------------------------+-------+
| Variable_name                                                | Value |
+--------------------------------------------------------------+-------+
| Gr_latest_primary_election_by_most_uptodate_member_timestamp |       |
+--------------------------------------------------------------+-------+
1 row in set (0.005 sec)

The database logs also tell exactly what criteria the primary member selected during failover.

2026-06-14T10:04:02.243809Z 0 [System] [MY-015575] [Repl] Plugin group_replication reported: 'Member with uuid 00021702-2222-2222-2222-222222222222 was elected primary since it was the most up-to-date member with 2755 transactions more than second most up-to-date member 00021703-3333-3333-3333-333333333333. In case of a tie member weight and then uuid lexical order was used over the most updated members.'

MySQL JSON duality views

With the introduction of JSON duality views, we can leverage a single unified JSON document for both relational and hierarchical JSON data. This provides a common, structured JSON format for the application, allowing it to perform both read and write operations.

Let’s see a quick scenario below on how it works.

Below are two relational tables from which we obtain aggregated information in JSON format. 

mysql> CREATE TABLE products (
  product_id INT PRIMARY KEY,
  product_type VARCHAR(100)
);

mysql> CREATE TABLE products_details (
  product_detail_id INT PRIMARY KEY,
  product_id INT,
  name VARCHAR(100),
  active varchar(10)
);

mysql> INSERT INTO products (product_id,product_type) VALUES (1,'IT'), (2,'TEL');
mysql> INSERT INTO products_details (product_detail_id,product_id,name,active) VALUES (1,1,'Laptop','Yes'), (2,2,'Mobile','Yes');

Here is the exact Json View which fetch the columns from the relation table based on the join condition. Each of those relational table columns is mapped with a JSON data structure (_id,v_product_type,v_product_type ), and the complete details of the product details table are fetched into the (product) array.

mysql> CREATE JSON RELATIONAL DUALITY VIEW view_product AS
SELECT JSON_DUALITY_OBJECT( WITH(INSERT,UPDATE,DELETE)
    '_id': product_id,
    'v_product_type': product_type,
    'product': (
        SELECT JSON_ARRAYAGG(
            JSON_DUALITY_OBJECT(WITH(INSERT,UPDATE,DELETE)
                'v_product_detail_id': product_detail_id,
                'v_name': name,
                'v_active': active
                
            )
        )
        FROM products_details
        WHERE products_details.product_id = products.product_id
    )
)
FROM products;

mysql> select * from view_product;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| data                                                                                                                                                                           |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"_id": 1, "product": [{"v_name": "Laptop", "v_active": "Yes", "v_product_detail_id": 1}], "_metadata": {"etag": "313642c2aa24f0571264332afa140715"}, "v_product_type": "IT"}  |
| {"_id": 2, "product": [{"v_name": "Mobile", "v_active": "Yes", "v_product_detail_id": 2}], "_metadata": {"etag": "3d229ada02ac660f9f6cac994b44831a"}, "v_product_type": "TEL"} |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.002 sec)

Once the duality view is created, we can perform both read/write operations.

Reading the duality view

mysql> select * from view_product;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| data                                                                                                                                                                           |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"_id": 1, "product": [{"v_name": "Laptop", "v_active": "Yes", "v_product_detail_id": 1}], "_metadata": {"etag": "313642c2aa24f0571264332afa140715"}, "v_product_type": "IT"}  |
| {"_id": 2, "product": [{"v_name": "Mobile", "v_active": "Yes", "v_product_detail_id": 2}], "_metadata": {"etag": "3d229ada02ac660f9f6cac994b44831a"}, "v_product_type": "TEL"} |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Writing the underlying table in the duality view

mysql> UPDATE view_product
SET data = JSON_SET(
    data,
    '$.product[0].v_name',
    'Notepad'
)
WHERE JSON_EXTRACT(data, '$._id') = 1;

mysql> select * from products_details;
+-------------------+------------+---------+--------+
| product_detail_id | product_id | name    | active |
+-------------------+------------+---------+--------+
|                 1 |          1 | Notepad | Yes    |
|                 2 |          2 | Mobile  | Yes    |
+-------------------+------------+---------+--------+

After performing the above write operations, we can see that the view now shows the updated data.

mysql > select * from view_product;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| data                                                                                                                                                                           |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"_id": 1, "product": [{"v_name": "Notepad", "v_active": "Yes", "v_product_detail_id": 1}], "_metadata": {"etag": "72c4368420cdc698842d0ab4bd9315ab"}, "v_product_type": "IT"} |
| {"_id": 2, "product": [{"v_name": "Mobile", "v_active": "Yes", "v_product_detail_id": 2}], "_metadata": {"etag": "3d229ada02ac660f9f6cac994b44831a"}, "v_product_type": "TEL"} |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Hypergraph Optimizer

With the Hypergraph Optimiser, we now have more advanced optimisation for complex queries and a broader set of Join plans than the older traditional method, missing earlier. By using “Join hypergraph”, the optimiser now has better reach to all tables in the join condition.

Hypergraph Optimiser is OFF

mysql> SET optimizer_switch='hypergraph_optimizer=off';

mysql> SELECT t1.k, COUNT(*) AS cnt
FROM sbtest1 t1
JOIN sbtest2 t2 ON t1.id = t2.id
JOIN sbtest3 t3 ON t1.id = t3.id
WHERE t1.k BETWEEN 200000 AND 500000
GROUP BY t1.k
ORDER BY cnt DESC
LIMIT 100;

Output:

| 498870 | 119 |
| 498729 | 119 |
| 497668 | 119 |
| 498076 | 119 |
+--------+-----+
100 rows in set (4.000 sec)

Explain output:

-> Limit: 100 row(s)
    -> Sort: cnt DESC, limit input to 100 row(s) per chunk
        -> Stream results  (cost=1.22e+6 rows=175136)
            -> Group aggregate: count(0)  (cost=1.22e+6 rows=175136)
                -> Nested loop inner join  (cost=1.1e+6 rows=493200)
                    -> Nested loop inner join  (cost=601547 rows=493200)
                        -> Filter: (t1.k between 200000 and 500000)  (cost=99122 rows=493200)
                            -> Covering index range scan on t1 using k_1 over (200000 <= k <= 500000)  (cost=99122 rows=493200)
                        -> Single-row covering index lookup on t2 using PRIMARY (id = t1.id)  (cost=0.919 rows=1)
                    -> Single-row covering index lookup on t3 using PRIMARY (id = t1.id)  (cost=0.919 rows=1)

Hypergraph Optimiser is ON

mysql> SET optimizer_switch='hypergraph_optimizer=on';

mysql> SELECT t1.k, COUNT(*) AS cnt
FROM sbtest1 t1
JOIN sbtest2 t2 ON t1.id = t2.id
JOIN sbtest3 t3 ON t1.id = t3.id
WHERE t1.k BETWEEN 200000 AND 500000
GROUP BY t1.k
ORDER BY cnt DESC
LIMIT 100;

Output:

| 499721 | 119 |
| 499052 | 119 |
| 498870 | 119 |
| 498384 | 119 |
+--------+-----+
100 rows in set (0.498 sec)

Explain output:

-> Sort: cnt DESC, limit input to 100 row(s) per chunk  (cost=1.96e+6..1.96e+6 rows=100)
    -> Table scan on <temporary>  (cost=1.87e+6..1.9e+6 rows=175136)
        -> Aggregate using temporary table  (cost=1.87e+6..1.87e+6 rows=175136)
            -> Inner hash join (t2.id = t3.id)  (cost=990754..1.44e+6 rows=493200)
                -> Covering index scan on t3 using k_1  (cost=0.312..308240 rows=986400)
                -> Hash
                    -> Inner hash join (t1.id = t2.id)  (cost=370988..824021 rows=493200)
                        -> Covering index scan on t2 using k_1  (cost=0.312..308240 rows=986400)
                        -> Hash
                            -> Filter: (t1.k between 200000 and 500000)  (cost=0.416..205287 rows=493200)
                                -> Covering index range scan on t1 using k_1 over (200000 <= k <= 500000)  (cost=0.359..176877 rows=493200)

We can see that with “hypergraph_optimizer=enabled”, the query execution time is almost 8x faster.

The performance difference might not be noticeable with a few joins or a smaller table’s data set, but with more complex joins, it can yield better performance. In the above example, we can see that when “hypergraph_optimizer=enabled”, the optimiser replaces “Nested loop inner join” with “Inner hash join”, which is generally better for large datasets. 

Higher version source allowed

Now, it’s possible that a lower version replica can connect to a higher version source when the major versions differ. That means we don’t have to rely on all replicas being upgraded in one go; we can just upgrade the source, verify it, and later perform rolling upgrades on lower-version replicas as per our own timelines and convenience.

Of course, we have to be cautious not to run any such feature or change on the source that doesn’t support lower-version replicas.

Please note – This won’t be applicable to previous releases, say (8.4, 8.0), as they didn’t restrict such replication connectivity. It would be useful for 9.7 or the next major release.

To enable this functionality, we need to ensure the following variable is enabled on the Replica. By default its enabled on 9.7

mysql> show variables like 'replica_allow_higher_version_source';
+-------------------------------------+-------+
| Variable_name                       | Value |
+-------------------------------------+-------+
| replica_allow_higher_version_source | ON    |
+-------------------------------------+-------+
1 row in set (0.008 sec)

Summary

The above discussion highlights key advancements in MySQL 9.7 LTS, ranging from some innovative or operational improvements to developer-centric features such as “JSON Duality” Views. Also, the “Hypergraph Optimiser” is now available for community release, which was previously exclusive to MySQL Heatwave/Enterprise.  As a Long-Term Support (LTS) release, MySQL 9.7 is structured to provide a stable and consistent environment, prioritising architectural reliability over frequent experimental changes.

One more important mention here: It’s suggested to use MySQL 9.7.1, or the next sub-releases, as 9.7.0 has some higer severity CVE’s. If you are using Percona Server for MySQL (PS), we skipped 9.7.0 and are shipping the fixed 9.7.1 version directly.

Still, it’s highly recommended to test any new component or changes in your lower/staging environment before deploying in production to better assess the overall impact on existing workload, queries, and database behaviour.

 

The post Inside MySQL 9.7 LTS Features appeared first on Percona.

Jul
10
2026
--

Running DuckDB as a MySQL 9.7 storage engine

ducksdb-mysql-engine is an experimental build of MySQL 9.7 where a table you mark ENGINE=DuckDB answers analytical queries from DuckDB instead of InnoDB. Same server, same connection, no second copy of the data. On TPC-H at scale factor 10, InnoDB times out on 6 of the 22 queries and burns 1317 seconds on the 16 it finishes. The DuckDB tables run all 22 in about 15 seconds.

It’s an experiment, not production software. It patches mysqld and has rough edges, which we list at the end. Source is on GitHub under GPLv2:     https://github.com/EvgeniyPatlan/ducksdb-mysql-engine.

Why we made it

MySQL is great for transactions and slow at analytics. A wide GROUP BY over a few hundred million rows, or a six-way join, takes minutes on InnoDB. The usual fix is to copy the data into a column store and keep it in sync, so now you’re running two systems and the pipeline between them.

We wanted the table itself to be the column store, with the heavy queries offloaded for you. Mark it ENGINE=DuckDB, query it the way you always have, and DuckDB does the analytical work.

What it actually is

DuckDB is an in-process columnar query engine, basically SQLite for OLAP. It stores data by column and it’s built for scans and aggregations, which is exactly what a row store is bad at.

We’re not the first to put it behind a relational table. Alibaba’s AliSQL has had a built-in DuckDB engine for a while. MariaDB shipped MariaDB DuckDB about a week ago, while we were already building ours. AliSQL got there first; MariaDB and we landed on the same idea independently, around the same time, them on MariaDB and us on stock MySQL 9.7. Their engine is the closest comparison to ours, so it’s in the benchmarks below.

How it hooks into MySQL

MySQL doesn’t have a select_handler, the API MariaDB uses to grab a whole SELECT and run it inside an engine. We added our own: a handlerton::pushdown_select hook.

The pushdown path. Either the whole query renders to DuckDB SQL and runs columnar, or it declines and the normal row path handles it.

The engine is compiled into mysqld, and each schema is one DuckDB file under the datadir. Three patches do the integration, and all three are generic, so they’ll fire for any engine that exposes the hook:

  • The hook runs at the end of JOIN::optimize(). If every base table in the block is one engine that has the hook, that engine looks at the optimized JOIN, and if it can translate the whole query it sets JOIN::override_executor_func (which the executor already checks in sql_union.cc). The query gets regenerated as DuckDB SQL, prepared once, run, and the aggregated result is staged into a temp table. EXPLAIN is left alone.
  • A server-side LOAD DATA INFILE goes into a DuckDB COPY instead of crawling through write_row row by row. At 600M rows that’s a 20-minute load instead of 80.
  • For single-engine statements we clear OPTIMIZER_SWITCH_SEMIJOIN in prepare, so IN, EXISTS, NOT IN and NOT EXISTS stay as subqueries the builder can render instead of getting rewritten into semijoin nests it can’t recognize.

The builder only renders a node when the output is provably identical to what MySQL would return. If it can’t, it declines and MySQL runs the query unchanged. Literals are bound as parameters. Collation, NULL ordering and decimal scale are matched on purpose, and an unmapped collation or a REAL literal is enough to make it back off. With that in place, all 22 TPC-H queries push down and match InnoDB row for row.

 

Getting started

The fastest way in is the image:

docker run -d --name mysql-duckdb -p 3306:3306 \
-e MYSQL_ROOT_PASSWORD=secret \
-v mysql-duckdb-data:/var/lib/mysql \
evgeniypatlan/test-images:mysql-9.7-duckdb-v0.2.0

Make a table, put a few rows in, run an aggregate:

CREATE DATABASE shop; USE shop;
CREATE TABLE sales (id INT PRIMARY KEY, region INT, amount DECIMAL(12,2)) ENGINE=DuckDB;
INSERT INTO sales VALUES (1,1,100),(2,1,200),(3,2,50);

SELECT region, SUM(amount) FROM sales GROUP BY region;
-- region | SUM(amount)
-- 1 | 300.00
-- 2 | 50.00

 

Nothing about that query is special, and that is the point. To check it actually went to DuckDB rather than down the row path, watch the Ducksdb_pushdown_count status variable:

SELECT region, SUM(amount) FROM sales GROUP BY region; -- offloaded
SHOW STATUS LIKE 'Ducksdb_pushdown_count'; -- counter goes +1

SELECT * FROM sales WHERE id = 3; -- point lookup
SHOW STATUS LIKE 'Ducksdb_pushdown_count'; -- counter unchanged

 

The single-row lookup stays on the row path deliberately. For one row an index seek beats spinning up a DuckDB result, so there is no reason to offload it. OLTP keeps its path, analytics get the column store, and you do not pick by hand.

If you would rather build it, you need the MySQL 9.7 tree under vendor/mysql-server/ and a DuckDB prefix, then:

ln -s ../../engine vendor/mysql-server/storage/duckdb
scripts/build-server.sh # applies the 3 patches, builds mysqld + clients

 

Does it actually go fast?

All 22 TPC-H queries, were executed in a Docker on one laptop (20 cores, 62 GiB RAM), the same data loaded into four engines: InnoDB, our MySQL+DuckDB, MariaDB+DuckDB, and standalone DuckDB as the reference. Warm wall-clock, minimum over a few runs, in seconds.

SF10, around 60 million lineitem rows

A handful of rows here; the whole table is in docs/tpch_engine_comparison.md:

Query InnoDB MySQL+DuckDB MariaDB +DuckDB Native DuckDB
Q1 >180 1.77 0.84 0.77
Q5 127.6 0.53 0.46 0.71
Q7 145.0 0.45 0.38 0.67
Q9 >180 1.52 2.30 1.78
Q18 101.7 1.35 1.39 1.31
Q19 120.4 0.15 0.67 0.83
All 22  Finished 16/22 15.1s 13.3s 16.3

 

SF10, all 22 queries, log scale (lower is better). Hatched InnoDB bars did not finish inside 180 s. The three DuckDB engines sit in a tight band near the floor.

InnoDB is somewhere between 100 and 340 times slower per query, and on 6 of the 22 it never finished inside the 180-second cap (the correlated subqueries and the heaviest scans). It burned 1317 seconds on just the 16 it did finish. The three DuckDB engines get through all 22 in about 15 seconds, and ours lands right between MariaDB and plain DuckDB. The gap is so big there is not much else to say about it.

SF100, around 600 million lineitem rows

At this size InnoDB is out of the running (a copy of the data alone is about 100 GB and queries run for hours), so it is the three DuckDB engines only, run one at a time:

Query MySQL+DuckDB MariaDB+DuckDB Native DuckDB
Q1 15.25 6.50 5.50
Q9 20.97 115.29 19.54
Q10 10.64 ERR 8.14
Q13 19.75 ERR 13.27
Q18 14.88 29.62 11.08
Q19 2.91 7.98 6.61
Correct 22/22 20/22 22/22

 

SF100, three DuckDB engines, log scale (lower is better). Q15 for our engine is shown at its matched-memory time (~4 s); the capped run measured 1309 s, explained below. MariaDB errored on Q10 and Q13.

At 600 million rows ours is still correct on all 22 and stays close to plain DuckDB. MariaDB’s engine drops two queries (Q10, and Q13 on its column-list syntax) and is a lot slower on the big joins – Q9 took 115 seconds against our 21 and native’s 20.

One honest word on Q15 at SF100, because in the full table it shows an ugly number for our engine. It is not a real loss. We capped DuckDB’s memory so it spills to disk instead of getting OOM-killed inside mysqld, and under that cap Q15’s CTE spills a lot. Give it the memory MariaDB had and it runs in about 4 seconds, like native. The answer was always right; only the clock was bad.

And one number we did not expect: loading those 600 million rows took about 20 minutes with our engine (the COPY shortcut) versus about 80 minutes with MariaDB, which loads row by row on a single core. Roughly four times faster to get the data in.

Try it, then tell us

If any of this sounds useful, pull the image and throw your own queries at it:

docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=secret \
evgeniypatlan/test-images:mysql-9.7-duckdb-v0.2.0

The source is on GitHub (GPLv2), patches and benchmark harness included: https://github.com/EvgeniyPatlan/ducksdb-mysql-engine. The full per-query benchmark and how we measured it live in docs/tpch_engine_comparison.md.

 

The post Running DuckDB as a MySQL 9.7 storage engine appeared first on Percona.

Jul
07
2026
--

Percona Operator for MySQL 1.2.0: Cross-Site Replication, Encrypted Backups, and Automatic Storage Scaling

 

Percona Operator for MySQL 1.2.0 is out, and it closes three gaps that platform teams hit once a MySQL deployment grows past a single cluster. Picture a fleet that has outgrown one region: you want a warm replica cluster in a second data center, backups in object storage that pass an auditor’s encryption check, and volumes that grow before they fill at 3 a.m. Until now, each of those meant scripting around the operator. This release brings all three into the custom resource.

The three headline features are cross-site replication for Group Replication, encrypted backups, and automatic storage scaling. Each one turns a manual, error-prone procedure into a declarative field you set once and let the operator reconcile.

The operator is open source and runs on any CNCF-certified Kubernetes distribution. Many of the changes in this release come straight from what users asked for on forums.percona.com and in the public issue tracker, from disaster-recovery topologies to backup encryption to storage that keeps up with data growth.

In this post, you’ll learn about:

  • Cross-site replication for Group Replication clusters
  • Encrypted backups to S3, GCS, and Azure
  • Automatic storage scaling for MySQL data volumes
  • Other improvements worth knowing about


Cross-site replication for Group Replication

 

Group Replication gives you a self-healing, multi-primary-capable cluster inside one Kubernetes cluster. What it does not give you on its own is a second site. If the region hosting your cluster goes down, Group Replication cannot fail over to hardware it does not know about. Teams have solved this by hand-wiring asynchronous replication between clusters and babysitting it, which is exactly the kind of stateful glue an operator should own.

 

Why it matters

A disaster-recovery topology is only useful if it is reproducible and observable. Hand-built replication links drift: someone changes a credential, a channel stalls, and nobody notices until the failover that was supposed to save you does not work. Declaring the topology as a Kubernetes object means the operator reconciles it continuously, and the same manifest recreates it in staging, in a runbook test, and in the real event.

 

How it works

The operator adds a new custom resource, PerconaServerMySQLClusterSet, that groups two or more Group Replication clusters into a single set with one primary. The operator drives MySQL Shell to build the InnoDB ClusterSet, wires the replica clusters to the primary, and tracks the topology in the resource’s status. A replica cluster provisions from the primary using a chosen recovery method, so you do not stage data manually.

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQLClusterSet
metadata:
  name: my-cluster-set
  finalizers:
    - percona.com/clusterset-dissolve
spec:
#  unsafeFlags:
#    forcedFailover: false
#    forcedClusterRemoval: false
  primaryCluster: pscluster1
  credentialsSecret:
    name: ps-cluster1-secrets
    key: clusterset
  sslMode: AUTO
  createReplicaClusterOptions:
    recoveryMethod: clone
  clusters:
    - innodbClusterName: pscluster1
      endpoints:
      - host: ps-cluster1-mysql-primary.default.svc.cluster.local
    - innodbClusterName: pscluster2
      endpoints:
      - host: ps-cluster2-mysql-0.ps-cluster2-mysql.default.svc.cluster.local
  mysqlshellRunner:
    image: percona/percona-server:8.4.10-10.1

primaryCluster names the source of truth. Each entry under clusters points at a Group Replication cluster by its InnoDB cluster name and reachable endpoints, so the clusters can live in separate namespaces or separate Kubernetes clusters joined by routable DNS. createReplicaClusterOptions.recoveryMethod: clone tells the replica to seed itself with a full clone. The percona.com/clusterset-dissolve finalizer ensures the operator tears the ClusterSet down cleanly instead of leaving orphaned replication channels behind.
 

Failover and cleanup

Once the set exists, the operator keeps the replica clusters attached to the primary and surfaces the topology in the resource’s status, so you can see which cluster is primary and whether every replica is connected without shelling into MySQL Shell. A planned switchover promotes a replica to primary. The unsafeFlags block gates the disruptive paths for when the primary is already gone.

Note: The unsafeFlags block gates disruptive operations such as forced failover and forced cluster removal. Leave these off for normal operation and reach for them only in a controlled recovery, since a forced failover can diverge history if the old primary comes back.

 

Encrypted backups

Backups are the copy of your data most likely to leave the cluster’s security boundary. They land in an object-storage bucket, get replicated across a provider’s regions, and often live longer than the database that produced them. If they are not encrypted before they leave the pod, a bucket misconfiguration or a leaked credential exposes the whole dataset. This release lets the operator encrypt backup data as it is

 

How it works

Backups in the operator run on Percona XtraBackup. XtraBackup encrypts the stream with its xbcrypt component before uploading, so the data is ciphertext at rest in the bucket and stays that way until you restore it with the same key. You supply the key through a Kubernetes Secret and reference that Secret from the backup configuration. The operator never bakes the key into a manifest or an image.

 

Wiring it up

Point a storage target at an encryption-key Secret with encryptionKeySecret:

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: cluster1
spec:
  backup:
    storages:
      s3-us-west:
        type: s3
        encryptionKeySecret:
          key: encryptionKey
          name: my-s3-encryption-key-secret
        s3:
          bucket: S3-BACKUP-BUCKET-NAME-HERE
          credentialsSecret: cluster1-s3-credentials
          region: us-west-2

The same encryptionKeySecret field works under S3, GCS, and Azure storage targets, so a multi-cloud backup policy uses one consistent mechanism. You can also set an encryptionKeySecret at the backup level to apply one key across every storage target instead of repeating it per bucket. The referenced Secret holds the key under the encryptionKey data field.

 

Restoring an encrypted backup

Encryption is transparent on the way back in. When you restore, the operator reads the same Secret, hands the key to XtraBackup, and decrypts the stream before it prepares the data directory. The only hard requirement is that the key still exists: the restore fails fast if the Secret is missing or holds a different key than the one that produced the backup. Encryption also composes with backup compression, so you keep the smaller footprint and the ciphertext-at-rest guarantee at the same time.

Note: Keep the encryption key safe and versioned outside the cluster. A backup encrypted with a key you have lost is not recoverable. Treat the key with the same care as the backups themselves.

 

Automatic storage scaling

Running out of disk is one of the fastest ways to take a database down, and it rarely happens at a convenient hour. The operator has supported manual volume expansion since an earlier release: you raise resources.requests.storage, apply, and the operator grows the PersistentVolumeClaim for you. That still requires a human to notice the trend and act. Version 1.2.0 adds automatic scaling that watches usage and grows the volume on its own.

 

Why it matters

Storage growth is predictable in aggregate and unpredictable in timing. A batch import, a retention change, or an unexpected traffic spike can eat headroom faster than an on-call engineer can respond. Letting the operator resize before the volume fills turns a page someone at 3 a.m. incident into a log line, as long as your storage class supports online expansion.

 

How it works

You enable volume expansion, then define an autoscaling policy. The operator monitors each data PVC and, when usage crosses the threshold, grows the volume by a fixed step up to a ceiling you set. Because it builds on the Kubernetes volume-expansion API, the underlying storage class must have AllowVolumeExpansion: true.

 

Wiring it up

apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: cluster1
spec:
  enableVolumeExpansion: true
  storageScaling:
    enableVolumeScaling: true
    autoscaling:
      enabled: true
      growthStep: 2Gi
      maxSize: 10Gi
      triggerThresholdPercent: 80

 

triggerThresholdPercent is the fill level that triggers a resize (default 80, allowed range 50 to 95). growthStep is how much capacity each resize adds (default 2Gi), and maxSize caps total growth so a runaway workload cannot expand a volume without bound. The operator validates the relationship between these fields: autoscaling cannot be enabled unless enableVolumeScaling is on. For teams that prefer an external controller to own resizing, enableExternalAutoscaling hands that responsibility off instead.

The operator records each resize in the cluster status under storageAutoscaling, including the count of resizes and the timestamp of the last one. That gives you an audit trail and a signal worth alerting on: a volume that keeps hitting its growthStep is telling you the workload has changed, and a volume approaching maxSize is telling you to plan capacity before the ceiling stops the next resize.

Note: PVC expansion is one-way. Kubernetes can grow a volume but cannot shrink it, so set maxSize deliberately. Confirm your storage class allows expansion before you rely on this in production.

 

Other improvements

Beyond the three headline features, 1.2.0 ships a set of enhancements that smooth day-two operations:

  • Dedicated root user Secret (K8SPS-689): the operator publishes root connection details in a dedicated Secret, so applications and tooling read one predictable object instead of parsing several.
  • Disable NodePort allocation for LoadBalancer Services (K8SPS-496): set allocateLoadBalancerNodePorts: false to stop Kubernetes from opening NodePorts you never use behind a cloud load balancer.
  • Custom cluster naming for PMM (K8SPS-627): give a cluster a stable display name so multi-region and multi-namespace fleets stay legible in Percona Monitoring and Management.
  • Vault encryption Secret validation (K8SPS-487): the operator validates the Vault Secret and reports problems in status immediately, instead of failing later during an operation.
  • Concurrent reconciliation (K8SPS-434): tune how many clusters the operator reconciles at once through an environment variable, which helps a single operator manage a larger fleet.
  • Independent mysql-monit sidecar resources (K8SPS-742): set CPU and memory for the monitoring sidecar separately from the database container.
  • Orchestrator API authentication (K8SPS-19): the Orchestrator API now requires valid credentials.
  • Binlog storage configuration in restore objects (K8SPS-716): point a restore at the binlog storage it needs for point-in-time recovery.

For the full list, including bug fixes, see the release notes linked below.

 

Conclusion

Percona Operator for MySQL 1.2.0 extends the operator across the parts of the lifecycle that used to need custom glue: replication across sites, encryption of data that leaves the cluster, and storage that keeps up with growth. Platform teams running MySQL fleets on Kubernetes get declarative control over disaster recovery, a cleaner path through a security review, and one less 3 a.m. page. If there is a topology or a control you still have to script around, tell us on the forum, since that feedback is where releases like this one come from.

 

Try Percona Operator for MySQL 1.2.0

 

The post Percona Operator for MySQL 1.2.0: Cross-Site Replication, Encrypted Backups, and Automatic Storage Scaling appeared first on Percona.

Jun
30
2026
--

Community Docker Images: keeping the operator open without a vendor registry lock-in

PostgreSQL community images address a real gap in how a Kubernetes database operator earns your trust. Running a database operator on Kubernetes means trusting two things: the code, and the container images the operator pulls. The code is on GitHub, easy to inspect, easy to fork. The container images, the registry that hosts them, and the license that governs them all sit with the vendor, and any of those three can change without the source repository changing at all. Starting with Percona Operator for PostgreSQL 3.0.0, you can run the operator against community images you build yourself from the official PostgreSQL packages on download.postgresql.org, in a registry you control.

 

TL;DR

  • Community Docker Images: tech preview in PGO 3.0.0, official in 3.1.0. Point the operator at upstream-built PostgreSQL images instead of the Percona Distribution images.
  • Build them yourself from the official PostgreSQL source. The Dockerfiles pull packages from download.postgresql.org (the PGDG repositories), so the trust chain runs from PGDG to your registry with no vendor in the middle.
  • There are limits. Anything Percona-specific (TDE in our distribution build, for example) does not exist in an upstream-built image. That trade is intentional.

In this post:

  • How open source gets diluted in practice
  • Why distributions exist anyway, honestly
  • How Community Docker Images work
  • Limits of the upstream path
  • What to try, what to tell us

 

 

How open source gets diluted

Open source has changed in the last few years, and not always for the better. Companies have learned that you can keep a project’s source code fully open and still capture most of the lock-in by quietly closing the parts that matter in production: the release artifacts, the container images, the supported OS list, the certified Kubernetes distributions, the marketplace listings.

 

Same project, closed artifacts

You can have a fully community CNCF project that does not appear on the Red Hat Marketplace except as a paid Enterprise edition. Similarly, you can have a vendor that ships one packaging in the community and a richer one in Enterprise with the features you actually need in production. The license still says “open source.” The practical experience says “you depend on us.” And the source repository’s license is not the only license that matters here: a vendor can change the license, the trademark policy, or the distribution terms on the container images alone, while leaving the source repository untouched. That has happened in the PostgreSQL operator space recently, and the community noticed.

 

Why the community is right to be wary

Nobody outside the vendor can predict when a license will change, when a feature will move behind a paywall, or when an external contribution will get rejected because it competes with an Enterprise feature. Recent history has plenty of examples and the PostgreSQL community has been paying attention. When this community resists vendor-controlled distributions, it is not nostalgia. It is a rational read of where things have gone before.

I work on Percona’s PostgreSQL operator, so I see this conversation from the vendor side. The skepticism is fair. The honest question for us is what to do about it.
 

Why distributions exist anyway

Acknowledging the community’s concerns does not mean distributions are pointless. There are real reasons to ship one, and pretending otherwise makes for bad blog posts.
 

What a distribution buys you

A vendor-built distribution lets the vendor:

  1. Control the build process, dependencies, and defaults so they fit a specific user shape.
  2. Ship hotfixes faster, because the whole release path sits in one place.
  3. Fork PostgreSQL itself when something the upstream community will not accept, or can take years to accept, matters to customers, such as Transparent Data Encryption.
  4. For a Kubernetes operator, ship images with exactly the tools and extensions the operator supports, and skip everything else. The CVE surface stays smaller.
  5. Give QA and Service teams a predictable environment. “We support extensions A, B, C and not D, X, Z” is only honest if QA actually exercises A, B, C and the Service team can work with them in the production environment.
  6. Give customers one accountable party for the full release cycle, from hotfix through package availability. Some teams explicitly need that contract for compliance and audit reasons.
  7. And yes, less positive reasons that we covered above also apply, which is exactly the part the community keeps pointing at.

 

The trade-off you accept

If you run the vendor distribution, you accept that the vendor’s registry, image policy, and supported-extension matrix become part of your stack. If the vendor changes any of that, your operator deployment changes with it. That is not hypothetical for users who have lived through it on other products.

So the real question is whether you can keep the benefits a distribution provides for the users who want them, while leaving an honest, supported door open for users who do not. That is the door PGO 3.0.0 opens.

 

Community PostgreSQL Images in PGO 3.0.0

Starting with Percona Operator for PostgreSQL 3.0.0, the operator can run against images built from upstream PostgreSQL packages, not just the Percona Distribution images. This is what we are calling Community PostgreSQL Images. In 3.0.0, the feature ships as a tech preview. In 3.1.0, these images become part of our official release cycle and are fully documented.

One of the main advantages of Community Docker Images is that the community can request or contribute any extension that does not exist in the official Percona PostgreSQL distribution. TimescaleDB and Citus are the first examples: the community asked for them, and we shipped both in the Community Images set from day one.

 

How to use “Community PostgreSQL images”

The operator does not care where the image came from, as long as the image meets the operator’s runtime expectations 

A typical CR using a community image looks like this:

apiVersion: pgv2.percona.com/v2
kind: PerconaPGCluster
metadata:
  name: cluster1
spec:
  image: registry.example.com/postgresql-community:18
  postgresVersion: 18
  proxy:
    pgBouncer:
      image: registry.example.com/pgbouncer-community:1.23
  backups:
    pgbackrest:
      image: registry.example.com/pgbackrest-community:2.51
  # other spec fields unchanged from a normal CR

The fields that change are spec.image, spec.proxy.pgBouncer.image, and spec.backups.pgbackrest.image. You can build and publish all three images under your own registry, with your own tags if that helps you track versions. The operator drives the rest of the deployment the same way it always has: instances, backups, replication, monitoring, all of it.

 

What ships are in each image

Each Community Docker Image is a thin layer over the chosen base (UBI9 or UBI8) plus the packages the operator needs for that role. Where you see {N}, substitute the PostgreSQL major you build for (17, 18, and so on).

postgres image (e.g. postgres17):

Package Role
postgresql{N}-server PostgreSQL server
postgresql{N}-contrib contrib modules
pg_repack_{N} online table/index reorganization
pgaudit_{N} audit logging
set_user_{N} privilege escalation control
pgvector_{N} vector similarity search
wal2json_{N} WAL to JSON logical decoding
pg_cron_{N} in-database cron scheduler
pgbackrest&lt;/code> backup/restore tool
patroni HA cluster manager
timescaledb-2-postgresql-{N} time-series extension (x86_64 only; EL9 only for PG18)
citus_{N} distributed PostgreSQL (PG16+ only)

pgbackrest image:

Package Role
pgbackrest backup/restore tool only

pgbouncer image:

Package Role
pgbouncer connection pooler only

 

The split is intentional. The postgres image ships the full operator-aware runtime. The backup and proxy images stay minimal. As a result, the operator’s components are in separate failure domains and shrink the attack surface of each container.

 

Limits worth being honest about

A community image is not a Percona Distribution image. Two practical consequences:

  • Distribution-only features will not work. Transparent Data Encryption, for example, lives in the Percona Distribution build. A community image built from upstream PostgreSQL does not include it. If you depend on TDE, run the distribution image.
  • Support boundaries are different. Percona Support is responsible for the Percona Distribution images and the operator code. A community image you built yourself

Ultimately, these are the right trade-offs. The point of community images is to give you transparency and control. Taking care of your own image is part of that deal. At the same time, we publish all three images under perconalab/percona-postgresql-operator on Docker Hub so you can evaluate the tech preview without standing up your own build pipeline first. perconalab is Percona’s non-production namespace, so use those images for testing. For production, build and sign your own.

UBI9 (EL9):

docker.io/perconalab/percona-postgresql-operator:main-postgres14-community
docker.io/perconalab/percona-postgresql-operator:main-postgres15-community
docker.io/perconalab/percona-postgresql-operator:main-postgres16-community
docker.io/perconalab/percona-postgresql-operator:main-postgres17-community
docker.io/perconalab/percona-postgresql-operator:main-postgres18-community
docker.io/perconalab/percona-postgresql-operator:main-pgbackrest-community
docker.io/perconalab/percona-postgresql-operator:main-pgbouncer-community
docker.io/perconalab/percona-postgresql-operator:main-upgrade-community

UBI8 (EL8):

docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres14-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres15-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres16-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres17-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres18-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-upgrade-community

 

How to build the images

The Dockerfile, the package list, and a sample CI job ship in percona-docker/postgresql-containers/community. The build is a regular make target on top of docker buildx, so you can run it on any multi-platform builder.

# Prerequisites: docker buildx with a multi-platform builder
docker buildx create --use --name multiarch

# Build and push all PostgreSQL community images (UBI9 / EL9)
git clone https://github.com/percona/percona-docker
cd percona-docker/postgresql-containers/community
make all TAG=1.0.0 REGISTRY=myrepo/percona-postgresql-operator

# Or a single image
make postgres17 TAG=1.0.0 REGISTRY=myrepo/percona-postgresql-operator

# UBI8 / EL8 variants
make all-ubi8 TAG=1.0.0-ubi8 REGISTRY=myrepo/percona-postgresql-operator

make all builds all three images (postgres, pgBouncer, pgBackRest) so they stay version-aligned. Override REGISTRY and TAG to point at your own namespace and tagging scheme. Once the images are in your registry, plug them into the CR fields shown earlier, and the operator picks them up.

Full build documentation: percona-docker/postgresql-containers/community/README.md.

 

How to contribute

Community images live in percona/percona-docker, and the build is driven by a transform.py generator that produces the Dockerfiles under build/. The files under build/ are regenerated on every sync, so contributions go through the generator, never through the generated files.

Full contribution guide: community/CONTRIBUTING.md.

 

How to provide feedback

Two channels, depending on the shape of the feedback:

  • GitHub issue on percona/percona-postgresql-operator with the community-images label. Use this for bug reports, missing extensions, build problems, and concrete requests. The label keeps all community-image reports in one filter the team watches.

 

What’s next

The first step was taking full engineering ownership of Percona Operator for PostgreSQL as an independent project, so the roadmap, the release cadence, and the governance live with one team that the community can talk to directly. Community PostgreSQL Images are the next step in that same commitment. If the community adopts this path, we have ideas for what to invest in next.

We will let the community tell us. If this is useful, we keep investing here. We are ready to add more features to the operator around Community Images. Conversely, if nobody adopts it, that is also a signal, and an honest one.

Try the tech preview in 3.0.0. Open an issue if the build flow is rougher than it should be. Tell us what you want next on the forum or directly on GitHub.

 

Try It Out

The post Community Docker Images: keeping the operator open without a vendor registry lock-in appeared first on Percona.

Jun
16
2026
--

Extending pt-archiver with a Partition-Aware Plug-in for Fast Retention Policy Enforcement

Managing data retention policies is one of the most common operational tasks in MySQL.

Applications continuously generate transactional, audit, logging, telemetry, and event data. Over time, these tables can grow to billions of rows, causing:

  • Larger backups
  • Longer recovery times
  • Reduced buffer pool efficiency
  • Slower index maintenance
  • Increased storage costs
  • Degraded query performance

To address these problems, organizations typically implement retention policies based on dates or timestamps. Examples include deleting events older than 90 days or purging session data older than 30 days and so forth. The deleted data can then eventually be archived somewhere else, like in another DBMS or on external files.

One of the most widely used tools for implementing these policies in MySQL ecosystems is pt-archiver, part of the Percona Toolkit.

This article provides a review of what pt-archiver is and how to use it, but in particular it focuses on the fact this tool is not partitioning aware, and this can make the deletion phase more costly. The article shows how to extend pt-archiver with a Perl plugin to make it aware of partitioning.

 

What is pt-archiver?

pt-archiver is a command-line utility from Percona Toolkit designed to:

  • Archive rows from MySQL tables
  • Purge rows from MySQL tables
  • Move data between tables into the local database or a remote one
  • Export rows into files

In a few words: implementing retention policies safely.

The tool processes rows incrementally in chunks, avoiding massive transactions and reducing impact on production systems.

Example:

pt-archiver \
  --source h=localhost,D=mydb,t=events \
  --where "created_at &lt; '2026-05-01'" \
  --purge \
  --limit 1000 \
  --commit-each

This command:

  • Scans rows matching the WHERE condition
  • Processes them in chunks of 1000 rows
  • Commits every chunk
  • Deletes matching rows from the source table

pt-archiver provides several advantages compared to ad-hoc DELETE statements.

Instead of running:

DELETE FROM events
WHERE created_at &lt; '2026-05-01';

which may:

  • Lock rows for a long time
  • Generate massive undo/redo logs
  • Create replication lag
  • Exhaust transaction logs

pt-archiver processes rows incrementally to make the process overhead less impactful for the database performance.

pt-archiver implementation permits flexible archival strategies

Rows can be copied to another table on a remote host, exported to files or removed completely

More details: ps://docs.percona.com/percona-toolkit/pt-archiver.html

Example: Copy rows to a remote archive table

The following example archives rows older than 90 days from a local table into an archive table hosted on a remote MySQL server:

pt-archiver \
  --source h=localhost,D=sales,t=orders,u=archiver,p=secret \
  --dest h=archive-server,D=archive,t=orders_archive,u=archiver,p=secret \
  --where "created_at &lt; '2026-05-01'" \
  --limit 1000 \
  --commit-each \
  --progress 10000 \
  --statistics

In this example:

  • –source defines the source table
  • –dest defines the remote archive destination
  • –where selects rows eligible for archival
  • –limit controls batch size
  • –commit-each commits every batch independently to reduce transaction overhead

-progress reports progress every 10,000 rows

If rows should be removed from the source table after being copied, add –purge

Example: Export rows to a file

The following example exports rows older than one year into a text file:

pt-archiver \
  --source h=localhost,D=sales,t=orders,u=archiver,p=secret \
  --where "created_at &lt; NOW() - INTERVAL 1 YEAR" \
  --file '/tmp/orders_archive_%Y-%m-%d.txt' \
  --output-format csv \
  --limit 1000 \
  --commit-each \
  --progress 10000 \
  --statistics

In this example:

  • –file specifies the output file
  • -output-format csv exports rows in CSV format
  • Date placeholders in the filename are expanded automatically

Rows can optionally be deleted from the source table by adding –purge

This allows pt-archiver to be used both for data retention and for offline archival workflows.

The Hidden Cost of DELETE Statements

Although pt-archiver is much safer than massive DELETE operations, it still fundamentally relies on DELETE statements.

This is a critical point.

Even when there are proper indexes, the rows are processed in chunks, and transactions are small; the large-scale DELETE operations remain expensive.

Deleting rows is expensive in InnoDB because it involves:

  • Locating rows via indexes
  • Modifying clustered indexes
  • Modifying secondary indexes
  • Generating undo logs
  • Generating redo logs
  • Purge thread processing
  • Replication event generation
  • Page fragmentation

When deleting billions of rows, the overhead becomes enormous.

Indexes help for sure, but only partially.

Consider:

DELETE FROM events
WHERE created_at &lt; '2024-01-01';

If created_at is indexed, MySQL can efficiently locate rows.

However, locating rows efficiently is only part of the cost. The actual delete operations still require all those things we mentioned above.

At considerable scale, this becomes expensive.

Why RANGE Partitioning is Superior for Retention Policies

For time-based retention policies, partitioning is often dramatically more efficient. In particular, RANGE partitioning is very useful for these cases.

Example:

CREATE TABLE events (
    id BIGINT NOT NULL,
    created_at DATETIME NOT NULL,
    payload JSON,
    PRIMARY KEY(id, created_at)
)

PARTITION BY RANGE (TO_DAYS(created_at)) (
    PARTITION p202604 VALUES LESS THAN (TO_DAYS('2026-05-01')),
    PARTITION p202605 VALUES LESS THAN (TO_DAYS('2026-06-01')),
    PARTITION p202606 VALUES LESS THAN (TO_DAYS('2026-07-01'))
);

With partitioning, dropping old data becomes:

ALTER TABLE events DROP PARTITION p202604;

This operation is dramatically faster than running a DELETE.

Dropping a partition:

  • Removes an entire physical partition
  • Avoids row-by-row DELETE
  • Avoids undo generation for each row
  • Avoids secondary index maintenance per row
  • Minimizes redo generation
  • Is nearly metadata-only

This can remove millions or billions of rows in a matter of seconds without the same large cost of DELETE.

The Problem: pt-archiver is Not Partition-Aware

Unfortunately, pt-archiver does not automatically understand partitioning strategies.

Even if the table is partitioned or the retention policy perfectly matches partition boundaries, pt-archiver still executes DELETE statements.

Example:

pt-archiver \
  --where "created_at &lt; NOW() - INTERVAL 90 DAY" \
  --purge

Internally, this still produces DELETE … instead of ALTER TABLE … DROP PARTITION …

This means organizations may lose the major operational benefits of partitioning, or they need to implement custom scripts for managing the selection of rows to copy using pt-archiver and then use DROP PARTITION separately from the tool. That is doable, and to be honest, not too complicated, but why not make pt-archiver aware of partitioning for some specific use cases?

Extending pt-archiver with Pulg-ins

Fortunately, pt-archiver supports Perl plug-ins.

A plug-in can do plenty of things. Like: inspect runtime conditions, interact with MySQL, override behaviors, and execute custom logic

This gives us an opportunity to implement partition-aware retention handling.

The plug-in can:

  1. Inspect partition definitions
  2. Analyze the WHERE condition
  3. Determine which partitions are fully expired
  4. Execute ALTER TABLE DROP PARTITION
  5. Prevent row-by-row DELETE processing

This approach combines the scheduling/orchestration power of pt-archiver with the efficiency of partition pruning.

Plug-in Design

Our plug-in will:

  • Connect using the pt-archiver DB handle
  • Inspect INFORMATION_SCHEMA.PARTITIONS
  • Identify partitions older than the retention cutoff
  • Issue DROP PARTITION statements
  • Log actions
  • Skip DELETE processing

Assumptions:

  • The table is RANGE partitioned
  • Partitions are DATETIME based using the TO_DAYS() function to define ranges
  • Partition naming convention contains dates
  • Retention policy aligns with partition boundaries; if the plugin cannot determine a specific boundary, pt-archiver does nothing

Full Perl Plug-in for pt-archiver

package pt_archiver_partition_drop;

use strict;
use warnings;

sub new {
    my ($class, %args) = @_;
    my $self = {
        dbh        =&gt; $args{dbh},
        db         =&gt; $args{db},
        tbl        =&gt; $args{tbl},
        statistics =&gt; {},
    };

    bless $self, $class;
    return $self;
}

sub statistics {
    my ($self) = @_;
    return $self-&gt;{statistics};
}


sub before_begin {
    my ($self) = @_;
    my $dbh = $self-&gt;{dbh} or die "Missing dbh from pt-archiver\n";
    my $db  = $self-&gt;{db}  or die "Missing db from pt-archiver plugin args\n";
    my $tbl = $self-&gt;{tbl} or die "Missing tbl from pt-archiver plugin args\n";
    my $where  = _get_cmdline_option('where');
    my $dryrun = $ENV{PT_PARTITION_DROP_DRY_RUN} ? 1 : 0;

    die "Missing --where from original command line\n" unless $where;

    print "PLUGIN before_begin called\n";
    print "DB=$db TABLE=$tbl\n";
    print "WHERE=$where\n";
    print "PLUGIN_DRY_RUN=$dryrun\n";

    my ($column, $cutoff_date) = _parse_where($where);

    my $partitions = _get_partitions($dbh, $db, $tbl);

    if (!@$partitions) {
        print "Table `$db`.`$tbl` is not partitioned. Refusing DELETE.\n";
        exit(0);
    }

    my $partition_expr = $partitions-&gt;[0]-&gt;{expression};
    die "Missing PARTITION_EXPRESSION\n"
        unless defined $partition_expr &amp;&amp; length $partition_expr;

    print "Partition expression: $partition_expr\n";

    my $cutoff_value = _evaluate_cutoff(
        $dbh,
        $partition_expr,
        $column,
        $cutoff_date,
    );

    print "Cutoff date: $cutoff_date\n";
    print "Cutoff boundary value: $cutoff_value\n";

    my $matched;

    for my $p (@$partitions) {
        next if !defined $p-&gt;{description};
        next if uc($p-&gt;{description}) eq 'MAXVALUE';

        if ($p-&gt;{description} == $cutoff_value) {
            $matched = $p;
            last;
        }
    }


    if (!$matched) {
        print "No exact partition boundary matches cutoff $cutoff_value. Refusing DELETE.\n";
        exit(0);
    }

    print "Matched boundary partition: $matched-&gt;{name}, position $matched-&gt;{position}\n";

    my @drop;

    for my $p (@$partitions) {
        next if !defined $p-&gt;{description};
        next if uc($p-&gt;{description}) eq 'MAXVALUE';

        if ($p-&gt;{position} &lt;= $matched-&gt;{position}) {
            push @drop, $p-&gt;{name};
            print "Eligible for DROP: $p-&gt;{name}, boundary $p-&gt;{description}\n";
        }
    }

    if (!@drop) {
        print "No partitions eligible for DROP. Refusing DELETE.\n";
        exit(0);
    }

    my $sql = sprintf(
        "ALTER TABLE %s.%s DROP PARTITION %s",
        _quote_ident($db),
        _quote_ident($tbl),
        join(", ", map { _quote_ident($_) } @drop),
    );

    print "SQL: $sql\n";

    if ($dryrun) {
        print "PT_PARTITION_DROP_DRY_RUN enabled. Not executing DROP PARTITION.\n";
    }
    else {
        $dbh-&gt;do($sql);
        print "Dropped partitions: " . join(", ", @drop) . "\n";
    }

    $self-&gt;{statistics}-&gt;{partitions_dropped} = scalar @drop;

    exit(0);
}


sub _parse_where {
    my ($where) = @_;

    $where =~ s/^\s+|\s+$//g;

    die "Only WHERE format supported: created_at &lt; 'YYYY-MM-DD'\n"
        unless $where =~ /^`?([A-Za-z0-9_]+)`?\s*&lt;\s*'(\d{4}-\d{2}-\d{2})'\s*$/;

    return ($1, $2);
}

sub _evaluate_cutoff {
    my ($dbh, $partition_expr, $column, $cutoff_date) = @_;

    my $expr = $partition_expr;
    $expr =~ s/`//g;

    die "Partition expression does not reference column `$column`: $partition_expr\n"
        unless $expr =~ /\b\Q$column\E\b/i;

    $expr =~ s/\b\Q$column\E\b/'$cutoff_date'/ig;

    die "Unsafe generated expression: $expr\n"
        unless $expr =~ /^[A-Za-z0-9_\s\(\)\+\-\*\/,\.'":]+$/;

    my $sql = "SELECT $expr";

    print "Boundary evaluation SQL: $sql\n";

    my ($value) = $dbh-&gt;selectrow_array($sql);

    die "Cannot evaluate cutoff expression: $sql\n"
        unless defined $value;

    return $value;
}

sub _get_partitions {
    my ($dbh, $db, $tbl) = @_;

    my $sql = q{
        SELECT
            PARTITION_NAME,
            PARTITION_DESCRIPTION,
            PARTITION_EXPRESSION,
            PARTITION_ORDINAL_POSITION
        FROM INFORMATION_SCHEMA.PARTITIONS
        WHERE TABLE_SCHEMA = ?
          AND TABLE_NAME = ?
          AND PARTITION_NAME IS NOT NULL
        ORDER BY PARTITION_ORDINAL_POSITION
    };

    my $sth = $dbh-&gt;prepare($sql);
    $sth-&gt;execute($db, $tbl);
    my @partitions;

    while (my $row = $sth-&gt;fetchrow_hashref()) {
        push @partitions, {
            name        =&gt; $row-&gt;{PARTITION_NAME},
            description =&gt; $row-&gt;{PARTITION_DESCRIPTION},
            expression  =&gt; $row-&gt;{PARTITION_EXPRESSION},
            position    =&gt; $row-&gt;{PARTITION_ORDINAL_POSITION},
        };
    }

    return \@partitions;
}


sub _get_cmdline_option {

    my ($name) = @_;

    my $opt = "--$name";

    for (my $i = 0; $i &lt; @ARGV; $i++) {
        if ($ARGV[$i] eq $opt &amp;&amp; defined $ARGV[$i + 1]) {
            return $ARGV[$i + 1];
        }

        if ($ARGV[$i] =~ /^\Q$opt\E=(.*)$/) {
            return $1;
        }
    }

    if (open my $fh, '&lt;', "/proc/$$/cmdline") {
        local $/;
        my $raw = &lt;$fh&gt;;
        close $fh;

        my @cmd = split /\0/, $raw;

        for (my $i = 0; $i &lt; @cmd; $i++) {
            if ($cmd[$i] eq $opt &amp;&amp; defined $cmd[$i + 1]) {
                return $cmd[$i + 1];
            }

            if ($cmd[$i] =~ /^\Q$opt\E=(.*)$/) {
                return $1;
            }
        }
    }

    return undef;
}



sub _quote_ident {

    my ($ident) = @_;

    die "Invalid identifier: $ident\n"
        unless defined $ident &amp;&amp; $ident =~ /^[A-Za-z0-9_]+$/;

    return "`$ident`";
}

1;

Create the file named  pt_archiver_partition_drop.pm into the /usr/local/share/perl5 path.

Also set the environment variable PERL5LIB to let pt-archiver where to find the Perl package

export PERL5LIB=/usr/local/share/perl5

Example Usage

First, create the partitioned table events and insert some fake data.

DROP TABLE IF EXISTS events;


CREATE TABLE events (
  id BIGINT NOT NULL,
  created_at DATETIME NOT NULL,
  payload JSON DEFAULT NULL,
  PRIMARY KEY (id, created_at)
)
PARTITION BY RANGE (TO_DAYS(created_at)) (
  PARTITION p202604 VALUES LESS THAN (TO_DAYS('2026-05-01')),
  PARTITION p202605 VALUES LESS THAN (TO_DAYS('2026-06-01')),
  PARTITION p202606 VALUES LESS THAN (TO_DAYS('2026-07-01')),
  PARTITION pmax VALUES LESS THAN MAXVALUE
);

INSERT INTO events (id, created_at, payload) VALUES

-- p202604
(1,  '2026-04-01 08:00:00', JSON_OBJECT('event', 'login',    'user', 'alice')),
(2,  '2026-04-03 09:15:00', JSON_OBJECT('event', 'view',     'page', 'home')),
(3,  '2026-04-05 10:30:00', JSON_OBJECT('event', 'click',    'button', 'signup')),
(4,  '2026-04-08 11:45:00', JSON_OBJECT('event', 'search',   'term', 'mysql')),
(5,  '2026-04-10 12:00:00', JSON_OBJECT('event', 'purchase', 'amount', 100)),
(6,  '2026-04-14 13:20:00', JSON_OBJECT('event', 'logout',   'user', 'alice')),
(7,  '2026-04-18 14:35:00', JSON_OBJECT('event', 'download', 'file', 'report.pdf')),
(8,  '2026-04-22 15:50:00', JSON_OBJECT('event', 'upload',   'file', 'image.png')),
(9,  '2026-04-26 16:05:00', JSON_OBJECT('event', 'click',    'button', 'buy')),
(10, '2026-04-30 23:59:59', JSON_OBJECT('event', 'month_end')),

-- p202605

(11, '2026-05-01 00:00:00', JSON_OBJECT('event', 'login',    'user', 'bob')),
(12, '2026-05-03 08:10:00', JSON_OBJECT('event', 'view',     'page', 'pricing')),
(13, '2026-05-06 09:20:00', JSON_OBJECT('event', 'search',   'term', 'percona')),
(14, '2026-05-09 10:30:00', JSON_OBJECT('event', 'purchase', 'amount', 250)),
(15, '2026-05-12 11:40:00', JSON_OBJECT('event', 'logout',   'user', 'bob')),
(16, '2026-05-16 12:50:00', JSON_OBJECT('event', 'download', 'file', 'backup.sql')),
(17, '2026-05-20 13:00:00', JSON_OBJECT('event', 'upload',   'file', 'data.csv')),
(18, '2026-05-24 14:10:00', JSON_OBJECT('event', 'click',    'button', 'subscribe')),
(19, '2026-05-28 15:20:00', JSON_OBJECT('event', 'view',     'page', 'docs')),
(20, '2026-05-31 23:59:59', JSON_OBJECT('event', 'month_end')),

-- p202606

(21, '2026-06-01 00:00:00', JSON_OBJECT('event', 'login',    'user', 'carol')),
(22, '2026-06-03 08:05:00', JSON_OBJECT('event', 'search',   'term', 'partitioning')),
(23, '2026-06-06 09:15:00', JSON_OBJECT('event', 'view',     'page', 'dashboard')),
(24, '2026-06-09 10:25:00', JSON_OBJECT('event', 'purchase', 'amount', 500)),
(25, '2026-06-12 11:35:00', JSON_OBJECT('event', 'logout',   'user', 'carol')),
(26, '2026-06-16 12:45:00', JSON_OBJECT('event', 'login',    'user', 'dave')),
(27, '2026-06-20 13:55:00', JSON_OBJECT('event', 'download', 'file', 'archive.zip')),
(28, '2026-06-24 14:05:00', JSON_OBJECT('event', 'upload',   'file', 'video.mp4')),
(29, '2026-06-28 15:15:00', JSON_OBJECT('event', 'click',    'button', 'checkout')),
(30, '2026-06-30 23:59:59', JSON_OBJECT('event', 'month_end')),

-- pmax
(31, '2026-07-01 00:00:00', JSON_OBJECT('event', 'login',    'user', 'eve')),
(32, '2026-07-05 08:30:00', JSON_OBJECT('event', 'view',     'page', 'future')),
(33, '2026-07-10 09:45:00', JSON_OBJECT('event', 'search',   'term', 'maxvalue')),
(34, '2026-08-01 10:00:00', JSON_OBJECT('event', 'purchase', 'amount', 750)),
(35, '2026-09-01 11:15:00', JSON_OBJECT('event', 'retained_future'));

 

Now you can run the following command to delete all rows before the 1st of May, which, by the way, matches the entire first partition in the table.

pt-archiver \
  --source h=localhost,D=mydb,t=events,m=pt_archiver_partition_drop \
  --where "created_at &lt; '2026-05-01'" \
  --purge

 

Notice the Perl plugin must be indicated with the m option in the DSN string.

In practice:

  • pt-archiver initializes
  • The plug-in runs
  • Partitions are dropped
  • No DELETE statements are executed

Here is what you get from the execution of the above command:

PLUGIN before_begin called
DB=mydb TABLE=events
WHERE=created_at &lt; '2026-05-01'
PLUGIN_DRY_RUN=0
Partition expression: to_days(`created_at`)
Boundary evaluation SQL: SELECT to_days('2026-05-01')
Cutoff date: 2026-05-01
Cutoff boundary value: 740102
Matched boundary partition: p202604, position 1
Eligible for DROP: p202604, boundary 740102
SQL: ALTER TABLE `mydb`.`events` DROP PARTITION `p202604`
Dropped partitions: p202604

You can simply verify the table has been managed correctly:

SELECT * FROM mydb.events;

SHOW CREATE TABLE mydb.events;

 

Now TRUNCATE the table and recreate the data and try now to specify the where conditions that match a RANGE that is not the first in the list of the boundaries.

pt-archiver \
  --source h=localhost,D=mydb,t=events,m=pt_archiver_partition_drop \
  --where "created_at &lt; '2026-06-01'" \
  --purge

You should get:

PLUGIN before_begin called
DB=mydb TABLE=events
WHERE=created_at &lt; '2026-06-01'
PLUGIN_DRY_RUN=0
Partition expression: to_days(`created_at`)
Boundary evaluation SQL: SELECT to_days('2026-06-01')
Cutoff date: 2026-06-01
Cutoff boundary value: 740133
Matched boundary partition: p202605, position 2
Eligible for DROP: p202604, boundary 740102
Eligible for DROP: p202605, boundary 740133
SQL: ALTER TABLE `mydb`.`events` DROP PARTITION `p202604`, `p202605`
Dropped partitions: p202604, p202605

In this case, two partitions have been identified and dropped.

 

Truncate the table and recreate the data again. Try now to provide a WHERE condition that does not match any of the boundaries in the RANGE.

pt-archiver \
  --source h=localhost,D=mydb,t=events,m=pt_archiver_partition_drop \
  --where "created_at &lt; '2026-04-25'" \
  --purge

 

You get the following:

PLUGIN before_begin called
DB=mydb TABLE=events
WHERE=created_at &lt; '2026-04-25'
PLUGIN_DRY_RUN=0
Partition expression: to_days(`created_at`)
Boundary evaluation SQL: SELECT to_days('2026-04-25')
Cutoff date: 2026-04-25
Cutoff boundary value: 740096
No exact partition boundary matches cutoff 740096. Refusing DELETE.

As expected, the tool now refuses to execute anything if it doesn’t find an exact match.

 

Operational Benefits

This approach provides major advantages.

Dropping partitions is vastly faster than deleting rows, and minimal binary logging is needed, compared to billions of row deletes. There is no massive transactional overhead for managing undo logs and purging. You get then a better InnoDB Buffer Pool stability because of less page churn.

In the end, retention jobs are completed quickly and consistently in a predictable way and at the minimal cost.

 

Important Caveats

Partition Boundaries Must Match Retention Policy

If partitions contain mixed retention windows, DROP PARTITION may remove too much data. For this reason, ensure correct partition design.

Recommended:

  • daily partitions
  • weekly partitions
  • monthly partitions

aligned with business retention requirements.

Metadata Locks

ALTER TABLE DROP PARTITION still acquires metadata locks.

Test carefully in production.

Backup Awareness

Ensure dropped partitions are no longer needed before removal or use pt-archiver to also copy the data into a remote server or dump the data into a CSV file before running the DROP PARTITION.

 

Possible Enhancements

The plug-in can be extended further.

Potential improvements:

  • Support for daily partitions
  • Support for UNIX timestamp partitions
  • Dry-run reporting
  • Automatic partition creation
  • Push Slack notifications
  • Export Prometheus metrics
  • Safety checks for replicas
  • GTID-aware orchestration
  • Integration with pt-online-schema-change workflows

These are just some ideas I had meanwhile doing my tests. What you can do by implementing a Perl plugin is only limited by your imagination and your real needs.

Conclusion

pt-archiver remains an excellent tool for implementing retention policies and archival workflows.

However, DELETE-based purging becomes increasingly expensive at scale, even with proper indexing and chunked processing.

For large time-series or historical datasets, RANGE partitioning is often a dramatically superior strategy.

The challenge is that pt-archiver does not natively leverage partition-level operations.

Fortunately, its Perl plug-in architecture allows advanced users to extend its behavior and implement partition-aware cleanup logic.

By combining:

  • pt-archiver orchestration
  • MySQL RANGE partitioning
  • Custom Perl plug-ins

Organizations can achieve:

  • Faster retention enforcement
  • Lower operational overhead
  • Smaller replication impact
  • Dramatically improved scalability

For large MySQL deployments, this hybrid approach can turn multi-hour purge operations into near-instant metadata operations.

The use case presented in this article is limited to a specific scenario, but you can reuse it or customize it if you have a different kind of RANGE partitioning, for example, not using TO_DAYS().

Take this as just an example of how you can extend pt-archiver. What you can do for real is driven by your needs and/or only limited by your imagination.

More info about extending pt-archiver:
https://docs.percona.com/percona-toolkit/pt-archiver.html#extending

 

The post Extending pt-archiver with a Partition-Aware Plug-in for Fast Retention Policy Enforcement appeared first on Percona.

Jun
09
2026
--

Percona Operator for MySQL (PXC) 1.20.0: Automatic Storage Resizing, TLS Certificate Rotation, and ARM64 Support

Percona Operator for MySQL PXC 1.20.0 is out today, and it addresses three long-requested operational headaches: storage that grows on its own before it fills up, TLS certificates that rotate without cluster downtime, and images that run natively on ARM64.

Disk-full incidents on PXC clusters often arrive at 2 AM when monitoring alerts fire, and someone has to manually expand PVCs before writes grind to a halt. Certificate rotations have traditionally meant a carefully timed series of kubectl edits with real downtime risk. And ARM64 hardware has been increasingly common in dev clusters and cost-optimized cloud node pools, where x86-only images created extra friction. 1.20.0 addresses all three in a single release.

The operator is open source and runs on any CNCF-conformant Kubernetes distribution, including GKE, EKS, AKS, and OpenShift. It supports Kubernetes 1.33 through 1.36 and PXC 8.4, 8.0, and 5.7.

 

In this post, you’ll learn about:

  • Automatic PVC storage resizing with configurable thresholds and a hard cap
  • Zero-downtime TLS certificate rotation via a new Secret naming convention
  • Native ARM64 support across all operator images
  • PITR validation that catches misconfigured targets before restores begin
  • Configurable leader election for high-latency or unstable networks
  • Other improvements in this release

 

Automatic Storage Resizing

 

Why it matters

A full data volume is the most common cause of unplanned maintenance on a PXC cluster. Until now, avoiding it required external monitoring, manual kubectl patch pvc steps, and waiting for the storage class to honor the resize. Even with good alerting, the operator itself had no mechanism to react: it could only expand PVCs when you changed the spec by hand.

1.20.0 introduces built-in storage autoscaling. The operator polls each PVC’s actual disk usage, and when usage crosses a configured threshold, it automatically expands the claim. You set the trigger percentage, the step size per resize event, and an optional upper bound. The operator handles everything else.

 

How it works

The autoscaler runs inside the normal reconcile loop. It reads status.capacity.storage from each PXC PVC, compares current usage against triggerThresholdPercent, and issues a PVC resize when the threshold is crossed. It sets a percona.com/pvc-resize-in-progress annotation on the CR while an expansion is active. This annotation blocks concurrent rolling restarts or upgrades from starting, so nothing disrupts the cluster mid-resize.

You can also set enableExternalAutoscaling: true if an external tool, such as KEDA, already manages PVC sizes for your cluster. When you enable external autoscaling, the built-in loop skips its resize check entirely to avoid conflicts.

 

Wiring it up

Add storageScaling to your PerconaXtraDBCluster spec:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: cluster1
spec:
  crVersion: 1.20.0
  storageScaling:
    enableVolumeScaling: true
    autoscaling:
      enabled: true
      triggerThresholdPercent: 80   # resize when a PVC is 80% full
      growthStep: 2Gi               # add 2Gi per resize event
      maxSize: 100Gi                # never grow beyond 100Gi per PVC
#     enableExternalAutoscaling: false

Any PVC expansion requires enableVolumeScaling: true, whether the autoscaler or a manual spec change triggers it. Setting autoscaling.enabled: true enables the threshold-based path on top of that. Leave the autoscaling block out if you only want to permit manual spec-driven resizes.

 

Caveats

Storage expansion requires a StorageClass with allowVolumeExpansion: true. Check before enabling:

kubectl get storageclass \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.allowVolumeExpansion}{"\n"}{end}'

Autoscaling applies only to PXC data volumes. If your storage class or CSI driver handles expansion externally, use enableExternalAutoscaling: true to prevent the two mechanisms from racing.

 

Automated TLS Certificate Rotation

Why it matters

Rotating TLS certificates on a live PXC cluster has always carried risk. The Galera protocol requires all nodes to trust each other’s CA simultaneously. Swap the CA on one node before the others accept it, and inter-node communication breaks. The safe approach requires a three-phase CA swap with rolling restarts between each phase: a process that is easy to get wrong under time pressure.

1.20.0 formalizes this into a first-class operator workflow. Create a Secret named <ssl-secret>-new containing the replacement credentials, and the operator runs the full three-phase rotation automatically, pausing for rolling restarts between each step.

 

How it works

The rotation proceeds in three steps that the operator coordinates:

  1. Combined CA phase. The old CA and new CA are merged into a single ca.crt and pushed to all nodes. Every node now trusts both roots.
  2. New leaf phase. The new tls.crt and tls.key are pushed node by node with a rolling restart. New leaf certs are signed by the new CA, and the combined CA means all nodes trust them.
  3. New CA only phase. The combined ca.crt is replaced with the new CA only. The old root is removed. Another rolling restart completes the rotation.

When step 3 completes, the operator automatically deletes the -new Secret. The cluster never loses TLS connectivity between nodes during the process.

 

Wiring it up

Given a cluster named cluster1 using the default SSL Secret cluster1-ssl, create the replacement:

kubectl create secret generic cluster1-ssl-new \
  --from-file=ca.crt=new-ca.crt \
  --from-file=tls.crt=new-server.crt \
  --from-file=tls.key=new-server.key

You do not need to change the PerconaXtraDBCluster CR. The operator detects the -new Secret on the next reconcile and starts the rotation. No kubectl patch on the CR, no operator restart.

 

Caveats

The operator does not yet surface rotation progress in .status.conditions. Monitor the rotation by watching PXC pods restart in sequence and checking that the -new Secret is eventually gone:

kubectl get pods -w -l app.kubernetes.io/component=pxc
kubectl get secret cluster1-ssl-new  # should 404 when rotation is complete

 

ARM64 Support

 

Why it matters

AWS Graviton3, Google Axion, and Azure Cobalt100 instances deliver better price-to-performance on memory-intensive workloads like PXC. Previously, running the operator on ARM64 nodes required cross-architecture scheduling workarounds or explicit node exclusions for operator pods. All PXC operator images now publish native linux/arm64 layers alongside nodeSelector

 

What is covered

Every image in the PXC operator stack ships multi-arch manifests in 1.20.0:

  • The operator manager image
  • The PXC xtrabackup sidecar
  • The log collector (Fluentbit-based)
  • The init container

This release also fixes a logrotate crash on ARM64 (K8SPXC-1821) that a missing dependency in the ARM64 container layer caused. 1.20.0 ships the fix.
 

Wiring it up

You do not need any configuration change. Pull the 1.20.0 operator image and Kubernetes schedules it on whichever architecture is available. To pin PXC pods explicitly to ARM64 nodes, add a nodeSelector or node affinity in the spec.pxc block:

spec:
  pxc:
    nodeSelector:
      kubernetes.io/arch: arm64

 

Other Improvements

  • PITR target validation before restore begins (K8SPXC-1318, K8SPXC-1634, K8SPXC-1635, K8SPXC-1793): The operator now validates PITR targets (type, GTID, timestamp) against available binary logs before starting a restore. It catches a misconfigured target before it pauses the cluster, rather than after.
  • Configurable leader election (K8SPXC-1805): Three new environment variables tune leader election timing for high-latency or flaky network environments.
  • SST retry limit (K8SPXC-1619): A new spec.pxc.sstRetryCount field caps the number of State Snapshot Transfer retry attempts, preventing a node that repeatedly fails SST from looping indefinitely.
  • Custom logrotate configuration (K8SPXC-1789): Supply a custom logrotate config via a ConfigMap reference in spec.logcollector.logRotate for fine-grained control over log rotation for PXC and utility containers.
  • Enhanced full cluster crash recovery (K8SPXC-1828): 1.20.0 hardens the crash recovery path to prevent potential data loss after sudden node power-offs.

 

Deprecation notice: PMM2 monitoring integration is deprecated in 1.20.0. Migrate to PMM 3 before version 1.22.0, when PMM2 support will be removed.

 

Conclusion

PXC Operator 1.20.0 turns three previously manual steps into operator-managed concerns: disk growth, certificate rotation, and ARM64 scheduling. Combined with PITR validation improvements and configurable leader election, this release reduces the operational surface area for clusters running under production pressure. If you run into edge cases with automatic storage resizing or TLS rotation, the community forum is the right place to share them.

 

Try It Out

 

The post Percona Operator for MySQL (PXC) 1.20.0: Automatic Storage Resizing, TLS Certificate Rotation, and ARM64 Support appeared first on Percona.

May
28
2026
--

Percona Operator for PostgreSQL 3.0.0: Hard Fork, OLM Scoping, Major Upgrades


The Percona Operator for PostgreSQL 3.0.0 is here. This is the release that completes the hard fork of the operator from the Crunchy Data PostgreSQL Operator into a fully independent project, with a dedicated upstream.pgv2.percona.com API group for the inherited CRDs, an automatic CRD-rename rollout for existing 2.x installs on upgrade, and a public roadmap that drives what comes next.

This release ships three headline changes that matter for production teams. The CRD renaming under a Percona-owned API group, which finally lets the Crunchy operator and the Percona operator coexist in the same Kubernetes cluster. Proper OLM namespace scoping for OpenShift installations. And the move to the official Percona Distribution image for major PostgreSQL version upgrades, aligning the upgrade path with the same binaries that run in your clusters.

 

All three land in service of the same goal: making 3.0.0 a clean, durable operational baseline for the operator’s next several years as an independent project. Future releases will be shaped by what the community asks for and contributes back. The public roadmap is the durable signal of that commitment.

In this post, you will learn about:

  • The hard fork and how the CRD rename unlocks coexistence with the Crunchy operator
  • OLM namespace-scoping improvements for OpenShift installations
  • The move to the official Percona Distribution image for major PostgreSQL version upgrades
  • Other improvements and the 2.7.0 deprecation
  • Supported PostgreSQL versions and platforms

 

Hard fork: CRDs renamed under upstream.pgv2.percona.com

The Percona Operator for PostgreSQL has, until now, been a soft fork. Custom Resources inherited from Crunchy PGO used the upstream postgres-operator.crunchydata.com API group. The two operators shared CRDs, which meant you could only run one of them in a given Kubernetes cluster. Installing both would lead to overlapping CRDs, conflicting webhooks, and finalizer collisions, so platform teams had to pick a side before they had finished evaluating.

Starting with 3.0.0, every inherited CRD is renamed into a new dedicated upstream.pgv2.percona.com API group (K8SPG-1007). Percona’s own native CRDs (such as PerconaPGCluster under pgv2.percona.com/v2) are unchanged. The change applies to the inherited resources: PostgresCluster, PGUpgrade, PGAdmin, and the rest.

 

Coexistence: running both operators in the same cluster

The practical effect is that the Crunchy Data PostgreSQL Operator and the Percona Operator for PostgreSQL can now run on the same Kubernetes cluster at the same time, even in the same namespaces, with no CRD or webhook conflict. That unlocks a few real workflows: evaluating both operators on the same staging cluster without spinning up a second cluster, running existing Crunchy-managed clusters in some namespaces while bringing up new Percona-managed clusters in others, or testing a new database version on the Percona side while production stays on Crunchy until you are confident. The choice between the two operators stops being all-or-nothing.

 

Upgrade behavior for existing 2.x installs

For an existing install, the upgrade to 3.0.0 is mechanically simple. The operator creates the new-API-group CRDs alongside the legacy ones, then runs a one-time migration that updates dependent objects (Secrets, certificates, finalizer references) to point at the new CRD instances. Existing custom resources keep working through the legacy CRDs during the transition, and once migration completes, all reconciliation moves to the new group.

Old PostgresCluster reference:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: cluster1


New (after upgrade to 3.0.0):

apiVersion: upstream.pgv2.percona.com/v1beta1
kind: PostgresCluster
metadata:
  name: cluster1

 

Day-to-day, your PerconaPGCluster Custom Resource (the one most teams interact with directly) is unchanged. The rename mostly matters in three situations: when a kubectl filter or a GitOps repository hard-codes the old API group, when a CI pipeline references the legacy CRD by name, and when you run the Percona and Crunchy operators side by side and need them not to collide.

Note: During the CRD migration on upgrade, the release notes report brief disruptions to pgBackRest operations (typically 1 to 2 minutes) while Kubernetes propagates certificate changes. Plan the upgrade during a maintenance window if backup continuity is critical, or pause scheduled backups during the upgrade.

Full details on the API-group change are in the Percona PostgreSQL operator documentation.

 

Improved OLM namespace scoping for OpenShift

OpenShift users install operators through the OpenShift Lifecycle Manager (OLM), and OLM enforces an OperatorGroup to scope which namespaces an operator watches. In practice, 2.x had quirks: teams that selected “Single namespace” mode would sometimes see the operator reconciling CRs in other namespaces, and teams in “All namespaces” mode would sometimes see incomplete coverage when CRs were created in newly-added namespaces.

3.0.0 fixes this by aligning the operator’s namespace watch list with the OperatorGroup that OLM applies. All-namespaces installs watch all namespaces. Single-namespace installs respect the targetNamespaces set on the OperatorGroup.

 

Why it matters in shared infrastructure

For an OpenShift platform team running shared infrastructure, this distinction matters operationally. A typical setup has the database operator installed once in a platform namespace (such as openshift-operators) but expected to serve PerconaPGCluster resources owned by individual application teams in their own namespaces. If the operator over-reaches into namespaces it should not watch, RBAC noise multiplies. If it under-reaches, application teams file tickets about clusters that never reconcile. The 3.0.0 alignment with OperatorGroup semantics removes both failure modes.

 

OperatorGroup wiring

For users installing through OLM via the OpenShift web console, the install flow is unchanged. The fix is in how the operator’s reconciler interprets the OLM-supplied namespace scope after install. For users who manage OperatorGroups directly, a single-namespace install looks like this:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: percona-pg-operator-group
  namespace: postgres-prod
spec:
  targetNamespaces:
    - postgres-prod

And an all-namespaces install:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: percona-pg-operator-group
  namespace: openshift-operators
spec: {}

The empty spec: {} (or an OperatorGroup with no targetNamespaces) means “watch all namespaces” by OLM convention. The 3.0.0 operator now honors that.

 

Note: After you upgrade an existing 2.x install to 3.0.0, the operator may begin reconciling PerconaPGCluster resources in namespaces it had previously ignored due to the prior scoping bug. Audit existing CRs across your cluster before upgrading, especially if you have stale test clusters in unintended namespaces. The release notes call this out explicitly.

Note for community vs certified bundle users: Community OLM bundles did not support cluster-wide (all-namespaces) mode in earlier versions, 3.0.0 adds it. Certified bundles already supported cluster-wide mode, but they used a separate stable-cw channel for it with 3.0.0 the channels are unified, so users upgrading from a certified stable-cw install need to switch their subscription channel to stable to receive the upgrade.

For the full install workflow on OpenShift, see the OpenShift installation documentation.


Major PostgreSQL version upgrades now use the official Percona Distribution image


Major-version upgrades (for example, PostgreSQL 17 to 18) require running pg_upgrade, which needs binaries for both the source and target versions in the same environment. The operator has supported major-version upgrades since 2.x, but it shipped its own dedicated upgrade image to do so. That worked, but it meant a Percona-specific image lived in the upgrade path, separate from the same Percona Distribution for PostgreSQL build that runs in your clusters.

 

Switching to the official Percona Distribution image

In 3.0.0, the operator switches to using the official Percona Distribution for PostgreSQL image for major-version upgrades: percona/percona-distribution-postgresql-upgrade (current tag: 18.4-17.10-16.14-15.18-14.23-1, which encodes the bundled major versions). The benefit is alignment: the binaries that run pg_upgrade are the same binaries that ship in the corresponding percona-distribution-postgresql image you already run in production, built from the same source, signed the same way, and patched on the same schedule. The operator orchestrates the upgrade through the PerconaPGUpgrade Custom Resource that names the source and target versions, the upgrade image, and the target component images (PostgreSQL, pgBouncer, pgBackRest).

 

Running an upgrade through the PerconaPGUpgrade CR

A PostgreSQL 17 to 18 upgrade looks like this:

apiVersion: pgv2.percona.com/v2
kind: PerconaPGUpgrade
metadata:
  name: cluster1-17-to-18
spec:
  postgresClusterName: cluster1
  image: docker.io/percona/percona-distribution-postgresql-upgrade:18.4-17.10-16.14-15.18-14.23-1
  fromPostgresVersion: 17
  toPostgresVersion: 18
  toPostgresImage: docker.io/percona/percona-distribution-postgresql:18.4-1
  toPgBouncerImage: docker.io/percona/percona-pgbouncer:1.25.2-1
  toPgBackRestImage: docker.io/percona/percona-pgbackrest:2.58.0-2

Apply it with kubectl apply -f upgrade.yaml -n <namespace>. The operator reconciles the upgrade as a controlled, observable process: it brings the cluster down for the upgrade window, runs pg_upgrade from the bundled image, brings the cluster back up on the target version, and updates pgBouncer and pgBackRest images in the same step.

Operationally, this matters for teams running on PostgreSQL’s annual major-version cadence. Every September brings a new major release; staying on a supported version means executing one major upgrade per cluster per year. Pulling the upgrade image from the same percona-distribution-postgresql registry path as the runtime image means image-signature verification, mirror-to-private-registry rules, and CVE-scanning policies you already have in place apply to the upgrade flow without any per-image exception.

Note: The pgaudit extension is not upgraded automatically. After the operator completes the major version upgrade, drop and recreate pgaudit manually in each database that uses it: DROP EXTENSION pgaudit; followed by CREATE EXTENSION pgaudit;. The release notes call this out as a required step (K8SPG-1022). Also worth scanning for collation-dependent indexes after the upgrade and refreshing collation metadata with ALTER DATABASE <name> REFRESH COLLATION VERSION; per the upstream PostgreSQL 18 release notes.

Full procedure, prerequisites, and rollback notes are in the major version upgrade documentation.

Other Improvements

Operational polish landed alongside the headline changes:

  • Go 1.26 update (K8SPG-1019): the operator binary is now built with Go 1.26, picking up performance optimizations, tooling improvements, and the security fixes that landed in the Go runtime since the previous release.
  • pgaudit upgrade documentation (K8SPG-1022): the major-version upgrade docs now include an explicit pgaudit drop-and-recreate procedure, surfacing the gotcha that previously caught users mid-upgrade.

The release also defaults the cluster-upgrade documentation to PostgreSQL 18 across all examples and tutorials.

 

Supported software and platforms

The Percona Operator for PostgreSQL 3.0.0 is developed and tested on:

  • PostgreSQL: 14.23-1, 15.18-1, 16.14-1, 17.10-1, 18.4-1 
  • pgBackRest: 2.58.0-2
  • pgBouncer: 1.25.2-1
  • Patroni: 4.1.3
  • PostGIS: 3.5.6
  • PMM Client: 2.44.1-1 and 3.7.1

 

Supported Kubernetes platforms:

  • Google Kubernetes Engine (GKE) 1.33 to 1.35
  • Amazon Elastic Kubernetes Service (EKS) 1.33 to 1.35
  • OpenShift 4.18 to 4.21
  • Azure Kubernetes Service (AKS) 1.33 to 1.35
  • Minikube 1.38.1 (Kubernetes v1.35.1) for local development

 

Deprecation: 2.7.0 support dropped

Support for Custom Resource Definitions from operator version 2.7.0 has been removed. If you are still on 2.7.0, upgrade to 2.8.x or 2.9.x first, then upgrade to 3.0.0. The CRD migration described above only handles 2.8.x and 2.9.x to 3.0.0 transitions cleanly.

 

Conclusion

3.0.0 is the release where the Percona Operator for PostgreSQL becomes a fully independent project. The CRD rename removes the last upstream coupling that mattered operationally. The OLM scoping fix removes a long-standing OpenShift quirk. The official major-version upgrade image removes one of the more painful operational gaps in earlier versions.

Beyond the technical work, 3.0.0 is also where Percona’s commitment to community-driven development moves from intent to mechanism. The public roadmap is open. The issue tracker is open. The images are freely redistributable. Future releases will be shaped by what the community asks for, files, and contributes back. If there is a feature you want to see in 3.1.0 or 3.2.0, open an issue or a PR, that is where the work happens now.

 

Try It Out

The post Percona Operator for PostgreSQL 3.0.0: Hard Fork, OLM Scoping, Major Upgrades appeared first on Percona.

May
27
2026
--

Migrate from Crunchy Data PostgreSQL Operator to Percona PostgreSQL Operator: Backup-Restore and PV Reuse


A Percona PostgreSQL operator pgBackRest restore is the simplest way to move off the Crunchy Data PostgreSQL Operator: take a full Crunchy backup, point the new Percona cluster’s dataSource at the existing pgBackRest archive, and the cluster bootstraps from it before its first start. This post covers that path, plus a second option, persistent-volume reuse, for cases where you want to skip the data copy entirely.

This is part 3 of a 3-part series on running PostgreSQL on Kubernetes with a fully open-source operator. Part 1 walked through the changing open-source landscape and announced the hard fork of the Crunchy Data PostgreSQL Operator into the fully independent Percona PostgreSQL Operator v3.0.0Part 2 covered the standby cluster method, the safest migration path when downtime budget is tight.

This post covers two simpler paths:

  • Backup and restore, the fastest if you can tolerate a short application-downtime window
  • Persistent volume reuse, when you want to skip the data copy entirely and keep the existing PGDATA

If you are landing here cold, start with part 1 for the why, then read Part 2 for the standby method. The rest of this post assumes you have already decided to migrate and want a tested playbook.

Tested with

Component Version
Crunchy Data PostgreSQL Kubernetes Operator v5.8.x (tested on v5.8.7)
Percona PostgreSQL Kubernetes Operator v3.x.x (tested on v3.0.0)
PostgreSQL 18 (must match between source and target)
Object storage SeaweedFS (Apache-2.0), or any S3-compatible service. Required for the backup-and-restore method, optional for PV reuse.
Tools kubectlhelm (v3)

Different versions may have slight differences in CR fields or behavior. Always consult the official documentation for the operator and PostgreSQL version you are running.

 

What this post does NOT cover

  • Application-side connection-string changes beyond updating to the new pgBouncer service
  • Schema-changing upgrades, major PostgreSQL version upgrades, or extension migrations
  • Crunchy enterprise-only features like TDE or pgBackRest custom encryption
  • Operating two operators against the same namespace before the hard fork. Use Percona PostgreSQL Operator v3.0.0 or higher.

 

1. Migration using backup and restore

This is often the fastest and simplest path, especially when you do not need a live standby. You take a full backup of the Crunchy source cluster, then create a Percona cluster that automatically restores from that backup before its first start.

Data written between the final backup and the application cutover is lost, so the migration window is the time between those two events. For a near-zero-downtime alternative, see part 2: standby cluster method.

 

Overview

Before you begin

Set the namespace once. Every command in this guide reads from this variable:

export MIGRATION_NS=postgres-migration
kubectl create namespace $MIGRATION_NS

 

Deploy SeaweedFS

Skip this step if you already have an S3-compatible repository (AWS S3, GCS, Ceph). Update the endpoint and credentials in the YAML examples accordingly.

SeaweedFS provides an S3-compatible object store that runs inside Kubernetes. Both operators will use it as the shared pgBackRest WAL archive.

TLS is required. pgBackRest always connects to S3 endpoints over HTTPS, even when repo1-s3-verify-tls: "n" is set (that flag skips certificate verification, it does not fall back to HTTP). The steps below generate a self-signed certificate and pass it to SeaweedFS via Helm values.

# Generate a self-signed TLS certificate for SeaweedFS S3
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout /tmp/seaweedfs.key \
  -out /tmp/seaweedfs.crt \
  -subj "/CN=seaweedfs-all-in-one"

kubectl -n $MIGRATION_NS create secret tls seaweedfs-s3-tls \
  --cert=/tmp/seaweedfs.crt \
  --key=/tmp/seaweedfs.key

helm repo add seaweedfs https://seaweedfs.github.io/seaweedfs/helm
helm repo update

helm install seaweedfs seaweedfs/seaweedfs \
  --namespace $MIGRATION_NS \
  --version 4.23.0 \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-backup-restore/examples/seaweedfs-values.yaml \
  --wait

The Helm values file in the repo creates the pg-migration bucket on first start, so no separate aws s3 mb step is needed.

Step 0. Create pgBackRest secrets

Both operators need credentials to read and write the shared SeaweedFS bucket. Apply the secrets from examples/01-pgbackrest-secrets.yaml:

# Copy and edit the file first to set your credentials.
kubectl apply -n $MIGRATION_NS \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-backup-restore/examples/01-pgbackrest-secrets.yaml

Both contain the same SeaweedFS credentials (pgmigration / pgmigration123). For AWS S3, replace those with your IAM access key ID and secret access key.

 

Step 1. Start with your existing Crunchy Data cluster

If you already have a running Crunchy cluster, ensure its pgBackRest repo1 points at the shared bucket. The repo1-path value must match the path that will be referenced in the Percona dataSource.pgbackrest.global.repo1-path field.

Optional: deploy the Crunchy operator for testing. The Helm install below is shown only as a quick way to reproduce this blog post’s example. The migration steps in the rest of this post do not depend on how you deployed the source operator.

helm install pgo \
  oci://registry.developers.crunchydata.com/crunchydata/pgo \
  -n $MIGRATION_NS \
  --version 5.8.7 \
  --set singleNamespace=true \
  --wait

To start a fresh source cluster for testing, apply examples/02-crunchy-source-cluster.yaml:

kubectl apply -n $MIGRATION_NS \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-backup-restore/examples/02-crunchy-source-cluster.yaml

The key pgBackRest settings:

global:
  repo1-path: /crunchy-to-percona/repo1   # source repo referenced in Percona dataSource
  repo1-s3-uri-style: path                # required for path-style S3 endpoints (SeaweedFS, MinIO)
  repo1-s3-verify-tls: "n"                # skip TLS verification for self-signed cert; remove for AWS S3
repos:
  - name: repo1
    s3:
      bucket: pg-migration
      endpoint: seaweedfs-all-in-one.postgres-migration.svc.cluster.local:8443
      region: us-east-1

Wait for the cluster and its pgBackRest stanza to be ready:

kubectl wait pod \
  --selector postgres-operator.crunchydata.com/cluster=crunchy-source,postgres-operator.crunchydata.com/data=postgres \
  -n $MIGRATION_NS \
  --for=condition=Ready \
  --timeout=300s

kubectl wait postgrescluster/crunchy-source \
  -n $MIGRATION_NS \
  --for=jsonpath='{.status.pgbackrest.repos[0].stanzaCreated}'=true \
  --timeout=300s

 

Step 2. Trigger a full backup (the migration cutover point)

This is the backup the Percona cluster will restore from. Stop accepting writes on the application side before triggering it to ensure a consistent snapshot, or accept that data written after this backup will be lost.

kubectl annotate postgrescluster crunchy-source \
  -n $MIGRATION_NS \
  postgres-operator.crunchydata.com/pgbackrest-backup="$(date +%s)"

kubectl wait job \
  --selector postgres-operator.crunchydata.com/pgbackrest-backup=manual,postgres-operator.crunchydata.com/cluster=crunchy-source \
  -n $MIGRATION_NS \
  --for=condition=Complete \
  --timeout=600s

 

Step 3. Deploy the Percona Operator

kubectl apply -n $MIGRATION_NS --server-side \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/tags/v3.0.0/deploy/bundle.yaml

kubectl wait deployment percona-postgresql-operator \
  -n $MIGRATION_NS \
  --for=condition=Available \
  --timeout=120s

Step 4. Create the Percona cluster from the backup

Apply examples/03-percona-restored-cluster.yaml:

kubectl apply -n $MIGRATION_NS \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-backup-restore/examples/03-percona-restored-cluster.yaml

The key section that bootstraps the cluster from the Crunchy backup:

dataSource:
  pgbackrest:
    stanza: db
    configuration:
      - secret:
          name: percona-pgbackrest-secret
    global:
      # Must match repo1-path in the Crunchy source cluster exactly.
      repo1-path: /crunchy-to-percona/repo1
      repo1-s3-uri-style: path
      repo1-s3-verify-tls: "n"
    repo:
      name: repo1
      s3:
        bucket: pg-migration
        endpoint: seaweedfs-all-in-one.postgres-migration.svc.cluster.local:8443
        region: us-east-1

The Percona cluster’s own backup repository must use a different path from the Crunchy source:

backups:
  pgbackrest:
    global:
      repo1-path: /percona-restored/repo1   # different from Crunchy's path

As soon as the Custom Resource is applied, the cluster is bootstrapped from the storage referenced in dataSource and then started. Once the cluster becomes ready, you can immediately create new backups; in this case, repo1 from the backups section will be used as the target repository.

Wait for the cluster to reach ready state:

kubectl wait perconapgcluster/percona-restored \
  -n $MIGRATION_NS \
  --for=jsonpath='{.status.state}'=ready \
  --timeout=600s

Verify the data was restored successfully:

PERCONA_PRIMARY=$(kubectl get pod -n $MIGRATION_NS \
  --selector postgres-operator.crunchydata.com/cluster=percona-restored,postgres-operator.crunchydata.com/role=primary \
  -o jsonpath='{.items[0].metadata.name}')

kubectl -n $MIGRATION_NS exec "${PERCONA_PRIMARY}" -c database -- \
  psql -t -c "SELECT pg_is_in_recovery();"

Expected output: f. The cluster is the primary and accepts writes.

Step 5. Verify the cluster is healthy

kubectl wait perconapgcluster/percona-restored \
  -n $MIGRATION_NS \
  --for=jsonpath='{.status.state}'=ready \
  --timeout=600s

kubectl wait perconapgcluster/percona-restored \
  -n $MIGRATION_NS \
  --for=jsonpath='{.status.pgbackrest.repos[0].stanzaCreated}'=true \
  --timeout=300s

Step 6. Take a post-migration backup

Apply examples/04-post-migration-backup.yaml:

kubectl apply -n $MIGRATION_NS \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-backup-restore/examples/04-post-migration-backup.yaml

kubectl wait perconapgbackup/post-migration-backup \
  -n $MIGRATION_NS \
  --for=jsonpath='{.status.state}'=Succeeded \
  --timeout=600s

This creates a clean recovery baseline on the Percona cluster’s own repository. All future PITR restores will use this backup, independent of the Crunchy archive.

Step 7. Reconnect your application

kubectl get service -n $MIGRATION_NS \
  --selector postgres-operator.crunchydata.com/cluster=percona-restored,postgres-operator.crunchydata.com/role=pgbouncer

Step 8. Clean up the Crunchy cluster

Once the migration is verified and your application is connected to the new cluster:

kubectl delete postgrescluster crunchy-source -n $MIGRATION_NS
helm uninstall pgo -n $MIGRATION_NS

 

Rollback

Until Step 8, rollback is straightforward: switch the application connection string back to the Crunchy pgBouncer service. The Crunchy primary still holds the authoritative state because no writes were directed at the Percona cluster during the cutover (you stopped writes before Step 2). Any writes the application sent to the Percona cluster after cutover will not be present on Crunchy and would need to be replayed manually.

After Step 8, rollback requires restoring the Crunchy cluster from a backup, which is feasible because the original repo1 is still in the bucket.

Troubleshooting

archive.info missing. The repo1-path in dataSource.pgbackrest.global must match the Crunchy source cluster’s repo1-path exactly:

kubectl get postgrescluster crunchy-source -n $MIGRATION_NS \
  -o jsonpath='{.spec.backups.pgbackrest.global.repo1-path}'

kubectl get perconapgcluster percona-restored -n $MIGRATION_NS \
  -o jsonpath='{.spec.dataSource.pgbackrest.global.repo1-path}'

 

Restore job fails with TLS errors. pgBackRest requires HTTPS even with repo1-s3-verify-tls: "n". Verify SeaweedFS is reachable:

kubectl run -i --rm s3-check \
  --image=perconalab/awscli \
  --restart=Never \
  -n $MIGRATION_NS \
  -- bash -c "
    AWS_ACCESS_KEY_ID=pgmigration \
    AWS_SECRET_ACCESS_KEY=pgmigration123 \
    AWS_DEFAULT_REGION=us-east-1 \
    aws --endpoint-url https://seaweedfs-all-in-one.${MIGRATION_NS}.svc.cluster.local:8443 \
        --no-verify-ssl \
        s3 ls s3://pg-migration
  "

 

Cluster stuck in restoring state. Check the pgBackRest restore job logs:

kubectl logs \
  --selector postgres-operator.crunchydata.com/cluster=percona-restored,postgres-operator.crunchydata.com/pgbackrest-restore=percona-restored \
  -n $MIGRATION_NS \
  -c pgbackrest

Data missing after restore. The restore captures data up to the latest backup. If post-backup data is critical, re-run the backup on the Crunchy cluster after quiescing writes, then delete and recreate the Percona cluster to restore from the newer backup.

2. Migration using existing persistent volumes

This method reuses the Crunchy primary’s PGDATA persistent volume directly. It avoids a full backup-restore cycle: you retain the Crunchy primary’s PV, delete the Crunchy cluster, then create a Percona cluster whose PVC binds to that same PV. PostgreSQL starts on the existing data directory without any restore step.

It is useful when:

  • you want to avoid copying data
  • your storage is very large
  • you must preserve the original data directory exactly
  • you removed the cluster but kept the PV

 

Overview

 

Before you begin

export MIGRATION_NS=postgres-migration
kubectl create namespace $MIGRATION_NS

Step 1. Deploy the Crunchy and Percona operators

Both operators run in the same namespace. Crunchy PGO is uninstalled during the migration once the PV is retained.

Note (Crunchy): The Helm install for Crunchy PGO below is shown only as a quick way to reproduce this blog post’s example. If you are running Crunchy PGO in production, follow the official Crunchy Data documentation for installation. The migration steps in the rest of this post do not depend on how you deployed the source operator.

Note (Percona): The kubectl apply of the Percona operator below uses defult configuration of v3.0.0 from the operator repo for reproducibility of this guide. For production deployments, follow the official Percona Operator for PostgreSQL installation documentation to ensure the cluster configuration is properly sized and configured for your workload and traffic requirements.

helm install pgo \
  oci://registry.developers.crunchydata.com/crunchydata/pgo \
  -n $MIGRATION_NS \
  --version 5.8.7 \
  --set singleNamespace=true \
  --wait

kubectl apply -n $MIGRATION_NS --server-side \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/tags/v3.0.0/deploy/bundle.yaml

kubectl wait deployment pgo \
  -n $MIGRATION_NS --for=condition=Available --timeout=120s

kubectl wait deployment percona-postgresql-operator \
  -n $MIGRATION_NS --for=condition=Available --timeout=120s

Step 2. Start the Crunchy source cluster

If you already have a running Crunchy cluster with replicas: 1, proceed to Step 3.

To start a fresh cluster for testing:

kubectl apply -n $MIGRATION_NS \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-pv/examples/01-crunchy-source-cluster.yaml

kubectl wait pod \
  --selector postgres-operator.crunchydata.com/cluster=crunchy-source,postgres-operator.crunchydata.com/role=master \
  -n $MIGRATION_NS \
  --for=condition=Ready \
  --timeout=300s

Step 3. Stop writes and identify the primary PV

Stop your application from writing to the database. This is the start of the downtime window. Then identify the primary pod, its PVC, and the backing PV:

PRIMARY=$(kubectl get pod -n $MIGRATION_NS \
  --selector postgres-operator.crunchydata.com/cluster=crunchy-source,postgres-operator.crunchydata.com/role=master \
  -o jsonpath='{.items[0].metadata.name}')

PVC_NAME=$(kubectl get pod -n $MIGRATION_NS "${PRIMARY}" \
  -o jsonpath='{.spec.volumes[?(@.name=="postgres-data")].persistentVolumeClaim.claimName}')

PV_NAME=$(kubectl get pvc -n $MIGRATION_NS "${PVC_NAME}" \
  -o jsonpath='{.spec.volumeName}')

echo "Primary pod: ${PRIMARY}"
echo "PVC:         ${PVC_NAME}"
echo "PV:          ${PV_NAME}"

Step 4. Configure the source cluster to retain PVs

If you want to delete the Crunchy source cluster but keep the persistent volumes, the PV reclaim policy must be set to Retain. For dynamically provisioned PersistentVolumes, the default reclaim policy is Delete, which removes the data once there are no more PersistentVolumeClaims associated with the PV.

kubectl patch pv "${PV_NAME}" \
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

kubectl get pv -n $MIGRATION_NS

Delete the Crunchy cluster and uninstall PGO:

kubectl patch postgrescluster crunchy-source -n $MIGRATION_NS \
  --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]' 2>/dev/null || true

kubectl delete postgrescluster crunchy-source -n $MIGRATION_NS
helm uninstall pgo -n $MIGRATION_NS

After the PVC is deleted, the PV enters Released state. A Released PV retains its old claimRef and cannot be claimed by a new PVC until it is cleared:

kubectl patch pv "${PV_NAME}" --type=json \
  -p='[{"op":"remove","path":"/spec/claimRef"}]'

kubectl wait pv "${PV_NAME}" \
  --for=jsonpath='{.status.phase}'=Available \
  --timeout=60s

Label the PV so the Percona PVC selector binds to it exclusively. This prevents accidental binding to another available volume:

kubectl label pv "${PV_NAME}" percona-pv-migration=migrated

 

Step 5. Create the Percona cluster with the retained volume

kubectl apply -n $MIGRATION_NS \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-pv/examples/02-percona-migrated-cluster.yaml

The key section that binds the PVC to the labelled PV:

instances:
  - name: instance1
    replicas: 1
    dataVolumeClaimSpec:
      selector:
        matchLabels:
          percona-pv-migration: migrated

The Percona Operator creates a PVC with that selector. The PVC binds to the labelled PV, and PostgreSQL starts on the existing PGDATA directory with no restore needed. pgBackRest uses a local PVC-backed repository (repo1.volume), so no S3 credentials or external storage are required, but you can use S3 storage as well.

Wait for the cluster to become ready and verify the data is intact:

kubectl wait perconapgcluster/percona-migrated \
  -n $MIGRATION_NS \
  --for=jsonpath='{.status.state}'=ready \
  --timeout=600s

PERCONA_PRIMARY=$(kubectl get pod -n $MIGRATION_NS \
  --selector postgres-operator.crunchydata.com/cluster=percona-migrated,postgres-operator.crunchydata.com/role=primary \
  -o jsonpath='{.items[0].metadata.name}')

kubectl -n $MIGRATION_NS exec "${PERCONA_PRIMARY}" -c database -- \
  psql -t -c "SELECT pg_is_in_recovery();"

Expected output: f. The cluster is the primary and accepts writes.

Step 6. Scale up replicas

The cluster started with a single replica to reuse the migrated PV. Once the primary is healthy, drop the PVC selector and scale out so the operator can provision fresh replica volumes from the storage class:

kubectl patch perconapgcluster percona-migrated \
  --namespace $MIGRATION_NS \
  --type=json \
  -p='[
    {"op":"remove","path":"/spec/instances/0/dataVolumeClaimSpec/selector"},
    {"op":"replace","path":"/spec/instances/0/replicas","value":3}
  ]'

kubectl wait perconapgcluster/percona-migrated \
  --namespace $MIGRATION_NS \
  --for=jsonpath='{.status.state}'=ready \
  --timeout=300s

Removing the selector here is important: leaving it in place would cause the new replica PVCs to fail provisioning because no other PV carries the migration label.

Step 7. Take a post-migration backup

kubectl apply -n $MIGRATION_NS \
  -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/refs/heads/migration-from-crunchy-guide/e2e-tests/tests/migration-from-crunchy-pv/examples/03-post-migration-backup.yaml

kubectl wait perconapgbackup/post-migration-backup \
  -n $MIGRATION_NS \
  --for=jsonpath='{.status.state}'=Succeeded \
  --timeout=600s

This creates the first backup on the Percona cluster’s local pgBackRest repository, establishing a baseline for future PITR restores.

Step 8. Reconnect your application

kubectl get service -n $MIGRATION_NS \
  --selector postgres-operator.crunchydata.com/cluster=percona-migrated,postgres-operator.crunchydata.com/role=pgbouncer

Step 9. Cleanup

After the migration is verified, remove the migration label from the PV (Step 6 already removed the PVC selector that depended on it):

kubectl label pv "${PV_NAME}" percona-pv-migration-

 

Rollback

PV migration is the least rollback-friendly of the three methods. Once the Percona cluster has started writing to the PGDATA directory, the original Crunchy timeline is gone. If you need a way back, take a Crunchy-side pgBackRest backup before Step 4 and treat that backup as your rollback point. Recovery is then a fresh Crunchy cluster restored from that backup.

Troubleshooting

PVC stays in Pending state. The PVC selector did not match the labelled PV. Verify the label and PV phase:

kubectl get pv "${PV_NAME}" --show-labels
kubectl get pv "${PV_NAME}" -o jsonpath='{.status.phase}'

PostgreSQL fails to start (data directory errors). Check the database container logs:

kubectl -n $MIGRATION_NS logs "${PERCONA_PRIMARY}" -c database

If the Crunchy cluster was shut down uncleanly, there may be incomplete WAL. Patroni will attempt crash recovery automatically; check the logs for progress.

PV was deleted before setting Retain. If the PV was deleted along with the PVC (default Delete policy), the data is gone and PV migration is no longer possible. Use the backup-and-restore migration above, restoring from the most recent pgBackRest backup.

 

Conclusion

Two more migration paths from the Crunchy Data PostgreSQL Operator to the fully open-source Percona PostgreSQL Operator. Combined with Part 2, the series gives you three production-tested options:

  • Standby cluster (part 2): near-zero downtime via streaming replication and pgBackRest standby
  • Backup and restore (this post): the simplest path, restoring directly from a Crunchy pgBackRest backup
  • Persistent volume reuse (this post): when you want to keep storage and skip the data copy

All three approaches are safe, predictable, and reversible, with the rollback caveats noted in each section. Because Percona’s operator, images, and tooling are 100 percent open source, you keep full control: you can always migrate back to the Crunchy operator, or out to another open-source operator (Zalando, StackGres, CloudNativePG) using the same patterns. That last journey is a topic for a future post.

This post covers basic deployment patterns and simplified configuration examples. If your environment uses custom images, Crunchy enterprise features, or otherwise needs tailored migration steps, contact the Percona team and we will help you plan and execute the move.

 

Try It Out

The post Migrate from Crunchy Data PostgreSQL Operator to Percona PostgreSQL Operator: Backup-Restore and PV Reuse appeared first on Percona.

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com