Editor’s Note: This article was originally authored by Inel Pandzic.
Migrating data between MongoDB clusters is rarely a five-minute job. A large initial clone followed by days of change-stream replication is normal, and for all that time, Percona ClusterSync for MongoDB (PCSM) is a critical piece of your infrastructure. Until now, it was also a single point of failure: if the host, container, or pod running PCSM crashed, replication stopped dead. Nobody took over. An operator had to notice the outage and manually recover the process before synchronization could resume.
That changes with PCSM 1.0.0, which introduces built-in active-standby high availability for the replication phase.
What’s new
You can now run multiple PCSM instances pointed at the same source and target clusters. They coordinate through a MongoDB-backed lease stored on the target cluster: exactly one instance holds the lease and is ACTIVE — it runs replication. The rest stay STANDBY and take over automatically when the lease expires. No external coordinator— the target MongoDB cluster you already have is the source of truth.
Best of all, HA is always on and requires zero configuration changes. A single instance simply wins the lease immediately and behaves exactly as before. If you want failover, just start a second (or third) instance with the same source and target, and you’re done.
How it works
Three mechanisms make failover safe: lease-based election, term fencing, and checkpoint-based recovery.
Lease election without split-brain
All coordination state lives in the percona_clustersync_mongodb database on the target. Election is a single atomic conditional write: an instance claims the lease only if the current one has expired, and expiry is evaluated against the server clock. Two racing standbys cannot both win, and clock skew between PCSM hosts is irrelevant — MongoDB’s clock is the only one that matters.
The lease itself is one small document – percona_clustersync_mongodb.lease:
The term field a monotonic fencing token that increments on every election (every ACTIVE->STANDBY change), and it’s stamped into every checkpoint (percona_clustersync_mongodb.checkpoints) write the ACTIVE makes. Consider the classic distributed-systems nightmare: an ACTIVE stalls (GC pause, network partition, pod killed and instance restarted before the lease expired), its lease expires, a standby is promoted — and then the old instance wakes up and keeps writing as if nothing happened. With term fencing, that “zombie” active’s checkpoint writes carry a stale term. The target rejects them, the deposed instance notices, and it self-demotes to STANDBY. The new ACTIVE’s state is never corrupted.
The failover path – Checkpoint-based Recovery
On promotion, the new ACTIVE resumes replication from the persisted checkpoint — the same crash-recovery mechanism PCSM has always used, now triggered automatically. Timings are fixed in this release: the lease TTL is 10 seconds, the ACTIVE renews every 3 seconds, and each instance heartbeats every 3 seconds. In practice, a hard-killed ACTIVE is replaced within roughly the lease TTL.
Failover during the initial clone
One boundary to know: HA covers the change-stream replication phase. The initial clone currently is not resumable. If the ACTIVE dies mid-clone, a standby is still promoted, detects the interrupted clone during recovery, and fails cleanly with an explicit reason surfaced in both the logs and /status: initial clone interrupted by failover and is not resumable; start a new run to re-clone from scratch Recovery is a plain /start on the new ACTIVE, which restarts the sync from scratch. Automatic failover kicks in once the clone completes and continuous replication begins.
Operating an HA deployment
Each instance maintains a liveness document (percona_clustersync_mongodb.members) on the target, and API responses include a cluster envelope so an operator hitting any node can see who is ACTIVE: me (the instance you reached), role, and the full member list with every live instance’s host, port, and role. If you send an operational command (/start, /pause, /finalize, …) to a STANDBY, it responds with HTTP 409 and error: "not_active" — with the same envelope in the body, so your client can immediately locate the ACTIVE and retry there:
No guessing, no stale DNS tricks.
Two operational notes:
The envelope appears only when more than one live member is observed. A lone instance returns responses byte-identical to the pre-HA API, so existing tooling keeps working unchanged. Treat the envelope as optional.
/metrics (and pprof) are served on every role.
For cleanup, two new direct commands clear HA state on the target while no server is running: pcsm reset members and pcsm reset lease.
Observability: watching a failover happen
Each instance exports HA metrics on /metrics, on every role:
percona_clustersync_mongodb_ha_term — gauge; current lease term
percona_clustersync_mongodb_ha_role_transitions_total — counter; role changes on this instance
With these few metrics you can build a dashboard that tells the whole HA story at a glance: a state timeline of ha_active per instance doubles as the deployment roster and shows who is ACTIVE and since when, ha_term shows the current fencing term, and the rate of ha_role_transitions_total per instance works as a flap detector (0 means stable; a climbing number means trouble). We ship exactly such a board with the project — grab the Grafana dashboard JSON from the repo and use it as a starter for your own monitoring.
Here is a healthy three-instance deployment: one ACTIVE, two STANDBYs, term 1, no transitions.
And the same deployment right after the ACTIVE was killed: the instance on port 2243 took over, the lease term incremented to 2, and its transition counter recorded the promotion. The old ACTIVE rejoined as a STANDBY.
One glance at the timeline tells you the full failover story — who held the lease, when it changed hands, and whether the deployment has been stable since.
Getting started
Upgrade note: replication state from 0.9.0 is not compatible with 1.0.0. Run pcsm reset against the target before starting replication with the new version.
Then simply start more than one instance:
Run the same command on a second host, check /status on either one, and you’ll see one ACTIVE and one STANDBY. Kill the ACTIVE and watch the standby pick up replication from the last checkpoint within seconds.
Join the Community
Percona thrives on community collaboration. As we continue to refine PCSM and celebrate its production-ready GA release, we invite you to get involved. We welcome bug reports, feature suggestions, and code contributions. Your input helps us build the tools the MongoDB community actually needs. We encourage you to deploy PCSM in your production environments today! Check out the PCSM repository and join the journey with us!
Ready to break free from vendor lock-in? Check out the PCSM Documentation to get started with your first sync today.
This RAM/Memory reduction is most useful for Valkey 9.1 environments that are key string heavy. They are used extensively in caching, sessions/authentication, real-time apps, ecomm, Ad tech/martech, and gaming.
These scenarios see a reduction of 37.5%, which has been attained in our testing environments versus version 7.2.. The biggest jump in the reduction came between 7.2 and 8.1 where the difference was 29.8%. Although the reduction doesn’t apply to every Valkey environment, those that are string heavy should experience 37.5% RAM reductions.
RAM usage scenario:
Starting point: 1 TB on Valkey 7.2
Valkey 7.2: 1,000 GB
Valkey 8.0: ~800 GB
Valkey 8.1: ~640 GB
Valkey 9.1: ~630 GB
This post is based on a rigorous benchmark and a structural teardown of the source, aimed at answering three questions:
For the same data, how much memory does each version actually save?
Where exactly does that saving come from at the data-structure level?
How do different data structures and encoding result in significantly different memory footprints?
1. Test Setup:
Five versions were tested — 7.2.14, 8.0.10, 8.1.9, 9.0.5, and 9.1.1 — all built from source, all using jemalloc 5.3.0 as the allocator, so version is the only variable.
Test Environment settings:
Standalone mode, not cluster, to rule out cluster-level overhead
–maxmemory 0, –save “” –appendonly no, to keep persistence and eviction out of the picture
valkey-benchmark -t set -n 1000000 -r 1000000 -d 16: one million SETs, 16-byte values
Both -n and -r are set to 1,000,000 on purpose — this causes random-key collisions, so only about 63.2% (~632K) land as unique keys. This matches the official Valkey benchmark methodology exactly
The baseline (empty instance) memory is subtracted for each version, giving the true marginal bytes per key
2. Results: Memory per Key, Version by Version
Version
Unique Keys
used_memory
Bytes / Key
vs Prev
vs 7.2.14
7.2.14
632,181
61.78 MB
102.48 B
—
baseline
8.0.10
632,565
57.00 MB
94.49 B
?7.8%
?7.8%
8.1.9
632,573
43.43 MB
71.98 B
?23.8%
?29.8%
9.0.5
631,672
43.36 MB
71.99 B
+0.0%
?29.8%
9.1.1
631,985
38.58 MB
64.02 B
?11.1%
?37.5%
We can see that across roughly 632K unique keys, the size of per key averages 102.48 bytes in version 7.2.14 down to 64.02 bytes in 9.1.1 — a cumulative drop of 37.5%. In other words, the same hardware stores roughly 60% more of these small keys on 9.1 than it did on 7.2. We can see 8.1 alone accounts for the single biggest jump, at -23.8%, because that’s where the hash table got rewritten. However 9.0 shows basically zero change, since that release simply wasn’t targeting memory.
3. Let us see how the change happens version by version.
7.2 ? 8.0: Embedding the Key into dictEntry
In 7.2, dictEntry is a classic three-pointer struct — *key, *val, *next — 24 bytes total. The catch is that the key itself is a pointer to a separately allocated SDS blob: every lookup pays for an extra malloc and an extra memory hop just to read the key.
8.0 introduces embeddedDictEntry: the key’s bytes are stored inline right after the struct. *val, *next, plus the inlined key data come to 16 bytes plus the key length — one allocation instead of two.
Result: ?8 bytes per key, one fewer memory hop on every lookup, automatic on upgrade, no config changes needed.
8.0 ? 8.1: A Cache-Line-Sized Hash Table
This is the single largest jump in the whole series. In 8.0, the dictionary is a classic chained hash table: reaching a value means “bucket ? entry ? key ? value” — four memory hops (plus two more per hash collision).
8.1 rewrites this as a new hashtable structure where one bucket is exactly 64 bytes — the size of a CPU cache line, — with the key and value embedded directly inside it.The bucket’s 8-byte metadata region packs a “has-child-bucket” bit, 7 filled-bits, and 7 bytes of secondary hash, letting most non-matching candidates be skipped without an extra memory access. Key and value are embedded in the same serverObject, shortening the path to roughly two hops — “bucket ? serverObject” — and a hash collision usually stays within the same cache line.
dictEntry as a struct disappears entirely. The result is roughly ?20 bytes per key-value pair (?30 bytes for keys with a TTL) — the single biggest jump in the entire 7.2?9.1 series.
9.0 ? 9.1: Reusing the Pointer Field Itself
In earlier versions, embedded strings (embstr) still wasted an 8-byte pointer field simply referencing the contiguous payload. 9.1’s fix: for short strings, the string bytes are stored directly inline in those same 8 bytes normally used for the pointer — no pointer needed, and no separate allocation for the value.
Result: up to a 20% memory reduction for strings under 128 bytes — and the benchmark’s “key + 16-byte value” pairs sit right in the middle of that range.
The three structural changes account for most of the improvement:
8.0 inlined the key directly into dictEntry, eliminating a separate allocation and one memory hop (?8 bytes/key); 8.1 rewrote the dictionary into a 64-byte, cache-line-aligned hashtable that embeds both key and value, collapsing the lookup path from four hops to about two and removing dictEntry altogether (roughly ?20 to ?30 bytes per entry); and 9.1 reused the embedded-string pointer field to store short string data inline instead of pointing to it, cutting up to 20% off strings under 128 bytes.
Together, these changes show that Valkey’s memory efficiency gains came from targeted, low-level restructuring of core data structures rather than a single sweeping rewrite.
4. Same 1 Million Records, Two Encodings, 54% Less Memory
Two approaches to the same data:
Approach A: 4 separate String keys per user (SET user:1:name Alice, etc.)
Approach B: 1 Hash key per user, 4 fields (HSET user:1 name Alice age 30 city Toronto score 100)
Same data, same record count. Approach A uses 220 MB; Approach B uses 100 MB — a 54% reduction. That gap comes from the fact that every key pays a fixed per-key overhead: four keys pay it four times, one key pays it once.
Beyond raw memory — where Hash wins because it shares per-key overhead, there’s per-field TTL, where String is native with EXPIRE per key, while Hash didn’t natively support field-level TTL before 7.4.
For atomic multi-field operations, Hash wins with a single HSET or HMGET, while String needs multiple round trips.
For cluster slot locality, String needs a manual {hashtag} to co-locate related keys, while a Hash is automatically one key, one slot.
So there’s no universally correct choice — it’s memory versus flexibility, and it interacts with your cluster routing strategy too.
Summary
Testing five Valkey versions (7.2.14 through 9.1.1) under identical conditions, per-key memory for a simple SET workload dropped from 102.48 bytes to 64.02 bytes — a cumulative 37.5% reduction, with 8.1 alone responsible for the largest single jump (?23.8%) and 9.0 showing essentially no change. Three targeted structural rewrites drove this: 8.0 inlined the key into dictEntry, removing a separate allocation (?8 bytes/key); 8.1 replaced the chained hash table with a 64-byte, cache-line-aligned hashtable that embeds key and value together, cutting the lookup path from four hops to two and eliminating dictEntry entirely (?20 to ?30 bytes/entry); and 9.1 reused the embedded-string pointer field to store short string data inline, saving up to 20% for strings under 128 bytes. Separately, how the same data is modeled also matters as much as the version: storing a user’s data as 4 individual String keys costs 220 MB, versus 100 MB for the same fields grouped into a single Hash — a 54% saving driven by paying the fixed per-key overhead once instead of four times. That memory advantage comes with trade-offs, though: Hash lags String on native per-field TTL support (pre-7.4) but wins on atomic multi-field operations and automatic cluster slot locality, so the right choice depends on whether memory efficiency or per-field flexibility matters more for the use case.
In many real production databases, we often see some tables receiving high-frequency updates (millions per day) and autovacuum running repeatedly (hundreds of times per day). Additionally, those tables can sometimes become heavily bloated. The end effect is poor performance, poor concurrency, and an unmanageable database.
In case we are using methods pg_gather for diagnosis, such anomalies are easy to spot by clicking on the table header to sort.
Very often, a closer look at these tables reveals that status-update workloads are driving these heavy updates. Many times, even the table names reveal their purpose through words like “task”, “queue”, or “bucket”. A discussion with application architects often reveals that these tables are used for specific workflows, with status fields tracking what is completed and what remains pending for further processing. There can also be multiple workflow stages. Sounds like use case for Kafka, right? Yes, Kafka is widely used to manage such workflows.
While this is happening on the database side, the buzz around “Just use PostgreSQL” is becoming increasingly popular. System designers and architects want to reduce the complexity of maintaining multiple components and dealing with operational issues. More complexity implies more potential failures. Recently, Chandan Shukla wrote a blog post titled PostgreSQL as a Workflow Engine: Building Reliable Long-Running AI Jobs Without Kafka, detailing trade-offs and considerations for building a simple yet reliable workflow engine using simple methods.
Yet, critical questions remain:
Is there a better way to architect database workflows in PostgreSQL?
Can we build a highly scalable system without the performance degradation caused by frequent updates and table bloat?
Is it possible to eliminate the complexity of external components like Kafka in favor of a purely PostgreSQL-centric workflow engine?
Yes, we can. Furthermore, when building modern enterprise applications where data arrives as events that trigger subsequent workflows, I strongly recommend modeling the architecture around those events from the very beginning. . This concept is not new in the PostgreSQL ecosystem, However it is important to periodically highlight wonderful community tools, especially since PostgreSQL projects lack a centralized marketing framework. The original PgQ was created by Marko Kreen as part of Skytools. Largely forgotten for over 20 years—perhaps due to limited documentation—it was brought back into the spotlight by community members like Alexander Kukushkin through talks at various PostgreSQL conferences, such as PostgreSQL queues done right with PgQ – POSETTE 2026.
Later, Nikolay Samokhvalov took on the effort to refresh the framework by modernizing the codebase as PgQue, providing better documentation, and delivering informative talks. PgQue does not require any binary extensions to be installed or added to the system.
Why this implementation is different
A generic queue implementation in PostgreSQL using native SQL features like FOR UPDATE SKIP LOCKED can lead to massive table bloat, concurrency bottlenecks, and performance degradation over a time. In contrast, a queue implementation leveraging PostgreSQL-specific features—such as storing snapshot information and identifying transactions completed between two snapshots—avoids these pitfalls entirely. Instead of updating event status flags in place, events are simply inserted into an append-only log (a partitioned table). This approach transfers the responsibility of tracking the position of consumption to the consumer, which is far cheaper to maintain and resolves the issues mentioned above. No more updates, bloat, or concurrency issues on the queue log! An additional advantage of this approach is that there can be multiple consumers working on the same event log/Queue.
PgQue provides all above mentioned advantages as a ready to use framework, such that noone need to spend time in reinventing all those good learnings and intricacies of PostgreSQL snapshots.
Despite the name, PgQue is not a queue in the usual sense: events are not removed when consumed. Ack simply advances that consumer’s cursor. The model is a shared, append-only log — closer to a Kafka topic, and closest of all to Pulsar, whose subscription model is almost the same. Every registered consumer keeps its own position over the same events and independently sees all of them; the events are stored once, with no copy per subscriber.
Two things Kafka has, PgQue does not. There are no partitions — so no per-key ordering guarantee. And the cursor is not an offset: it points at a tick (a time slice), not at a message, so a consumer commits “through tick 900”, never “through event 4712”. Retention differs too — space is reclaimed by rotating and truncating tables, but only once the slowest consumer has moved past them. The guaranteed rewind window is one rotation period 2 hours by default, and a stalled consumer blocks reclamation rather than losing data. But the PgQue offers queue semantics Kafka leaves to the application: per-consumer redelivery – a failed event is re-appended as a private copy visible only to the consumer that “nack”ed it, a built-in dead-letter queue (DLQ), and competing consumers via the cooperative-consumer API — Pulsar’s Shared subscription, and marked experimental as of writing this blog.
Events are batched, which means that a single fetch request by a consumer can return a list of events in a batch. This significantly reduces back-and-forth network interactions with the server. Once the consumer finishes processing those events, it sends an acknowledgment to the server, which increments the pointer to the next batch. Yes, Please note that acknowledgement is per batch, not per message. So if the processing of a specific message fails, that message needs to be negatively acknowledged, before the batch is acknowledged.
Batch boundaries are marked by a ticker (a clock ticker), which ticks at scheduled intervals—such as every 1/10th of a second or every second. Only transactions completed between the previous batch and the current batch are visible to consumers.
What is new in PgQue
PgQ’s core mechanics—snapshot-based batching, tick generation, TRUNCATE-based table rotation, and per-consumer cursors—are carried over entirely rather than cherry-picked.
What PgQue changes is everything that previously made the engine difficult to deploy: C extensions, external daemons, overly permissive default roles, and the requirement for users to explicitly manage ticks.
pgque.ticker_loop() is a procedure, allowing it to commit between iterations. pg_cron triggers it once per second, and within that second, it ticks at the specified interval. The configuration parameter pgque.config.tick_period_ms sets this rate—100 ms by default, tunable to any exact divisor of 1000 ms.
Some of the main features include:
LISTEN/NOTIFY on every tick: The ticker emits pg_notify(‘pgque_<queue>’, tick_id) after producing a tick, allowing consumers to wait on notifications rather than polling.
Clear and separate roles: PgQue provides distinct security roles for producers and consumers via pgque_writer and pgque_reader, alongside an administrative role named pgque_admin.
Send and receive APIs: Core APIs providing essential queue functionality, including send(), send_batch(), subscribe(), unsubscribe(), receive(), ack(), and nack().
Dead-letter queue: After a sufficient number of failed retry attempts, messages can be moved to a dead-letter queue using dlq_inspect(), dlq_replay(), dlq_replay_all(), dlq_purge(), and event_dead().
Idempotent producers: Deduplicates publishes by key so that retrying producers do not double-publish messages, using send_idem() and maint_idem().
Cooperative consumers: Enables multiple workers to share a single logical consumer cursor, allowing fan-out and work-sharing on the same queue via register_subconsumer(), subscribe_subconsumer(), receive_coop(), and touch_subconsumer().
Management control: Control queue state, scheduler registration, and configuration in a single operation via start(), stop(), status(), version(), set_tick_period_ms(), and uninstall().
Observability: Provides queue and consumer statistics, health scoring, stuck-consumer detection, and an OpenTelemetry-compatible metrics view via queue_stats(), consumer_stats(), queue_health(), otel_metrics(), stuck_consumers(), throughput(), and error_rate().
Delayed delivery: Schedule events for future execution, promoted into the queue when due via send_at() and maint_d
How to use PgQue in applications
An application client can use raw SQL APIs provided by PgQue or leverage official client libraries. As of writing this article, client libraries exist for Python, Go, TypeScript, and Ruby under the “client” directory of the repository. If you are using a programming language other than these four, the existing client libraries can serve as a reference implementation.
I am going to mention only how to use it. But if you are interested in how internally it works, I would recommend this blog and concepts
Producer
In case, you want to test producers and consumers from a psql commandline prompt, the necessary steps are covered with detailed explanations in the tutorial.
The producer simply needs to publish events to the queue using PL/pgSQL functions provided by the PgQue as follows:
where “orders” is the queue name and the JSONB string represents the message payload.
If this is an active queue, no additional steps are needed. However, on lower-activity queues (default 500 events since the last tick), you can manually trigger a tick using the pgque.force_next_tick() function as follows, because ticking is throttled to avoid wastages.
select pgque.force_next_tick('orders');
The background ticker will then assign this event to the next batch.
Consumer
On the consumer side, call the pgque.receive() function:
select * from pgque.receive('orders', 'processor', 100);
In lower-activity queues, consumers may encounter empty batches (batches containing no events). Empty batches require no acknowledgment; calling receive() automatically completes them and advances the consumer cursor. Conversely, non-empty batches must be acknowledged via ack(); otherwise, subsequent calls will repeatedly return the same unacknowledged batch. The standard consumer processing loop follows this pattern:
receive batch ? if messages exist, process and acknowledge ? repeat.
following SQL snippet demonstrates this logic when testing from a psql session:
do $$
declare
v_head bigint;
v_msgs pgque.message[];
m pgque.message;
v_prev bigint;
v_now bigint;
v_pass int := 0;
begin
select last_tick_id into v_head from pgque.get_queue_info('orders');
loop
v_pass := v_pass + 1;
select last_tick into v_prev
from pgque.get_consumer_info('orders', 'processor');
select coalesce(array_agg(r), '{}')
into v_msgs
from pgque.receive('orders', 'processor', 1000) as r;
if array_length(v_msgs, 1) > 0 then
foreach m in array v_msgs loop
raise notice 'msg_id=% type=% payload=%', m.msg_id, m.type, m.payload;
end loop;
perform pgque.ack(v_msgs[1].batch_id);
raise notice 'batch % acked, % messages, pass %',
v_msgs[1].batch_id, array_length(v_msgs, 1), v_pass;
exit; -- drop this line to drain everything
end if;
commit; -- one batch per transaction
select last_tick into v_now
from pgque.get_consumer_info('orders', 'processor');
exit when v_now >= v_head; -- caught up to where we started
exit when v_now = v_prev; -- cursor stalled: no further tick
exit when v_pass > 2000; -- safety guard
end loop;
raise notice 'passes=% last_tick=%', v_pass, v_now;
end $$
Now, consider a common scenario: a consumer receives a batch of events, but while some events process successfully, others fail. How should this be handled?
This is where negative acknowledgments (nack) apply. Any events that fail processing must be negative-acknowledged before acknowledging the entire batch. Here is an example statement to nack message ID 2004 within batch ID 8:
select pgque.nack(8, m, '30 seconds', 'downstream 503')
from pgque.receive('orders', 'processor', 100) as m
where m.msg_id = 2004;
Once all failed messages in a batch have been negative-acknowledged, the consumer can safely acknowledge the batch:
select pgque.ack(8);
Execution order is critical here because calling ack() advances the consumer cursor past the batch window. Negative-acknowledged messages are moved to a retry queue and re-enqueued into the main queue after the specified delay (30 seconds in this example).
If a consumer repeatedly fails to process an event, will it retry indefinitely? No. Similar to Kafka, PgQue implements a Dead Letter Queue (DLQ). Once the configured retry threshold is exceeded, the event moves to the DLQ. You can configure the maximum retry count using:
The zero-bloat behavior pioneered by PgQ has proven effective in production since 2007. PgQue inherits these benefits along with its core architectural strengths: snapshot-based batching, tick generation, TRUNCATE-based table rotation, and consumer cursor tracking. System performance, stability, and scalability heavily depend on solid design. PgQue offers a simple, well-architected framework to implement workflow management entirely inside PostgreSQL, eliminating the need for external tools like Kafka.
As shown in the graph, there are no spikes in the XID horizon or consumer latency. Metric lines for both PgQue and PgQ remain consistently low and near-flat during testing.
Percona Toolkit’s pt-online-schema-change (pt-osc) has long been the preferred solution for performing online schema changes with minimal downtime. Its chunk-based copy algorithm is designed to adapt dynamically to the workload, making it suitable for very large tables in production environments.
However, under specific conditions, one of its optimization mechanisms can become counterproductive.
During a customer engagement involving Percona XtraDB Cluster (PXC), we investigated a case where running pt-online-schema-change with a selective –where clause caused severe Galera Flow Control and general overhead, effectively stalling the cluster for an extended period. The issue was not caused by the schema change itself, but by the interaction between:
adaptive chunk auto-resizing
highly selective copying of the newest rows
clustered replication in Galera
InnoDB checkpointing
This article explains why it happens and proposes a set of practical mitigations that eliminate the root cause.
The Scenario
Consider a large table containing hundreds of millions of rows.
Instead of rebuilding the entire table, we only want to copy a subset of rows and apply a schema change like adding a new index.
Unfortunately, it exposes an interesting corner case in pt-online-schema-change.
How Chunk Auto-Resize Works
During the copy phase, pt-osc continuously adjusts the chunk size in order to keep each chunk copy close to the configured execution time (–chunk-time), 0.5 seconds by default.
Simplified, the algorithm works like this:
if chunks execute too quickly, increase the next chunk size
if chunks become slower, decrease the size
repeat until reaching a stable value
For ordinary table scans this works remarkably well because row density is relatively uniform.
The problem appears when most scanned rows are discarded by –where.
The Hidden Effect of a Selective WHERE Clause
Imagine a table like this with 1 billion rows:
Rows
Match WHERE
990 million
No
10 million
Yes
Suppose the matching rows are located at the end of the clustered index — and that was exactly the case here. Keep in mind that chunk selection always relies on the Primary Key boundaries starting from minimum value, but in this case those boundaries were combined (AND) with the condition passed via –where. Since the created_at values increased in step with the Primary Key, all the matching rows ended up clustered at the end of the index.
For a long time pt-osc scans chunks where almost every row is rejected. The copy operation becomes extremely fast.
The adaptive algorithm interprets this as: “Chunks are too small.” It therefore keeps increasing the chunk size.
Eventually, when the scan reaches the first matching rows, the chunk size has already grown dramatically. Instead of copying a few thousand rows, pt-osc may suddenly copy hundreds of thousands—or even a million—of rows in a single transaction.
Nothing is technically wrong. But Galera now has a completely different problem.
Why Galera Suffers
Every copied chunk is replicated as a write set.
Very large transactions produce:
large write sets
long certification phases
increased apply queues
longer replication latency
Flow Control activation
Once Flow Control starts, cluster throughput decreases dramatically.
Application commits begin waiting.
In extreme situations the cluster appears almost frozen until the oversized transaction has been fully applied by every node.
Ironically, the adaptive algorithm—designed to optimize throughput—ends up creating exactly the workload pattern that Galera dislikes the most.
Why InnoDB Suffers too
As soon as the first large chunks began processing, the following messages appeared in the PXC writer node’s error log:
[Warning] [MY-014084] [InnoDB] Threads are unable to reserve space in redo log which can't be reclaimed
due to the 'log_checkpointer' consumer still lagging behind at LSN = 852886039088.
Consider increasing innodb_redo_log_capacity.
This indicates that InnoDB received a very large transaction, and the writing threads could not find enough free space in the redo logs because the log_checkpointer consumer was falling behind. In practice, this forces the server to trigger checkpoints more frequently and on a larger scale, which increases overall overhead.
After Galera Replication propagated the write set to the other nodes, the same type of message was observed on them as well, during the apply phase of the replicated transaction.
Symptoms
Typical symptoms include:
Flow Control percentage rapidly increasing
Growing wsrep_local_recv_queue
High replication lag between nodes
Application commits waiting
CPU utilization increasing
Large spikes in transaction apply time
More frequent InnoDB checkpoints on all the nodes
The schema change itself may still complete successfully, but the impact on production traffic can be severe. In the worst scenario this can lead to a complete stall of the cluster due to excessive flow control. This actually happened in production, and the cluster had to be restarted after remaining locked for dozens of minutes.
The Proposed Fixes
Here are a few options, starting with the simplest and most effective one: disable adaptive chunk resizing whenever a selective –where clause is used.
Instead of continuously increasing chunk sizes, keep them fixed throughout the operation.
Simply include the following options in the command:
--chunk-time = 0
--max-flow-ctl = 0
If –chunk-time option is set to zero, the chunk size doesn’t auto-adjust, so query times will vary, but query chunk sizes will not. Another way to do the same thing is to specify a value for –chunk-size explicitly, instead of leaving it at the default, and omit the option –chunk-time.
The adaptive algorithm is bypassed. Chunk execution becomes predictable. Transaction size remains bounded. Galera never receives unexpectedly massive write sets. The overall schema change may take slightly longer, but cluster stability improves dramatically. For production environments this is usually the preferable trade-off.
Another parameter you should consider is –max-flow-ctl.
With –max-flow-ctl=PCT, pt-osc periodically checks wsrep_flow_control_paused (the percentage of time the node has spent paused for flow control) and automatically pauses row copying whenever it exceeds the threshold, resuming once the cluster drops back below it.
Setting –-max-flow-ctl to zero (or very close to it, e.g. 1) makes pt-online-schema-change stop copying rows at the very first sign that any node has entered flow control, rather than tolerating some pause time before backing off — which is the safest possible stance for a production cluster, especially one that’s already showing signs of stress (like the checkpointer-lag warnings, where large transactions were pushing InnoDB’s redo log reclamation to its limit): any additional write pressure from the migration’s chunked copy could tip an already struggling node into a much bigger stall, so a near-zero threshold trades migration speed for safety, letting the tool fully defer to the cluster’s real-time capacity and only advance when there’s genuinely no contention, which is generally the right call for critical, latency-sensitive production workloads where a slower-but-safe schema change beats a fast one that risks a cluster-wide write freeze.
A related lever worth mentioning is –chunk-index. By default, pt-online-schema-change chunks the table using the primary key (or the most appropriate unique index it can find), which is precisely why, in our case, chunk boundaries tracked the same monotonically increasing column as created_at and produced the long stretch of near-empty chunks described above. If the table has a secondary index whose ordering is not correlated with created_at, forcing the tool to chunk on that index instead (–chunk-index=idx_name) can spread matching and non-matching rows more evenly across chunks, so the adaptive algorithm never sees the extended low-density run that drives chunk size to unsafe levels in the first place. That said, this is not a substitute for –chunk-time=0 and –max-flow-ctl: pt-osc’s own documentation warns that a poorly chosen chunking index can hurt performance, since it forces query plans, and few tables have a convenient index that is both selective enough to chunk on and uncorrelated with the –where condition. In practice, we recommend treating –chunk-index as a secondary, table-specific optimization, with fixed chunk sizes and flow-control awareness remaining the primary safeguard.
A third option is to avoid the –where clause altogether and split the task into two phases instead. First, run pt-archiver solely to delete the rows you no longer need; then run pt-online-schema-change to alter the table. This two-step procedure avoids any risk stemming from the non-uniform distribution of the matching rows.
Conclusion
pt-online-schema-change remains one of the best tools available for online schema migrations.
Nevertheless, when using –where to copy only a small subset of rows on Percona XtraDB Cluster, adaptive chunk resizing can unintentionally generate oversized transactions that trigger excessive Galera Flow Control and temporarily stall the cluster.
Keeping chunk sizes fixed eliminates this feedback loop, producing smoother replication behaviour and reducing the operational risk of online schema changes. Other fixes are also available.
Sometimes, predictability is a better optimization than aggressiveness.
Today we are talking about GovHub, Drupal in Government, and Why Governments Love Drupal with guest Jasmyne Epps. We’ll also cover Convivial Gov Site Template as our module of the week.
Have you ever wanted to stand up a polished, accessible government website in Drupal (with components, content types, SEO, and cookie consent all wired up) without writing any code? There’s a site template for that.
Like the Haven site template we talked about a couple of weeks ago, Convivial Gov gives you a curated stack plus demo content, and in this case hands you a robust, ready-to-customize government site.
Because it’s built on Drupal CMS, you get all the latest Drupal tooling: Canvas for visual page building, Single Directory Components, and Recipes.
The front end is Morpht’s Morphos theme, built on Tailwind and DaisyUI, so you get dark mode, multiple colour palettes, and a big library of editor-friendly components out of the box. It’s worth mentioning that using the Morphos theme on a production site requires a paid license
The provided components are sorted into six buckets: container, content, child, element, background, and behavior. They include fun ones like scroll reveal and a colour palette switching behavior
The content model is broad. You get seven content types: Page, Section, Article, Publication, Resource, Topic, and Audience. And, they come with a stack of teaser and card view modes to display them.
The whole point is no-code: a site builder can compose sophisticated pages in Canvas without ever touching a template.
One thing to watch: the default timezone is Australia/Sydney out of the box
It’s also worth comparing Convivial Gov to another site template called Local. Both dropped in March 2026, both are Canvas-based Drupal CMS site templates for the public sector, and both lean on ECA for automation — so there’s real common ground. The difference is scope and mechanism. Local, from Annertech, is narrowly purpose-built for local councils and community-service directories: it ships a specific service information architecture — Service and Service Landing content types — with ECA wired so section pages stay in sync when service pages get published or updated, taking its cues from the gov.uk design system. Convivial Gov goes the other way — it’s design-system-led and general-purpose, a broad component library and content model meant for any government, agency, or marketing site rather than one particular workflow.
Percona Server for MySQL 9.7.2-2 now supports DISTANCE() for vector similarity scoring directly in SQL (COSINE, EUCLIDEAN, MANHATTAN, DOT metrics). This is the compute primitive you need to rank or filter embeddings by similarity directly in SQL. ANN indexing (e.g. HNSW, IVF) is the next milestone for fast large-scale similarity search; and this function provides the scoring layer that indexing strategies will further accelerate.
Why we’re adding this
MySQL’s native DISTANCE() and VECTOR_DISTANCE() functions are only available in HeatWave MySQL on OCI, not included in Community or Commercial MySQL, and limited to three metrics (COSINE, DOT, EUCLIDEAN). Percona is bringing the same capability to anyone running Percona Server for MySQL on any supported platform.
MySQL 9.7 already supports the VECTOR data type (TO_VECTOR() and FROM_VECTOR() functions) for storing embeddings. DISTANCE() is the natural next step: it lets you query by similarity directly in SQL, ranking or filtering rows based on vector distance, without leaving MySQL.
Background on Vectors & Distance Metrics
What’s a vector?
A fixed-length list of numbers, an “embedding” produced by an ML model to represent something (text, an image, a product, a user preference). The key insight: “similar” things end up with numerically close vectors, so you can compute their distance to rank or filter them.
What’s a distance metric?
A mathematical function that turns two vectors into a single number describing similarity. Here are the five metrics this release supports:
EUCLIDEAN (L2): Straight-line distance, the intuitive notion of geometric distance. General-purpose; works well when magnitude matters.
EUCLIDEAN_SQUARED: Same ranking as EUCLIDEAN, but skips the square root. Faster for comparisons and ORDER BY clauses when you only care about ordering, not the actual value.
MANHATTAN (L1): Grid-style distance (like taxicab distance on a city grid). More robust to outliers than Euclidean in some applications.
COSINE similarity: Measures the angle between two vectors. Cosine distance is computed as 1 minus the cosine similarity so that smaller value = more similar.
DOT (inner product): Raw dot product sum(a[i] * b[i]). Used by models trained specifically for dot-product similarity (e.g., some matrix-factorization systems). Normally higher values mean more similar but since it is common in the industry to multiply by -1 and flip the scale: now smaller values = more similar, in line with the other metrics above.
What’s SIMD?
Single Instruction, Multiple Data: a CPU capability that performs the same arithmetic on many numbers at once instead of one at a time. Distance math over hundreds of dimensions (embedding size) is exactly the kind of repetitive arithmetic that SIMD accelerates. The Percona implementation automatically detects and uses the best SIMD tier available on your hardware at startup, no recompilation needed per target architecture.
Meet DISTANCE() / VECTOR_DISTANCE()
Function signature:
DISTANCE(vector1, vector2, metric)
Where: – vector1 and vector2 are VECTOR data type or binary string literals (via TO_VECTOR()). – metric is a fixed literal string (case-insensitive, not a column nor an expression): ‘EUCLIDEAN’, ‘EUCLIDEAN_SQUARED’, ‘MANHATTAN’, ‘COSINE’, or ‘DOT’. – Returns a DOUBLE representing the distance/similarity score.
Synonym:VECTOR_DISTANCE() is an alias with identical behavior.
Choosing a metric: Match the metric to how your embedding model was trained. Most modern embedding models (OpenAI, Cohere, etc.) are trained with COSINE similarity, so use ‘COSINE’. If magnitude carries meaning (e.g., you’re working with raw feature vectors, not normalized embeddings), use ‘EUCLIDEAN’ or ‘EUCLIDEAN_SQUARED’. When in doubt, check your embedding model’s documentation.
All dimensions must match: Both vectors must have the same dimensionality, or the function returns an error.
If either input is NULL, the result is NULL.
Getting started: Step-by-step example
1. Create a table with vector embeddings
CREATE TABLE products (
id INT PRIMARY KEY,
name VARCHAR(255),
embedding VECTOR(1536) -- Example: 1536-dimensional embedding
);
Find the top-5 products most similar to a query embedding:
SELECT id, name, DISTANCE(embedding, TO_VECTOR('[0.12, 0.22, 0.32, ..., 0.382]'), 'EUCLIDEAN') AS similarity_score
FROM products
ORDER BY similarity_score
LIMIT 5;
Results ordered by highest similarity first (best/closest matches first) :
id | name | similarity_score
---|------------|------------------
2 | Product B | 0.5432109
1 | Product A | 0.9654321
3 | Product C | 0.9876543
Under the hood: SIMD dispatch that adapts to your hardware
The engineering story behind the performance claim: instead of compiling the binary once for a specific CPU (with -march=native), Percona’s implementation detects the CPU’s capabilities at startup and selects the best available SIMD tier:
SSE4.2 (x86_64) or NEON (aarch64): 128-bit SIMD, ~2–3× speedup over scalar.
AVX2 (x86_64): 256-bit SIMD, ~4–6× speedup.
AVX-512F (x86_64): 512-bit SIMD, ~8–12× speedup on CPUs that support it.
SVE2 (aarch64): scalable SIMD on ARM, adapts to the CPU’s vector width.
Scalar: unoptimized fallback, works everywhere.
Dimension-aware kernel selection: Distance calculations on small vectors (<16 dimensions) uses the narrower 128-bit tier, because the overhead of setting up larger SIMD registers outweighs the benefit. Larger vectors automatically use the widest available tier.
Unaligned loads by design:VECTOR column data isn’t guaranteed to be cache-line aligned (and shouldn’t require alignment). All SIMD kernels use unaligned-load intrinsics as modern CPUs have identical throughput for aligned and unaligned loads when data is in cache. By always using unaligned loads, we avoid faulting on misaligned input without sacrificing performance.
Current limitations and what’s coming next
DISTANCE() is a scalar function that computes the distance between two vectors and returns a single number. Without an approximate-nearest-neighbor (ANN) index, a query like ORDER BY DISTANCE(…) LIMIT k over a large table is a full table scan: you call the distance function on every row, then sort. It’s correct and SIMD-accelerated per row, but it scales as O(n), not O(log n).
The natural next step: ANN indexing. To make large-scale similarity search fast (e.g., finding the 10 nearest neighbors in a table of 1 million vectors in milliseconds), you need an approximate-nearest-neighbor index: HNSW (Hierarchical Navigable Small World), IVF (Inverted File with refinement), or similar. These are graph-based or clustering-based structures that prune the search space and return approximate results much faster. This is the direction the vector feature is building toward, and it’s on the roadmap. For now, DISTANCE() is the scoring primitive that those indexing strategies will accelerate.
While ANN indexing delivers speed, it relies on approximate results. Should your application require exact precision instead of estimates, the DISTANCE() function is available in Percona Server for MySQL 9.7.2-2.
We want your feedback
Try it out and let us know what you think: – Report bugs or feature ideas onJIRA. – Join the conversation on Percona community forum. – Questions about usage or performance? Reach out to us.
Your feedback shapes the roadmap, especially use cases you’d like to see (e.g., specific ANN index strategies, embedding model integrations, performance tuning for your workload).
Written by Catalin Besleaga. Reviewed by Dennis Kittrell and Peter Zaitsev. Percona® is a registered trademark of Percona LLC. MySQL® is a registered trademark of Oracle Corporation.
This week’s module is Formdazzle, a developer tool that makes theming Drupal forms easier.
Drupal’s Form API is a powerful abstraction, but when you want to target one specific field, label, button, or form wrapper, the default Twig template suggestions can be limited.
The module works by taking information Drupal already knows about the form, like the form ID, element type, and element name, and using that to generate more targeted Twig template suggestions.
For example, in a Drupal View with exposed filters, you may want to style the Reset button differently from the Submit button. By default, Drupal renders both buttons through the same input–submit.html.twig template, which makes it difficult to customize them independently. This module lets you assign different templates to individual form buttons—such as Submit, Reset, or Filter—based on their action, type, and other properties.
This module has no configuration. Just enable the module and it starts working and look at the twig debug comments including extra template suggestions.
Large language models are increasingly capable of performing practical systems-administration tasks. I wanted to understand how well they could handle something more specialized: database administration.
To explore this, I developed a harness for evaluating the ability of different LLMs to execute real DBA tasks on remote systems.
My primary focus is on open-weight models because of their deployment flexibility and cost efficiency. Their token costs are often substantially lower than those of closed frontier models, which makes them especially interesting for automation workloads where an agent may require dozens of interactions to complete a task.
Models Tested
The models I am currently evaluating fall into two broad categories.
Heavy, Full-Size Models
qwen/qwen3.8-max
moonshotai/kimi-k3
z-ai/glm-5.3
deepseek/deepseek-v4-pro-0813
Lighter or Flash Models
deepseek/deepseek-v4-flash-0731
qwen/qwen3.8-27b
meta/muse-glimmer-30b
google/gemma-4-31b-it
I use OpenRouter to manage requests and responses across the different models.
The harness receives a task, for example:Install Percona Server for MySQL 8.4
It sends the task to the selected LLM and asks the model to translate the request into command-line operations or Bash scripts.
The harness executes those commands over SSH on the target system.
Command output, errors, and system state are sent back to the LLM.
The model evaluates the result and generates the next commands.
The process continues until one of several conditions is reached:
The task is completed.
A timeout occurs.
The maximum number of steps is reached.
The configured cost limit is reached.
Harness Workflow
The DBA AI benchmark harness converts natural-language DBA tasks into commands, executes them remotely over SSH, returns the results to the LLM, and continues iterating until the task completes or a configured limit is reached.
Example
Here is an example of running the harness against two hosts:
<code class="language-bash">uv run dba.py \
-m qwen/qwen3.8-max \
--host 165.22.191.129 \
--host 68.183.121.153 \
--task "Install MySQL 9.7 Replication with encrypted traffic" \
--max-steps 200 \
--mode unattended
</code>
The model is responsible for deciding how to interpret the request, what commands to execute, how to react to errors, and when the task has been completed.
Some Interesting Findings
So far, almost all models have successfully executed the tasks, with only a few individual exceptions.
One of the more interesting findings is that smaller and cheaper models are often just as capable of completing the task as much larger models.
The main difference is usually the number of steps.
Smaller models may require more iterations to arrive at the correct configuration, but the overall cost can still be dramatically lower.
In several tests, smaller models completed the same work at roughly one-tenth of the cost of larger models.
One particularly interesting behavior appears when the requested task cannot be completed exactly as specified.
For example, I asked deepseek/deepseek-v4-pro-0813 to:
Install MySQL 9.7
The model responded:
“MySQL 8.4 is available (not 9.7 as requested, but 8.4 is what Ubuntu provides; will use it). Installing MySQL server on node1 (primary).”
Instead of simply failing, the model interpreted the situation, selected an available version, and continued with the task.
Whether this behavior is desirable depends on the automation scenario, but it demonstrates an important characteristic of LLM-driven infrastructure automation: the model may make operational decisions rather than blindly execute instructions.
That makes validation and guardrails particularly important.
Batch Runner for Comparing Models
Running one model at a time is useful during development, but it becomes inconvenient when comparing many models and configurations.
One test asks the models to perform the following task:
These two servers are identical and freshly built. Install Percona Server 8.4 on both and set up replication between them: one primary taking writes, one replica following it.
Decide which server takes which role and say which you chose. Replication traffic must go over the private network the two servers share, not over the public internet, and the replica must refuse writes of its own.
meta/muse-glimmer-30b required 74 steps but cost only $0.0479.
qwen/qwen3.8-max completed the task in 22 steps but cost $0.4200.
deepseek/deepseek-v4-flash-0731 completed the task in 35 steps for only $0.0082.
This illustrates an important point for agentic infrastructure workloads:
The model that uses the fewest steps is not necessarily the most economical model.
A smaller model can require significantly more iterations while still being dramatically cheaper overall.
Handling Impossible Tasks
Another useful test is to deliberately give the models an impossible request.
For example:
Install Percona Server for MySQL 9.7.2.
That release does not exist.
The responses were interesting.
Most models decided to install the closest available version, 9.7.1, instead.
DeepSeek behaved differently and completed the run without performing an installation.
These cases are useful because they test more than a model’s ability to generate shell commands. They expose how the model reasons about ambiguity, unavailable software, conflicting requirements, and whether it should modify the user’s intent.
For production automation, those behaviors may be just as important as raw task-completion rates.
More Than MySQL
The examples above focus primarily on MySQL and Percona Server, but the framework is not limited to MySQL.
The same approach can be used for:
MySQL
Percona Server for MySQL
PostgreSQL
MongoDB
Other database and Linux administration tasks
In practice, the harness is testing whether an LLM can act as an iterative systems operator:
plan ? execute ? observe ? adjust ? verify
What I Am Learning
The early results suggest several things.
First, modern LLMs are already surprisingly capable of performing multi-step database administration tasks when they have access to command execution and system feedback.
Second, model size is not necessarily the best predictor of practical usefulness.
A smaller model that requires twice as many steps may still be significantly more economical than a larger model.
Third, the ability of models to exercise judgment introduces both opportunities and risks. A model that substitutes an unavailable software version may be helpful in one situation and unacceptable in another.
This means future evaluation should look beyond simple task completion.
Useful dimensions include:
Task completion rate
Number of steps
Total inference cost
Execution time
Correctness of the final configuration
Number of unnecessary operations
Recovery from command failures
Handling of ambiguous requirements
Handling of impossible requirements
Security of generated configurations
Whether the final system actually matches the user’s intent
I expect these characteristics to become increasingly important as LLMs move from answering DBA questions to actually operating database infrastructure.
There are many interesting areas to explore, including additional database platforms, more complex failure scenarios, security evaluation, cost optimization, model comparison, and better automated scoring.
If you are interested in cooperating, testing additional models, adding DBA workloads, or improving the benchmark framework, please let me know.
A while ago, we discussed the cross-site replication feature of the Percona PXC operator. Recently, a similar cross-site replication feature was introduced in the Percona (PS MySQL) operator v1.2.0, a topology based on Group Replication/InnoDB Cluster.
In this blog post, we will explore how to add a DR Cluster to an existing DC Cluster to form a ClusterSet environment, which provides a seamless switchover between DC/DR members. The good part is that all the complexity and configuration will be managed by the Percona operator, with very few setup steps required.
Group Replication (DC-DR) using Innodb ClusterSet
Let’s discuss how we can implement a cross-site replication (DC-DR) topology across two separate database clusters.
Environment used for Demonstration
Two separate database clusters (pscluster1 and pscluster2) are deployed within a single GCP/Kubernetes environment, each managed by a dedicated operator.
Percona Operator for MySQL( based on Percona Server for MySQL) has a version. 1.2.0
shell> kubectl get ps -n ps
NAME REPLICATION ENDPOINT STATE MYSQL ORCHESTRATOR HAPROXY ROUTER AGE
ps-cluster1 group-replication ps-cluster1-haproxy.ps ready 3 3 28m
Step 2: Retrieve the DC’s InnoDB cluster name, which will be required later to construct the ClusterSet.
shell> kubectl get ps ps-cluster1 -n ps -o jsonpath='{.status.innodbClusterName}{"\n"}'
pscluster1
Step 3: Get a service endpoint which is used when setting up the ClusterSet environment. We will use ps-cluster1-mysql-primary, which is mapped to the current Primary node in the existing cluster.
Step 4: The DR site should have the same credentials as DC. We need to export the DC secret containing the credentials and import it into the DR cluster.
Note: We set spec.mysql.bootstrap.mode to manual so Pod mysql-0 does not form a Group Replication group until the ClusterSet adopts it and references the secret file we copied from the DC.
It is also expected that after applying the Custom Resource file cr.yaml, the Pod mysql-0 starts but stays NotReady. The cluster ps-cluster2 reports the Initialising state and the AwaitingExternalBootstrap condition in status. Pods mysql-1 and mysql-2 do not start until Pod mysql-0 joins the Group Replication.
Once we deploy the custom resource, we can notice the status below. Please note that the complete Pods will be ready only once DR successfully syncs and joins the DC cluster.
shell> kubectl get ps -n ps-dr
NAME REPLICATION ENDPOINT STATE MYSQL ORCHESTRATOR HAPROXY ROUTER AGE
ps-cluster2 group-replication ps-cluster2-haproxy.ps-dr initializing 17m
shell> kubectl get pods -n ps-dr
NAME READY STATUS RESTARTS AGE
percona-server-mysql-operator-6b78d5f68c-4gsm9 1/1 Running 0 28m
ps-cluster2-mysql-0
Step 2: We need to note down the DR InnoDB cluster name, which will later be used by the ClusterSet.
shell> kubectl get ps ps-cluster2 -n ps -o jsonpath='{.status.innodbClusterName}{"\n"}'
pscluster2
Step 3: We need to identify the DR cluster endpoint that is reachable from the DC. This endpoint will be used to feed the clone and perform other management operations.
We will use the local FQDN ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local of the ps-cluster2-mysql-0 Pod, which is currently initialised and awaiting the ready state.
shell> kubectl get pods -n ps-dr
NAME READY STATUS RESTARTS AGE
percona-server-mysql-operator-6b78d5f68c-4gsm9 1/1 Running 0 48m
ps-cluster2-mysql-0 1/2 Running 0 37m
By connecting to the DC Primary Pod, we can verify cross-communication between the DC and the DR.
shell> kubectl exec -it ps-cluster1-mysql-0 -n ps -- sh
sh-5.1$ curl -v ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local:3306
* Trying 10.60.1.7:3306...
* Connected to ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local (10.60.1.7) port 3306 (#0)
> GET / HTTP/1.1
> Host: ps-cluster2-mysql-0.ps-cluster2-mysql.ps-dr.svc.cluster.local:3306
> User-Agent: curl/7.76.1
> Accept: */*
Data Restoration Part
By default, when the DR site initialises and attempts to join the ClusterSet, it receives data from the source/DC via the clone recovery method.
The Operator uses MySQL Shell to create a physical snapshot of the dataset from the Source/DC and transfer it to the DR Replica.
Alternatively, we can manually perform a backup and restore on the target cluster. This strategy helps when integrating the DR site into the ClusterSet, as pre-populating the data and GTID history allows it to apply only delta changes rather than performing a full initial synchronisation. For large datasets or to prevent excessive load and performance degradation on the donor node, a backup-and-restore approach can be considered.
Here, we will use the clone recovery method to sync the DR cluster.
Next, we will deploy the ClusterSet configurations so that DR joins the DC cluster via a data clone and connects via Asynchronous Replication to stay in sync.
Below is the clusterset.yaml file where we pass the various information like (Secrets, DC/DR Endpoints, Cluster Name etc) , which we fetched in some of the above steps earlier.
Please note: Member “ps-cluster2-mysql-0” on the DR side will connect to the DC via asynchronous replication. While the local members of the DR will join via the Group Replication mechanism.
This action will also start a separate backend job on DC, which creates an associated Pod to perform the ClusterSet activity.
shell> kubectl get jobs -n ps
NAME STATUS COMPLETIONS DURATION AGE
my-cluster-set-pscluster2-add-replica Complete 1/1 23s 53s
….
shell> kubectl get pods -n ps
NAME READY STATUS RESTARTS AGE
my-cluster-set-pscluster2-add-replica-wk7xd 1/1 Running 0 3s
As soon as the process completes successfully, the job and associated Pod will be removed from the list.
There is one extra pod that can be noticed, and it persists. This provides a utility/client for running MySQL Shell commands against the MySQL ClusterSet.
shell> kubectl get pods -n ps
NAME READY STATUS RESTARTS AGE
my-cluster-set-runner-5d68fbcc74-svmpw 1/1 Running 0 10h
If the job fails or errors occur, we need to investigate the exact problem using the information below.
Once the switchover finishes successfully, all new writes now go to the new Primary cluster ps-cluster2 and the old DC “ps-cluster1” will automatically become the Async Replica. The application should connect with a load balancer (Haproxy/MySQL Router) endpoint, e.g., (ps-cluster2-haproxy), which will forward all requests to the backend node (by default) to the Primary member.
shell> kubectl get svc ps-cluster2-haproxy -n ps-dr
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ps-cluster2-haproxy ClusterIP 34.118.226.22 <none> 3306/TCP,3307/TCP,3309/TCP,33060/TCP,33062/TCP 12h
Key Takeaways
Configuring and managing a multi-region topology manually or on virtual machines can be quite challenging. By packaging these components within the Percona PS Operator/K8S, setting up cross-region environments becomes far simpler. This streamlined deployment is critical for establishing robust disaster recovery solutions, offloading production workloads, or scaling read operations. Furthermore, built-in support for seamless DC-DR switchovers and ad hoc failovers adds significant value to the architecture.
Before deploying these topologies into production environments, it is strongly advised to thoroughly test and validate their behaviour in lower or non-production environments. Proceed with production deployment only after achieving full confidence in the setup.
A few thousand gcache.page.* files in a Percona XtraDB Cluster (PXC) data directory is not something you see every day. We came across a case where these files had been accumulating over time and slowly consuming disk space. So, let’s dig into what happened.
At first glance, it looked like GCache had simply stopped cleaning itself up. The investigation started by answering two simple questions: when did the files start appearing, and what changed in the cluster at that time?
Finding the starting point
The oldest files showed the issue started on July 9.
[hostx] percona@hostx: ~ $ ls -lh /var/lib/mysql/mysql-data/gcache.page.*
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000000
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000001
-rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000002
...
-rw-r----- 1 mysql mysql 128M Jul 26 07:45 /var/lib/mysql/mysql-data/gcache.page.005976
-rw-r----- 1 mysql mysql 128M Jul 26 07:50 /var/lib/mysql/mysql-data/gcache.page.005977
-rw-r----- 1 mysql mysql 128M Jul 26 07:56 /var/lib/mysql/mysql-data/gcache.page.005978
The newest files showed they stopped being created on July 26, which immediately provided a timeline to investigate.
The creation of GCache page files wasn’t random. It started at a specific point in time and stopped after the next MySQL restart.
Can large transactions cause this?
Normally, you don’t see thousands of GCache page files unless Galera cannot reclaim old pages or an exceptionally large writeset forces additional page allocation. In this environment, the GCache ring file was around 60 GB, making the large writeset theory very unlikely – actually Impossible! Because there’s a hard limit of the largest transaction size at 2GB.
That pushed the investigation toward the error log, where Galera was found logging:
2026-07-09T21:41:04.617132Z 91699 [Note] [MY-000000] [Galera] Freezing gcache purge at 16198595494
2026-07-09T21:41:04.622051Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/mysql-data/gcache.page.000000 of size 134217728 bytes
This was the first strong clue. When gcache.freeze_purge_at_seqno is active, Galera stops reclaiming old GCache pages. As replication continues, new page files are allocated while existing ones remain on disk.
Correlating with cluster activity
Looking a few seconds later in the error log revealed a cluster partition. The timing is difficult to ignore: Galera froze GCache purging, created the first page file, and then recorded the membership change.
2026-07-09T21:41:04.617132Z 91699 [Note] [MY-000000] [Galera] Freezing gcache purge at 16198595494
2026-07-09T21:41:04.622051Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/mysql-data/gcache.page.000000 of size 134217728 bytes
2026-07-09T21:41:14.537019Z 53033 [Note] [MY-010559] [Repl] Multi-threaded replica statistics for channel '': seconds elapsed = 122; events assigned = 110401704; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 7659954362300 waited (count) when Workers occupied = 47189196 waited when Workers occupied = 22964040282100
2026-07-09T21:42:02.139966Z 91770 [Note] [MY-010914] [Server] Aborted connection 91770 to db: 'unconnected' user: 'percona' host: '10.1.163.3' (Got an error reading communication packets).
2026-07-09T21:42:17.061513Z 0 [Note] [MY-000000] [Galera] forgetting c3acab8a-8a74 (ssl://10.1.21.5:4567)
2026-07-09T21:42:17.061596Z 0 [Note] [MY-000000] [Galera] Node 7e3c8f1e-ad75 state primary
2026-07-09T21:42:17.061615Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,7e3c8f1e-ad75,11)
memb {
7e3c8f1e-ad75,1
}
joined {
}
left {
}
partitioned {
c3acab8a-8a74,1
}
)
Although the logs do not explicitly state why purge was frozen, the sequence of events strongly suggests that Galera retained the writesets so the partitioned node could potentially perform an IST when it rejoined.
How could the purge have been frozen?
Further investigation into the codebase and documentation hinted that it is practically impossible that Galera can actually invoke the gcache pages purge pause and only practical way to do it is using:
SET GLOBAL wsrep_provider_options='gcache.freeze_purge_at_seqno=XYZ';
Rather than stopping with a theory, Peter Sylvester (SoS) reproduced the behavior in the lab by manually setting gcache.freeze_purge_at_seqno and generating workload with Sysbench. The result matched what we observed in the production.
Additionally, even though gcache.keep_pages_count=3, Galera continued creating additional page files because purging was frozen.
mysql> show global status like 'wsrep_last_committed';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| wsrep_last_committed | 94559 |
+----------------------+-------+
1 row in set (0.00 sec)
mysql> SET GLOBAL wsrep_provider_options="gcache.freeze_purge_at_seqno=94559";
Query OK, 0 rows affected (0.00 sec)
Sysbench was then run on the cluster’s source host to generate logs…
[root@CENTOS9-1 ~]# sysbench oltp_read_write --db-driver=mysql --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --mysql-port=3306 --table_size=1000 --tables=4 --threads=4 --rand-type=uniform --range_size=100 --time=0 --rate=0 --report_interval=5 run
WARNING: Both event and time limits are disabled, running an endless test
sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 4
Report intermediate results every 5 second(s)
Initializing random number generator from current time
Initializing worker threads...
Threads started!
[ 5s ] thds: 4 tps: 311.80 qps: 6246.62 (r/w/o: 4373.81/1248.40/624.40) lat (ms,95%): 16.71 err/s: 0.00 reconn/s: 0.00
Following page files are present in datadir
[root@CENTOS9-3 ~]# ls -lh /var/lib/mysql/*cache*
-rw-r----- 1 mysql mysql 11M Aug 6 14:14 /var/lib/mysql/galera.cache
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000000
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000001
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000002
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000003
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000004
-rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000005
-rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000006
After the load completed, the mysqld was restarted to observe if the page files were then cleared! But they were not.
[root@CENTOS9-3 ~]# systemctl stop mysql
[root@CENTOS9-3 ~]# systemctl start mysql
[root@CENTOS9-3 ~]# ls -lh /var/lib/mysql/*cache*
-rw-r----- 1 mysql mysql 11M Aug 6 14:17 /var/lib/mysql/galera.cache
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000000
-rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000001
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000002
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000003
-rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000004
-rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000005
-rw-r----- 1 mysql mysql 10M Aug 6 14:17 /var/lib/mysql/gcache.page.000006
Another interesting observation
Later the MySQL was restarted, expecting the startup to reclaim the unused pages. It didn’t. Every page file remained on disk after a restart. Even after the explicit configuration of gcache.freeze_purge_at_seqno=-1.
This suggests that startup recovery does not automatically remove these accumulated page files simply because gcache.freeze_purge_at_seqno has been cleared. At least in testing, once purge has been frozen and page files accumulate, restarting MySQL alone is not enough to reclaim the space.
We have a new bug in place for getting this behaviour sorted: PXC-5323
Can the files be deleted?
To answer that, MySQL was stopped, the local GCache files were removed and the node was started again.
Galera recreated the required cache structures automatically. More importantly, the node successfully completed an IST; and in our tests, deleting the local page files did not force an SST.
Before removing the files, make sure the node is stopped and the cluster has another healthy node that can provide the required writesets. Always validate this behavior in your own environment before using it operationally. Our testing showed that an IST was sufficient.
Conclusion
Based on both the production logs and our lab testing, the accumulation of gcache.page.* files was caused by GCache purging being frozen. Restarting MySQL did not reclaim the accumulated files in our testing. The behavior where a PXC node continuously creates new GCache page files is already fixed under: PXC-4495.
The practical workaround is to stop MySQL, remove the local galera.cache and gcache.page.* files, and start the node again. In our testing, the node rejoined the cluster using IST without requiring a full SST. As always, make sure another healthy node has the required writesets before performing this cleanup.
