Jun
30
2026
--

Community Docker Images: keeping the operator open without a vendor registry lock-in

PostgreSQL community images address a real gap in how a Kubernetes database operator earns your trust. Running a database operator on Kubernetes means trusting two things: the code, and the container images the operator pulls. The code is on GitHub, easy to inspect, easy to fork. The container images, the registry that hosts them, and the license that governs them all sit with the vendor, and any of those three can change without the source repository changing at all. Starting with Percona Operator for PostgreSQL 3.0.0, you can run the operator against community images you build yourself from the official PostgreSQL packages on download.postgresql.org, in a registry you control.

 

TL;DR

  • Community Docker Images: tech preview in PGO 3.0.0, official in 3.1.0. Point the operator at upstream-built PostgreSQL images instead of the Percona Distribution images.
  • Build them yourself from the official PostgreSQL source. The Dockerfiles pull packages from download.postgresql.org (the PGDG repositories), so the trust chain runs from PGDG to your registry with no vendor in the middle.
  • There are limits. Anything Percona-specific (TDE in our distribution build, for example) does not exist in an upstream-built image. That trade is intentional.

In this post:

  • How open source gets diluted in practice
  • Why distributions exist anyway, honestly
  • How Community Docker Images work
  • Limits of the upstream path
  • What to try, what to tell us

 

 

How open source gets diluted

Open source has changed in the last few years, and not always for the better. Companies have learned that you can keep a project’s source code fully open and still capture most of the lock-in by quietly closing the parts that matter in production: the release artifacts, the container images, the supported OS list, the certified Kubernetes distributions, the marketplace listings.

 

Same project, closed artifacts

You can have a fully community CNCF project that does not appear on the Red Hat Marketplace except as a paid Enterprise edition. Similarly, you can have a vendor that ships one packaging in the community and a richer one in Enterprise with the features you actually need in production. The license still says “open source.” The practical experience says “you depend on us.” And the source repository’s license is not the only license that matters here: a vendor can change the license, the trademark policy, or the distribution terms on the container images alone, while leaving the source repository untouched. That has happened in the PostgreSQL operator space recently, and the community noticed.

 

Why the community is right to be wary

Nobody outside the vendor can predict when a license will change, when a feature will move behind a paywall, or when an external contribution will get rejected because it competes with an Enterprise feature. Recent history has plenty of examples and the PostgreSQL community has been paying attention. When this community resists vendor-controlled distributions, it is not nostalgia. It is a rational read of where things have gone before.

I work on Percona’s PostgreSQL operator, so I see this conversation from the vendor side. The skepticism is fair. The honest question for us is what to do about it.
 

Why distributions exist anyway

Acknowledging the community’s concerns does not mean distributions are pointless. There are real reasons to ship one, and pretending otherwise makes for bad blog posts.
 

What a distribution buys you

A vendor-built distribution lets the vendor:

  1. Control the build process, dependencies, and defaults so they fit a specific user shape.
  2. Ship hotfixes faster, because the whole release path sits in one place.
  3. Fork PostgreSQL itself when something the upstream community will not accept, or can take years to accept, matters to customers, such as Transparent Data Encryption.
  4. For a Kubernetes operator, ship images with exactly the tools and extensions the operator supports, and skip everything else. The CVE surface stays smaller.
  5. Give QA and Service teams a predictable environment. “We support extensions A, B, C and not D, X, Z” is only honest if QA actually exercises A, B, C and the Service team can work with them in the production environment.
  6. Give customers one accountable party for the full release cycle, from hotfix through package availability. Some teams explicitly need that contract for compliance and audit reasons.
  7. And yes, less positive reasons that we covered above also apply, which is exactly the part the community keeps pointing at.

 

The trade-off you accept

If you run the vendor distribution, you accept that the vendor’s registry, image policy, and supported-extension matrix become part of your stack. If the vendor changes any of that, your operator deployment changes with it. That is not hypothetical for users who have lived through it on other products.

So the real question is whether you can keep the benefits a distribution provides for the users who want them, while leaving an honest, supported door open for users who do not. That is the door PGO 3.0.0 opens.

 

Community PostgreSQL Images in PGO 3.0.0

Starting with Percona Operator for PostgreSQL 3.0.0, the operator can run against images built from upstream PostgreSQL packages, not just the Percona Distribution images. This is what we are calling Community PostgreSQL Images. In 3.0.0, the feature ships as a tech preview. In 3.1.0, these images become part of our official release cycle and are fully documented.

One of the main advantages of Community Docker Images is that the community can request or contribute any extension that does not exist in the official Percona PostgreSQL distribution. TimescaleDB and Citus are the first examples: the community asked for them, and we shipped both in the Community Images set from day one.

 

How to use “Community PostgreSQL images”

The operator does not care where the image came from, as long as the image meets the operator’s runtime expectations 

A typical CR using a community image looks like this:

apiVersion: pgv2.percona.com/v2
kind: PerconaPGCluster
metadata:
  name: cluster1
spec:
  image: registry.example.com/postgresql-community:18
  postgresVersion: 18
  proxy:
    pgBouncer:
      image: registry.example.com/pgbouncer-community:1.23
  backups:
    pgbackrest:
      image: registry.example.com/pgbackrest-community:2.51
  # other spec fields unchanged from a normal CR

The fields that change are spec.image, spec.proxy.pgBouncer.image, and spec.backups.pgbackrest.image. You can build and publish all three images under your own registry, with your own tags if that helps you track versions. The operator drives the rest of the deployment the same way it always has: instances, backups, replication, monitoring, all of it.

 

What ships are in each image

Each Community Docker Image is a thin layer over the chosen base (UBI9 or UBI8) plus the packages the operator needs for that role. Where you see {N}, substitute the PostgreSQL major you build for (17, 18, and so on).

postgres image (e.g. postgres17):

Package Role
postgresql{N}-server PostgreSQL server
postgresql{N}-contrib contrib modules
pg_repack_{N} online table/index reorganization
pgaudit_{N} audit logging
set_user_{N} privilege escalation control
pgvector_{N} vector similarity search
wal2json_{N} WAL to JSON logical decoding
pg_cron_{N} in-database cron scheduler
pgbackrest</code> backup/restore tool
patroni HA cluster manager
timescaledb-2-postgresql-{N} time-series extension (x86_64 only; EL9 only for PG18)
citus_{N} distributed PostgreSQL (PG16+ only)

pgbackrest image:

Package Role
pgbackrest backup/restore tool only

pgbouncer image:

Package Role
pgbouncer connection pooler only

 

The split is intentional. The postgres image ships the full operator-aware runtime. The backup and proxy images stay minimal. As a result, the operator’s components are in separate failure domains and shrink the attack surface of each container.

 

Limits worth being honest about

A community image is not a Percona Distribution image. Two practical consequences:

  • Distribution-only features will not work. Transparent Data Encryption, for example, lives in the Percona Distribution build. A community image built from upstream PostgreSQL does not include it. If you depend on TDE, run the distribution image.
  • Support boundaries are different. Percona Support is responsible for the Percona Distribution images and the operator code. A community image you built yourself

Ultimately, these are the right trade-offs. The point of community images is to give you transparency and control. Taking care of your own image is part of that deal. At the same time, we publish all three images under perconalab/percona-postgresql-operator on Docker Hub so you can evaluate the tech preview without standing up your own build pipeline first. perconalab is Percona’s non-production namespace, so use those images for testing. For production, build and sign your own.

UBI9 (EL9):

docker.io/perconalab/percona-postgresql-operator:main-postgres14-community
docker.io/perconalab/percona-postgresql-operator:main-postgres15-community
docker.io/perconalab/percona-postgresql-operator:main-postgres16-community
docker.io/perconalab/percona-postgresql-operator:main-postgres17-community
docker.io/perconalab/percona-postgresql-operator:main-postgres18-community
docker.io/perconalab/percona-postgresql-operator:main-pgbackrest-community
docker.io/perconalab/percona-postgresql-operator:main-pgbouncer-community
docker.io/perconalab/percona-postgresql-operator:main-upgrade-community

UBI8 (EL8):

docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres14-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres15-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres16-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres17-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-postgres18-community
docker.io/perconalab/percona-postgresql-operator:main-ubi8-upgrade-community

 

How to build the images

The Dockerfile, the package list, and a sample CI job ship in percona-docker/postgresql-containers/community. The build is a regular make target on top of docker buildx, so you can run it on any multi-platform builder.

# Prerequisites: docker buildx with a multi-platform builder
docker buildx create --use --name multiarch

# Build and push all PostgreSQL community images (UBI9 / EL9)
git clone https://github.com/percona/percona-docker
cd percona-docker/postgresql-containers/community
make all TAG=1.0.0 REGISTRY=myrepo/percona-postgresql-operator

# Or a single image
make postgres17 TAG=1.0.0 REGISTRY=myrepo/percona-postgresql-operator

# UBI8 / EL8 variants
make all-ubi8 TAG=1.0.0-ubi8 REGISTRY=myrepo/percona-postgresql-operator

make all builds all three images (postgres, pgBouncer, pgBackRest) so they stay version-aligned. Override REGISTRY and TAG to point at your own namespace and tagging scheme. Once the images are in your registry, plug them into the CR fields shown earlier, and the operator picks them up.

Full build documentation: percona-docker/postgresql-containers/community/README.md.

 

How to contribute

Community images live in percona/percona-docker, and the build is driven by a transform.py generator that produces the Dockerfiles under build/. The files under build/ are regenerated on every sync, so contributions go through the generator, never through the generated files.

Full contribution guide: community/CONTRIBUTING.md.

 

How to provide feedback

Two channels, depending on the shape of the feedback:

  • GitHub issue on percona/percona-postgresql-operator with the community-images label. Use this for bug reports, missing extensions, build problems, and concrete requests. The label keeps all community-image reports in one filter the team watches.

 

What’s next

The first step was taking full engineering ownership of Percona Operator for PostgreSQL as an independent project, so the roadmap, the release cadence, and the governance live with one team that the community can talk to directly. Community PostgreSQL Images are the next step in that same commitment. If the community adopts this path, we have ideas for what to invest in next.

We will let the community tell us. If this is useful, we keep investing here. We are ready to add more features to the operator around Community Images. Conversely, if nobody adopts it, that is also a signal, and an honest one.

Try the tech preview in 3.0.0. Open an issue if the build flow is rougher than it should be. Tell us what you want next on the forum or directly on GitHub.

 

Try It Out

The post Community Docker Images: keeping the operator open without a vendor registry lock-in appeared first on Percona.

Jun
30
2026
--

Debugging with Ephemeral Containers

Debugging applications in Kubernetes can be tricky. Containers are designed to be small, immutable, and purpose-built. That is great for production, but not always ideal when something breaks. Many production images are minimal or distroless. They may not include tools that are useful for troubleshooting.

In some cases, the application container may already be crashing, which means kubectl exec is not useful. In other cases, accessing the nodes may not be possible. Since Pods are immutable, it is impossible to add another container for troubleshooting.

This is where ephemeral containers help.

What Are Ephemeral Containers?

In Kubernetes, ephemeral containers are a special type of container designed to run temporarily inside an existing Pod. Their primary purpose is to help administrators and developers troubleshoot, inspect, and debug live applications without disrupting the running service. They are not meant to run application workloads. Instead, they are designed for operational debugging.

An ephemeral container is not added to a Pod by editing spec.containers. Kubernetes treats it differently from regular containers. When an ephemeral container is created, a request is sent to the Kubernetes API server using the Pod’s ephemeralcontainers subresource. This distinction is important because most of a Pod’s spec is immutable after creation; Kubernetes does not allow you to simply edit a running Pod and append a normal container to spec.containers.

spec:
  ephemeralContainers:
  - name: <debug container-name>
    image: <image>
    targetContainerName: <> # (Optional)If ephemeral container needs to have same pid namespace of a running container.

Key Characteristics of Ephemeral Containers

Unlike standard containers, ephemeral containers have the following characteristics:

  1. They do not have guaranteed CPU or memory resources (limits or requests).
  2. If an ephemeral container crashes or completes its task, it will never restart.
  3. They do not include fields such as ports, livenessProbe, or readinessProbe.

Examples

For testing, the Percona Operator for MySQL based on XtraDB Cluster is installed. The installation steps can be found here. Once the installation is complete, the running pods should look similar to the following:

# kubectl get po
NAME                                              READY   STATUS      RESTARTS   AGE
cluster1-haproxy-0                                2/2     Running     0          17h
cluster1-haproxy-1                                2/2     Running     0          17h
cluster1-haproxy-2                                2/2     Running     0          16h
cluster1-pxc-0                                    3/3     Running     0          17h
cluster1-pxc-1                                    3/3     Running     0          17h
cluster1-pxc-2                                    3/3     Running     0          16h
percona-xtradb-cluster-operator-6b5f75f65-fpjxr   1/1     Running     0          17h
xb-cron-cluster1-fs-pvc-20266250025-372f8-hmrmh   0/1     Completed   0          7h9m

NOTE: It is important to note that this behavior depends on your environment, specifically whether you have the required privileges or Security Context Constraints (SCC) in place. The commands below are for demonstration purposes and do not necessarily follow all best practices, such as avoiding generic ubuntu images, long-running shells like bash, or running containers as root.Always run ephemeral containers with a strong focus on security, especially in production systems.

Let’s look at some examples of how ephemeral containers can be useful.

1. Ephemeral Container with shared network namespace

When an ephemeral container is created in a Pod, it shares the network namespace of all other containers in that Pod. This is particularly useful for troubleshooting network-related issues.

Let’s create an ephemeral container named debug-1 in the MySQL pod cluster1-pxc-0:

% kubectl debug pod/cluster1-pxc-0 --image=ubuntu --container=debug-1 -ti -- bash
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.

When we check the processes in the container, only the bash process that was started when the container was created is running.

root@cluster1-pxc-0:/# ps -elf
F S UID          PID    PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root           1       0  0  80   0 -  1192 do_wai 07:16 pts/0    00:00:00 bash
4 R root           9       1  0  80   0 -  1701 -      07:16 pts/0    00:00:00 ps -elf

However, port 3306 is open because the MySQL process in the primary container shares the same network namespace as our debug container.

root@cluster1-pxc-0:/# netstat -tlnp | grep :3306
tcp        0      0 10.42.0.18:33062        0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::33060                :::*                    LISTEN      -

This capability is highly effective for analyzing network traffic, egress connectivity, and local service availability.Let’s examine the Pod spec and status to see how ephemeral containers are represented.

The following is the spec of the ephemeral container. Note the securityContext, which we will discuss later in this post:

% kubectl get po cluster1-pxc-0 -oyaml | yq .spec.ephemeralContainers    
- command:
    - bash
  image: ubuntu
  imagePullPolicy: Always
  name: debug-1
  resources: {}
  securityContext:
    capabilities:
      add:
        - SYS_PTRACE
  stdin: true
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  tty: true

Status of ephemeral containers

% kubectl get po cluster1-pxc-0 -oyaml | yq .status.ephemeralContainerStatuses
- containerID: containerd://4f656ada1679595109a9ac4c5bae916dc35404d7a45818eacef83aabd6d8509b
  image: docker.io/library/ubuntu:latest
  imageID: docker.io/library/ubuntu@sha256:53958ec7b67c2c9355df922dd08dbf0360611f8c3cdb656875e81873db9ffdba
  lastState: {}
  name: debug-1
  ready: false
  resources: {}
  restartCount: 0
  state:
    running:
      startedAt: "2026-06-25T07:16:22Z"
  user:
    linux:
      gid: 0
      supplementalGroups:
        - 0
        - 1001
      uid: 0

2. Ephemeral Container with shared network namespace, pid namespace

While sharing the network namespace is useful, sharing the PID namespace to inspect running processes can be beneficial in many debugging scenarios.

Let’s create an ephemeral container named debug-2 in the cluster1-pxc-0 Pod, specifically targeting the PID namespace of the pxc container:

% kubectl debug pod/cluster1-pxc-0 --image=ubuntu --container=debug-2 --target=pxc -ti -- bash
Targeting container "pxc". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.

A key change from the previous command is the addition of the --target=pxc flag. This creates an ephemeral container that shares the PID namespace of the pxc container.

When we check the processes in the container, we can see the MySQL processes.

root@cluster1-pxc-0:/# ps -elf
F S UID          PID    PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S 1001           1       0  4  80   0 - 832724 do_pol Jun24 ?       00:50:54 mysqld --wsrep_start_position=9d4e0c3e-6fd5-11f1-aab4-1a8a26ce607c:28
4 S 1001          90       1  0  80   0 - 306929 futex_ Jun24 ?       00:00:00 /var/lib/mysql/mysql-state-monitor
1 Z 1001        2999       1  0  80   0 -     0 -      Jun24 ?        00:00:00 [wsrep_sst_xtrab] <defunct>
4 S root      173053       0  0  80   0 -  1192 do_wai 08:01 pts/0    00:00:00 bash
4 R root      173081  173053  0  80   0 -  1701 -      08:01 pts/0    00:00:00 ps -elf

We can also verify this by network stats

root@cluster1-pxc-0:/# netstat -anlp | grep 3306
tcp        0      0 10.42.0.18:33062        0.0.0.0:*               LISTEN      1/mysqld            
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      1/mysqld            
tcp        0      0 10.42.0.18:59330        10.42.0.18:33062        TIME_WAIT   -                   
tcp        0      0 10.42.0.18:33062        10.42.1.4:52464         TIME_WAIT   -                   
tcp        0      0 10.42.0.18:36008        10.42.0.18:33062        TIME_WAIT   -                   
tcp        0      0 10.42.0.18:33062        10.42.1.4:52448         TIME_WAIT   -                   
tcp        0      0 10.42.0.18:33756        10.42.0.18:33062        TIME_WAIT   -                   
tcp        0      0 10.42.0.18:43086        10.42.0.18:33062        TIME_WAIT   -                   
tcp        0      0 10.42.0.18:43206        10.42.0.18:33062        TIME_WAIT   -                   
tcp        0      0 10.42.0.18:44256        10.42.0.18:33062        TIME_WAIT   -                   
tcp        0      0 10.42.0.18:43220        10.42.0.18:33062        TIME_WAIT   -                   
tcp        0      0 10.42.0.18:44258        10.42.0.18:33062        TIME_WAIT   -                   
tcp6       0      0 :::33060                :::*                    LISTEN      1/mysqld

3. Ephemeral Container with shared network namespace, shared pid namespace, shared volume

In some cases, you may need to collect dumps, check database files, or inspect logs, which requires access to the filesystem. However, each container has its own mount namespace, and mount namespaces cannot be shared directly.A volume mounted in a Pod, however, can be shared across containers, including ephemeral containers.

Let’s check the volumes present in the cluster1-pxc-0 pod and how they are mounted to the pxc container.

% kubectl get po cluster1-pxc-0 -o yaml | yq '.spec.volumes[] | select(.name == "datadir")'
name: datadir
persistentVolumeClaim:
  claimName: datadir-cluster1-pxc-0

% kubectl get po cluster1-pxc-0 -o yaml | yq '.spec.containers[] | select(.name == "pxc")| .volumeMounts[] | select(.name == "datadir")' 
mountPath: /var/lib/mysql
name: datadir

As seen above, the persistent volume holding the database files is mounted at /var/lib/mysql.

Let’s create an ephemeral container named debug-3 in the cluster1-pxc-0 pod, sharing the PID namespace of the pxc container and mounting the datadir volume at /db-mount:

% kubectl patch po cluster1-pxc-0 --subresource=ephemeralcontainers -p '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debug-3",
                "command": ["bash"],
                "image": "ubuntu",
                "targetContainerName": "pxc",
                "stdin": true,
                "tty": true,
                "volumeMounts": [{
                    "mountPath": "/db-mount",
                    "name": "datadir",
                    "readOnly": true
                }]
            }
        ]
    }
}'
pod/cluster1-pxc-0 patched

The command above creates the ephemeral container in the pod. To access it, we will attach to the running debug container:

% kubectl attach -ti pod/cluster1-pxc-0 -c debug-3
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
root@cluster1-pxc-0:/# cd /db-mount
root@cluster1-pxc-0:/db-mount# ls
'#ib_16384_0.dblwr'                 binlog.000001        gvwstate.dat                 mysql                     mysqlx.sock.lock       pxc-entrypoint.sh
'#ib_16384_1.dblwr'                 binlog.000002        ib_buffer_pool               mysql-state-monitor       notify.sock            readiness-check.sh
 '#innodb_redo'                     binlog.000003        ibdata1                      mysql-state-monitor.log   peer-list              sys
 '#innodb_temp'                     binlog.index         ibtmp1                       mysql.ibd                 performance_schema     undo_001
 audit_filter.20260624T140457.log   cluster1-pxc-0.pid   innobackup.backup.full.log   mysql.state               pmm-prerun.sh          undo_002
 audit_filter.log                   galera.cache         innobackup.backup.log        mysql_upgrade_history     private_key.pem        version_info
 auth_plugin                        get-pxc-state        liveness-check.sh            mysqld-error.log          public_key.pem         wsrep_cmd_notify_handler.sh
 auto.cnf                           grastate.dat         logrotate.status             mysqlx.sock               pxc-configure-pxc.sh

As demonstrated, the database files located at /var/lib/mysql are now accessible at /db-mount within the debug-3 container.

Since the volume was mounted with the “readOnly”: true parameter, no writes can be performed.

root@cluster1-pxc-0:/db-mount# touch test
touch: cannot touch 'test': Read-only file system

If you require write permissions, simply omit the “readOnly”: true flag.

% kubectl patch po cluster1-pxc-0 --subresource=ephemeralcontainers -p '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debug-4",
                "command": ["bash"],
                "image": "ubuntu",
                "targetContainerName": "pxc",
                "stdin": true,
                "tty": true,
                "volumeMounts": [{
                    "mountPath": "/db-mount",
                    "name": "datadir"
                }]
            }
        ]
    }
}'
pod/cluster1-pxc-0 patched

Now, attach to the container and create a file in the volume mount:

% kubectl attach -ti pod/cluster1-pxc-0 -c debug-4                  
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
root@cluster1-pxc-0:/# touch /db-mount/test
root@cluster1-pxc-0:/# ls /db-mount/test
/db-mount/test

4. Ephemeral Container on a Kubernetes node’s namespace and filesystem

If you need to examine system logs (such as kernel logs or dmesg) but do not have SSH access to the nodes, you can run an ephemeral container directly on the node’s namespace and filesystem.

Let’s check the nodes of the Kubernetes cluster and run an ephemeral container directly on one:

% kubectl get nodes
NAME                 STATUS   ROLES                AGE   VERSION
chetan-1-36-node-1   Ready    control-plane,etcd   6d    v1.36.1+k3s1
chetan-1-36-node-2   Ready    control-plane,etcd   6d    v1.36.1+k3s1
chetan-1-36-node-3   Ready    control-plane,etcd   6d    v1.36.1+k3s1

Run an ephemeral container:

% kubectl debug node/chetan-1-36-node-1 --image=ubuntu --container=debug-5 -ti -- bash
Creating debugging pod node-debugger-chetan-1-36-node-1-hfgms with container debug-5 on node chetan-1-36-node-1.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
root@chetan-1-36-node-1:/#

Node’s filesystem can be accessed at /host.

root@chetan-1-36-node-1:/#  ls /host
bin                boot  etc   lib                lib64       media  opt   root  sbin                snap  swapfile  tmp  var
bin.usr-is-merged  dev   home  lib.usr-is-merged  lost+found  mnt    proc  run   sbin.usr-is-merged  srv   sys       usr
root@chetan-1-36-node-1:/#  ls /host/var/log/dmesg 
/host/var/log/dmesg

Behind the scenes, a Pod with the name node-debugger-<node-name>-<hash> is created.

% kubectl get po -l app.kubernetes.io/managed-by=kubectl-debug
NAME                                     READY   STATUS      RESTARTS   AGE
node-debugger-chetan-1-36-node-1-hfgms   0/1     Completed   0          33m

Let’s check the spec of the debug pod

% kubectl get po -l app.kubernetes.io/managed-by=kubectl-debug
NAME                                     READY   STATUS      RESTARTS   AGE
node-debugger-chetan-1-36-node-1-hfgms   0/1     Completed   0          33m

Let’s check the spec of the debug pod

% kubectl get po node-debugger-chetan-1-36-node-1-hfgms -oyaml | yq .spec
containers:
  - command:
      - bash
    image: ubuntu
    imagePullPolicy: Always
    name: debug-5
    resources: {}
    stdin: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
      - mountPath: /host
        name: host-root
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-4lj9h
        readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostIPC: true
hostNetwork: true
hostPID: true
nodeName: chetan-1-36-node-1
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
  - operator: Exists
volumes:
  - hostPath:
      path: /
      type: ""
    name: host-root
  - name: kube-api-access-4lj9h
    projected:
      defaultMode: 420
      sources:
        - serviceAccountToken:
            expirationSeconds: 3607
            path: token
        - configMap:
            items:
              - key: ca.crt
                path: ca.crt
            name: kube-root-ca.crt
        - downwardAPI:
            items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
                path: namespace

Some key observations from the spec are the following: 

hostIPC: true  -> Share host’s IPC namespace
hostNetwork: true  -> Share Host Node Network
hostPID: true  -> Share host’s PID namespace

volumes:
  - hostPath:
      path: /      -> Use Host’s file system 
      type: ""
    name: host-root

    volumeMounts:
      - mountPath: /host
        name: host-root

The specifications above indicate excessive permissions for a regular application; consequently, these might be disabled by your system administrator via security policies.

Profile of an Ephemeral Container

An interesting option for the kubectl debug command is --profile.

The official documentation provides the following:

--profile string     Default: "general"
Options are "general", "baseline", "restricted", "netadmin" or "sysadmin". Defaults to "general"

These profiles define the capabilities associated with the SecurityContext and determine how the host’s filesystem and namespaces are accessed. The security context details for each profile are listed below; further details are available in the source code.

Profile Debug Pod(SecurityContext) Debug Node(SecurityContext)
general Add SYS_PTRACE cap Attach host “/” filesystem.

No SecurityContext

Use HostNetwork, HostPID Namespace,HostIPC Namespace

baseline No SecurityContext No SecurityContext
restricted Drop ALL capabilities

runAsNonRoot: true

allowPrivilegeEscalation: false

seCompProfile: RuntimeDefault

Drop ALL capabilities

runAsNonRoot: true

allowPrivilegeEscalation: false

seCompProfile: RuntimeDefault

netadmin Add NET_ADMIN, NET_RAW Cap Add NET_ADMIN, NET_RAW Cap

Use HostNetwork, HostPID Namespace,HostIPC Namespace

sysadmin privileged: true

Use HostNetwork, HostPID Namespace,HostIPC Namespace

privileged: true

Use HostNetwork, HostPID Namespace,HostIPC Namespace

Attach host “/” filesystem.

 

 

The pod’s execution behavior depends on the profile chosen when running the kubectl debug command.

Caveats with Ephemeral Containers

Even though ephemeral containers are useful, there are several caveats and potential security risks that users should be aware of:

  1. Ephemeral containers may be able to inspect processes, network traffic, environment variables, mounted volumes, or service account context depending on the Pod and cluster configuration.
  2. They can bypass the security benefits of distroless or minimal images.
  3. Ephemeral containers can expose secrets
  4. Once an ephemeral container is added, it remains part of the Pod spec; it is not possible to remove it.
  5. As long as the ephemeral container’s main process is running, you can attach to it using kubectl attach.
  6. Ephemeral containers behaviour is unpredictable, containers might terminate abruptly depending on the resource configuration and utilization of the pod.

Guardrails and Best practices with Ephemeral Containers

Ephemeral containers are powerful, but they can create operational and security risks if used carelessly. Adhering to the following guardrails and best practices can help mitigate these risks:

  1. Control ephemeral containers access through RBAC. Only necessary users should have the privilege.
  2. Avoid using the sysadmin profile when possible. The restricted profile is better suited for maintaining a strong security posture.
  3. Always use approved, scanned images that contain only the tools required for debugging, rather than generic images like ubuntu.
  4. Avoid running debug containers with long-running processes like sleep infinity or bash. Instead, run specific tools or commands that perform the required action and then terminate.
  5. Audit ephemeral containers usage.
  6. Recycle Pods that contain ephemeral containers whenever possible (e.g., during maintenance windows), especially if the ephemeral process has not terminated.

Conclusion

Ephemeral containers are a powerful troubleshooting tool for Kubernetes workloads, especially when application images are minimal, distroless, or missing debugging utilities. They allow engineers to inspect a running Pod without rebuilding the image or restarting the workload.

However, they should be treated as controlled operational access, not as a default debugging shortcut. Ephemeral containers can expose sensitive runtime details such as processes, environment variables, mounted volumes, and network state.

Always restrict usage with proper RBAC. Use approved debug images, prefer the least-privileged debug profile, and reserve powerful profiles such as sysadmin for “break-glass” scenarios only.

Debug sessions should be short-lived, intentional, and tied to a real troubleshooting need. In short, ephemeral containers improve debuggability, but they must be used with clear security, audit, and operational guardrails.

The post Debugging with Ephemeral Containers appeared first on Percona.

Jun
30
2026
--

Why I haven’t run my databases on Kubernetes

A few years ago, if there was a discussion on “Should we run databases on Kubernetes?”, there were more people saying no than yes. One of the common answers was, “No. Kubernetes is for stateless workloads. Keep your databases outside.”

Thankfully, today the discussion is no longer about whether we should run databases on Kubernetes, but how we can run them better on Kubernetes.

In this post, we will look at some of the common arguments brought up against running databases on Kubernetes.

1. Kubernetes was designed for stateless workloads

By far the most common concern

Even though Kubernetes was initially used mainly for stateless workloads, many additions have been made since its initial version to accommodate stateful workloads. Features like StatefulSet and PersistentVolumes were introduced, and storage usage has been streamlined through the Container Storage Interface (CSI). Furthermore, the platform itself has evolved to support system-level capabilities like Swap space to better manage memory-intensive databases.

Today, a large number of users are successfully running stateful workloads on K8s. According to the latest CNCF Annual Survey Report, stateful containers have become a standard practice, with 79% of Innovators now running stateful applications in production.

2. Is my data safe on Kubernetes?

Is my data safe on Kubernetes even when pods, nodes, or the entire Kubernetes cluster go down?

This sounds terrifying at first, but cloud-native architecture is built with failure in mind. Pods die, nodes crash, and entire clusters can fail. The golden rule of running databases on Kubernetes is that your data must never depend on the lifecycle of the compute layer.

This is where the absolute decoupling of compute and storage becomes critical. The underlying storage infrastructure exists entirely independent of the Pod, Node, and even the Kubernetes cluster itself. As long as your storage backend is architected correctly, your data is preserved regardless of what happens to the compute environment.

For a database, this tiered isolation changes everything:

  • If a Pod or Node dies: Kubernetes automatically schedules a replacement Pod and reattaches it to the existing, intact storage volume.
  • If the entire Kubernetes Cluster goes down: Because the data lives safely outside the cluster boundary, you can spin up a completely new Kubernetes cluster, connect it to the existing storage backend, and restore your database operations.

For replicated databases, a Kubernetes Operator can automate this resilience, detecting failures, promoting replicas, and reconciling your state across nodes, or even helping orchestrate disaster recovery across entirely different clusters.

3. Won’t My Database Experience Downtime When Pods or Nodes Go Down?

What if a Pod or a node running a database goes down? Will the database experience downtime?

While failover concepts remain similar to traditional environments, Kubernetes transforms this into an automated, declarative process via Operators. Kubernetes automatically schedules a replacement Pod and reattaches it to the existing, intact storage volume. If the database is properly configured with a redundant, highly available configuration (recommended configuration), the remaining healthy Pods will continue to serve traffic. When utilizing Kubernetes solutions, you are delivering a data service powered by specific database technology, where high availability is maintained through a cluster of N Pods. The critical shift in mindset here is to focus on the service’s resilience rather than the survival of individual Pods; our goal is to optimize the overall service, not the individual Pod, a distinction that is often misunderstood.

4. I have already built a lot of custom automation in our in-house environment. Why should I switch to Operators?

While custom scripts might work well today, proprietary automation always comes with heavy long-term maintenance overhead. To scale effectively, automation should be split into two distinct layers: infrastructure and database management.

Because Kubernetes is widely adopted and continuously updated by the global community, it handles the infrastructure layer (compute, networking, and storage) out of the box. By adopting a good Operator, complex database-specific logic is offloaded. Ultimately, using Operators prevents you from reinventing the wheel and lets the teams focus on application value rather than maintaining custom infrastructure code.

5. Managing Pods, PVCs and configuration is painful

Manually managing StatefulSets, PersistentVolumeClaims (PVCs), and various other Kubernetes objects to run a database on Kubernetes can be tedious.

This is where database operators emerge as game-changers.

An operator is a Kubernetes-native controller that watches custom resources and continuously works to move the actual system toward the desired state. As a user, you describe something like: “I want a database cluster with three instances, backups enabled, this storage, these resources, this version, monitoring enabled, and this replication setup.” The operator then handles everything under the hood, including managing Kubernetes objects and implementing the operational logic.

Instead of playing the role of a mechanic who has to assemble all the parts and make everything run, you simply get into a car built by expert mechanics and drive it. The operator abstracts away the complexity so you can focus on the outcome rather than the implementation details.

6. I can configure bare-metal or VM nodes exactly how I want for database workloads, but Kubernetes makes this level of customization impossible.

A very valid concern, especially for database engineers accustomed to carefully tuned virtual machines or bare-metal servers. Running databases inside Pods does introduce an additional abstraction layer. There may be a very small category of specialized, legacy “pet” databases that require highly specific bare-metal tuning and extreme isolation.

That said, the vast majority of configurations are achievable on Kubernetes. There are a few exceptions; for example, certain low-level storage tuning options may be limited by CSI driver abstractions. However, these are edge cases that rarely affect modern production deployments and do not impact the vast majority of database workloads.

For almost all modern production use cases, Kubernetes is more than capable of running databases reliably and efficiently.

7. Managed databases are way better than running on kubernetes

Managed databases are a great fit for many use cases, as they remove significant complexity and operational overhead.

However, they do come with a few caveats:

  • Cost at Scale: Managed databases can become incredibly expensive compared to running databases yourself using Kubernetes Operators, especially as your data scales.
  • Vendor Lock-in: When you rely on a cloud provider’s managed service, you are typically locked into their ecosystem, making it difficult and costly to migrate away.
  • Configuration Limits: In many cases, cloud vendors restrict your control, making it impossible to apply deep custom configurations or install specific database extensions that your application might require.

In short, many of the concerns raised are no longer relevant today or do not apply to the majority of use cases.

Conclusion

It’s safe to say that databases run and run well on Kubernetes if configured well. In many cases, the shift is cultural rather than technical.

Running databases on Kubernetes offers the advantage of preventing vendor lock-in. Even though database migration tools exist, cloud migrations remain difficult in practice. This is especially true with major cloud vendors, where your data layer is often tightly integrated with a complex web-native ecosystem of services.

Operators excel at automating complex Day-2 operations like failovers, backups, and rolling updates. Running both your application and data layers under a single Kubernetes cluster provides a unified control plane for automation. This setup seamlessly aligns with GitOps workflows and becomes incredibly efficient when managing a large fleet of databases at scale.

We have seen users derive incredible value by running databases on Kubernetes with operators. Some use it to manage standard database instances, while others have built their own fully fledged, internal DBaaS (Database-as-a-Service) platforms, and the list is long. Selecting the right Operator makes all the difference on this journey. Opting for free and open-source operators like the Percona Operators not only provides enterprise-grade databases but also saves your teams significant time and money.

 

The post Why I haven’t run my databases on Kubernetes appeared first on Percona.

Jun
29
2026
--

Talking Drupal #559 – Marketing Drupal

Today we are talking about Marketing, AI, and Drupal with guest Paul Johnson. We’ll also cover Curated Colors as our module of the week.

For show notes visit: https://www.talkingDrupal.com/559

Topics

  • Paul’s Current Projects
  • Enterprise AI Summit Details
  • Marketing the AI Initiative
  • Partnering on Event Booths
  • Drupal’s Outside Perception
  • What’s Working Now
  • Growing the Marketing Team
  • How to Contribute
  • Outside In Storytelling
  • Case Study Examples
  • AI Initiative Impact
  • Roadmap and Launch Planning
  • Finding New Adopters
  • Where Pros Research
  • Conference Pitch Story
  • Local Event Playbook
  • Funnel and Webinars
  • Industry Guides and Demos
  • SEO and AI Search
  • Why Agents Avoid Drupal
  • High Leverage Contributions
  • Measuring AI Mentions
  • Vibe Coders to Governance
  • Fixing Misconceptions

Resources

Guests

Paul Johnson – pdjohnson

Hosts

Nic Laflin – nLighteneddevelopment.com nicxvan John Picozzi – epam.com johnpicozzi Scott Falconer – managing-ai.com scott-falconer

MOTW Correspondent

Martin Anderson-Clutz – mandclu.com mandclu

  • Brief description:
    • Have you ever wanted to allow editors on your Drupal site to choose styling from a brand-approved color palette? There’s a module for that.
  • Module name/project name:
  • Brief history
    • How old: created in Apr 2026 by Kyle Einecker (ctrladel) of True Summit
    • Versions available: 1.0.0 which works with Drupal 10.3, 11, and 12
  • Maintainership
    • Actively maintained
    • Security coverage
    • Test coverage
    • Documentation – in-depth README
    • Number of open issues: 2 open issues, neither of which are bugs
  • Usage stats:
    • 27 sites
  • Module features and usage
    • Curated Colors enforces brand consistency by replacing generic color text inputs or wide-open color pickers with a curated, visual swatch popover containing only pre-approved, named options
    • It streamlines rebranding by storing abstract keys (such as brand-primary) instead of raw hex values (e.g., #0678be) in the database. That means updating a brand color in the future only requires a CSS or configuration change rather than a massive data migration
    • Curated Colors is also extensible beyond colors. It functions as a generic visual variant selector. Site builders can repurpose it to let editors pick card layouts, button styles (like primary, outline, or danger), hero text alignments, or icon themes
    • Editors can pick from neatly organized groups with human-readable labels and see a live preview swatch of their selection before saving
    • Palettes are managed as exportable Drupal configuration. Each entry maps a machine key to a label, administrative hex preview, and optional custom CSS
    • The module provides a curated_color field type and an accompanying swatch-based popover widget that can be restricted to specific palette groups. It also features a native curated_color_picker Form API element and integrates with the Canvas module via SDC annotations
    • The field exposes properties like value, hex, style, and css, making it simple to output selections as classes, inline styles, or raw codes in Twig templates
    • Finally, Curated Colors includes an example submodule providing a working SDC component and sample palette templates so you can see exactly how it’s meant to be used
Jun
29
2026
--

Skipping Percona Server for MySQL 8.4.9 and 9.7.0

Upstream MySQL published an out-of-schedule release this week with two high-severity CVE fixes. We’ve pulled those fixes into our next builds and are skipping the two versions we had already queued: Percona Server for MySQL 8.4.9 and 9.7.0.

These fixes arrived through Oracle’s new monthly Critical Security Patch Updates (CSPUs), which Oracle announced begin May 28, 2026. CSPUs ship targeted high-severity fixes between Oracle’s quarterly Critical Patch Updates. For MySQL, these updates are issued as needed rather than on a fixed monthly schedule, so out-of-schedule security fixes like these may become more common.

We’ve handled a skip like this before. When MySQL Community Server 8.4.2 followed 8.4.1 by only a few weeks, we skipped 8.4.1 and shipped its contents in 8.4.2-2. This is the same approach.

What’s happening

The code for 8.4.9 and 9.7.0 was already ready for packaging when the CVE fixes landed. Rather than ship those builds and follow immediately with a security patch, we applied the fixes, re-tested, and re-tagged. Percona Server for MySQL 8.4.10 and 9.7.1 will carry everything 8.4.9 and 9.7.0 would have contained, plus the upstream high-severity CVE fixes.

These fixes come from Oracle’s June 2026 Critical Security Patch Update; the specific CVE identifiers will be listed in the 8.4.10 and 9.7.1 release notes. No action is required on your part. The fixes reach you in 8.4.10 and 9.7.1, expected within days. If your security policy requires faster remediation, contact Percona Support to discuss interim options.

8.4.9 and 9.7.0 will not appear in the package repositories. A normal upgrade moves you straight to 8.4.10 or 9.7.1, which carry the skipped versions’ content.

Who this affects

If you were waiting specifically for 8.4.9 or 9.7.0, those versions won’t be published. Point your upgrade at the next releases instead, which include the same content and the CVE fixes. The delay is a few days, not weeks. If you weren’t tracking a specific version number, nothing changes for you.

What to do

Nothing urgent. Upgrade to the next Percona Server for MySQL releases as you normally would once they’re published. We’ll announce them through release notes and the Percona Blog. For questions about timing or the security content, reach out to Percona Support or post in the Percona Community Forum.

What to expect going forward

Oracle’s monthly CSPUs mean out-of-schedule fixes will happen more often. Our approach stays consistent: we evaluate every upstream release, and when high-severity fixes land between our scheduled releases, we fold them into the next release rather than shipping a separate build for each one. Your LTS support commitments don’t change. We’re watching how often Oracle uses the monthly cadence and will adjust release planning if the volume warrants it.

The post Skipping Percona Server for MySQL 8.4.9 and 9.7.0 appeared first on Percona.

Jun
22
2026
--

Talking Drupal #558 – Agent Management System

Today we are talking about AI, Agents, and A System to manage them with guest Luke McCormick. We’ll also cover AI Auto-reference as our module of the week.

For show notes visit: https://www.talkingDrupal.com/558

Topics

  • Introducing Agent Management
  • Origin Story Claude Credits
  • Scrum Meets AI Retention
  • Handoff Protocol Filesystem
  • Why Handoffs Work So Well
  • Examples and Human Loop
  • Agent Roles and Model Costs
  • Choosing Models by Task
  • Not Drupal Specific
  • Works With Any Model
  • Scrum Sprints For Agents
  • Human Cognitive Overload
  • Tuning Autonomy Levels
  • Setup And Handoff File
  • Updating Customized AMS
  • Persistent Memory Artifacts
  • Demand Better Summaries
  • Solo Power With Agents
  • Roadmap And AMS Trio

Resources

Guests

Luke McCormick – cellear

Hosts

Nic Laflin – nLighteneddevelopment.com nicxvan John Picozzi – epam.com johnpicozzi

MOTW Correspondent

Martin Anderson-Clutz – mandclu.com mandclu

  • Brief description:
    • Have you ever wanted to use AI to suggest related content on your Drupal site? There’s a module for that.
  • Module name/project name:
  • Brief history
    • How old: created in June 2023 by Scott Euser (scott_euser) or Soapbox
    • Versions available: 1.0.0-rc4
  • Maintainership
    • Actively maintained
    • Security coverage – opted in, needs stable release
    • Test coverage
    • Number of open issues: 4 open issues, 1 of which is a bug
  • Usage stats:
    • 19 sites
  • Module features and usage
    • AI Auto-reference works with any reference fields, so it could find suitable taxonomy terms, nodes, etc
    • It does that by rendering a specified view mode, so it should with any kind of complex layout approach you may have implemented on your site
    • It will also automatically shorten your content to fit within your AI model’s token window, which you can also configure
    • The module extends Drupal’s main AI module, which means you can select which model to use, and probably means you can also use guardrails, and all the other powerful features that come with that ecosystem
    • Ai Auto-reference comes with default prompts, but you can also edit those if you really want to make sure you’re squeezing out every drop of relevance
    • You can also choose for which fields in each content type you want to generate suggestions, as well as whether you want the suggestions should be automatically applied, or whether you want them manually reviewed
    • As mentioned on the project page, you can already have AI suggest things like tags using the AI module without this project, but this may be a better choice if you want to make sure the recommendations stick to an existing set
Jun
17
2026
--

Security advisory: CVE-2026-9740 and CVE-2026-11933 in Percona Server for MongoDB

TL;DR: This advisory covers the two most important high-severity memory-safety vulnerabilities affecting MongoDB Community and our downstream Percona Server for MongoDB – CVE-2026-11933 and CVE-2026-9740. Both will be addressed in a single coordinated patch release, bundled with other recently revealed lower-scored CVE fixes: CVE-2026-9753, CVE-2026-9752, CVE-2026-9751, CVE-2026-9750, CVE-2026-9749, CVE-2026-9748, CVE-2026-9747, CVE-2026-9746, CVE-2026-9743, and CVE-2026-9741.

Fixes land in Percona Server for MongoDB patch window starting next week. The first high-vulnerability issue has nothing between it and your mongod process except your firewall. The second has a configuration off-switch you can flip during a maintenance window.  Read on to understand why, how, and what.

CVE-2026-9740 — the one that does not need credentials

A stack overflow in the BSON validator, specifically in the BSONColumn interleaved-reference handling. The validator’s depth tracking resets on mutual recursion between validation functions, so a sufficiently nested input exhausts the thread’s stack before any explicit limit fires. The result: mongod crashes.

CVSS 8.7. High severity. The reason it lands in High instead of merely Medium is the prerequisite for exploitation – there is none.

The attacker needs network reachability to a mongod listener. No credentials, no prior session, and no application interaction. One crafted message over the wire and the process is down. Repeated crashes are trivially repeatable, so an attacker who can reach the port can keep the instance offline for as long as they keep that reachability. The urgency of this issue comes from the audience – everyone with a TCP route to your database.

Upstream tracking: SERVER-125063. Affected versions are Percona Server for MongoDB 8.0 ? 8.0.23-10 and PSMDB 7.0 ? 7.0.34-19. The vulnerable BSONColumn code path was introduced in 7.0, so 6.0 and earlier are not in scope for this one.

CVE-2026-11933 — the one that does need credentials and permissions to read

The vulnerable code path is inside MongoDB Server’s server-side JavaScript engine, specifically in the BSON-to-array conversion routine. When a BSON document is materialized as a JavaScript array for use inside a server-side script, the engine can reach a state where it accesses memory that has already been freed. An attacker who can submit input that flows into that conversion path can shape what happens at the point of access.

Server-side JavaScript is reachable from the following surfaces:

  1. The $where query operator (deprecated in 8.0).
  2. The $function aggregation expression (deprecated in 8.0).
  3. The $accumulator aggregation expression (deprecated in 8.0).
  4. The mapReduce command (deprecated since 5.0).
  5. JavaScript functions stored in system.js.

MongoDB logs a warning when you run deprecated functions.


Prerequisites for exploitation:

  1. The attacker must be authenticated to MongoDB.
  2. The attacker must hold any role that permits running queries or aggregations against a collection. The built-in read role on a single database is sufficient.
  3. Server-side JavaScript must be enabled on the mongod instance. This is the default; many production deployments leave it enabled even when they do not use it.

CVSS 8.8. High severity. Two demonstrated outcomes:

  • Information disclosure (reading other content out of the mongod process memory) and
  • Denial of Service (crashing it).

Upstream tracking: SERVER-128125. Affected versions: every supported and End of Life Percona Server for MongoDB major from 4.4 through 8.0.

The good news and bad news

CVE-2026-11933 has a configuration off-switch. If your application does not use server-side JavaScript — $where, $function, $accumulator, mapReduce, or stored system.js functions — you can disable server-side JavaScript on the server, removing the attack surface entirely until you patch.

How to check whether your applications use server-side JavaScript before disabling:

  1. Enable MongoDB profiling at level 2 (all operations) on a representative mongod server for a representative time window. See details in Manage the database profiler.
  2. Search the system.profile collection for operations that include $where, $function, $accumulator, or mapReduce.
  3. Inspect application code paths and stored aggregation pipelines for the same operators. Check system.js in each database for stored functions.
  4. If any usage exists, treat disabling as not viable for those deployments and rely on patching plus the defense-in-depth controls below.

How to disable server-side JavaScript:

Add to your configuration file for mongodand mongos:

security: 
  javascriptEnabled: false

Or pass --noscripting on the command line. See the reference documentation for details about MongoDB Setting:  security.javascriptEnabled.

After a restart, any operation that reaches for server-side JavaScript will return an error. That is the catch: if your application does use one of those operators, this is not a viable mitigation for you, and you have to wait for the patch. If you are not sure whether your application uses them, turn on the database profiler at level 2 on a representative replica for a window long enough to be representative, then grep the profile collection for the operator names. Several teams have done this exercise in the last forty-eight hours and learned the answer is “no, we don’t actually use any of that.” The cost of disabling is then the cost of a mongod or mongos restart.

That was good news. Now the bad news: CVE-2026-9740 has no equivalent off-switch. The BSON validator is core to every client message; it cannot be disabled. Patch and network controls are the only options.

What is shipping, and when

The fixes for both CVEs will land in a single coordinated patch release for each supported major:

  • Percona Server for MongoDB 7.0 series — fix targeted for  June 23, 2026.
  • Percona Server for MongoDB 8.0 series — fix targeted for  June 25, 2026.
  • Percona Server for MongoDB 6.0 series — fix targeted for  June 25, 2026 (for CVE-2026-11933).

All dates are targets, not commitments. Plan one upgrade window covering all CVEs.

Percona is not building binary packages for the 5.x line. We’re being upfront about that — the calculus on extended support has a limit, and 5.x is past it for us. If you have a hard requirement on 5.x and the time pressure to meet it, the source is available for building. Percona customers on 5.x can open a ticket, and we’ll work on the case individually.

As usual, you can download patches from your package manager or Percona Software Downloads page.

On Kubernetes via the Percona Operator for MongoDB: same drill as usual. When the patched image is published, edit the image tag in your PerconaServerMongoDB custom resource and let the operator roll the cluster. Don’t wait for the June operator release to do it for you. See details in our documentation on how to Upgrade Percona Server for MongoDB. You do not need to wait for an operator release to apply a security fix.

What to do this week

In order of urgency, for most deployments:

  1. Confirm your mongod or mongos listeners are not reachable from any source you would not trust with a shell on the host. If you find an exposure, fix that first. CVE-2026-9740 turns any such exposure into a DoS primitive.
  2. For deployments that do not use server-side JavaScript, disable it. Full mitigation for CVE-2026-11933 within a single mongod restart.
  3. Plan your upgrade window for the week the relevant fixed release lands. One window. Both CVEs. Plus, the others scored lower.
  4. Audit which roles in your deployment can run ad-hoc queries or aggregations. The bar for CVE-2026-11933 is the standard read role, so the population of potential attackers is larger than for most memory-safety defects.

One closing point, because it has come up several times in customer conversations this week. For a deployment behind tight network controls, the post-authenticated bug is the more urgent one. For a deployment reachable from broader networks — public cloud, shared internal LANs, multi-tenant infrastructure — the pre-authenticated bug is. Triage by your exposure, not by their CVSS.


Questions, or a deployment you’re not sure how to triage? Find us on the Percona Forum, or, for customers, in the support portal.

Reviewed by Ivan Groenewold. Vetted for technical accuracy as of June 17, 2026.

The post Security advisory: CVE-2026-9740 and CVE-2026-11933 in Percona Server for MongoDB appeared first on Percona.

Jun
16
2026
--

Extending pt-archiver with a Partition-Aware Plug-in for Fast Retention Policy Enforcement

Managing data retention policies is one of the most common operational tasks in MySQL.

Applications continuously generate transactional, audit, logging, telemetry, and event data. Over time, these tables can grow to billions of rows, causing:

  • Larger backups
  • Longer recovery times
  • Reduced buffer pool efficiency
  • Slower index maintenance
  • Increased storage costs
  • Degraded query performance

To address these problems, organizations typically implement retention policies based on dates or timestamps. Examples include deleting events older than 90 days or purging session data older than 30 days and so forth. The deleted data can then eventually be archived somewhere else, like in another DBMS or on external files.

One of the most widely used tools for implementing these policies in MySQL ecosystems is pt-archiver, part of the Percona Toolkit.

This article provides a review of what pt-archiver is and how to use it, but in particular it focuses on the fact this tool is not partitioning aware, and this can make the deletion phase more costly. The article shows how to extend pt-archiver with a Perl plugin to make it aware of partitioning.

 

What is pt-archiver?

pt-archiver is a command-line utility from Percona Toolkit designed to:

  • Archive rows from MySQL tables
  • Purge rows from MySQL tables
  • Move data between tables into the local database or a remote one
  • Export rows into files

In a few words: implementing retention policies safely.

The tool processes rows incrementally in chunks, avoiding massive transactions and reducing impact on production systems.

Example:

pt-archiver \
  --source h=localhost,D=mydb,t=events \
  --where "created_at &lt; '2026-05-01'" \
  --purge \
  --limit 1000 \
  --commit-each

This command:

  • Scans rows matching the WHERE condition
  • Processes them in chunks of 1000 rows
  • Commits every chunk
  • Deletes matching rows from the source table

pt-archiver provides several advantages compared to ad-hoc DELETE statements.

Instead of running:

DELETE FROM events
WHERE created_at &lt; '2026-05-01';

which may:

  • Lock rows for a long time
  • Generate massive undo/redo logs
  • Create replication lag
  • Exhaust transaction logs

pt-archiver processes rows incrementally to make the process overhead less impactful for the database performance.

pt-archiver implementation permits flexible archival strategies

Rows can be copied to another table on a remote host, exported to files or removed completely

More details: ps://docs.percona.com/percona-toolkit/pt-archiver.html

Example: Copy rows to a remote archive table

The following example archives rows older than 90 days from a local table into an archive table hosted on a remote MySQL server:

pt-archiver \
  --source h=localhost,D=sales,t=orders,u=archiver,p=secret \
  --dest h=archive-server,D=archive,t=orders_archive,u=archiver,p=secret \
  --where "created_at &lt; '2026-05-01'" \
  --limit 1000 \
  --commit-each \
  --progress 10000 \
  --statistics

In this example:

  • –source defines the source table
  • –dest defines the remote archive destination
  • –where selects rows eligible for archival
  • –limit controls batch size
  • –commit-each commits every batch independently to reduce transaction overhead

-progress reports progress every 10,000 rows

If rows should be removed from the source table after being copied, add –purge

Example: Export rows to a file

The following example exports rows older than one year into a text file:

pt-archiver \
  --source h=localhost,D=sales,t=orders,u=archiver,p=secret \
  --where "created_at &lt; NOW() - INTERVAL 1 YEAR" \
  --file '/tmp/orders_archive_%Y-%m-%d.txt' \
  --output-format csv \
  --limit 1000 \
  --commit-each \
  --progress 10000 \
  --statistics

In this example:

  • –file specifies the output file
  • -output-format csv exports rows in CSV format
  • Date placeholders in the filename are expanded automatically

Rows can optionally be deleted from the source table by adding –purge

This allows pt-archiver to be used both for data retention and for offline archival workflows.

The Hidden Cost of DELETE Statements

Although pt-archiver is much safer than massive DELETE operations, it still fundamentally relies on DELETE statements.

This is a critical point.

Even when there are proper indexes, the rows are processed in chunks, and transactions are small; the large-scale DELETE operations remain expensive.

Deleting rows is expensive in InnoDB because it involves:

  • Locating rows via indexes
  • Modifying clustered indexes
  • Modifying secondary indexes
  • Generating undo logs
  • Generating redo logs
  • Purge thread processing
  • Replication event generation
  • Page fragmentation

When deleting billions of rows, the overhead becomes enormous.

Indexes help for sure, but only partially.

Consider:

DELETE FROM events
WHERE created_at &lt; '2024-01-01';

If created_at is indexed, MySQL can efficiently locate rows.

However, locating rows efficiently is only part of the cost. The actual delete operations still require all those things we mentioned above.

At considerable scale, this becomes expensive.

Why RANGE Partitioning is Superior for Retention Policies

For time-based retention policies, partitioning is often dramatically more efficient. In particular, RANGE partitioning is very useful for these cases.

Example:

CREATE TABLE events (
    id BIGINT NOT NULL,
    created_at DATETIME NOT NULL,
    payload JSON,
    PRIMARY KEY(id, created_at)
)

PARTITION BY RANGE (TO_DAYS(created_at)) (
    PARTITION p202604 VALUES LESS THAN (TO_DAYS('2026-05-01')),
    PARTITION p202605 VALUES LESS THAN (TO_DAYS('2026-06-01')),
    PARTITION p202606 VALUES LESS THAN (TO_DAYS('2026-07-01'))
);

With partitioning, dropping old data becomes:

ALTER TABLE events DROP PARTITION p202604;

This operation is dramatically faster than running a DELETE.

Dropping a partition:

  • Removes an entire physical partition
  • Avoids row-by-row DELETE
  • Avoids undo generation for each row
  • Avoids secondary index maintenance per row
  • Minimizes redo generation
  • Is nearly metadata-only

This can remove millions or billions of rows in a matter of seconds without the same large cost of DELETE.

The Problem: pt-archiver is Not Partition-Aware

Unfortunately, pt-archiver does not automatically understand partitioning strategies.

Even if the table is partitioned or the retention policy perfectly matches partition boundaries, pt-archiver still executes DELETE statements.

Example:

pt-archiver \
  --where "created_at &lt; NOW() - INTERVAL 90 DAY" \
  --purge

Internally, this still produces DELETE … instead of ALTER TABLE … DROP PARTITION …

This means organizations may lose the major operational benefits of partitioning, or they need to implement custom scripts for managing the selection of rows to copy using pt-archiver and then use DROP PARTITION separately from the tool. That is doable, and to be honest, not too complicated, but why not make pt-archiver aware of partitioning for some specific use cases?

Extending pt-archiver with Pulg-ins

Fortunately, pt-archiver supports Perl plug-ins.

A plug-in can do plenty of things. Like: inspect runtime conditions, interact with MySQL, override behaviors, and execute custom logic

This gives us an opportunity to implement partition-aware retention handling.

The plug-in can:

  1. Inspect partition definitions
  2. Analyze the WHERE condition
  3. Determine which partitions are fully expired
  4. Execute ALTER TABLE DROP PARTITION
  5. Prevent row-by-row DELETE processing

This approach combines the scheduling/orchestration power of pt-archiver with the efficiency of partition pruning.

Plug-in Design

Our plug-in will:

  • Connect using the pt-archiver DB handle
  • Inspect INFORMATION_SCHEMA.PARTITIONS
  • Identify partitions older than the retention cutoff
  • Issue DROP PARTITION statements
  • Log actions
  • Skip DELETE processing

Assumptions:

  • The table is RANGE partitioned
  • Partitions are DATETIME based using the TO_DAYS() function to define ranges
  • Partition naming convention contains dates
  • Retention policy aligns with partition boundaries; if the plugin cannot determine a specific boundary, pt-archiver does nothing

Full Perl Plug-in for pt-archiver

package pt_archiver_partition_drop;

use strict;
use warnings;

sub new {
    my ($class, %args) = @_;
    my $self = {
        dbh        =&gt; $args{dbh},
        db         =&gt; $args{db},
        tbl        =&gt; $args{tbl},
        statistics =&gt; {},
    };

    bless $self, $class;
    return $self;
}

sub statistics {
    my ($self) = @_;
    return $self-&gt;{statistics};
}


sub before_begin {
    my ($self) = @_;
    my $dbh = $self-&gt;{dbh} or die "Missing dbh from pt-archiver\n";
    my $db  = $self-&gt;{db}  or die "Missing db from pt-archiver plugin args\n";
    my $tbl = $self-&gt;{tbl} or die "Missing tbl from pt-archiver plugin args\n";
    my $where  = _get_cmdline_option('where');
    my $dryrun = $ENV{PT_PARTITION_DROP_DRY_RUN} ? 1 : 0;

    die "Missing --where from original command line\n" unless $where;

    print "PLUGIN before_begin called\n";
    print "DB=$db TABLE=$tbl\n";
    print "WHERE=$where\n";
    print "PLUGIN_DRY_RUN=$dryrun\n";

    my ($column, $cutoff_date) = _parse_where($where);

    my $partitions = _get_partitions($dbh, $db, $tbl);

    if (!@$partitions) {
        print "Table `$db`.`$tbl` is not partitioned. Refusing DELETE.\n";
        exit(0);
    }

    my $partition_expr = $partitions-&gt;[0]-&gt;{expression};
    die "Missing PARTITION_EXPRESSION\n"
        unless defined $partition_expr &amp;&amp; length $partition_expr;

    print "Partition expression: $partition_expr\n";

    my $cutoff_value = _evaluate_cutoff(
        $dbh,
        $partition_expr,
        $column,
        $cutoff_date,
    );

    print "Cutoff date: $cutoff_date\n";
    print "Cutoff boundary value: $cutoff_value\n";

    my $matched;

    for my $p (@$partitions) {
        next if !defined $p-&gt;{description};
        next if uc($p-&gt;{description}) eq 'MAXVALUE';

        if ($p-&gt;{description} == $cutoff_value) {
            $matched = $p;
            last;
        }
    }


    if (!$matched) {
        print "No exact partition boundary matches cutoff $cutoff_value. Refusing DELETE.\n";
        exit(0);
    }

    print "Matched boundary partition: $matched-&gt;{name}, position $matched-&gt;{position}\n";

    my @drop;

    for my $p (@$partitions) {
        next if !defined $p-&gt;{description};
        next if uc($p-&gt;{description}) eq 'MAXVALUE';

        if ($p-&gt;{position} &lt;= $matched-&gt;{position}) {
            push @drop, $p-&gt;{name};
            print "Eligible for DROP: $p-&gt;{name}, boundary $p-&gt;{description}\n";
        }
    }

    if (!@drop) {
        print "No partitions eligible for DROP. Refusing DELETE.\n";
        exit(0);
    }

    my $sql = sprintf(
        "ALTER TABLE %s.%s DROP PARTITION %s",
        _quote_ident($db),
        _quote_ident($tbl),
        join(", ", map { _quote_ident($_) } @drop),
    );

    print "SQL: $sql\n";

    if ($dryrun) {
        print "PT_PARTITION_DROP_DRY_RUN enabled. Not executing DROP PARTITION.\n";
    }
    else {
        $dbh-&gt;do($sql);
        print "Dropped partitions: " . join(", ", @drop) . "\n";
    }

    $self-&gt;{statistics}-&gt;{partitions_dropped} = scalar @drop;

    exit(0);
}


sub _parse_where {
    my ($where) = @_;

    $where =~ s/^\s+|\s+$//g;

    die "Only WHERE format supported: created_at &lt; 'YYYY-MM-DD'\n"
        unless $where =~ /^`?([A-Za-z0-9_]+)`?\s*&lt;\s*'(\d{4}-\d{2}-\d{2})'\s*$/;

    return ($1, $2);
}

sub _evaluate_cutoff {
    my ($dbh, $partition_expr, $column, $cutoff_date) = @_;

    my $expr = $partition_expr;
    $expr =~ s/`//g;

    die "Partition expression does not reference column `$column`: $partition_expr\n"
        unless $expr =~ /\b\Q$column\E\b/i;

    $expr =~ s/\b\Q$column\E\b/'$cutoff_date'/ig;

    die "Unsafe generated expression: $expr\n"
        unless $expr =~ /^[A-Za-z0-9_\s\(\)\+\-\*\/,\.'":]+$/;

    my $sql = "SELECT $expr";

    print "Boundary evaluation SQL: $sql\n";

    my ($value) = $dbh-&gt;selectrow_array($sql);

    die "Cannot evaluate cutoff expression: $sql\n"
        unless defined $value;

    return $value;
}

sub _get_partitions {
    my ($dbh, $db, $tbl) = @_;

    my $sql = q{
        SELECT
            PARTITION_NAME,
            PARTITION_DESCRIPTION,
            PARTITION_EXPRESSION,
            PARTITION_ORDINAL_POSITION
        FROM INFORMATION_SCHEMA.PARTITIONS
        WHERE TABLE_SCHEMA = ?
          AND TABLE_NAME = ?
          AND PARTITION_NAME IS NOT NULL
        ORDER BY PARTITION_ORDINAL_POSITION
    };

    my $sth = $dbh-&gt;prepare($sql);
    $sth-&gt;execute($db, $tbl);
    my @partitions;

    while (my $row = $sth-&gt;fetchrow_hashref()) {
        push @partitions, {
            name        =&gt; $row-&gt;{PARTITION_NAME},
            description =&gt; $row-&gt;{PARTITION_DESCRIPTION},
            expression  =&gt; $row-&gt;{PARTITION_EXPRESSION},
            position    =&gt; $row-&gt;{PARTITION_ORDINAL_POSITION},
        };
    }

    return \@partitions;
}


sub _get_cmdline_option {

    my ($name) = @_;

    my $opt = "--$name";

    for (my $i = 0; $i &lt; @ARGV; $i++) {
        if ($ARGV[$i] eq $opt &amp;&amp; defined $ARGV[$i + 1]) {
            return $ARGV[$i + 1];
        }

        if ($ARGV[$i] =~ /^\Q$opt\E=(.*)$/) {
            return $1;
        }
    }

    if (open my $fh, '&lt;', "/proc/$$/cmdline") {
        local $/;
        my $raw = &lt;$fh&gt;;
        close $fh;

        my @cmd = split /\0/, $raw;

        for (my $i = 0; $i &lt; @cmd; $i++) {
            if ($cmd[$i] eq $opt &amp;&amp; defined $cmd[$i + 1]) {
                return $cmd[$i + 1];
            }

            if ($cmd[$i] =~ /^\Q$opt\E=(.*)$/) {
                return $1;
            }
        }
    }

    return undef;
}



sub _quote_ident {

    my ($ident) = @_;

    die "Invalid identifier: $ident\n"
        unless defined $ident &amp;&amp; $ident =~ /^[A-Za-z0-9_]+$/;

    return "`$ident`";
}

1;

Create the file named  pt_archiver_partition_drop.pm into the /usr/local/share/perl5 path.

Also set the environment variable PERL5LIB to let pt-archiver where to find the Perl package

export PERL5LIB=/usr/local/share/perl5

Example Usage

First, create the partitioned table events and insert some fake data.

DROP TABLE IF EXISTS events;


CREATE TABLE events (
  id BIGINT NOT NULL,
  created_at DATETIME NOT NULL,
  payload JSON DEFAULT NULL,
  PRIMARY KEY (id, created_at)
)
PARTITION BY RANGE (TO_DAYS(created_at)) (
  PARTITION p202604 VALUES LESS THAN (TO_DAYS('2026-05-01')),
  PARTITION p202605 VALUES LESS THAN (TO_DAYS('2026-06-01')),
  PARTITION p202606 VALUES LESS THAN (TO_DAYS('2026-07-01')),
  PARTITION pmax VALUES LESS THAN MAXVALUE
);

INSERT INTO events (id, created_at, payload) VALUES

-- p202604
(1,  '2026-04-01 08:00:00', JSON_OBJECT('event', 'login',    'user', 'alice')),
(2,  '2026-04-03 09:15:00', JSON_OBJECT('event', 'view',     'page', 'home')),
(3,  '2026-04-05 10:30:00', JSON_OBJECT('event', 'click',    'button', 'signup')),
(4,  '2026-04-08 11:45:00', JSON_OBJECT('event', 'search',   'term', 'mysql')),
(5,  '2026-04-10 12:00:00', JSON_OBJECT('event', 'purchase', 'amount', 100)),
(6,  '2026-04-14 13:20:00', JSON_OBJECT('event', 'logout',   'user', 'alice')),
(7,  '2026-04-18 14:35:00', JSON_OBJECT('event', 'download', 'file', 'report.pdf')),
(8,  '2026-04-22 15:50:00', JSON_OBJECT('event', 'upload',   'file', 'image.png')),
(9,  '2026-04-26 16:05:00', JSON_OBJECT('event', 'click',    'button', 'buy')),
(10, '2026-04-30 23:59:59', JSON_OBJECT('event', 'month_end')),

-- p202605

(11, '2026-05-01 00:00:00', JSON_OBJECT('event', 'login',    'user', 'bob')),
(12, '2026-05-03 08:10:00', JSON_OBJECT('event', 'view',     'page', 'pricing')),
(13, '2026-05-06 09:20:00', JSON_OBJECT('event', 'search',   'term', 'percona')),
(14, '2026-05-09 10:30:00', JSON_OBJECT('event', 'purchase', 'amount', 250)),
(15, '2026-05-12 11:40:00', JSON_OBJECT('event', 'logout',   'user', 'bob')),
(16, '2026-05-16 12:50:00', JSON_OBJECT('event', 'download', 'file', 'backup.sql')),
(17, '2026-05-20 13:00:00', JSON_OBJECT('event', 'upload',   'file', 'data.csv')),
(18, '2026-05-24 14:10:00', JSON_OBJECT('event', 'click',    'button', 'subscribe')),
(19, '2026-05-28 15:20:00', JSON_OBJECT('event', 'view',     'page', 'docs')),
(20, '2026-05-31 23:59:59', JSON_OBJECT('event', 'month_end')),

-- p202606

(21, '2026-06-01 00:00:00', JSON_OBJECT('event', 'login',    'user', 'carol')),
(22, '2026-06-03 08:05:00', JSON_OBJECT('event', 'search',   'term', 'partitioning')),
(23, '2026-06-06 09:15:00', JSON_OBJECT('event', 'view',     'page', 'dashboard')),
(24, '2026-06-09 10:25:00', JSON_OBJECT('event', 'purchase', 'amount', 500)),
(25, '2026-06-12 11:35:00', JSON_OBJECT('event', 'logout',   'user', 'carol')),
(26, '2026-06-16 12:45:00', JSON_OBJECT('event', 'login',    'user', 'dave')),
(27, '2026-06-20 13:55:00', JSON_OBJECT('event', 'download', 'file', 'archive.zip')),
(28, '2026-06-24 14:05:00', JSON_OBJECT('event', 'upload',   'file', 'video.mp4')),
(29, '2026-06-28 15:15:00', JSON_OBJECT('event', 'click',    'button', 'checkout')),
(30, '2026-06-30 23:59:59', JSON_OBJECT('event', 'month_end')),

-- pmax
(31, '2026-07-01 00:00:00', JSON_OBJECT('event', 'login',    'user', 'eve')),
(32, '2026-07-05 08:30:00', JSON_OBJECT('event', 'view',     'page', 'future')),
(33, '2026-07-10 09:45:00', JSON_OBJECT('event', 'search',   'term', 'maxvalue')),
(34, '2026-08-01 10:00:00', JSON_OBJECT('event', 'purchase', 'amount', 750)),
(35, '2026-09-01 11:15:00', JSON_OBJECT('event', 'retained_future'));

 

Now you can run the following command to delete all rows before the 1st of May, which, by the way, matches the entire first partition in the table.

pt-archiver \
  --source h=localhost,D=mydb,t=events,m=pt_archiver_partition_drop \
  --where "created_at &lt; '2026-05-01'" \
  --purge

 

Notice the Perl plugin must be indicated with the m option in the DSN string.

In practice:

  • pt-archiver initializes
  • The plug-in runs
  • Partitions are dropped
  • No DELETE statements are executed

Here is what you get from the execution of the above command:

PLUGIN before_begin called
DB=mydb TABLE=events
WHERE=created_at &lt; '2026-05-01'
PLUGIN_DRY_RUN=0
Partition expression: to_days(`created_at`)
Boundary evaluation SQL: SELECT to_days('2026-05-01')
Cutoff date: 2026-05-01
Cutoff boundary value: 740102
Matched boundary partition: p202604, position 1
Eligible for DROP: p202604, boundary 740102
SQL: ALTER TABLE `mydb`.`events` DROP PARTITION `p202604`
Dropped partitions: p202604

You can simply verify the table has been managed correctly:

SELECT * FROM mydb.events;

SHOW CREATE TABLE mydb.events;

 

Now TRUNCATE the table and recreate the data and try now to specify the where conditions that match a RANGE that is not the first in the list of the boundaries.

pt-archiver \
  --source h=localhost,D=mydb,t=events,m=pt_archiver_partition_drop \
  --where "created_at &lt; '2026-06-01'" \
  --purge

You should get:

PLUGIN before_begin called
DB=mydb TABLE=events
WHERE=created_at &lt; '2026-06-01'
PLUGIN_DRY_RUN=0
Partition expression: to_days(`created_at`)
Boundary evaluation SQL: SELECT to_days('2026-06-01')
Cutoff date: 2026-06-01
Cutoff boundary value: 740133
Matched boundary partition: p202605, position 2
Eligible for DROP: p202604, boundary 740102
Eligible for DROP: p202605, boundary 740133
SQL: ALTER TABLE `mydb`.`events` DROP PARTITION `p202604`, `p202605`
Dropped partitions: p202604, p202605

In this case, two partitions have been identified and dropped.

 

Truncate the table and recreate the data again. Try now to provide a WHERE condition that does not match any of the boundaries in the RANGE.

pt-archiver \
  --source h=localhost,D=mydb,t=events,m=pt_archiver_partition_drop \
  --where "created_at &lt; '2026-04-25'" \
  --purge

 

You get the following:

PLUGIN before_begin called
DB=mydb TABLE=events
WHERE=created_at &lt; '2026-04-25'
PLUGIN_DRY_RUN=0
Partition expression: to_days(`created_at`)
Boundary evaluation SQL: SELECT to_days('2026-04-25')
Cutoff date: 2026-04-25
Cutoff boundary value: 740096
No exact partition boundary matches cutoff 740096. Refusing DELETE.

As expected, the tool now refuses to execute anything if it doesn’t find an exact match.

 

Operational Benefits

This approach provides major advantages.

Dropping partitions is vastly faster than deleting rows, and minimal binary logging is needed, compared to billions of row deletes. There is no massive transactional overhead for managing undo logs and purging. You get then a better InnoDB Buffer Pool stability because of less page churn.

In the end, retention jobs are completed quickly and consistently in a predictable way and at the minimal cost.

 

Important Caveats

Partition Boundaries Must Match Retention Policy

If partitions contain mixed retention windows, DROP PARTITION may remove too much data. For this reason, ensure correct partition design.

Recommended:

  • daily partitions
  • weekly partitions
  • monthly partitions

aligned with business retention requirements.

Metadata Locks

ALTER TABLE DROP PARTITION still acquires metadata locks.

Test carefully in production.

Backup Awareness

Ensure dropped partitions are no longer needed before removal or use pt-archiver to also copy the data into a remote server or dump the data into a CSV file before running the DROP PARTITION.

 

Possible Enhancements

The plug-in can be extended further.

Potential improvements:

  • Support for daily partitions
  • Support for UNIX timestamp partitions
  • Dry-run reporting
  • Automatic partition creation
  • Push Slack notifications
  • Export Prometheus metrics
  • Safety checks for replicas
  • GTID-aware orchestration
  • Integration with pt-online-schema-change workflows

These are just some ideas I had meanwhile doing my tests. What you can do by implementing a Perl plugin is only limited by your imagination and your real needs.

Conclusion

pt-archiver remains an excellent tool for implementing retention policies and archival workflows.

However, DELETE-based purging becomes increasingly expensive at scale, even with proper indexing and chunked processing.

For large time-series or historical datasets, RANGE partitioning is often a dramatically superior strategy.

The challenge is that pt-archiver does not natively leverage partition-level operations.

Fortunately, its Perl plug-in architecture allows advanced users to extend its behavior and implement partition-aware cleanup logic.

By combining:

  • pt-archiver orchestration
  • MySQL RANGE partitioning
  • Custom Perl plug-ins

Organizations can achieve:

  • Faster retention enforcement
  • Lower operational overhead
  • Smaller replication impact
  • Dramatically improved scalability

For large MySQL deployments, this hybrid approach can turn multi-hour purge operations into near-instant metadata operations.

The use case presented in this article is limited to a specific scenario, but you can reuse it or customize it if you have a different kind of RANGE partitioning, for example, not using TO_DAYS().

Take this as just an example of how you can extend pt-archiver. What you can do for real is driven by your needs and/or only limited by your imagination.

More info about extending pt-archiver:
https://docs.percona.com/percona-toolkit/pt-archiver.html#extending

 

The post Extending pt-archiver with a Partition-Aware Plug-in for Fast Retention Policy Enforcement appeared first on Percona.

Jun
15
2026
--

Talking Drupal #557 – Test-Driven Drupal eBook

Today we are talking about Test Driven Development, ebooks, and Drupal with guest Oliver Davies. We’ll also cover Juicer Social Feed as our module of the week.

For show notes visit: https://www.talkingDrupal.com/557

Topics

  • What Is Test Driven Drupal
  • Why Automated Tests Matter
  • How TDD Works
  • AI and Test Quality
  • Balancing Test Coverage
  • When to Write Tests
  • Why Write the Book
  • Why Write an Ebook
  • From Email Course to Ebook
  • Ebook vs Print Tradeoffs
  • Who the Book Helps
  • What You Will Learn
  • Keeping Content Updated
  • Publishing Tools Workflow
  • Lessons and Drupal Changes
  • Podcast and Future Books
  • Mob Programming Explained
  • Free Ebook and Wrap Up

Resources

Guests

Oliver Davies – oliverdavies.uk opdavies

Hosts

Nic Laflin – nLighteneddevelopment.com nicxvan John Picozzi – epam.com johnpicozzi Scott Falconer – managing-ai.com scott-falconer

MOTW Correspondent

Martin Anderson-Clutz – mandclu.com mandclu

  • Brief description:
    • Have you ever wanted to embed social feeds into your Drupal website? There’s a module for that.
  • Module name/project name:
  • Brief history
    • How old: created in Mar 2026 by Denis Omerovi? (drupalchille)
    • Versions available: 1.0.2, that works with Drupal 10.3 or 11
  • Maintainership
    • Actively maintained (version released today!)
    • No open issues
  • Usage stats:
    • 4 sites
  • Module features and usage
    • This module embeds an aggregated social media feed from Juicer.io directly into Drupal as a configurable block. It natively supports content from Instagram, LinkedIn, Facebook, X (Twitter), TikTok, Bluesky, YouTube, and more.
    • Traditionally, displaying feeds from platforms like Facebook, X, or Instagram requires creating developer accounts, managing rotating OAuth tokens, and keeping up with constantly shifting API restrictions. Juicer handles all API authentication on its platform, shielding your website from sudden breaking changes by individual social networks.
    • To use this module, you will need an active account on Juicer.io. They offer both free and paid tiers depending on how many sources you want to aggregate and how frequently you need the feed to sync.
    • The module is created and maintained by the official Juicer.io team. That should ensure that the module is closely aligned with the product’s features and any potential API changes over time.
    • The embedded feed is made available as a Drupal block, to make it easy to control where it should appear on your site.
    • When placing the Juicer block, the UI exposes several user-friendly settings:
    • Feed Slug: Just paste your unique Juicer feed ID to establish the connection.
    • Post Limit: Control exactly how many items populate initially.
    • Source Filtering: If your Juicer account aggregates five networks, but you only want to show LinkedIn posts on a specific page, you can filter down to a single network right inside the block settings.
    • SEO/Semantic Control: You can set titles/subtitles and choose the exact heading level hierarchy ( through ) to ensure your pages remain semantically correct and accessible.
    • I did get a chance to test out the module and the service today, and I can tell you from experience, it’s a huge improvement on having to create and pull in feeds directly. I did notice that the block didn’t show up in the Drupal Canvas component library, but I was able to determine that two lines of code to declare the block as FullyValidatable were all that was needed. So I opened a Feature Request to add that, and it was merged in and a new release cut in less than an hour. So it’s now Drupal Canvas compatible too!
    • It’s worth pointing out that the standard Juicer’s embed script loads HTMX, which conflicts with the version of HTMX included in Drupal 11 core. As a result, the module fetches feed HTML directly from the Juicer API and includes a minimal HTMX shim to prevent errors.
    • John, you nominated this module, why don’t you start us off by telling us about how you got started using it?
Jun
14
2026
--

Group Replication VS Percona XtraDB Cluster: The True Cost of Consistency

Overview

When building high-availability MySQL environments, the choice between MySQL Group Replication (GR) and Percona XtraDB Cluster (PXC) often comes down to how they handle the eternal database dilemma: data consistency versus performance.       

While both provide “synchronous-like” replication, they approach the problem of stale reads—reading data that has been committed on one node but not yet applied on another—in distinct ways. Understanding these differences, and the performance penalties associated with fixing them, is critical for any production environment.

Technology Overviews

MySQL Group Replication (GR)

Group Replication is the native, albeit more recent, high-availability solution built by Oracle for MySQL. It is based on a distributed state machine architecture and uses the Paxos consensus protocol.

  • Mechanism: When a transaction is committed, it is sent to all group members. The members must agree (consensus) on the order of transactions. Once a majority agrees, the transaction is “certified” and committed on the originator.
  • Replication Type: Virtually synchronous. The consensus ensures the data is received and ordered across nodes, but the actual applying of the data to the database happens asynchronously in the background.

Percona XtraDB Cluster (PXC)

PXC is an open-source enterprise solution based on Percona Server for MySQL and the Galera Replication library, which is the first and most mature virtually synchronous solution for MySQL.

  • Mechanism: When a node commits a transaction, it sends it to all other members of the Primary component (active group). All nodes must certify the transaction (check for conflicts), this is done on each node in the cluster, including the node that originates the write-set, before the originating node can finalize the commit.
  • Replication Type: Strictly synchronous (up to the certification level), asynchronous afterward. If the certification test fails, the node drops the write-set and the cluster rolls back the original transaction. If the test succeeds, however, the transaction commits and the write-set is applied to the rest of the cluster.

The Battle Against “Stale Reads”: Why It Matters

The most critical distinction for developers is whether a SELECT query on Node B will immediately see the INSERT just performed on Node A.

In a distributed system, there is a microsecond-to-millisecond gap between a transaction being globally ordered (everyone knows it happened) and being locally applied (the data is physically readable in the table). Reading executed on a secondary during this gap results in a stale read.

Why is avoiding stale reads so critical?

While a stale read might just mean a user temporarily sees their old profile picture after updating it, in many business cases, it breaks the application’s core logic:

  1. Financial Transactions: A user deposits $100 on the Primary node and immediately refreshes their balance page, which reads from a Replica. If the read is stale, the balance hasn’t updated. The user panics, thinking their money is lost.
  2. E-commerce & Inventory: A customer buys the last item in stock. The next user immediately loads the product page. A stale read tells the second user the item is still available, leading to a cancelled order and a frustrated customer.
  3. Security & Access: A user changes their password or updates a critical permission. If the next authentication request hits a node lagging by just a fraction of a second, their valid login might be rejected, or a revoked session might still be active.

To prevent these scenarios, we must tell the database to enforce strict consistency. But how do GR and PXC handle this, and what does it cost?

Consistency Controls Comparison

Both Group Replication and Percona XtraDB Cluster provide built-in mechanisms to enforce consistency and eliminate stale reads when your application demands it. However, they approach this problem using entirely different variables and distinct levels of granularity. The table below breaks down the specific controls each technology offers, highlighting exactly what it takes to force a node to serve fresh data.

Feature MySQL Group Replication Percona XtraDB Cluster
Default Behavior Reads on secondaries may be stale because the applier thread might be lagging after consensus. Reads on secondaries may be stale due to asynchronous background applying.
Stale Read Fix Uses the group_replication_consistency variable. Uses the wsrep-sync-wait variable.
Consistency Levels Offers EVENTUAL, BEFORE, AFTER, and BEFORE_AND_AFTER. Offers granular levels from 0 (default, no checks) up to 7 (checks on all READ, UPDATE, DELETE, INSERT, and REPLACE statements).
The Fix Setting to AFTER ensures the next read is fresh. Setting to 7 ensures we have a comparable scenario with GR. However in PXC setting wsrep_sync_wait = 1 will be enough to avoid stale reads.

The True Cost of Being Consistent

If we know stale reads are bad, why don’t we just enforce strict consistency everywhere? 

An image can help to understand:

Because in distributed databases, consistency is incredibly expensive. To test this, we used a 3-node internal lab environment to run a Sysbench-based TPC-C derivative test (50/50 read/write split, running for 600 seconds, scaling from 1 to 1024 threads).

You can find the detailed machine specifications here. The benchmarks were executed using a TPC-C derivative test based on sysbench. Finally—and crucially—you can review the configuration files used for the tests. I maintained the same baseline MySQL configuration across the board, only adjusting the parameters specific to each replication technology.

 

Scenario 1: Default (Relaxed) Consistency

(GR = EVENTUAL, PXC = wsrep-sync-wait 0)

I want to remind, that MySQL CE and Percona Server are running using Group Replication, while PXC is using galera.

With default settings, both systems allow stale reads.

Both technologies scales well up to 128 threads:

  • Group Replication performs exceptionally well, handling up to 15K operations/sec before dropping off after 128 threads.
  • PXC (Galera) is slightly less efficient at peak but scales very nicely and predictably.

At this level, the lag between the moment of commit and the moment the server returns the answer is minimal. But we are entirely exposed to stale reads.

Scenario 2: Enforced Consistency (The Cost)

(GR = AFTER, PXC = wsrep-sync-wait 7)

When we configure the servers to prevent stale reads, the systems must wait for transactions to be fully applied before returning a read. This is where the architectural differences become glaringly apparent:

  • PXC (Galera): Performance drops but not too much from a peak of ~9K ops/sec (in the previous test)  to roughly ~8.5K ops/sec. This is a hit but not huge and the database remains highly functional and stable.
  • Group Replication: Performance catastrophically drops from ~15K ops/sec (in the previous test) to a staggering ~3.8K ops/sec.

This is the crucial takeaway

Enforcing strict consistency in Group Replication results in a massive ~75% performance penalty. The latency between the commit and the server response increases significantly compared to PXC. 

The intermediate way

There is another approach which is to inject the higher consistency only when it is really needed.

The Solution: Session-Level Consistency You do not need, and should not use, full consistency at the global level for general cases. Instead, force consistency only when and where it is critical.

While for Group Replication there is no support for SQL injection hints like SELECT /*+ SET_VAR(…) */, you can enforce this at the session level right before a critical read:

SET SESSION group_replication_consistency = 'AFTER';
-- OR for PXC:
SET SESSION wsrep_sync_wait = 7;

 

To note that  PXC offers more flexibility and you can use hints:

select /*+ SET_VAR(wsrep_sync_wait=7) */ @@session.wsrep_sync_wait ,@@global.wsrep_sync_wait;
+---------------------------+--------------------------+
| @@session.wsrep_sync_wait | @@global.wsrep_sync_wait |
+---------------------------+--------------------------+
|                         7 |                        0 |
+---------------------------+--------------------------+

 

 

By isolating these variables to specific sessions (like the immediate redirect after a password change or a checkout process), you ensure data integrity exactly where the business requires it, while allowing the rest of your application to enjoy the high-speed performance of relaxed consistency. 

PXC: The performance drop is minimal and the solution is able to provide a consistent delivery with nice scalability up to 256 threads.

Group Replication: The solution suffers from a significant drop, not as if we set the AFTER condition at global level, but still we see a drop of ~52%. 

Comparing the two solutions we can see that PXC is able to deal with the additional requested consistency better. 

 

Additional differences

But these are not the only differences we can immediately see.
Performing a comparison about resources utilization, we can see that while both solutions move the same amount of data as IO operations:

 

Yes, for exactly the same load and traffic Group Replication consumes 8GB more than PXC, which in this environment represents 26% memory more, over total available.

Cost that is reflected also as CPU utilization.

 

Conclusion: How to Survive the Cost

How impactful is enforcing strict consistency at a global level in a production environment? Massively. If you blindly enforce strict consistency globally without understanding your architecture, you will decimate your database throughput. Here is the reality of how the two solutions handle that tax:

  • The Group Replication Reality: By default (using EVENTUAL consistency), MySQL Group Replication behaves essentially as semi-synchronous replication paired with an automated topology manager (see The Failover Brownout: Rethinking High Availability in MySQL Group Replication). The Primary is allowed to forge ahead and serve traffic even if the Secondaries are lagging significantly behind. The moment you demand strict consistency, the Primary is violently tethered back to the rest of the cluster, and its performance drops off a cliff as it waits for the slowest node.
  • The PXC Advantage: Percona XtraDB Cluster (PXC) absorbs the “consistency penalty” much more gracefully. While varying consistency levels exist in PXC, adjusting them does not cause the same dramatic throughput shock seen in MGR. This is because PXC enforces a virtually synchronous, high-consistency baseline from the start. It simply does not allow the node receiving writes to deviate too far from the rest of the cluster. You pay a baseline performance tax upfront, but in exchange, you get guaranteed, ironclad High Availability out of the box.

The Final Verdict Modifying consistency values at the global server level should only be done after rigorous load testing and a complete understanding of the performance tax you are about to pay.

Ultimately, it comes down to choosing the right tool for your specific SLA:

  • If your architecture demands a true, virtually synchronous solution with strict High Availability out of the box, PXC is the purpose-built engine for the job.
  • If you are looking for a highly automated, semi-synchronous solution, Group Replication delivers excellent default performance—but tuning it to mimic PXC’s strict consistency will cost you heavily in throughput.

 

References

https://www.google.com/url?q=https://mariadb.com/docs/galera-cluster/galera-architecture/certification-based-replication&sa=D&source=docs&ust=1777342808813139&usg=AOvVaw3SAf2g7NO9d681ZJ0VVEMB

https://docs.percona.com/percona-xtradb-cluster/5.7/wsrep-system-index.html#wsrep_sync_wait

The post Group Replication VS Percona XtraDB Cluster: The True Cost of Consistency appeared first on Percona.

Written by in: Zend Developer |

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com