A customer recently asked if our Percona Distribution for PostgreSQL Operator supports the deployment of a standby cluster, which they need as part of their Disaster Recovery (DR) strategy. The answer is yes – as long as you are making use of an object storage system for backups, such as AWS S3 or GCP Cloud Storage buckets, that can be accessed by the standby cluster. In a nutshell, it works like this:
- The primary cluster is configured with pgBackRest to take backups and store them alongside archived WAL files in a remote repository;
- The standby cluster is built from one of these backups and it is kept in sync with the primary cluster by consuming the WAL files that are copied from the remote repository.
Note that the primary node in the standby cluster is not a streaming replica from any of the nodes in the primary cluster and that it relies on archived WAL files to replicate events. For this reason, this approach cannot be used as a High Availability (HA) solution. Even though the primary use of a standby cluster in this context is DR, it can be also employed for migrations as well.
So, how can we create a standby cluster using the Percona operator? We will show you next. But first, let’s create a primary cluster for our example.
Creating a Primary PostgreSQL Cluster Using the Percona Operator
You will find a detailed procedure on how to deploy a PostgreSQL cluster using the Percona operator in our online documentation. Here we want to highlight the main steps involved, particularly regarding the configuration of object storage, which is a crucial requirement and should better be done during the initial deployment of the cluster. In the following example, we will deploy our clusters using the Google Kubernetes Engine (GKE) but you can find similar instructions for other environments in the previous link.
Considering you have a Google account configured as well as the gcloud (from the Google Cloud SDK suite) and kubectl command-line tools installed, authenticate yourself with gcloud auth login, and off we go!
Creating a GKE Cluster and Basic Configuration
The following command will create a default cluster named “cluster-1” and composed of three nodes. We are creating it in the us-central1-a zone using e2-standard-4 VMs but you may choose different options. In fact, you may also need to indicate the project name and other main settings if you do not have your gcloud environment pre-configured with them:
gcloud container clusters create cluster-1 --preemptible --machine-type e2-standard-4 --num-nodes=3 --zone us-central1-a
Once the cluster is created, use your IAM identity to control access to this new cluster:
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account)
Finally, create the pgo namespace:
kubectl create namespace pgo
and set the current context to refer to this new namespace:
kubectl config set-context $(kubectl config current-context) --namespace=pgo
Creating a Cloud Storage Bucket
Remember for this setup we need a Google Cloud Storage bucket configured as well as a Service Account created with the necessary privileges/roles to access it. The respective procedures to obtain these vary according to how your environment is configured so we won’t be covering them here. Please refer to the Google Cloud Storage documentation for the exact steps. The bucket we created for the example in this post was named cluster1-backups-and-wals.
Likewise, please refer to the Creating and managing service account keys documentation to learn how to create a Service Account and download the corresponding key in JSON format – we will need to provide it to the operator so our PostgreSQL clusters can access the storage bucket.
Creating the Kubernetes Secrets File to Access the Storage Bucket
Create a file named my-gcs-account-secret.yaml with the following structure:
| apiVersion: v1 kind: Secret metadata: name: cluster1-backrest-repo-config type: Opaque data: gcs-key: <VALUE> |
replacing the <VALUE> placeholder by the output of the following command according to the OS you are using:
Linux:
base64 --wrap=0 your-service-account-key-file.json
macOS:
base64 your-service-account-key-file.json
Installing and Deploying the Operator
The most practical way to install our operator is by cloning the Git repository, and then moving inside its directory:
git clone -b v1.1.0 https://github.com/percona/percona-postgresql-operator cd percona-postgresql-operator
The following command will deploy the operator:
kubectl apply -f deploy/operator.yaml
We have already prepared the secrets file to access the storage bucket so we can apply it now:
kubectl apply -f my-gcs-account-secret.yaml
Now, all that is left is to customize the storages options in the deploy/cr.yaml file to indicate the use of the GCS bucket as follows:
| storages: my-gcs: type: gcs bucket: cluster1-backups-and-wals |
We can now deploy the primary PostgreSQL cluster (cluster1):
kubectl apply -f deploy/cr.yaml
Once the operator has been deployed, you can run the following command to do some housekeeping:
kubectl delete -f deploy/operator.yaml
Creating a Standby PostgreSQL Cluster Using the Percona Operator
After this long preamble, let’s look at what brought you here: how to deploy a standby cluster, which we will refer to as cluster2, that will replicate from the primary cluster.
Copying the Secrets Over
Considering you probably have customized the passwords you use in your primary cluster and that they differ from the default values found in the operator’s git repository, we need to make a copy of the secrets files, adjusted to the standby cluster’s name. The following procedure facilitates this task, saving the secrets files under /tmp/cluster1-cluster2-secrets (you can choose a different target directory):
| NOTE: make sure you have the yq tool installed in your system. |
mkdir -p /tmp/cluster1-cluster2-secrets/
export primary_cluster_name=cluster1
export standby_cluster_name=cluster2
export secrets="${primary_cluster_name}-users"
kubectl get secret/$secrets -o yaml \
| yq eval 'del(.metadata.creationTimestamp)' - \
| yq eval 'del(.metadata.uid)' - \
| yq eval 'del(.metadata.selfLink)' - \
| yq eval 'del(.metadata.resourceVersion)' - \
| yq eval 'del(.metadata.namespace)' - \
| yq eval 'del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration")' - \
| yq eval '.metadata.name = "'"${secrets/$primary_cluster_name/$standby_cluster_name}"'"' - \
| yq eval '.metadata.labels.pg-cluster = "'"${standby_cluster_name}"'"' - \
>/tmp/cluster1-cluster2-secrets/${secrets/$primary_cluster_name/$standby_cluster_name}
Deploying the Standby Cluster: Fast Mode
Since we have already covered the procedure used to create the primary cluster in detail in a previous section, we will be presenting the essential steps to create the standby cluster below and provide additional comments only when necessary.
| NOTE: the commands below are issued from inside the percona-postgresql-operator directory hosting the git repository for our operator. |
Deploying a New GKE Cluster Named cluster-2
This time using the us-west1-b zone here:
gcloud container clusters create cluster-2 --preemptible --machine-type e2-standard-4 --num-nodes=3 --zone us-west1-b kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account) kubectl create namespace pgo kubectl config set-context $(kubectl config current-context) --namespace=pgo kubectl apply -f deploy/operator.yaml
Apply the Adjusted Kubernetes Secrets:
export standby_cluster_name=cluster2
export secrets="${standby_cluster_name}-users"
kubectl create -f /tmp/cluster1-cluster2-secrets/$secrets
The list above does not include the GCS secret file; the key contents remain the same but the backrest-repo pod name needs to be adjusted. Make a copy of that file:
cp my-gcs-account-secret.yaml my-gcs-account-secret-2.yaml
then edit the copy to indicate “cluster2-” instead of “cluster1-”:
| name: cluster2-backrest-repo-config |
You can apply it now:
kubectl apply -f my-gcs-account-secret-2.yaml
The cr.yaml file of the Standby Cluster
Let’s make a copy of the cr.yaml file we customized for the primary cluster:
cp deploy/cr.yaml deploy/cr-2.yaml
and edit the copy as follows:
1) Change all references (that are not commented) from cluster1 to cluster2 – including current-primary but excluding the bucket reference, which in our example is prefixed with “cluster1-”; the storage section must remain unchanged. (We know it’s not very practical to replace so many references, we still need to improve this part of the routine).
2) Enable the standby option:
| standby: true |
3) Provide a repoPath that points to the GCS bucket used by the primary cluster (just below the storages section, which should remain the same as in the primary cluster’s cr.yaml file):
| repoPath: “/backrestrepo/cluster1-backrest-shared-repo” |
And that’s it! All that is left now is to deploy the standby cluster:
kubectl apply -f deploy/cr-2.yaml
With everything working on the standby cluster, do some housekeeping:
kubectl delete -f deploy/operator.yaml
Verifying it all Works as Expected
Remember that the standby cluster is created from a backup and relies on archived WAL files to be continued in sync with the primary cluster. If you make a change in the primary cluster, such as adding a row to a table, that change won’t reach the standby cluster until the WAL file it has been recorded to is archived and consumed by the standby cluster.
When checking if all is working with the new setup, you can force the rotation of the WAL file (and subsequent archival of the previous one) in the primary node of the primary cluster to accelerate the sync process by issuing:
psql> SELECT pg_switch_wal();
The Percona Kubernetes Operators automate the creation, alteration, or deletion of members in your Percona Distribution for MySQL, MongoDB, or PostgreSQL environment.
Nowadays we are seeing a lot of customers starting to use our 
As we all know, security is very important these days and we read about many data leaks. Security has many aspects, but one of the most important is securing data since it is a vital asset to companies. When we speak about data, it can be encrypted at rest (transparent data encryption – TDE, full disk encryption, column/field-level encryption) or in transit (TLS).
We have a 
? Preparing nodes 
? Starting control-plane 
? Installing CNI 
? Installing StorageClass 
? Joining worker nodes 
Set kubectl context to "kind-k8s-playground"
You can now use your cluster with:
kubectl cluster-info --context kind-k8s-playground
There are cases where multiple teams, customers, or applications run in the same Kubernetes cluster. Such an environment is called multi-tenant and requires some preparation and management. Multi-tenant Kubernetes deployment allows you to utilize the economy of scale model on various levels:
The cloud-native landscape matures every day, and new great tools and products continue to appear. We are starting a series of blog posts that are going to focus on new tools in the container and cloud-native world, and provide a holistic view through the prism of Percona products.
Description: CPU and memory resources should have a limit set for every container to prevent resource exhaustion. This control identifies all the Pods without resource limit definition.
Namespace default
Deployment - percona-server-mongodb-operator
StatefulSet - my-cluster-name-cfg
StatefulSet - my-cluster-name-rs0
Summary - Passed:3 Excluded:0 Failed:3 Total:6
Remediation: Define LimitRange and ResourceQuota policies to limit resource usage for namespaces or nodes.
This blog post is the last in the series of articles on migrating databases to Kubernetes with Percona Operators. Two previous posts can be found here:
As per the glossary, Disaster Recovery (DR) protocols are an organization’s method of regaining access and functionality to its IT infrastructure in events like a natural disaster, cyber attack, or even business disruptions related to the COVID-19 pandemic. When we talk about data, storing backups on remote servers is enough to pass DR compliance checks for some companies. But for others, Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) are extremely tight and require more than just a backup/restore procedure.
There are plenty of ways to run ProxySQL in Kubernetes (K8S). For example, we can deploy sidecar containers on the application pods, or run a dedicated ProxySQL service with its own pods.