Personal blog of Yzmir Ramirez

Recently I listened to Lenny Rachitsky’s podcast, where he invited Shreyas Doshi for the second time. The session was titled “4 questions Shreyas Doshi wishes he’d asked himself sooner”. One of the questions Shreyas brought up was, “Do I actually have a good taste?”. This is an interesting question to ask for an experienced product […]

This is the second part of the series of blog posts unmasking the complexity of MongoDB cluster exposure in Kubernetes with Percona Operator for MongoDB. In the first part, we focused heavily on split horizons and a single replica set. In this part, we will expose a sharded cluster and a single replica set with Istio, […]

Running and managing MongoDB clusters in Kubernetes is made easy with the Percona Operator for MongoDB. Some aspects are just easy to grasp as they are well defined in the operator custom resources and documentation, but some are often considered to be a hidden craft. Network exposure in cases of sharded clusters is quite straightforward, […]

The evolution of cloud-native technology has been nothing short of revolutionary. As we step into 2024, the cornerstone of cloud-native technology, Kubernetes, will turn ten years old. It continues to solidify its position and is anticipated to reach USD 5575.67 million by 2028, with a forecasted Compound Annual Growth Rate (CAGR) of 18.51% in the coming years, as reported by Industry Research Biz. 

The Cloud Native landscape continues to encompass both micro-trends and IT macro-trends, influencing and transforming the way businesses operate and deliver value to their customers.

As we at Percona wind down 2023 and look into what the next year holds, our attention is drawn to the cloud-native landscape and how it is maturing, growing, and evolving. 

KubeCon NA 2023 recap

The theme for KubeCon NA was very clear — AI and Large Language Models (LLMs). Keynotes were focused on how Kubernetes and Cloud Native help businesses embrace the new AI era. And it is understandable, as Kubernetes slowly becomes what it is intended to be – the Platform.

The field of Platform Engineering has witnessed significant advancements, as evidenced by the publication of the CNCF platform whitepaper and the introduction of a dedicated Platform Engineering day at the upcoming KubeCon event. At Percona, we observe a growing trend among companies utilizing Kubernetes as a means to offer services to their teams, fostering expedited software delivery and driving business growth.

Declarative GitOps management, with ArgoCD and Flux, is the community way of adding orchestration on top of orchestration. In our conversations with developers and engineers during the conference, we confirmed the CNCF GItOps Microsurvey data – 91% are already using GitOps.

According to the Dynatrace Kubernetes in the Wild 2023 report, a significant 71% (with 48% year-over-year growth!) of respondents are currently utilizing databases in Kubernetes (k8s).  This finding aligns with the observations made at the Data on Kubernetes (DoK) day, where discussions surrounding this topic transitioned from niche, tech-oriented conversations a year ago to more widespread, enterprise-level interest in adopting diverse use cases. These indicators suggest that the adoption of databases on k8s is in its early stages and is likely to continue growing in the future.

Predictions

Multi-cloud is a requirement

While this wave has been building for years, in 2024, we expect it to peak. According to a 2023 Forrester survey commissioned by Hashicorp, 61% of respondents had implemented, were expanding, or were upgrading their multi-cloud strategy. We expect that number to rise higher in 2024.

Nearly every vendor at Kubecon and every person we spoke to had some form of a multi-cloud requirement or strategy. Sometimes, this comes from necessity through acquisition or mergers. Oftentimes, it is a pillar of modern infrastructure strategy to avoid cloud vendor lock-in. At this point, it is ubiquitous, and if it is not part of your strategy, you are falling behind.

The business value of adopting this strategy is multi-fold:

• Freedom from vendor lock-in, which leads to increased negotiating power
• Agility in capitalizing on cloud-vendor advancements to innovate faster
• Increased application and database architecture RPO and RTO
• Adhering to security and governance requirements of customers

Percona’s Operators for MySQL, MongoDB, and PostgreSQL are designed with this value in mind. We want adopters of our technology to be able to deploy their critical open source databases and applications across any public or private cloud environment. All of the database automation for running a highly available, resilient, and secure database is built into the operator to simplify the operation and management of your clusters. 

Simplify and secure

Looking through various State of Kubernetes reports (VMWare, RedHat, SpectroCloud), it becomes clear that Complexity and Security are the top concerns for platform engineering teams.  

Simplification might come from different angles. Deployment is mostly solved already, whereas management and operations are still not. We expect to see various tooling and core patches to automate scaling, upgrades, migrations, troubleshooting, and more. 

Operators are an integral part of solving the complexity problem, where they take away the need for learning k8s primitives and application configuration internals. They also remove toil and allow engineers to focus on application development vs platform engineering work. Not only will new operators appear, but existing operators will mature and provide capabilities that meet or exceed managed services that users can get on public clouds. 

The latest report on Kubernetes adoption, security, and market trends in 2023 revealed that 67% reported delaying or slowing down deployment due to Kubernetes security concerns. Additionally, 37% of respondents experienced revenue or customer loss due to a container/Kubernetes security incident.

Considering the open source software vulnerability as one of the top concerns and the rapid increase in supply chain attacks (the SolarWinds attack and vulnerabilities like Log4Shell and Spring4Shell), along with container and Kubernetes strategies, there’s a growing emphasis on cybersecurity and operational understanding in development. 

Another significant issue within security concerns is the escalating complexity of modern systems, especially in platforms like Kubernetes, which highlights the need for unified threat models and scanning tools to address vulnerabilities. Standardization and collaboration are key to sharing common knowledge and patterns across teams and infrastructures. Creating repositories for memory-safe patterns in cloud systems to improve overall security.

A majority of RedHat’s security research respondents have a DevSecOps initiative underway. Most organizations are embracing DevSecOps, a term that covers processes and tooling enabling security to be integrated into the application development life cycle rather than treated as a separate process. However, 17% of organizations operate security separately from DevOps, lacking any DevSecOps initiatives. Consequently, they might miss out on the benefits of integrating security into the SDLC, such as enhanced efficiency, speed, and quality in software delivery.

AI and MLOps

Kubernetes has become a new web server for many production AI workloads, focusing on facilitating the development and deployment of AI applications, including model training. The newly formed Open Source AI Alliance, led by META and IBM, promises to support open-source AI. It comprises numerous organizations from various sectors, including software, hardware, nonprofit, public, and academic. The goal is to collaboratively develop tools and programs facilitating open development and run scalable and distributed training jobs for popular frameworks such as PyTorch, TensorFlow, MPI, MXNet, PaddlePaddle, and XGBoost.

While integrating AI and machine learning into cloud-native architectures, there’s an increasing demand from users for AI to be open and collaborative. The emergence of trends stemming from ‘AI Advancements and Ethical Concerns’ cannot be ignored.

Addressing ethical concerns and biases will necessitate the implementation of transparent AI frameworks and ethical guidelines during application development. Customers will increasingly prioritize AI efficiency and education to tackle legal and ethical concerns. This marks the end of an era of chaos, paving the way for efficiency gains, quicker innovation, and standardized practices.

Conclusion

At Percona, we prioritize staying ahead of market trends by adhering to industry best practices and leveraging our team’s expertise.

We’ve always made sure to focus on security in our software development, and weaving multi-cloud deployment into our products has been a crucial part of our strategy. Our commitment to open source software drives us to take additional precautions, ensuring operational security through best practices and principles, such as of least privilege, security in layers, and separation of roles/responsibilities through policy and software controls. And with multi-cloud in mind, we consistently incorporate new sharding functionalities into our roadmaps, such as the upcoming Shard-per-location support in the Percona Operator for MongoDB.

At the same time, we are not hesitating to rock the cloud-native community by incorporating top-notch features to address any new rising trends. You mentioned ‘More Simple Kubernetes’? Well, here we are – with storage autoscaling for databases in Kubernetes, slated for release in Q1, 2024 after a year of hard work. This fully automated scaling and tuning will enable a serverless-like experience in our Operators and Everest. Developers will receive the endpoint without needing to consider resources and tuning at all. It’s worry-free and doesn’t require human intervention.

Finally, the rising popularity of generative AI and engines like OpenAI or Bard has prompted our team to bring vector-handling capabilities to Percona-powered database software by adding support for the pgvector extension.

Our team always focuses on innovation to accelerate progress for everyone, and we will continue to push the boundaries further for our community and the rest of the world.

The Percona Kubernetes Operators automate the creation, alteration, or deletion of members in your Percona Distribution for MySQL, MongoDB, or PostgreSQL environment.

 

Learn More About Percona Kubernetes Operators

Percona, a leading provider of open-source database software and services, announced the general availability of Percona Operator for PostgreSQL version 2. The solution is 100% open source and automates the deployment and management of PostgreSQL clusters on Kubernetes. Percona is also offering 24/7 support and managed services for paying customers.

As more and more organizations move their workloads to Kubernetes, managing PostgreSQL clusters in this environment can be challenging. Kubernetes provides many benefits, such as automation and scalability, but it also introduces new complexities when it comes to managing databases. IT teams must ensure high availability, scalability, and security, all while ensuring that their PostgreSQL clusters perform optimally.

Percona Operator for PostgreSQL version 2 simplifies the deployment and management of PostgreSQL clusters on Kubernetes. The solution automates tasks such as scaling, backup and restore, upgrades, and more, and supports Patroni-based PostgreSQL clusters. Additionally, Percona Operator for PostgreSQL version 2 includes expanded options for customizing backup and restore operations, improved monitoring and alerting capabilities, and support for PostgreSQL 15.

For organizations that need additional support and managed services, Percona is offering 24/7 support and managed services for paying customers. This includes access to a dedicated team of PostgreSQL experts who can help with everything from installation and configuration to ongoing management and support.

Percona’s commitment to open source ensures that the Percona Operator for PostgreSQL version 2 remains a flexible and customizable solution that can be tailored to meet the unique needs of any organization.

Below you will find a short FAQ about the new operator and a comparison to version 1.x.

What is better in version 2 compared to version 1?

Architecture

Operator SDK is now used to build and package the Operator. It simplifies the development and brings more contribution friendliness to the code, resulting in better potential for growing the community. Users now have full control over Custom Resource Definitions that Operator relies on, which simplifies the deployment and management of the operator.

In version 1.x, we relied on Deployment resources to run PostgreSQL clusters, whereas in 2.0 Statefulsets are used, which are the de-facto standard for running stateful workloads in Kubernetes. This change improves the stability of the clusters and removes a lot of complexity from the Operator.

Backups

One of the biggest challenges in version 1.x is backups and restores. There are two main problems that our users faced:

• Not possible to change the backup configuration for the existing cluster
• Restoration from backup to the newly deployed cluster required workarounds

In this version, both these issues are fixed. In addition to that:

• Run up to four pgBackrest repositories
• Bootstrap the cluster from the existing backup through Custom Resource
• Azure Blob Storage support

Operations

Deploying complex topologies in Kubernetes is not possible without affinity and anti-affinity rules. In version 1.x, there were various limitations and issues, whereas this version comes with substantial improvements that enable users to craft the topology of their choice.

Within the same cluster, users can deploy multiple instances. These instances are going to have the same data but can have different configurations and resources. This can be useful if you plan to migrate to new hardware or need to test the new topology.

Each PostgreSQL node can have sidecar containers now to provide integration with your existing tools or expand the capabilities of the cluster.

Will Percona still support v1?

Percona Operator for PostgreSQL version 1 moves to maintenance mode and will go End-of-Life after one year – June 2024. We are going to provide bug and security fixes but will not introduce new features and improvements.

Customers with a contract with Percona will still have operational support until Operator goes into EoL stage.

I’m running version 1 now; how can I upgrade to version 2?

We have prepared detailed guidelines for migrating from version 1 to version 2 with zero or minimal downtime. Please refer to our documentation.

The Percona Operator for PostgreSQL version 2 is available now, and we invite you to try it out for yourself. Whether you’re deploying PostgreSQL for the first time or looking for a more efficient way to manage your existing environment, Percona Operator for PostgreSQL has everything you need to get the job done. We look forward to seeing what you can do with it!

For more information, visit Percona Operator for PostgreSQL v2 documentation page. For commercial support, please visit our contact page.

 

Learn more about Percona Operator for PostgreSQL v2

Developers need an efficient, reliable way to run their Django applications with a robust PostgreSQL. Percona Operator for PostgreSQL offers a powerful solution for managing and scaling PostgreSQL databases in a Kubernetes environment, making it an ideal choice for developer use cases. In this blog post, we’ll explore what it takes to run Django on Kubernetes with Percona Operator.

Set up PostgreSQL

Use your favorite way to deploy PostgreSQL Operator. I will use the regular kubectl approach:

$ kubectl apply --server-side -f deploy/bundle.yaml

Django application would require its own user and database. Alter cluster custom resource (cr.yaml) manifest spec.users section. The following example will create the cluster
appdb,  the user
djangoapp and database
pgtest. The user will have access to
pgtest database only:

metadata:
  name: appdb
spec:
  users:
    - name: djangoapp
      databases:
        - pgtest

Now you can apply the manifest to provision the cluster:

$ kubectl apply -f deploy/cr.yaml

The Operator will generate the secret for the user called
CLUSTERNAME–pguser–djangoapp. Read on to learn how to get credentials and connection string from it.

PostgreSQL 15 public schema

PostgreSQL 15 removes the global write privilege from the public schema. As a result, you might see the following error when running migration in Django:

django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (permission denied for schema public
LINE 1: CREATE TABLE "django_migrations" ("id" bigint NOT NULL PRIMA...
                     ^
)

To fix that, it is necessary to explicitly allow
djangoapp permissions to public schema. To do that, you need to connect to the cluster with superuser and run grants:

pgtest=# GRANT ALL ON SCHEMA public TO djangoapp;
GRANT

Learn how to connect with a superuser in our documentation.

Django and PostgreSQL

psycopg2

Psycopg is a PostgreSQL database adapter for the Python programming language. Django uses it to connect to the database. You will see the following error if you don’t have it installed and trying to connect to PostgreSQL:

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 or psycopg module

Install
psycopg2 by following its documentation. If you are installing it with pip, be aware that it might look for
pg_config  to build:

$ pip3 install psycopg2
…
Error: pg_config executable not found.


      pg_config is required to build psycopg2 from source.  Please add the directory
      containing pg_config to the $PATH or specify the full executable path with the
      option:


          python setup.py build_ext --pg-config /path/to/pg_config build ...

The easiest way to fix it is to install the
psycopg2–binary instead:

$ pip3 install psycopg2-binary

settings.py

settings.py  is the main configuration file for Django applications. It is where you should configure the database – provide credentials, set the correct engine, and more. You can read more about it in Django’s documentation.

There is nothing super specific about configuring Django and Percona Operator. First, get the connection string and credentials from the Operator. For the cluster
appdb and the user
djangoapp they are stored in a secret
appdb–pguser–djangoapp. The following commands will get you what is needed:

$ kubectl get secret appdb-pguser-djangoapp --template='{{.data.password | base64decode}}'
4<_R|8O/@:.2>PnO+DyEW1Kd

$ kubectl get secret appdb-pguser-djangoapp --template='{{index .data "pgbouncer-host" | base64decode}}'
appdb-pgbouncer.default.svc

Your settings.py DATABASES section will look the following way:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'pgtest',
        'USER': 'djangoapp',
        'PASSWORD': '4<_R|8O/@:.2>PnO+DyEW1Kd',
        'HOST': 'appdb-pgbouncer.default.svc’,
        'PORT': '5432',
    }
}

settings.py and Kubernetes

The recommended way to pass credentials to containers is through environment variables. In Kubernetes, it will be additionally wrapped into a Secret resource.

To make it work, we will put our database URI into an environment variable. You can get the database URI from the secret as well:

$ kubectl get secret appdb-pguser-djangoapp --template='{{index .data "pgbouncer-uri" | base64decode}}'
postgresql://djangoapp:UZihjfPtvfNTuIdVzhAUT%7B%5Bq@appdb-pgbouncer.default.svc:5432/pgtest

The recommended way for Django is to store environment variables in
.env  file:

DATABASE_URL=postgresql://djangoapp:UZihjfPtvfNTuIdVzhAUT%7B%5Bq@appdb-pgbouncer.default.svc:5432/pgtest

For using the database URL, use dj_database_url. Install it as usual with pip:

$ pip3 install dj_database_url

Now you can have something like this in your settings.py:

import dj_database_url
import os

if os.environ.get('DATABASE_URL'):
  DATABASES['default'] = dj_database_url.config(default=os.environ['DATABASE_URL'])

DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql'

Note the
ENGINE, as
dj_database_url does not set it.

You can also avoid using
dj_database_url  and pass each variable separately through
os.environ.

Passing a variable in Kubernetes

In Kubernetes, you can pass the
DATABASE_URL to a container through a Secret. You can mount a separate Secret or reuse the one that the Operator manages. The recommended way is to have a separate Secret object, as your application might be in a separate namespace, and you might not have enough permissions to mount the one that is managed by the Operator. The Secret and Deployment might look as follows:

apiVersion: v1
kind: Secret
metadata:
  name: my-db-secret
stringData:
  pgbouncer-uri: postgresql://djangoapp:UZihjfPtvfNTuIdVzhAUT%7B%5Bq@appdb-pgbouncer.default.svc:5432/pgtest
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-deploy
spec:
  replicas: 3
    spec:
      containers:
      - name: mydjango
        image: mydjangoapp:1.2.3
        ports:
        - containerPort: 8000
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: my-db-secret
              key: pgbouncer-uri

Conclusion: Deploying Django with Percona Operator for PostgreSQL

Running Django on Kubernetes with Percona Operator for PostgreSQL offers developers an efficient and scalable solution for managing their database needs in a Kubernetes environment. While there are some caveats and potential issues to be aware of, the configuration examples and explanations provided in this blog post will help developers overcome any challenges they may encounter. With Percona Operators, developers can focus on building and delivering their applications with confidence and ease.

Learn more about Percona Operator for PostgreSQL in our documentation.

You can get early access to new product features, invite-only “ask me anything” sessions with Percona Kubernetes experts, and monthly swag raffles. Interested? Fill in the form at percona.com/k8s.

As more companies look at migrating away from Oracle or implementing new databases alongside their applications, PostgreSQL is often the best option for those who want to run on open source databases.

Read Our New White Paper:

Why Customers Choose Percona for PostgreSQL

Operators hide the complexity of the application and Kubernetes. Instead of dealing with Pods, StatefulSets, tons of YAML manifests, and various configuration files, the user talks to Kubernetes API to provision a ready-to-use application. An Operator automatically provisions all the required resources and exposes the application. Though, there is always a risk that the user would want to do something manual that can negatively affect the application and the Operator logic.

In this blog post, we will explain how to limit access scope for the user to avoid manual changes for database clusters deployed with Percona Operators. To do so, we will rely on Kubernetes Role-based Access Control (RBAC).

The goal

We are going to have two roles: Administrator and Developer. Administrator will deploy the Operator, create necessary roles, and service accounts. Developers will be able to:

1. Create, modify, and delete Custom Resources that Percona Operators use
2. List all the resources – users might want to debug the issues

Developers will not be able to:

1. Create, modify, or delete any other resource

As a result, the Developer will be able to deploy and manage database clusters through a Custom Resource, but will not be able to modify any operator-controlled resources, like Pods, Services, Persistent Volume Claims, etc.

Action

We will provide an example for Percona Operator for MySQL based on Percona XtraDB Cluster (PXC), which just had version 1.12.0 released. 

Administrator

Create a dedicated namespace

We will allow users to manage clusters in a single namespace called

prod-dbs

:

$ kubectl create namespace prod-dbs

Deploy the operator

Use any of the ways described in our documentation to deploy the operator into the namespace. My personal favorite will be with simple

kubectl

command:

$ kubectl apply -f https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/v1.12.0/deploy/bundle.yaml

Create ClusterRole

ClusterRole resource defines the permissions the user will have for a specific resource in Kubernetes. You can find the YAML in this github repository.

    - apiGroups: ["pxc.percona.com"]
      resources: ["*"]
      verbs: ["*"]
    - apiGroups: [""]
      resources:
      - pods
      - pods/exec
      - pods/log
      - configmaps
      - services
      - persistentvolumeclaims
      - secrets
      verbs:
      - get
      - list
      - watch

As you can see we allow any operations for

pxc.percona.com

resources, but restrict others to get, list, and watch.

$ kubectl apply -f https://github.com/spron-in/blog-data/blob/master/rbac-operators/clusterrole.yaml

Create ServiceAccount

We are going to generate a kubeconfig for this service account. This is what the user is going to use to connect to the Kubernetes API.

apiVersion: v1
kind: ServiceAccount
metadata:
    name: database-manager
    namespace: prod-dbs

$ kubectl apply -f https://raw.githubusercontent.com/spron-in/blog-data/master/rbac-operators/serviceaccount.yaml

Create ClusterRoleBinding

We need to assign the

ClusterRole

to the

ServiceAccount

.

ClusterRoleBinding

 acts as a relation between these two.

$ kubectl apply -f https://github.com/spron-in/blog-data/blob/master/rbac-operators/clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
    name: percona-database-manager-bind
    namespace: prod-dbs
roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: percona-pxc-rbac
subjects:
    - kind: ServiceAccount
      name: database-manager
      namespace: prod-dbs

Developer

Verify

can-i

command allows you to verify if the service account can really do what we want it to do. Let’s try:

$ kubectl auth can-i create perconaxtradbclusters.pxc.percona.com --as=system:serviceaccount:prod-dbs:database-manager
yes

$ kubectl auth can-i delete service --as=system:serviceaccount:prod-dbs:database-manager
no

All good. I can create Percona XtraDB Clusters, but I can’t delete Services. Please note, that sometimes it might be useful to allow Developers to delete Pods to force the cluster recovery. If you feel that it is needed, please modify the ClusterRole.

Apply

There are multiple ways to use this service account. You can read more about it in Kubernetes documentation. For a quick demonstration, we are going to generate a kubeconfig that we can share with our user. 

Run this script to generate the config. What it does:

1. Gets the service account secret resource name
2. Extracts Certificate Authority (ca.crt) contents from the secret
3. Extracts the token from the secret
4. Gets the endpoint of the Kubernetes API
5. Generates the kubeconfig using all of the above

$ bash generate-kubeconfig.sh > tmp-kube.config

Now let’s see if it works as expected:

$ KUBECONFIG=tmp-kube.config -n prod-dbs apply -f https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/v1.12.0/deploy/cr.yaml
perconaxtradbcluster.pxc.percona.com/cluster1 created

$ KUBECONFIG=tmp-kube.config kubectl -n prod-dbs get pods
NAME                                               READY   STATUS    RESTARTS   AGE
cluster1-haproxy-0                                 2/2     Running   0          15m
cluster1-haproxy-1                                 2/2     Running   0          14m
cluster1-haproxy-2                                 2/2     Running   0          14m
cluster1-pxc-0                                     3/3     Running   0          15m
cluster1-pxc-1                                     3/3     Running   0          14m
cluster1-pxc-2                                     3/3     Running   0          13m
percona-xtradb-cluster-operator-77bf8b9df5-qglsg   1/1     Running   0          52m

I was able to create the Custom Resource and a cluster. Let’s try to delete the Pod:

$ KUBECONFIG=tmp-kube.config kubectl -n prod-dbs delete pods cluster1-haproxy-0
Error from server (Forbidden): pods "cluster1-haproxy-0" is forbidden: User "system:serviceaccount:prod-dbs:database-manager" cannot delete resource "pods" in API group "" in the namespace "prod-dbs"

What about the Custom Resource?

$ KUBECONFIG=tmp-kube.config kubectl -n prod-dbs delete pxc cluster1
perconaxtradbcluster.pxc.percona.com "cluster1" deleted

Conclusion

Percona Operators automate the deployment and management of the databases in Kubernetes. Least privilege principle should be applied to minimize the ability of inexperienced users to affect availability and data integrity. Kubernetes comes with sophisticated Role-Based Access Control capabilities which allow you to do just that without the need to reinvent it in your platform or application.

Try Percona Operators

This blog post continues the series of comparisons of solutions to run databases on Kubernetes. Previous blog posts:

• Run MongoDB in Kubernetes: Solutions, Pros and Cons
• Run PostgreSQL in Kubernetes: Solutions, Pros and Cons

The initial release of MySQL was in 1995 and Kubernetes was released 19 years later in 2014. According to DB-engines, MySQL today is the most popular open source relational database, whereas Kubernetes is the de-facto standard for running containerized workloads. The popularity of these two technologies among engineers drove companies to create solutions to run them together. In this blog post, we will review various Kubernetes Operators for MySQL, see how different they are, and what capabilities they provide for developers and operations teams.

• Bitpoke MySQL Operator (former Presslabs)
• Oracle MySQL Operator
• Moco by Cybozu
• Vitess Operator by PlanetScale
• Percona Operator for MySQL
  • Based on Percona XtraDB Cluster
  • Based on Percona Server for MySQL

The summary and comparison table can be found in our documentation.

Notable mentions

Before reviewing the operators, I need to mention numerous interesting solutions to run MySQL in k8s.

KubeDB

KubeDB is a swiss-army knife operator, which can deploy and manage multiple databases, including MySQL. The thing is that it is working in an open core model, where the most interesting features are not available in the free version, so I cannot easily try them out. Rest assured it is a viable solution, but in this blog post, I want to focus on open source offerings.

Helm or manual deployment

The desire to run MySQL on Kubernetes was there before the Operator SDK appeared. To address this, the community was very creative and developed numerous ways how to do it, ranging from regular manual deployments to more sophisticated helm charts (ex Bitnami Helm Chart).

They do the job – you can deploy a MySQL server. With some digging and tuning it is even possible to have a cluster. But what these solutions have in common is that they lack the ability to perform different day-2 operations: backups, scaling, upgrades, etc. For databases, it might be crucial, because data consistency and safety are at stake. Applying methods that worked on legacy environments, might not be safe on Kubernetes.

This is where Operators come into play.

Bitpoke MySQL Operator

Bitpoke is a company that provides tools for WordPress self-hosting on Kubernetes, including MySQL and WordPress operators. Their team developed one of the first MySQL operators and shared it with the community. Developed initially within Presslabs, since 2021 the team and operator have moved to Bitpoke. The operator is used in production by numerous companies. 

It is Apache 2.0 licensed. Interestingly enough, it uses Percona Server for MySQL under the hood “because of backup improvements (eg. backup locks), monitoring improvements, and some serviceability improvements (eg. utility user)”.

Deployment

I followed the documentation:

helm repo add bitpoke https://helm-charts.bitpoke.io
helm install mysql-operator bitpoke/mysql-operator

The operator is up and running. Deploying the cluster:

kubectl apply -f https://raw.githubusercontent.com/bitpoke/mysql-operator/master/examples/example-cluster-secret.yaml
kubectl apply -f https://raw.githubusercontent.com/bitpoke/mysql-operator/master/examples/example-cluster.yaml

This deploys MySQL 5.7 cluster with asynchronous replication – one main and one replica node.

Features

Bitpoke operator allows you to back up, restore, scale, and upgrade MySQL on Kubernetes. So regular day-2 operations are available. 

Let’s start with the pros:

• It is the oldest open source Operator for MySQL. It was battle-tested by Bitpoke and widely adopted by the community and other companies. This gives assurance that there are no critical issues. But do read ahead to the Cons section.
• Replication lag and mitigation takes lagging nodes out of rotation when lag is above a set threshold. It is important for asynchronous replication to present the real data for the application.

As for cons, it seems that the operator is not actively developed with 15 commits for the last year. 

Oracle MySQL Operator

This is not the first time Oracle created the Operator for MySQL, but the difference now is that this Operator made it to the General Availability stage. Operator is distributed under the unusual “Universal Permissive License (UPL)”, but it is really permissive and close to the MIT license.

Deployment

Standard deployment for the operator with helm, no surprises:

helm repo add mysql-operator https://mysql.github.io/mysql-operator/
helm repo update
helm install mysql-operator mysql-operator/mysql-operator --namespace mysql-operator --create-namespace

Now the cluster:

export NAMESPACE=default
helm install my-mysql-innodbcluster mysql-operator/mysql-innodbcluster -n $NAMESPACE \
        --version 2.0.7 \
        --set credentials.root.password=">-0URS4F3P4SS" \
        --set tls.useSelfSigned=true

This deploys a MySQL cluster with three nodes with Group Replication and a single MySQL router Pod in front of it for query routing. 

Features

Even though the operator was promoted to GA, some basic capabilities are not there or should be implemented by the user. For example, upgrades, monitoring, and topology flexibility.

• Operator uses MySQL Shell underneath and its codebase is in python (versus regular golang). Backups and restores also rely on MySQL Shell and use the dumpInstance() method. It is a logical backup and might be problematic for big databases.
• Operator supports only MySQL 8.0 and only Group Replication (or marketed InnoDB Cluster). Which I think is a logical move.
• You can use MySQL Community container images, but there is a certain trend from Oracle to push users towards Enterprise. At least it is visible in the two last release notes, where new additions are all-around features available in the Enterprise edition only.

Moco

Similar to the Bitpoke operator, Moco was created by Cybozu for its internal needs and later open-sourced. It goes under Apache 2.0 license, is written in Golang, and has a good release cadence.

Deployment

As usual, let’s try a quick start guide. Note that a cert-manager is required (curiosity peaked from the start!). 

Install cert-manager and deploy the operator:

curl -fsLO https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
kubectl apply -f cert-manager.yaml

helm repo add moco https://cybozu-go.github.io/moco/
helm repo update
helm install --create-namespace --namespace moco-system moco moco/moco

Create the cluster from an example folder:

kubectl apply -f https://raw.githubusercontent.com/cybozu-go/moco/main/examples/loadbalancer.yaml

This deploys a cluster with three nodes and semi-sync replication exposed with a load balancer.

Features

Moco is quite feature-rich and enables users to execute various management tasks. Refreshing solutions and ideas:

• Documentation. For an in-house grown product, the operator has pretty good and detailed documentation. From design and architecture decisions to how-tos and common use cases.
• Similar to Oracle’s operator, this one relies on MySQL Shell for backups and restores, but at the same time supports incremental backups. Binary logs are copied to an S3 bucket at the time of backup, so it is not a continuous binlog copy that can deliver point-in-time recovery, but only a one-time dump. 
• Operator has its own kubectl plugin to smoothen the onboarding and usage. I’m curious though if these plugins are widely used in production. Please share your thoughts in a comment.

There are some concerns that I have regarding this operator:

• It works with semi-sync replication which is not a good fit for workloads that have strong data-consistency requirements. See Face to Face with Semi-Synchronous Replication for more details.
• It does not have official support, so for businesses, it is a “use at your own risk” type of situation.

Vitess

Vitess is a database clustering system for horizontal scaling of MySQL and was initially developed in YouTube (and it was widely used there). Now it is a CNCF project and is actively developed by PlanetScale and the community. It is open source, but there are some features in Vitess itself that are only available for PlanetScale customers. Interesting fact: Vitess Operator serves as a core component of the PlanetScale DBaaS. So it is a production-grade and battle-tested operator.

Deployment

Going with a quickstart: 

git clone https://github.com/vitessio/vitess
cd vitess/examples/operator

kubectl apply -f operator.yaml

Operator is ready. Let’s deploy an intro cluster:

kubectl apply -f 101_initial_cluster.yaml

This deploys the following:

• Three etcd Pods
• Various Vitess Pods:
  • Two vttablet
  • vtadmin
  • vtctld
  • vtgate
  • vtorc 

You can read more about Vitess concepts and pieces in architecture documentation.

Features

Sharding is one of the biggest pros of this operator. The only competitor I can think of is TiDB (which should be MySQL protocol compatible, but not MySQL). There are no other solutions for MySQL sharding in the open source space. But at the same time, it all comes with a price – complexity, which Kubernetes for sure helps to masquerade. Getting familiar with all vt-* components can be overwhelming, especially for users who never used Vitess before. 

Operator provides users with all the regular management operations. The only downside is that these operations are not well documented and you have to discover them through various blog posts, reference docs, and other artifacts. For example, this blog post covers some basics for backups and restores, whereas this document covers basic Vitess operations.

Percona

At Percona we have two operators for MySQL:

1. Based on Percona XtraDB Cluster (PXC)
2. Based on Percona Server for MySQL (PS)

In Percona Operator for MySQL – Alpha Release, we explain why we decided to create the new operator. Both operators are fully open source as the components they are based on. The one based on PXC is production-ready, whereas PS is getting there.

Deployment

For Percona Kubernetes Operators we maintain helm charts for ease of onboarding. Deployment is a two-step process.

Deploy the operator:

helm repo add percona https://percona.github.io/percona-helm-charts/
helm install my-op percona/pxc-operator

And the database:

helm install my-db percona/pxc-db

Features

For features, I will focus on PXC as it is production-ready, and we are aiming for PS Operator to reach parity in the nearest future. 

• Proxy integration. Along with the database, the operator also deploys proxies. Users can choose from HAProxy and ProxySQL. The choice depends on the use case.
• Operator allows users to have multi-cluster or multi-regional MySQL deployments. This is useful for migrations or disaster recovery.
• There are no other operators for MySQL that provide a true point-in-time recovery. We developed a solution that continuously stores backups on object storage. Read more about architecture decisions in Point-In-Time Recovery in Percona Operator for MySQL Based on Percona XtraDB Cluster – Architecture Decisions.
• Operator provides automated and safe upgrades for minor versions of MySQL and its components (like proxies). It is extremely useful to quickly fix critical vulnerabilities and roll out bug fixes with zero downtime.

Percona is committed to providing open source products to the community, but we also provide exceptional services for our customers: managed and professional services and support. We have an ecosystem of products — Percona Platform — that brings together our software and services offerings.

As cloud-native apps continue to become increasingly central to how organizations operate, a startup founded by the creators of a popular open-source tool to manage authorization for cloud-native application environments is announcing some funding to expand its efforts at commercializing the opportunity.

Styra, the startup behind Open Policy Agent, has picked up $40 million in a Series B round of funding led by Battery Ventures. Also participating are previous backers A. Capital, Unusual Ventures and Accel; and new backers CapitalOne Ventures, Citi Ventures and Cisco Investments. Styra has disclosed CapitalOne is also one of its customers, along with e-commerce site Zalando and the European Patent Office.

Styra is sitting on the classic opportunity of open source technology: scale and demand.

OPA — which can be used across Kubernetes, containerized and other environments — now has racked up some 75 million downloads and is adding some 1 million downloads weekly, with Netflix, Capital One, Atlassian and Pinterest among those that are using OPA for internal authorization purposes. The fact that OPA is open source is also important:

“Developers are at the top of the food chain right now,” CEO Bill Mann said in an interview, “They choose which technology on which to build the framework, and they want what satisfies their requirements, and that is open source. It’s a foundational change: if it isn’t open source it won’t pass the test.”

But while some of those adopting OPA have hefty engineering teams of their own to customize how OPA is used, the sheer number of downloads (and potential active users stemming from that) speak to the opportunity for a company to build tools to help manage that and customize it for specific use cases in cases where those wanting to use OPA may lack the resources (or appetite) to build and scale custom implementations themselves.

As with many of the enterprise startups getting funded at the moment, Styra has proven itself in particular over the last year, with the switch to remote work, workloads being managed across a number of environments, and the ever-persistent need for better security around what people can and should not be using. Authorization is a particularly acute issue when considering the many access points that need to be monitored: as networks continue to grow across multiple hubs and applications, having a single authorization tool for the whole stack becomes even more important.

Styra said that some of the funding will be used to continue evolving its product, specifically by creating better and more efficient ways to apply authorization policies by way of code; and by bringing in more partners to expand the scope of what can be covered by its technology.

“We are extremely impressed with the Styra team and the progress they’ve made in this dynamic market to date,” said Dharmesh Thakker, a general partner at Battery Ventures. “Everyone who is moving to cloud, and adopting containerized applications, needs Styra for authorization—and in the light of today’s new, remote-first work environment, every enterprise is now moving to the cloud.” Thakker is joining the board with this round.

It was easy to wonder what would become of Docker after it sold its enterprise business in 2019, but it regrouped last year as a cloud native container company focused on developers, and the new approach appears to be bearing fruit. Today, the company announced a $23 million Series B investment.

Tribe Capital led the round with participation from existing investors Benchmark and Insight Partners. Docker has now raised a total of $58 million including the $35 million investment it landed the same day it announced the deal with Mirantis.

To be sure, the company had a tempestuous 2019 when they changed CEOs twice, sold the enterprise division and looked to reestablish itself with a new strategy. While the pandemic made 2020 a trying time for everyone, Docker CEO Scott Johnston says that in spite of that, the strategy has begun to take shape.

“The results we think speak volumes. Not only was the strategy strong, but the execution of that strategy was strong as well,” Johnston told me. He indicated that the company added 1.7 million new developer registrations for the free version of the product for a total of more than 7.3 million registered users on the community edition.

As with any open-source project, the goal is to popularize the community project and turn a small percentage of those users into paying customers, but Docker’s problem prior to 2019 had been finding ways to do that. While he didn’t share specific numbers, Johnston indicated that annual recurring revenue (ARR) grew 170% last year, suggesting that they are beginning to convert more successfully.

Johnston says that’s because they have found a way to turn a certain class of developer in spite of a free version being available. “Yes, there’s a lot of upstream open-source technologies, and there are users that want to hammer together their own solutions. But we are also seeing these eight to 10 person ‘two-pizza teams’ who want to focus on building applications, and so they’re willing to pay for a service,” he said.

That open-source model tends to get the attention of investors because it comes with that built-in action at the top of the sales funnel. Tribe’s Arjun Sethi, whose firm led the investment, says his company actually was a Docker customer before investing in the company and sees a lot more growth potential.

“Tribe focuses on identifying N-of-1 companies — top-decile private tech firms that are exhibiting inflection points in their growth, with the potential to scale toward outsized outcomes with long-term venture capital. Docker fits squarely into this investment thesis [ … ],” Sethi said in a statement.

Johnston says as they look ahead post-pandemic, he’s learned a lot since his team moved out of the office last year. After surveying employees, they were surprised to learn that most have been happier working at home, having more time to spend with family, while taking away a grueling commute. As a result, he sees going virtual first, even after it’s safe to reopen offices.

That said, he is planning to offer a way to get teams together for in-person gatherings and a full company get-together once a year.

“We’ll be virtual first, but then with the savings of the real estate that we’re no longer paying for, we’re going to bring people together and make sure we have that social glue,” he said.

---

Early Stage is the premier “how-to” event for startup entrepreneurs and investors. You’ll hear firsthand how some of the most successful founders and VCs build their businesses, raise money and manage their portfolios. We’ll cover every aspect of company building: Fundraising, recruiting, sales, product-market fit, PR, marketing and brand building. Each session also has audience participation built in — there’s ample time included for audience questions and discussion. Use code “TCARTICLE at checkout to get 20% off tickets right here.