Personal blog of Yzmir Ramirez

With its Kubernetes Engine (GKE), Google Cloud has long offered a managed service for running containers on its platform. Kubernetes users tend to have a variety of needs, but so far, Google only offered a single tier of GKE that wasn’t necessarily geared toward the high-end enterprise users the company is trying to woo. Today, however, the company announced a new advanced edition of GKE that introduces a number of new features and an enhanced financially backed SLA, additional security tools and new automation features. You can think of GKE Advanced as the enterprise version of GKE.

The new service will launch in the second quarter of the year and hasn’t yet announced pricing. The regular version of GKE is now called GKE Standard.

Google says the service builds upon the company’s own learnings from running a complex container infrastructure internally for years.

For enterprise customers, the financially backed SLA is surely a nice bonus. The promise here is 99.95 percent guaranteed availability for regional clusters.

Most users who opt for a managed Kubernetes environment do so because they don’t want to deal with the hassle of managing these clusters themselves. With GKE Standard, there’s still some work to be done with regard to scaling the clusters. Because of this, GKE Advanced includes a Vertical Pod Autoscaler that keeps on eye on resource utilization and adjusts it as necessary, as well as Node Auto Provisioning, an enhanced version of cluster autoscaling in GKE Standard.

In addition to these new GKE Advanced features, Google is adding GKE security features like the GKE Sandbox, which is currently in beta and will come exclusively to GKE Advanced once it’s launched, and the ability to enforce that only signed and verified images are used in the container environment.

The Sandbox uses Google’s gVisor container sandbox runtime. With this, every sandbox gets its own user-space kernel, adding an additional layer of security. With Binary Authorization, GKE Advanced users also can ensure that all container images are signed by a trusted authority before they are put into production. Somebody could theoretically still smuggle malicious code into the containers, but this process, which enforces standard container release practices, for example, should ensure that only authorized containers can run in the environment.

GKE Advanced also includes support for GKE usage metering, which allows companies to keep tabs on who is using a GKE cluster and charge them according. This feature, too, will be exclusive to GKE Advanced.

The OpenStack project, which powers more than 75 public and thousands of private clouds, launched the 19th version of its software this week. You’d think that after 19 updates to the open-source infrastructure platform, there really isn’t all that much new the various project teams could add, given that we’re talking about a rather stable code base here. There are actually a few new features in this release, though, as well as all the usual tweaks and feature improvements you’d expect.

While the hype around OpenStack has died down, we’re still talking about a very active open-source project. On average, there were 155 commits per day during the Stein development cycle. As far as development activity goes, that keeps OpenStack on the same level as the Linux kernel and Chromium.

Unsurprisingly, a lot of that development activity focused on Kubernetes and the tools to manage these container clusters. With this release, the team behind the OpenStack Kubernetes installer brought the launch time for a cluster down from about 10 minutes to five, regardless of the number of nodes. To further enhance Kubernetes support, OpenStack Stein also includes updates to Neutron, the project’s networking service, which now makes it easier to create virtual networking ports in bulk as containers are spun up, and Ironic, the bare-metal provisioning service.

All of that is no surprise, given that according to the project’s latest survey, 61 percent of OpenStack deployments now use both Kubernetes and OpenStack in tandem.

The update also includes a number of new networking features that are mostly targeted at the many telecom users. Indeed, over the course of the last few years, telcos have emerged as some of the most active OpenStack users as these companies are looking to modernize their infrastructure as part of their 5G rollouts.

Besides the expected updates, though, there are also a few new and improved projects here that are worth noting.

“The trend from the last couple of releases has been on scale and stability, which is really focused on operations,” OpenStack Foundation executive director Jonathan Bryce told me. “The new projects — and really most of the new projects from the last year — have all been pretty oriented around real-world use cases.”

The first of these is Placement. “As people build a cloud and start to grow it and it becomes more broadly adopted within the organization, a lot of times, there are other requirements that come into play,” Bryce explained. “One of these things that was pretty simplistic at the beginning was how a request for a resource was actually placed on the underlying infrastructure in the data center.” But as users get more sophisticated, they often want to run specific workloads on machines with certain hardware requirements. These days, that’s often a specific GPU for a machine learning workload, for example. With Placement, that’s a bit easier now.

It’s worth noting that OpenStack had some of this functionality before. The team, however, decided to uncouple it from the existing compute service and turn it into a more generic service that could then also be used more easily beyond the compute stack, turning it more into a kind of resource inventory and tracking tool.

Then, there is also Blazer, a reservation service that offers OpenStack users something akin to AWS Reserved Instances. In a private cloud, the use case for a feature is a bit different, though. But as some of the private clouds got bigger, some users found that they needed to be able to guarantee resources to run some of their regular, overnight batch jobs or data analytics workloads, for example.

As far as resource management goes, it’s also worth highlighting Sahara, which now makes it easier to provision Hadoop clusters on OpenStack.

In previous releases, one of the focus areas for the project was to improve the update experience. OpenStack is obviously a very complex system, so bringing it up to the latest version is also a bit of a complex undertaking. These improvements are now paying off. “Nobody even knows we are running Stein right now,” Vexxhost CEO Mohammed Nasar, who made an early bet on OpenStack for his service, told me. “And I think that’s a good thing. You want to be least impactful, especially when you’re in such a core infrastructure level. […] That’s something the projects are starting to become more and more aware of but it’s also part of the OpenStack software in general becoming much more stable.”

As usual, this release launched only a few weeks before the OpenStack Foundation hosts its bi-annual Summit in Denver. Since the OpenStack Foundation has expanded its scope beyond the OpenStack project, though, this event also focuses on a broader range of topics around open-source infrastructure. It’ll be interesting to see how this will change the dynamics at the event.

Google’s Cloud Services Platform for managing hybrid clouds that span on-premise data centers and the Google cloud is coming out of beta today. The company is also changing the product’s name to Anthos, a name that either refers to a lost Greek tragedy, the name of an obscure god in the Marvel universe or rosemary. That by itself would be interesting, but minor news. What makes this interesting is that Google also today announced that Anthos will run on third-party clouds, as well, including AWS and Azure.

“We will support Anthos and AWS and Azure as well, so people get one way to manage their application and that one way works across their on-premise environments and all other clouds,” Google’s senior VP for its technical infrastructure, Urs Hölzle, explained in a press conference ahead of today’s announcement.

So with Anthos, Google will offer a single managed service that will let you manage and deploy workloads across clouds, all without having to worry about the different environments and APIs. That’s a big deal and one that clearly delineates Google’s approach from its competitors’. This is Google, after all, managing your applications for you on AWS and Azure.

“You can use one consistent approach — one open-source based approach — across all environments,” Hölzle said. “I can’t really stress how big a change that is in the industry, because this is really the stack for the next 20 years, meaning that it’s not really about the three different clouds that are all randomly different in small ways. This is the way that makes these three cloud — and actually on-premise environments, too — look the same.”

Anthos/Google Cloud Services Platform is based on the Google Kubernetes Engine, as well as other open-source projects like the Istio service mesh. It’s also hardware agnostic, meaning that users can take their current hardware and run the service on top of that without having to immediately invest in new servers.

Why is Google doing this? “We hear from our customers that multi-cloud and hybrid is really an acute pain point,” Hölzle said. He noted that containers are the enabling technology for this but that few enterprises have developed a unifying strategy to manage these deployments and that it takes expertise in all major clouds to get the most out of them.

Enterprises already have major investments in their infrastructure and created relationships with their vendors, though, so it’s no surprise that Google is launching Anthos with more than 30 major hardware and software partners that range from Cisco to Dell EMC, HPE and VMware, as well as application vendors like Confluent, Datastax, Elastic, Portworx, Tigera, Splunk, GitLab, MongoDB and others.

Robin.io, a data management service that offers a hyper-converged storage platform based on Kubernetes, also tells me that it worked closely with Google to develop the Anthos Storage API. “Robin Storage offers bare metal performance, powerful data management capabilities and Kubernetes-native management to support running enterprise applications on Google Cloud’s Anthos across on-premises data centers and the cloud,” said Premal Buch, CEO of Robin.io.

Anthos is a subscription-based service, with the list prices starting at $10,000/month per 100 vCPU block. Enterprise prices will then be up for negotiation, though, so many customers will likely pay less.

It’s one thing to use a service like this for new applications, but many enterprises already have plenty of line-of-business tools that they would like to bring to the cloud as well. For them, Google is launching the first beta of Anthos Migrate today. This service will auto-migrate VMs from on-premises or other clouds into containers in the Google Kubernetes Engine. The promise here is that this is essentially an automatic process and once the container is on Google’s platform, you’ll be able to use all of the other features that come with the Anthos platform, too.

Google’s Hölzle noted that the emphasis here was on making this migration as easy as possible. “There’s no manual effort there,” he said.

Two of the biggest trends in applications development in recent years have been the rise of serverless and containerization. Today at Google Cloud Next, the company announced a new product called Cloud Run that is designed to bring the two together. At the same time, the company also announced Cloud Run for GKE, which is specifically designed to run on Google’s version of Kubernetes.

Oren Teich, director of product management for serverless, says these products came out of discussions with customers. As he points out, developers like the flexibility and agility they get using serverless architecture, but have been looking for more than just compute resources. They want to get access to the full stack, and to that end the company is announcing Cloud Run.

“Cloud Run is introducing a brand new product that takes Docker containers and instantly gives you a URL. This is completely unique in the industry. We’re taking care of everything from the top end of SSL provisioning and routing, all the way down to actually running the container for you. You pay only by the hundred milliseconds of what you need to use, and it’s end-to-end managed,” Teich explained.

As for the GKE tool, it provides the same kinds of benefits, except for developers running their containers on Google’s GKE version of Kubernetes. Keep in mind, developers could be using any version of Kubernetes their organizations happen to have chosen, so it’s not a given that they will be using Google’s flavor of Kubernetes.

“What this means is that a developer can take the exact same experience, the exact same code they’ve written — and they have G Cloud command line, the same UI and our console and they can just with one-click target the destination they want,” he said.

All of this is made possible through yet another open-source project the company introduced last year called Knative. “Cloud Run is based on Knative, an open API and runtime environment that lets you run your serverless workloads anywhere you choose — fully managed on Google Cloud Platform, on your GKE cluster or on your own self-managed Kubernetes cluster,” Teich and Eyal Manor, VP of engineering, wrote in a blog post introducing Cloud Run.

Serverless, as you probably know by now, is a bit of a misnomer. It’s not really taking away servers, but it is eliminating the need for developers to worry about them. Instead of loading their application on a particular virtual machine, the cloud provider, in this case, Google, provisions the exact level of resources required to run an operation. Once that’s done, these resources go away, so you only pay for what you use at any given moment.

As we continue the development of our Percona Operators to simplify database deployment in Kubernetes (Percona Server for MongoDB Operator 0.3.0 and Percona XtraDB Cluster Operator 0.3.0), one very popular question I get is: how does deployment in Kubernetes affect MySQL performance? Is there a big performance penalty? So I plan to look at how to measure and compare the performance of Kubernetes deployments to bare metal deployments. Kubernetes manages a lot of infrastructure resources like network, storage, cpu, and memory, so we need to look individually at different components.

To begin: I plan to run a single MySQL (Percona Server) instances in a Kubernetes deployment, and use local storage (fast NMVe device). I also want to customize my MySQL configuration, as the one that is supplied in public images are pretty much all set to defaults.

Let’s take a look at how we can customize it.

1. We are going to use a public Percona Server docker image “percona:ps-8.0.15-5”, it will deploy the latest version (at the time of writing) Percona Server for MySQL 8.0.15
2. We will deploy this on a specific node and will assign specific local storage to use for MySQL data
3. We’ll set up a custom configuration for MySQL.

Setting up Kubernetes

Here’s an example yaml file:

=====================
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  spec:
  selector:
    matchLabels:
      app: mysql
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: mysql
      spec:
        nodeSelector:
          kubernetes.io/hostname: smblade01
        volumes:
          - name: mysql-persistent-storage
            hostPath:
              path: /mnt/fast/mysql
              type: Directory
        containers:
        - image: percona:ps-8.0.15-5
          name: mysql
          env:
            # Use secret in real usage
          - name: MYSQL_ROOT_PASSWORD
            value: password
          ports:
          - containerPort: 3306
            name: mysql
          volumeMounts:
          - name: mysql-persistent-storage
            mountPath: /var/lib/mysql
===============

There is a lot of typical Kubernetes boilerplate to create a deployment, but the most important parts to note:

• We choose the node where to deploy with nodeSelector (lines 28–29).
• We allocate the local storage for MySQL volume with hostPath (lines 31–34).

After deploying this, we make sure the Pod is running

Kubectl get pods
NAME                      READY    STATUS     RESTARTS   AGE     IP             NODE         NOMINATED NODE    READINESS GATES
mysql-d74d5d459-b6zrs     1/1      Running    0          3m8s    192.168.1.15   smblade01    <none>            <none>

Set up MySQL to access fast storage and modify the default config for performance

Now as we are running MySQL on a dedicated node with fast storage, we want to customize the MySQL configuration to allocate a big buffer pool and adjust its IO settings.

As I said, a downloaded image will most likely run with default settings and there is no straightforward way to pass our custom my.cnf to deployment. I’ll show you how to resolve this now.

The default my.cnf contains the directive

!includedir /etc/my.cnf.d

So the solution for the custom my.cnf is as follows:

• Create a Kubernetes configmap from our custom my.cnf. Here’s how to achieve that:

kubectl create configmap mysql-config --from-file=my.cnf

• Define yaml to load the configmap into the volume that mapped to /etc/my.cnf.d (nb lines 23–26 and 40-41).

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
    app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      nodeSelector:
        kubernetes.io/hostname: smblade01
      volumes:
        - name: mysql-persistent-storage
          hostPath:
            path: /mnt/fast/mysql
            type: Directory
       - name: config-volume
          configMap:
            name: mysql-config
            optional: true
      containers:
      - image: percona:ps-8
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
        - name: config-volume
          mountPath: /etc/my.cnf.d

And here’s our example my.cnf:

[mysqld]
skip-log-bin
ssl=0
table_open_cache = 200000
table_open_cache_instances=64
back_log=3500
max_connections=4000
innodb_file_per_table
innodb_log_file_size=10G
innodb_log_files_in_group=2
innodb_log_buffer_size=64M
innodb_open_files=4000
innodb_buffer_pool_size= 100G
innodb_buffer_pool_instances=8
innodb_flush_log_at_trx_commit = 1
innodb_doublewrite=1
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_io_capacity=2000
innodb_io_capacity_max=4000
innodb_flush_neighbors = 0
innodb_use_native_aio=1
join_buffer_size=256K
sort_buffer_size=256K

When we deploy this yaml, we will have a MySQL instance running on a dedicated box with fast local storage, big log files, and 100GB allocated for its InnoDB buffer pool.

Now we’re set to proceed to our performance measurements. Stay tuned!

—
Photo by Joseph Barrientos on Unsplash

Aqua Security, a startup that helps customers launch containers securely, announced a $62 million Series C investment today led by Insight Partners.

Existing investors Lightspeed Venture Partners, M12 (Microsoft’s venture fund), TLV Partners and Shlomo Kramer also participated. With today’s investment, the startup’s investments since inception now total over $100 million, according to the company.

Early investors took a chance on the company when it was founded in 2015. Containers were barely a thing back then, but the founders had a vision of what was coming down the pike and their bet has paid off in a big way as the company now has first-mover advantage. As more companies turn to Kubernetes and containers, the need for a security product built from the ground up to secure this kind of environment is essential.

While co-founder and CEO Dror Davidoff says the company has 60 Fortune 500 customers, he’s unable to share names, but he can provide some clues like five of the world’s top banks. As companies like that turn to new technology like containers, they aren’t going to go whole hog without a solid security option. Aqua gives them that.

“Our customers are all taking very dramatic steps towards adoption of those new technologies, and they know that existing security tools that they have in place will not solve the problems,” Davidoff told TechCrunch. He said that most customers have started small, but then have expanded as container adoption increases.

You may thank that an ephemeral concept like a container would be less of a security threat, but Davidoff says that the open nature of containerization actually leaves them vulnerable to tampering. “Container lives long enough to be dangerous,” he said. He added, “They are structured in an open way, making it simple to hack, and once in, to do lateral movement. If the container holds sensitive info, it’s easy to have access to that information.”

Aqua scans container images for malware and makes sure only certified images can run, making it difficult for a bad actor to insert an insecure image, but the ephemeral nature of containers also helps if something slips through. DevOp can simply take down the faulty container and put a newly certified clean one quickly.

The company has 150 employees with offices in the Boston area and R&D in Tel Aviv in Israel. With the new influx of cash, the company plans to expand quickly, growing sales and marketing, customer support and expanding the platform into areas to cover emerging areas like serverless computing. Davidoff says the company could double in size in the next 12-18 months and he’s expecting 3x to 4x customer growth.

All of that money should provide fuel to grow the company as containerization spreads and companies look for a security solution to keep containers in production safe.

Densify, a Toronto company that helps customers optimize their cloud resources to control usage and spending, announced a new tool today specifically designed to optimize container usage in the cloud.

Company CEO Gerry Smith says that as containerization proliferates, it’s getting more difficult to track and control cloud infrastructure resource usage as software development and deployment happens with increasing speed.

“The whole basis upon which people buy and use cloud and container resources has become wildly expensive because of the lack of a resource management system,” Smith said.

The Densify solution looks at the consumption and for ways to cut costs and usage. “We have analytics in the cloud, any of various common cloud services that you can connect to, and then we use machine learning to analyze the resources and your cloud and container consumption,” he said.

Densify continuously makes recommendations on how to make better use of resources and to find the cheapest computing, whether that’s reserved instances, spot instances or other discounted cloud resources.

What’s more, it can help you identify whether you are providing too few resources to accommodate the number of containers you are deploying, as well as too many.

This may sound a bit like what Spotinst and Cloudyn, the company Microsoft bought a couple of years ago, do in terms of helping control costs in the cloud, but Smith says for his company it’s more about understanding the resources than pure cost.

“We look at ourselves as a resource management platform. So what we do is characterize the applications, demands of CPU and all the other resources, and use machine learning to predict what it’s going to need at any given minute, at any given day of a week of the year, so that we can then better predictively match the right supply,” Smith explained.

It’s providing information about each container at a highly detailed level, including “what’s running, what resources are being allocated, and the true utilization of an organization’s Kubernetes environment at a cluster, namespace and container level,” according to the company. All of this information should help DevOps teams better understand the resources required by their container deployments.

The company has actually been around since 2006 under the name Cirba. In its early guise it helped companies manage VMware installations. In 2016, it pivoted to cloud resource management and changed the company name to Densify. It has raised around $60 million since inception, with about half of that coming after the company changed to Densify in 2016.

The company is based in Toronto, but has offices in London and Melbourne, as well.

Cloud Foundry, the open-source platform-as-a-service project that more than half of the Fortune 500 companies use to help them build, test and deploy their applications, launched well before Kubernetes existed. Because of this, the team ended up building Diego, its own container management service. Unsurprisingly, given the popularity of Kubernetes, which has become somewhat of the de facto standard for container orchestration, a number of companies in the Cloud Foundry ecosystem starting looking into how they could use Kubernetes to replace Diego.

The result of this is Project Eirini, which was first proposed by IBM. As the Cloud Foundry Foundation announced today, Project Eirini now passes the core functional tests the team runs to validate the software releases of its application runtime, the core Cloud Foundry service that deploys and manages applications (if that’s a bit confusing, don’t even think about the fact that there’s also a Cloud Foundry Container Runtime, which already uses Kubernetes, but which is mostly meant to give enterprise a single platform for running their own applications and pre-built containers from third-party vendors).

“That’s a pretty big milestone,” Cloud Foundry Foundation CTO Chip Childers told me. “The project team now gets to shift to a mode where they’re focused on hardening the solution and making it a bit more production-ready. But at this point, early adopters are also starting to deploy that [new] architecture.”

Childers stressed that while the project was incubated by IBM, which has been a long-time backer of the overall Cloud Foundry project, Google, Pivotal and others are now also contributing and have dedicated full-time engineers working on the project. In addition, SUSE, SAP and IBM are also active in developing Eirini.

Eirini started as an incubation project, and while few doubted that this would be a successful project, there was a bit of confusion around how Cloud Foundry would move forward now that it essentially had two container engines for running its core service. At the time, there was even some concern that the project could fork. “I pushed back at the time and said: no, this is the natural exploration process that open-source communities need to go through,” Childers said. “What we’re seeing now is that with Pivotal and Google stepping in, that’s a very clear sign that this is going to be the go-forward architecture for the future of the Cloud Foundry Application Runtime.”

A few months ago, by the way, Kubernetes was still missing a few crucial pieces the Cloud Foundry ecosystem needed to make this move. Childers specifically noted that Windows support — something the project’s enterprise users really need — was still problematic and lacked some important features. In recent releases, though, the Kubernetes team fixed most of these issues and improved its Windows support, rendering those issues moot.

What does all of this mean for Diego? Childers noted that the community isn’t at a point where it’ll hold developing that tool. At some point, though, it seems likely that the community will decide that it’s time to start the transition period and make the move to Kubernetes official.

It’s worth noting that IBM today announced its own preview of Eirini in its Cloud Foundry Enterprise Environment and that the latest version of SUSE’s Cloud Foundry-based Application Platform includes a similar preview as well.

In addition, the Cloud Foundry Foundation, which is hosting its semi-annual developer conference in Philadelphia this week, also announced that it has certified it first to systems integrators, Accenture and HCL as part of its recently launched certification program for companies that work in the Cloud Foundry ecosystem and have at least 10 certified developers on their teams.

Percona announces the availability of the Percona Server for MongoDB Operator 0.3.0 early access release.

The Percona Server for MongoDB Operator simplifies the deployment and management of Percona Server for MongoDB in a Kubernetes or OpenShift environment. It extends the Kubernetes API with a new custom resource for deploying, configuring and managing the application through the whole life cycle.

You can install the Percona Server for MongoDB Operator on Kubernetes or OpenShift. While the operator does not support all the Percona Server for MongoDB features in this early access release, instructions on how to install and configure it are already available along with the operator source code in our Github repository.

The Percona Server for MongoDB Operator is an early access release. Percona doesn’t recommend it for production environments.

New Features

• Advanced nodes assignment implemented in this version allows to run containers with Percona Server for MongoDB nodes on different hosts, availability zones, etc. to achieve higher availability and fault tolerance.
• Private container registries like those in OpenShift are now supported by Percona Server for MongoDB Operator so that internet access is not required for the deployment anymore.
• Percona Monitoring and Management node can be now included in the cluster to provide its superior quality Percona Server for MongoDB monitoring.
• The Operator can now optionally allow Percona Server for MongoDB Pods to use the local storage of the host.
• Percona Server for MongoDB 4.0 is now supported by the Operator  in addition to Percona Server for MongoDB 3.6.

Improvements

• The OpenShift Platform 4.0 is now supported by the Percona Server for MongoDB Operator.
• The Operator now supports Pod Disruption Budget by default to limit the number of Pods that can go down simultaneously.

Fixed Bugs

• CLOUD-141: Operator failed to rescale cluster after self-healing.
• CLOUD-151: Dashboard upgrade in Percona Monitoring and Management caused loop due to no write access.
• CLOUD-152: Percona Server for MongoDB crash took place in case of no backup section in the Operator configuration file.
• CLOUD-91: The Operator was throwing error messages with Arbiters disabled in the deploy/cr.yaml configuration file.

Percona Server for MongoDB is an enhanced, open source and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition. It supports MongoDB® protocols and drivers. Percona Server for MongoDB extends MongoDB Community Edition functionality by including the Percona Memory Engine, as well as several enterprise-grade features. It requires no changes to MongoDB applications or code.

In this blog post, I’m going to show you how to test Kubernetes locally on OSX/macOS. Testing Kubernetes without having access to a cloud operator in a local lab is not as easy as it sounds. I’d like to share some of my experiences in this adventure. For those who have already experienced in Virtualbox & Vagrant combination, I can tell you that it doesn’t work. Since Kubernetes will require virtualization, setting another virtual environment within another VirtualBox has several issues. After trying to bring up a cluster for a day or two, I gave up my traditional lab and figured out that Kubernetes has an alternate solution called minikube.

Installation

If your OSX/macOS doesn’t have brew I strongly recommend installing it. My OSX/macOS version at the time of this post was macOS 10.14.3 (18D109).

$ brew update && brew install kubectl && brew cask install docker minikube virtualbox

Once minikube is installed, we’ll need to start the virtual environment that is required to run our operator.

I’m starting my minikube environment with 4Gb memory since our Percona Xtradb(PXC) Cluster will have 3 MySQL nodes + 1 ProxySQL pod.

$ minikube start --memory 4096
  minikube v0.35.0 on darwin (amd64)
  Creating virtualbox VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
  "minikube" IP address is 192.168.99.100
  Configuring Docker as the container runtime ...
  Preparing Kubernetes environment ...
  Pulling images required by Kubernetes v1.13.4 ...
  Launching Kubernetes v1.13.4 using kubeadm ...
  Waiting for pods: apiserver proxy etcd scheduler controller addon-manager dns
  Configuring cluster permissions ...
  Verifying component health .....
  kubectl is now configured to use "minikube"
  Done! Thank you for using minikube!

We’re now ready to install Install Percona XtraDB Cluster on Kubernetes.

Setup

Clone and download Kubernetes Operator for MySQL.

$ git clone -b release-0.2.0 https://github.com/percona/percona-xtradb-cluster-operator
Cloning into 'percona-xtradb-cluster-operator'...
remote: Enumerating objects: 191, done.
remote: Counting objects: 100% (191/191), done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 10321 (delta 73), reused 138 (delta 67), pack-reused 10130
Receiving objects: 100% (10321/10321), 17.04 MiB | 3.03 MiB/s, done.
Resolving deltas: 100% (3526/3526), done.
Checking out files: 100% (5159/5159), done.
$ cd percona-xtradb-cluster-operator

Here we have to make the following modifications for this operator to work on OSX/macOS.

1. Reduce memory allocation for each pod.
2. Reduce CPU usage for each pod.
3. Change the topology type (because we want to run all PXC instances on one node).

$ sed -i.bak 's/1G/500m/g' deploy/cr.yaml
$ grep "memory" deploy/cr.yaml
        memory: 500m
      #   memory: 500m
        memory: 500m
      #   memory: 500m
$ sed -i.bak 's/600m/200m/g' deploy/cr.yaml
$ grep "cpu" deploy/cr.yaml
        cpu: 200m
      #   cpu: "1"
        cpu: 200m
      #   cpu: 700m
$ grep "topology" deploy/cr.yaml
      topologyKey: "kubernetes.io/hostname"
    #   topologyKey: "failure-domain.beta.kubernetes.io/zone"
$ sed -i.bak 's/kubernetes\.io\/hostname/none/g' deploy/cr.yaml
$ grep "topology" deploy/cr.yaml
      topologyKey: "none"
    #   topologyKey: "failure-domain.beta.kubernetes.io/zone"

We’re now ready to deploy our PXC via the operator.

$ kubectl apply -f deploy/crd.yaml
customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com created
customresourcedefinition.apiextensions.k8s.io/perconaxtradbbackups.pxc.percona.com created
$ kubectl create namespace pxc
namespace/pxc created
$ kubectl config set-context $(kubectl config current-context) --namespace=pxc
Context "minikube" modified.
$ kubectl apply -f deploy/rbac.yaml
role.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created
rolebinding.rbac.authorization.k8s.io/default-account-percona-xtradb-cluster-operator created
$ kubectl apply -f deploy/operator.yaml
deployment.apps/percona-xtradb-cluster-operator created
$ kubectl apply -f deploy/secrets.yaml
secret/my-cluster-secrets created
$ kubectl apply -f deploy/configmap.yaml
configmap/pxc created
$ kubectl apply -f deploy/cr.yaml
perconaxtradbcluster.pxc.percona.com/cluster1 created

Here we’re ready to monitor the progress of our deployment.

$ kubectl get pods
NAME                                               READY   STATUS              RESTARTS   AGE
cluster1-pxc-node-0                                0/1     ContainerCreating   0          86s
cluster1-pxc-proxysql-0                            1/1     Running             0          86s
percona-xtradb-cluster-operator-5857dfcb6c-g7bbg   1/1     Running             0          109s

If any of the nodes is having difficulty passing any STATUS to Running state

$ kubectl describe pod cluster1-pxc-node-0
Name:               cluster1-pxc-node-0
Namespace:          pxc
Priority:           0
.
..
...
Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  3m47s (x14 over 3m51s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         3m47s                   default-scheduler  Successfully assigned pxc/cluster1-pxc-node-0 to minikube
  Normal   Pulling           3m45s                   kubelet, minikube  pulling image "perconalab/pxc-openshift:0.2.0"
  Normal   Pulled            118s                    kubelet, minikube  Successfully pulled image "perconalab/pxc-openshift:0.2.0"
  Normal   Created           117s                    kubelet, minikube  Created container
  Normal   Started           117s                    kubelet, minikube  Started container
  Warning  Unhealthy         89s                     kubelet, minikube  Readiness probe failed:
At this stage we’re ready to verify our cluster as soon as we see following output (READY 1/1):
$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
cluster1-pxc-node-0                                1/1     Running   0          7m38s
cluster1-pxc-node-1                                1/1     Running   0          4m46s
cluster1-pxc-node-2                                1/1     Running   0          2m25s
cluster1-pxc-proxysql-0                            1/1     Running   0          7m38s
percona-xtradb-cluster-operator-5857dfcb6c-g7bbg   1/1     Running   0          8m1s

In order to connect to this cluster, we’ll need to deploy a client shell access.

$ kubectl run -i --rm --tty percona-client --image=percona:5.7 --restart=Never -- bash -il
If you don't see a command prompt, try pressing enter.
bash-4.2$ mysql -h cluster1-pxc-proxysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3617
Server version: 5.5.30 (ProxySQL)
Copyright (c) 2009-2019 Percona LLC and/or its affiliates
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> \s
--------------
mysql  Ver 14.14 Distrib 5.7.25-28, for Linux (x86_64) using  6.2
Connection id:		3617
Current database:	information_schema
Current user:		root@cluster1-pxc-proxysql-0.cluster1-pxc-proxysql.pxc.svc.cluste
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.5.30 (ProxySQL)
Protocol version:	10
Connection:		cluster1-pxc-proxysql via TCP/IP
Server characterset:	latin1
Db     characterset:	utf8
Client characterset:	latin1
Conn.  characterset:	latin1
TCP port:		3306
Uptime:			14 min 1 sec
Threads: 1  Questions: 3  Slow queries: 0
--------------

A few things to remember:

• Secrets for this setup are under deploy/secrets.yaml, you can decode via

$ echo -n '{secret}' |base64 -D

• To reconnect shell

$ kubectl run -i --tty percona-client --image=percona:5.7 -- sh

• To redeploy the pod delete first and repeat above steps without configuration changes

$ kubectl delete -f deploy/cr.yaml

• To stop and delete  minikube virtual environment

$ minikube stop

$ minikube delete

References

• Percona Xtradb Cluster
• Percona Xtradb Cluster Operator
• Kubectl Cheat Sheet

Credits

• Mykola Marzhan – Director of Server Engineering (Percona)

—
Photo by frank mckenna on Unsplash