Personal blog of Yzmir Ramirez

Coroot, an open source observability tool powered by eBPF, went generally available with version 1.0 last week. As this tool is cloud-native, we were curious to know how it can help troubleshoot databases on Kubernetes.In this blog post, we will see how to quickly debug PostgreSQL with Coroot and Percona Operator for PostgreSQL.PrepareInstall CorootThe easiest […]

To follow up on my previous Kubernetes articles:

• Installed in part one (see Using Percona Kubernetes Operators With K3s Part 1: Installation) and
• Have Percona Server for MySQL running (Using Percona Kubernetes Operators With K3s Part 2: Percona Server for MySQL Operator)
• Monitoring with Percona Monitoring and Management (PMM): Using Percona Kubernetes Operators With K3s Part 3: Monitoring and PMM

I would like to introduce a project which brings Kubernetes observability to the next level and provides major help in visualizing and diagnosing Kubernetes deployments.

The project is Coroot and it comes with the Community version Coroot Community Edition (source code at https://github.com/coroot/coroot) which you can install on your Kubernetes. The installation using helm charts is trivial:

helm repo add coroot https://coroot.github.io/helm-charts
helm repo update
helm install --namespace coroot --create-namespace coroot coroot/coroot

Enabling port forwarding to access Coroot through the web:

kubectl port-forward -n coroot service/coroot 8080:8080 --address='0.0.0.0'

The first you can see is the graph representing Kubernetes deployment. Why is it helpful? Let’s review what we can see in the plain Kubernetes:

List of pods:

kubectl get pods   
NAME                                             READY   STATUS    RESTARTS       AGE
percona-server-mysql-operator-7bb68f7b6d-8mg4j   1/1     Running   0              2d20h
cluster1-orc-0                                   2/2     Running   0              2d5h
cluster1-orc-1                                   2/2     Running   0              2d5h
cluster1-mysql-1                                 3/3     Running   1 (2d5h ago)   2d5h
cluster1-orc-2                                   2/2     Running   0              2d5h
cluster1-haproxy-0                               2/2     Running   0              2d5h
cluster1-haproxy-1                               2/2     Running   0              2d5h
cluster1-haproxy-2                               2/2     Running   0              2d5h
cluster1-mysql-2                                 3/3     Running   1 (2d5h ago)   2d5h
cluster1-mysql-0                                 3/3     Running   1 (23h ago)    23h
my-op-psmdb-operator-855dbfff5d-s8js5            1/1     Running   0              69m
cluster2-orc-0                                   2/2     Running   0              58m
cluster2-orc-1                                   2/2     Running   0              58m
cluster2-mysql-0                                 3/3     Running   0              58m
cluster2-haproxy-0                               2/2     Running   0              57m
cluster2-haproxy-1                               2/2     Running   0              57m
cluster2-haproxy-2                               2/2     Running   0              57m
cluster2-mysql-1                                 3/3     Running   1 (57m ago)    58m
cluster2-orc-2                                   2/2     Running   0              57m
cluster2-mysql-2                                 3/3     Running   1 (56m ago)    57m

And the list of services:

kubectl get svc  
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
kubernetes               ClusterIP   10.43.0.1       <none>        443/TCP                                 2d20h
cluster1-mysql-primary   ClusterIP   10.43.252.22    <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   2d5h
cluster1-mysql-unready   ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   2d5h
cluster1-mysql           ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   2d5h
cluster1-orc             ClusterIP   None            <none>        3000/TCP,10008/TCP                      2d5h
cluster1-orc-0           ClusterIP   10.43.236.23    <none>        3000/TCP,10008/TCP                      2d5h
cluster1-orc-1           ClusterIP   10.43.12.1      <none>        3000/TCP,10008/TCP                      2d5h
cluster1-orc-2           ClusterIP   10.43.12.67     <none>        3000/TCP,10008/TCP                      2d5h
cluster1-haproxy         ClusterIP   10.43.196.248   <none>        3306/TCP,3307/TCP,3309/TCP              2d5h
cluster2-mysql-primary   ClusterIP   10.43.219.150   <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   59m
cluster2-mysql-unready   ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   59m
cluster2-mysql           ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP,6033/TCP   59m
cluster2-orc             ClusterIP   None            <none>        3000/TCP,10008/TCP                      59m
cluster2-orc-0           ClusterIP   10.43.226.204   <none>        3000/TCP,10008/TCP                      59m
cluster2-orc-1           ClusterIP   10.43.122.217   <none>        3000/TCP,10008/TCP                      59m
cluster2-orc-2           ClusterIP   10.43.221.34    <none>        3000/TCP,10008/TCP                      59m
cluster2-haproxy         ClusterIP   10.43.41.95     <none>        3306/TCP,3307/TCP,3309/TCP              59m

I do not know how this is for you, but I have a hard time processing these lists, while Coroot offers visual help. Coroot is a web application, so there will be a lot of screenshots.

There is a glance overview of my deployments, including physical nodes:

And from there we can drill down into individual components, for example, cluster1-mysql:

And there you can highlight important relations, like for HAProxy:

We can see that it handles pod mysql-1 and mysql-2 right now, even for the MySQL setup itself:

We can see mysql-2 is the primary and mysql-0 and mysql-1 are replicas. There is also a network layout:

This shows network connections between different physical nodes and pods mapping to nodes. If we run a scenario with MySQL primary failure (similar to what we did in Using Percona Kubernetes Operators With K3s Part 2: Percona Server for MySQL Operator), it also can be visualized.

Node failure:

Primary failover from mysql-0 to mysql-2:

In general, Coroot offers visibility into a variety of problems you may encounter in Kubernetes, and there are more details on their Failurepedia page.

Summary

In my view, Coroot brings the visibility of Kubernetes deployments to the next level and is a must-have tool for the diagnostics and troubleshooting of Kubernetes workloads. If you run your application in Kubernetes, I recommend you give it a try!