
While it is convenient to view the log of MySQL or MongoDB pods with
kubectl logs
, sometimes the log is purged when the pod is deleted, which makes searching historical logs a bit difficult. Grafana Loki, an aggregation logging tool from Grafana, can be installed in the existing Kubernetes environment to help store historical logs and build a comprehensive logging stack.
What Is Grafana Loki
Grafana Loki is a set of tools that can be integrated to provide a comprehensive logging stack solution. Grafana-Loki-Promtail is the major component of this solution. The Promtail agent collects and ships the log to the Loki datastore and then visualizes logs with Grafana. While collecting the logs, Promtail can label, convert and filter the log before sending. Next step, Loki receives the logs and indexes the metadata of the log. At this step, the logs are ready to be visualized in the Grafana and the administrator can use Grafana and Loki’s query language, LogQL, to explore the logs.

Installing Grafana Loki in Kubernetes Environment
We will use the official helm chart to install Loki. The Loki stack helm chart supports the installation of various components like
promtail
,
fluentd
,
Prometheus
and
Grafana
.
$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm repo update
$ helm search repo grafana
NAME CHART VERSION APP VERSION DESCRIPTION
grafana/grafana 6.29.2 8.5.0 The leading tool for querying and visualizing t...
grafana/grafana-agent-operator 0.1.11 0.24.1 A Helm chart for Grafana Agent Operator
grafana/fluent-bit 2.3.1 v2.1.0 Uses fluent-bit Loki go plugin for gathering lo...
grafana/loki 2.11.1 v2.5.0 Loki: like Prometheus, but for logs.
grafana/loki-canary 0.8.0 2.5.0 Helm chart for Grafana Loki Canary
grafana/loki-distributed 0.48.3 2.5.0 Helm chart for Grafana Loki in microservices mode
grafana/loki-simple-scalable 1.0.0 2.5.0 Helm chart for Grafana Loki in simple, scalable...
grafana/loki-stack 2.6.4 v2.4.2 Loki: like Prometheus, but for logs.
For a quick introduction, we will install only
loki-stack
and
promtail
. We will use the Grafana pod, that is deployed by Percona Monitoring and Management to visualize the log.
$ helm install loki-stack grafana/loki-stack --create-namespace --namespace loki-stack --set promtail.enabled=true,loki.persistence.enabled=true,loki.persistence.size=5Gi
Let’s see what has been installed:
$ kubectl get all -n loki-stack
NAME READY STATUS RESTARTS AGE
pod/loki-stack-promtail-xqsnl 1/1 Running 0 85s
pod/loki-stack-promtail-lt7pd 1/1 Running 0 85s
pod/loki-stack-promtail-fch2x 1/1 Running 0 85s
pod/loki-stack-promtail-94rcp 1/1 Running 0 85s
pod/loki-stack-0 1/1 Running 0 85s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/loki-stack-headless ClusterIP None <none> 3100/TCP 85s
service/loki-stack ClusterIP 10.43.24.113 <none> 3100/TCP 85s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/loki-stack-promtail 4 4 4 4 4 <none> 85s
NAME READY AGE
statefulset.apps/loki-stack 1/1 85s
Promtail
and
loki-stack
pods have been created, together with a service that
loki-stack
will use to publish the logs to Grafana for visualization.
You can see the promtail pods are deployed by a
daemonset
which spawns a promtail pod in every node. This is to make sure the logs from every pod in all the nodes are collected and shipped to the
loki-stack
pod for centralizing storing and management.
$ kubectl get pods -n loki-stack -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
loki-stack-promtail-xqsnl 1/1 Running 0 2m7s 10.42.0.17 phong-dinh-default <none> <none>
loki-stack-promtail-lt7pd 1/1 Running 0 2m7s 10.42.1.12 phong-dinh-node1 <none> <none>
loki-stack-promtail-fch2x 1/1 Running 0 2m7s 10.42.2.13 phong-dinh-node2 <none> <none>
loki-stack-promtail-94rcp 1/1 Running 0 2m7s 10.42.3.11 phong-dinh-node3 <none> <none>
loki-stack-0 1/1 Running 0 2m7s 10.42.0.19 phong-dinh-default <none> <none>
Integrating Loki With PMM Grafana
Next, we will add Loki as a data source of PMM Grafana, so we can use PMM Grafana to visualize the logs. You can do it from the GUI or with kubectl CLI.
Below is the step to add data source from PMM GUI:
Navigate to
https://<PMM-IP-addres>:9443/graph/datasources
then select
Add data source

Then Select Loki

Next, in the Settings, in the HTTP URL box, enter the DNS records of the
loki-stack
service

You can also use the below command to add the data source. Make sure to specify the name of PMM pod, in this command,
monitoring-0
is the PMM pod.
$ kubectl -n default exec -it monitoring-0 -- bash -c "curl 'http://admin:verysecretpassword@127.0.0.1:3000/api/datasources' -X POST -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{ \"orgId\": 1, \"name\": \"Loki\", \"type\": \"loki\", \"typeLogoUrl\": \"\", \"access\": \"proxy\", \"url\": \"http://loki-stack.loki-stack.svc.cluster.local:3100\", \"password\": \"\", \"user\": \"\", \"database\": \"\", \"basicAuth\": false, \"basicAuthUser\": \"\", \"basicAuthPassword\": \"\", \"withCredentials\": false, \"isDefault\": false, \"jsonData\": {}, \"secureJsonFields\": {}, \"version\": 1, \"readOnly\": false }'"
Exploring the Logs in PMM Grafana
Now, you can explore the pod logs in Grafana UI, navigate to Explore, and select Loki in the dropdown list as the source of metrics:

In the Log Browser, you can select the appropriate labels to form your first LogQL query, for example, I select the following attributes
{app="percona-xtradb-cluster", component="pxc", container="logs",pod="cluster1-pxc-1"}

Click Show logs and you can see all the logs of
cluster-pxc-1
pod

You can perform a simple filter with |= “message-content”. For example, filtering all the messages related to State transfer by
{app="percona-xtradb-cluster", component="pxc", container="logs",pod="cluster1-pxc-1"} |= "State transfer"

Conclusion
Deploying Grafana Loki in the Kubernetes environment is feasible and straightforward. Grafana Loki can be integrated easily with Percona Monitoring and Management to provide both centralized logging and comprehensive monitoring when running Percona XtraDB Cluster and Percona Server for MongoDB in Kubernetes.
It would be interesting to know if you have any thoughts while reading this, please share your comments and thoughts.