In this article, we will take a look at how to run Percona Monitoring and Management (PMM) Server in a container without root privileges.
Some of the concerns companies have about using Docker relate to the security risks that exist due to the requirement for root privileges in order to run the service and therefore the containers. If processes inside the container are running as root then they are also running as such outside of the container, which means that you should take measures to mitigate privilege escalation issues that could occur from container breakout. Recently, Docker added experimental support for running in rootless mode which requires a number of extra changes.
From a personal perspective, it seems that Docker tries to do it “all in one”, which has its issues. While I was looking to overcome some gripes with Docker, reduce requirements, and also run containers as a chosen user regardless of the process users inside the container, podman and buildah caught my attention, especially as podman also supports Kubernetes. To quote the docs:
Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode.
Configuring user namespaces
In order to use containers without the need for root privileges, some initial configuration is likely to be needed to set the mapping of namespaces, as well as a suitable limit. Using the shadow-utils package on CentOS, or uidmap package on Debian, it is possible to assign subordinate user (man subuid) and group IDs (man subgid) to allow the mapping of IDs inside a container to a dedicated set of IDs outside the container. This allows us to resolve the issue where a process is running as root both inside and outside of a container.
The following is an Ansible task snippet that, on a dedicated CentOS machine, makes the necessary changes, although you may need/want to adjust this if already using namespaces:
- name: install packages
yum:
name:
- runc
- podman
- podman-docker
- buildah
- shadow-utils
- slirp4netns
update_cache: yes
state: latest
become: yes
tags:
- packages
- name: set user namespace
copy:
content: '{{ ansible_user }}:100000:65536'
dest: '/etc/{{ item }}'
become: yes
loop:
- subuid
- subgid
register: set_user_namespace_map
tags:
- namespace
- name: set sysctl user.max_user_namespaces
sysctl:
name: user.max_user_namespaces
value: 100000
state: present
sysctl_file: /etc/sysctl.d/01-namespaces.conf
sysctl_set: yes
reload: yes
become: yes
register: set_user_namespace_sysctl
tags:
- namespace
- name: reboot to apply
reboot:
post_reboot_delay: 60
become: yes
when: set_user_namespace_map.changed or set_user_namespace_sysctl.changed
tags:
- namespace
This set of tasks will:
- Install some packages that are either required or useful when using podman.
- Configure the 2 files used to map IDs (
/etc/subuid
and
/etc/subgid
); we allow the user to have 65536 user IDs mapped, starting the mapping from 100000 and likewise for group IDs.
- Set
user.max_user_namespaces
to ensure that you can allocate sufficient IDs, making it persistent after a reboot. If you have needed to configure the namespaces then it is likely that you need to reboot to make sure that the changes are active.
Look Ma, PMM — and no root!
OK, so if all is well, then the system is configured to support user namespaces and we can run a container without needing to be root (or a member of a special group). The commands for podman are identical, or very close to those that you would use for Docker, and you can even set
alias docker=podman
if you wish! Here is how you could test PMM Server (v2) out, mapping port 8443 to the NGINX port inside the container:
$ whoami percona $ podman run -d --name pmm2-test -p 8443:443 docker.io/percona/pmm-server:2
In the previous command, the path to the registry is explicitly stated as being a Docker one, but if you were to simply specify percona/pmm-server:2 then by default a number of registries are checked and the first match will win. The list and order of the registries differ per distro, e.g.
- Ubuntu:
registries = ['quay.io', 'docker.io']
- RedHat:
registries = ['registry.redhat.io', 'quay.io', 'docker.io']
Checking the processes
The container for PMM server is now running as the user that executed the podman command, so we can take a look and see what the processes look like. First, let’s take a peek inside the container:
$ podman exec pmm-server bash -c "ps -e -o pid,ppid,pgrp,user,comm --forest"
PID PPID PGRP USER COMMAND
862 0 862 root ps
1 0 1 root supervisord
9 1 9 postgres postgres
88 9 88 postgres \_ postgres
98 9 98 postgres \_ postgres
100 9 100 postgres \_ postgres
101 9 101 postgres \_ postgres
102 9 102 postgres \_ postgres
103 9 103 postgres \_ postgres
104 9 104 postgres \_ postgres
154 9 154 postgres \_ postgres
173 9 173 postgres \_ postgres
183 9 183 postgres \_ postgres
10 1 10 root clickhouse-serv
11 1 11 grafana grafana-server
12 1 12 root nginx
23 12 12 nginx \_ nginx
13 1 13 root crond
14 1 14 pmm prometheus
17 1 17 root pmm-managed
51 17 17 root \_ supervisorctl
18 1 18 root pmm-agent
172 18 18 root \_ node_exporter
174 18 18 root \_ postgres_export
153 1 153 pmm percona-qan-api
We can see all of the processes that run as part of PMM, plus the ps command that was executed to look at the processlist.
Now, let’s take a look outside the container to see what these processes show up as:
$ pgrep -f 'postgres|supervisord|clickhouse-serv|pmm-managed' | xargs ps -o pid,ppid,pgrp,user,comm --forest PID PPID PGRP USER COMMAND 32020 32003 32003 percona node_exporter 23532 23358 23358 percona postgres_export 23530 23358 23358 percona node_exporter 23332 23321 23332 percona supervisord 23349 23332 23349 100025 \_ postgres 23440 23349 23440 100025 | \_ postgres 23456 23349 23456 100025 | \_ postgres 23458 23349 23458 100025 | \_ postgres 23459 23349 23459 100025 | \_ postgres 23460 23349 23460 100025 | \_ postgres 23461 23349 23461 100025 | \_ postgres 23462 23349 23462 100025 | \_ postgres 23512 23349 23512 100025 | \_ postgres 23531 23349 23531 100025 | \_ postgres 23541 23349 23541 100025 | \_ postgres 23350 23332 23350 percona \_ clickhouse-serv 23357 23332 23357 percona \_ pmm-managed
Great! As we can see from the processlist, none of the processes that we checked are running as root, the ones that are running as root inside the container are running as percona, and the remainder are using unknown user IDs generated by the subordinate mapping.
Using persistent volumes for your data
The documented way to use PMM with persistent volumes does not work in the same manner with podman. However, it is easily fixed as follows (plus we get to try out buildah):
$ cat <<EOS > PMM2.Dockerfile FROM docker.io/percona/pmm-server:2 VOLUMES ['/srv'] EOS $ buildah bud -f PMM2.Dockerfile --tag localhost/pmm-server:custom . $ podman create --name pmm-data -v /srv localhost/pmm-server:custom $ podman run -d --name pmm-server -v pmm-data:/srv -p 8443:443 localhost/pmm-server:custom
Summary
It is easy to get PMM server running in a container with podman and buildah while also increasing the security from the outside by restricting the user privileges for the processes.
And while you may not be able to replace Docker usage in certain circumstances, taking a look at podman and buildah is definitely recommended… and you can still use the Docker registry if that is the only location for the image that you want to run.
If you haven’t tried PMM2 yet then you can easily test it out or take a look at the online demo.