Recently, I wrote a blog post showing how to enforce SELinux with Percona XtraDB Cluster (PXC). The Linux distributions derived from RedHat use SELinux. There is another major mandatory discretionary access control (DAC) system, AppArmor. Ubuntu, for example, installs AppArmor by default. If you are concerned by computer security and use PXC on Ubuntu, you should enforce AppArmor. This post will guide you through the steps of creating a profile for PXC and enabling it. If you don’t want to waste time, you can just grab my profile, it seems to work fine. Adapt it to your environment if you are using non-standard paths. Look at the section “Copy the profile” for how to install it. For the brave, let’s go!
Install the tools
In order to do anything with AppArmor, we need to install the tools. On Ubuntu 18.04, I did:
apt install apparmor-utils
The apparmor-utils package provides the tools we need to generate a skeleton profile and parse the system logs.
Create a skeleton profile
AppArmor is fairly different from SELinux. Instead of attaching security tags to resources, you specify what a given binary can access, and how, in a text file. Also, processes can inherit permissions from their parent. We will only create a profile for the mysqld_safe script and it will cover the mysqld process and the SST scripts as they are executed under it. You create the skeleton profile like this:
root@BlogApparmor2:~# aa-autodep /usr/bin/mysqld_safe
Writing updated profile for /usr/bin/mysqld_safe.
On Ubuntu 18.04, there seems to be a bug. I reported it and apparently I am not the only one with the issue. If you get a “KeyError” error with the above command, try:
root@BlogApparmor2:~# echo "#include <abstractions>" > /etc/apparmor.d/scripts
root@BlogApparmor2:~# aa-autodep /usr/bin/mysqld_safe
The aa-autodep command creates the profile “usr.bin.mysqld_safe” in the /etc/apparmor.d directory. The initial content is:
root@BlogApparmor2:~# cat /etc/apparmor.d/usr.bin.mysqld_safe
# Last Modified: Wed Jul 25 18:56:31 2018
#include <tunables/global>
/usr/bin/mysqld_safe flags=(complain) {
#include <abstractions/base>
#include <abstractions/bash>
/bin/dash ix,
/lib/x86_64-linux-gnu/ld-*.so mr,
/usr/bin/mysqld_safe r,
}
I suggest you add, ahead of time, things you know are needed. In my case, I added:
/etc/mysql/** r,
/usr/bin/innobackupex mrix,
/usr/bin/wsrep_sst_xtrabackup-v2 mrix,
/usr/lib/galera3/* r,
/usr/lib/mysql/plugin/* r,
/usr/sbin/mysqld mrix,
/var/log/mysqld.log w,
owner /tmp/** rw,
owner /var/lib/mysql/** rwk,
This will save time on redundant questions later. Those entries are permissions granted to mysqld_safe. For example,
/etc/mysql** r
allows to read everything in
/etc/mysql
and its subdirectories. These lines need to go right after the
/usr/bin/mysqld_safe r,
line. Once done, parse and load the profile with:
root@BlogApparmor2:~# apparmor_parser -r /etc/apparmor.d/usr.bin.mysqld_safe
Get a well behaved SST script
If you read my previous blog post on SELinux, you may recall the
wsrep_sst_xtrabackup-v2
script does not behave well, security wise. The Percona developers have released a fixed version but it may not be available yet in a packaged form. In the meantime, you can download it from github.
Start iterating
My initial thought was to put the profile in complain mode, generate activity and parse the logs with aa-logprof to get entries to add to the profile. Likely there is something I am doing wrong but in complain mode, aa-logprof detects nothing. In order to get something I had to enforce the profile with:
root@BlogApparmor2:~# aa-enforce /etc/apparmor.d/usr.bin.mysqld_safe
Then, I iterated many times—like more than 20—over the following sequence:
- rm -rf /var/lib/mysql/* # optional
- systemctl start mysql &
- tail -f /var/log/mysqld.log /var/log/kern.log
- systemctl stop mysql
- ps fax | egrep ‘mysqld_safe|mysqld’ | grep -v grep | awk ‘{print $1}’ | xargs kill -9 # sometimes
- aa-logprof
- if something was not right, jump back to step 1
See the next section for how to run aa-logprof. Once that sequence worked well, I tried SST (joiner/donor) roles and IST.
Parse the logs with aa-logprof
Now, the interesting part begins, parsing the logs. Simply begin the process with:
root@BlogApparmor2:~# aa-logprof
and answer the questions. Be careful, I made many mistakes before I got it right, remember I am more a DBA than a Sysadmin. For example, you’ll get questions like:
Profile: /usr/sbin/mysqld
Path: /etc/hosts.allow
New Mode: r
Severity: unknown
[1 - #include <abstractions/lxc/container-base>]
2 - #include <abstractions/lxc/start-container>
3 - /etc/hosts.allow r,
(A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / Abo(r)t / (F)inish
AppArmor asks you how it should provide read access to the
/etc/hosts.allow
file. If you answer right away with “A”, it will add
#include <abstractions/lxc/container-base>
to the profile. With all the dependencies pulled by the lxc-related includes, you basically end up allowing nearly everything. You must first press “3” to get:
Profile: /usr/sbin/mysqld
Path: /etc/hosts.allow
New Mode: r
Severity: unknown
1 - #include <abstractions/lxc/container-base>
2 - #include <abstractions/lxc/start-container>
[3 - /etc/hosts.allow r,]
(A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / Abo(r)t / (F)inish
Notice the “[ ]” have moved to the bottom entry and then, press “A”. You’ll also get questions like:
Profile: /usr/bin/mysqld_safe
Execute: /bin/sed
Severity: unknown
(I)nherit / (C)hild / (N)amed / (X) ix On / (D)eny / Abo(r)t / (F)inish
For such a question, my answer is “I” for inherit. After a while, you’ll get through all the questions and you’ll be asked to save the profile:
The following local profiles were changed. Would you like to save them?
[1 - /usr/bin/mysqld_safe]
(S)ave Changes / Save Selec(t)ed Profile / [(V)iew Changes] / View Changes b/w (C)lean profiles / Abo(r)t
Writing updated profile for /usr/bin/mysqld_safe.
Revise the profile
Do not hesitate to edit the profile if you see, for example, many similar file entries which could be replaced by a “*” or “**”. If you manually modify the profile, you need to parse it to load your changes:
root@BlogApparmor2:~# apparmor_parser -r /etc/apparmor.d/usr.bin.mysqld_safe
Copy the profile
Once you have a server running with AppArmor enforced on PXC, simply copy the profile to the other servers and parse it. For example:
root@BlogApparmor3:~# cd /etc/apparmor.d
root@BlogApparmor3:/etc/apparmor.d# scp ubuntu@10.0.4.76:/etc/apparmor.d/usr.bin.mysqld_safe .
ubuntu@10.0.4.76's password:
usr.bin.mysqld_safe 100% 2285 3.0MB/s 00:00
root@BlogApparmor3:/etc/apparmor.d# aa-enforce usr.bin.mysqld_safe
Setting /etc/apparmor.d/usr.bin.mysqld_safe to enforce mode.
You can always verify if the profile is enforced with:
root@BlogApparmor3:/etc/apparmor.d# aa-status
apparmor module is loaded.
42 profiles are loaded.
20 profiles are in enforce mode.
/sbin/dhclient
...
/usr/bin/mysqld_safe
...
man_groff
Once enforced, I strongly advise to monitor the log files on a regular basis to see if anything has been overlooked. Similarly if you encounter a strange and unexpected behavior with PXC. Have the habit of checking the logs, it might save a lot of frustrating work.
Conclusion
As we have just seen, enabling AppArmor with PXC is not a difficult task, it just requires some patience. AppArmor is an essential component of a layered security approach. It achieves similar goals as the other well known DAC framework, SELinux. With the rising security concerns and the storage of sensitive data in databases, there are compelling reasons to enforce a DAC framework. I hope these two posts will help DBAs and Sysadmins to configure and enable DAC for PXC.
The post Lock Down: Enforcing AppArmor with Percona XtraDB Cluster appeared first on Percona Database Performance Blog.
To resolve a critical regression, Percona announces the release of Percona XtraDB Cluster 5.7.23-31.31.2 on October 2, 2018 Binaries are available from the 
Join Percona Chief Evangelist Colin Charles as he covers happenings, gives pointers and provides musings on the open source database community.
It’s 2018. Maybe now is the time to start migrating your network to