If you have set up per-process metrics in Percona Monitoring and Management, you may have found yourself in need of tuning it further to not only group processes together, but to display some of them in isolation. In this blogpost we will explore how to modify the rules for grouping processes, so that you can make the most out of this awesome PMM integration.
Let’s say you have followed the link above on how to set up the per-process metrics integration on PMM, and you have imported the dashboard to show these metrics. You will see something like the following:
This is an internal testing server we use, in which you can see a high number of VBoxHeadless (29) and mysqld (99) processes running. All the metrics in the dashboard will be grouped by the name of the command used. But, what if we want to see metrics for only one of these processes in isolation? As things stand, we will not be able to do so. It may not make sense to do so in a testing environment, but if you are running multiple mysqld processes (or mongos, postgres, etc) bound to different ports, you may want to see metrics for each of them separately.
Modifying the configuration file
Enter all.yaml!
In the process-exporter documentation on using a configuration file, we can see the following:
The general format of the -config.path YAML file is a top-level process_names section, containing a list of name matchers. […] A process may only belong to one group: even if multiple items would match, the first one listed in the file wins.
This means that even if we have two rules that would match a process, only the first one will be taken into account. This will allow us to both list processes by themselves, and not miss any non-grouped process. How? Let’s imagine we have the following processes running:
mysqld --port=1 mysqld --port=2 mysqld --port=3 mysqld --port=4
And we wanted to be able to tell apart the instances running in ports 1 and 2 from the other ones, we could use the following rules:
- name: "mysqld_port_1"
cmdline:
- '.*mysqld.*port=1.*'
- name: "mysqld_port_2"
cmdline:
- '.*mysqld.*port=2.*'
- name: "{{.Comm}}"
cmdline:
- '.+'
In cmdline we will need the regular expression against which to match the process command running. In this case, we made use of the fact that they were using different ports, but any difference in the command strings can be used. The last rule is the one that will default to “anything else” (with the regular expression that matches anything).
The default rule at the end will make sure you don’t miss any other process, so unless you want only some processes metrics collected, you should always have a rule for it.
A real life working example of configuring per-process metrics
In case all these generic information didn’t make much sense, we will present a concrete example, hoping that it will make everything fit together nicely.
In this example we want to have the mysqld instance using the mysql_sandbox16679.sock socket isolated from all the others, and the VM with ID finishing in 97eafa2795da listed by their own. All other processes are to be grouped together by using the basename of the executable.
You can check the output from ps aux to see the full command used. For instance:
shell> ps aux | grep 97eafa2795da agustin+ 27785 0.7 0.2 5619280 542536 ? Sl Nov28 228:24 /usr/lib/virtualbox/VBoxHeadless --comment centos_node1_1543443575974_22181 --startvm a0151e29-35dd-4c14-8e37-97eafa2795da --vrde config
So, we can use the following regular expression for it (we use .* to match any string):
.*VBoxHeadless.*97eafa2795da.*
The same applies to the regular expression for the mysqld process.
The configuration file will end up looking like:
shell> cat /etc/process-exporter/all.yaml
process_names:
- name: "Custom VBox"
cmdline:
- '.*VBoxHeadless.*97eafa2795da.*'
- name: "Custom MySQL"
cmdline:
- '.*mysqld.*mysql_sandbox16679.sock.*'
- name: "{{.Comm}}"
cmdline:
- '.+'
Let’s restart the service, so that new changes apply, and we will check the graphs after five minutes, to see new changes. Note that you may have to reload the page for the changes to apply.
shell> systemctl restart process-exporter
After refreshing, we will see the new list of processes in the drop-down list:
And after we select them, we will be able to see data for those processes in particular:
Thanks to the default configuration at the end, we are still capturing data from all the other mysqld processes. However, they will have their own group, as mentioned before:
Do you need to modify the metrics collected from Linux by