Percona Monitoring and Management 2.13 (PMM) introduced the Integrated Alerting feature as a technical preview. It adds a user-friendly way to set up and manage alerts for your databases. You can read more about this feature usage in our announcement blog post and in our documentation, while in this article we will be focusing on design and implementation details.
Entities
There are four basic entities used for IA: Alert Rule Template, Alert Rule, Alert, and Notification Channel.
Everything starts from the alert rule template. You can see its YAML representation below:
---
templates:
- name: pmm_mongodb_high_memory_usage
version: 1
summary: Memory used by MongoDB
expr: |-
sum by (node_name) (mongodb_ss_mem_resident * 1024 * 1024)
/ on (node_name) (node_memory_MemTotal_bytes)
* 100
> [[ .threshold ]]
params:
- name: threshold
summary: A percentage from configured maximum
unit: "%"
type: float
range: [0, 100]
value: 80
for: 5m
severity: warning
labels:
cultom_label: demo
annotations:
summary: MongoDB high memory usage ({{ $labels.service_name }})
description: |-
{{ $value }}% of memory (more than [[ .threshold ]]%) is used
by {{ $labels.service_name }} on {{ $labels.node_name }}.
A template serves as the base for alert rules. It defines several fields, let’s look at them:
- name: uniquely identifies template (required)
- version: defines template format version (required)
- summary: a template description (required)
- expr: a MetricsQL query string with parameter placeholders. MetricsQL is backward compatible with PromQL and provides some additional features. (required)
- params: contains parameter definitions required for the query. Each parameter has a name, type, and summary. It also may have a unit, available range, and default value.
- for: specifies the duration of time the expression must be met for; The alert query should return true for this period of time at which point the alert will be fired (required)
- severity: specifies default alert severity level (required)
- labels: are additional labels to be added to generated alerts (optional)
- annotations: are additional annotations to be added to generated alerts. (optional)
A template is designed to be re-used as the basis for multiple alert rules so from a single pmm_node_high_cpu_load template you can have alerts for production vs non-production, warning vs critical, etc.
Register for Percona Live ONLINE
A Virtual Event about Open Source Databases
Users can create alert rules from templates. An alert rule is what’s actually executed against metrics and what produces an alert. The rule can override default values specified in the template, add filters to apply the rule to only required services/nodes/etc, and specify target notification channels, such as email, Slack, PagerDuty, or Webhooks. If the rule hasn’t any associated notification channels its alerts will be available only via PMM UI. It’s useful to note that after creation rule keeps its relation with the template and any change in the template will affect all related rules.
Here is an alert rule example:
---
groups:
- name: PMM Integrated Alerting
rules:
- alert: /rule_id/c8e5c559-ffba-43ed-847b-921f69c031a9
rule: test
expr: |-
sum by (node_name) (mongodb_ss_mem_resident * 1024 * 1024)
/ on (node_name) (node_memory_MemTotal_bytes)
* 100
> 40
for: 5s
labels:
ia: "1"
rule_id: /rule_id/c8e5c559-ffba-43ed-847b-921f69c031a9
severity: error
template_name: pmm_mongodb_high_memory_usage
cultom_label: demo
annotations:
description: |-
{ { $value } }% of memory (more than 40%) is used
by {{ $labels.service_name }} on {{ $labels.node_name }}.
summary: MongoDB high memory usage ({{ $labels.service_name }})
It has a Prometheus alert rule format.
How it Works
Integrated Alerting feature built on top of Prometheus Alertmanager, VictoriaMetrics TimescaleDB (TSDB), and VMAlert.
VictoriaMetrics TSDB is the main metrics storage in PMM, VMalert responsible for alert rules execution, and Prometheus Alertmanager responsible for alerts delivery. VMAlert runs queries on VM TSDB, checks if they are positive for the specified amount of time (example: MySQL is down for 5 minutes), and triggers alerts. All alerts forwarded to the PMM internal Alertmanager but also can be duplicated to some external Alertmanager (it can be set up on the PMM Settings page).
There are four available templates sources:
- Built-in templates, shipped with PMM distribution. They are embedded into the managed binary (core component on PMM).
- Percona servers. It’s not available yet, but it will be similar to the STT checks delivery mechanism (HTTPS + files signatures).
- Templates created by the user via PMM UI. We persist them in PMM’s database.
- Templates created by the user as files in the
/srv/ia/templatesdirectory.
During PMM startup, managed loads templates from all sources into the memory.
Alert rules can be created via PMM UI or just by putting rule files in the /srv/prometheus/rules directory. Alert rules created via UI persist in PMM’s internal PostgreSQL database. For each alert rule from DB, managed binary creates a YAML file in /etc/ia/rules/ and asks VMalert to reload the configuration and reread rule files. VMAlert executes query from each loaded alert rule every minute, once the rule condition is met (query is positive for the specified amount of time) VMAlert produces an alert and passes it to the Alertmanager. Please note that /etc/ia/rules/ controlled by managed and any manual changes in that directory will be lost.
Managed generates configuration for Alertmanager and updates it once any related entity changes.
Managed goes through the list of the existing rules and collects unique notification channel combinations. For example, if we have two rules and each of them has assigned channels a,b, and c it will be the one unique channel combination. For each rule managed generates a route and for each unique channel combination, it generates a receiver in the Alertmanager configuration file. Each route has a target receiver and filter by rule id, also it can contain user-defined filters. If a rule hasn’t assigned notification channels, then a special empty receiver will be used. Users can redefine an empty receiver with Alertmanagers base configuration file /srv/alertmanager/alertmanager.base.yml. When some Notification Channel is disabled, managed recollects unique channel combinations excluding disabled channels and regenerates receivers and routing rules. If the rule has only one specified channel and it was disabled then a special disabled receiver will be used for that. Unlike empty receiver, disabled can’t be redefined by the user and always means “do nothing”. It prevents unexpected behavior after channels disabling. After each Alertmanager configuration update, managed asks Alermanager to reload it.
When Alertmanager receives an alert from VMAlert, it uses routes to find an appropriate receiver and forward alerts to destination channels. The user also can observe alerts via PMM UI. In that case, managed gets all available alerts from Alertmanager API and applies required filters before showing them.
Conclusion
The Integrated Alerting feature has many moving parts, and functionally it’s more about managing configuration for different components and making them work together. It provides a really nice way to be aware of important events in your system. While it’s still in tech preview state, it’s already helpful. With built-in templates, it’s easy to try without diving into documentation about Prometheus queries and other stuff. So please try it and tell us about your experience. What parameters of a system you would like to have covered with templates? What use cases do you have for alerting? We will happy to any feedback.
The 
This week we release 
I try to use Open Source when a good solution exists, so for my home firewall, I’m using 
Optimizing MySQL performance and troubleshooting MySQL problems are two of the most critical and challenging tasks for MySQL DBAs. The databases powering applications need to be able to handle changing traffic workloads while remaining responsive and stable in order to deliver an excellent user experience. Further, DBAs are also expected to find cost-efficient means of solving these issues.
In this blog post, I’ll look at how to import custom dashboards into 
It means that for dashboard importing, you have to specify the data source names assigned by your installation.
In November of 2020, we announced that in early 2021 Percona was slated to release a version of 
More and more companies are adopting Kubernetes, but after some time they see an unexpected growth around cloud costs. Engineering teams did their part in setting up auto-scalers, but the cloud bill is still growing. Today we are going to see how 
One piece of feedback I often hear from users of