Say Hello to Libcoredumper – A New Way to Generate Core Dumps, and Other Improvements


LibcoredumperIn a perfect world, we expect all software to run flawlessly and never have problems such as bugs and crashes. We also know that this perfect world doesn’t exist and we better be as prepared as possible to troubleshoot those types of situations. Historically, generating core dumps has been a task delegated to the kernel. If you are curious about how to enable it via Linux kernel, you can check out Getting MySQL Core file on Linux. There are a few drawbacks that pose either a limitation or a huge strain to get it working, such as:

  • System-wide configuration required. This is not something DBA always has access to.
  • Inability or very difficult to enable it for a specific binary only. Standards ways enable it for every software running on the box.
  • Nowadays, with cloud and containers, this task has become even more difficult because it sometimes requires containers to be running on privileged mode and host OS to be properly configured by the provider.

The above issues have driven exploration of alternative ways to do create a core dump to help troubleshooting bugs and crashes. More details can be found at PS-7114 .

The Libcoredumper

The libcoredumper is a fork of the archived project google-coredumper. Percona has forked it under Percona-Lab Coredumper, cherry-picked a few improvements from other forks of the same project, and enhanced it to compile and work on newer versions of Linux as well on newer versions of GCC and CLANG.

This project is a Tech Preview, as you may infer from the repository name (Percona Lab). We might not maintain compatibility with future kernel versions and/or patches. One should test the core dumper on their environment before putting this tool into production. We have tested on kernel versions up to 5.4.

This functionality is present on all versions of Percona Server for MySQL and Percona XtraDB Cluster starting from 5.7.31 and 8.0.21. If you compile PS/PXC from source, you can control if the library will be compiled by switching -DWITHCOREDUMPER to ON/OFF (default is ON).

How To Configure It

A new variable named coredumper has been introduced. One should include it under the [mysqld] section of my.cnf and it works independently of the older configuration core-file. This new variable can either be a boolean (no value specified) or with value. It follows a few rules:

  • No value – core dump will have saved under MySQL datadir and will be named core.
  • A path ending with  /  – core dump will be saved under the specified directory and will be named core.
  • A full path with filename  – core dump will be saved under the specified directory and will use the specified name.

Every core file will end with the timestamp of the crash instead of PID, for two main reasons:

  • Make it easier to correlate a core dump with a crash, as MySQL always print a Zulu/UTC timestamp on the logs when it crashes:
    10:02:09 UTC - mysqld got signal 11 ;
  • Operators / Containers will always be running MySQL (or whatever application it is running) as PID 1. If MySQL has crashed multiple times, we don’t want to core-dump to get overwritten by the last crash.

How To Know If I Am Using libcoredumper

When MySQL attempts to write a core file it stamps the log saying it will write a core file. When it does it delegating the action to Linux kernel, you always see a message like below:

. . .
Writing a core file

The above behavior remains the same, however, when MySQL is using libcoredumper to generate the core file, one should see that message informing that the library will be responsible for the action:

. . .
Writing a core file using lib coredumper

Other Improvements

Apart from libcoredumper, starting from the same 5.7 and 8.0 releases a stack trace will also:

  • Print binary BuildID – This information is very useful for support/development people in case the MySQL binary that crashed is a stripped binary. Stripped binaries are a technique to remove part of the binaries that are not essential for it to run, making the binary occupy less space in disk and in memory. When computers had a restriction on memory, this technique was widely used. Nowadays this doesn’t pose a limitation anymore on most of the hardware, however, it is becoming popular once again with containers where image size matters. Stripping the binary removed the binary symbols table, which is required to resolve a stack trace and lets you read the core dump. BuildID is how we can link things together again.
  • Print the server Version – This information is also useful to have at glance. Recent versions of MySQL/Percona Server for MySQL have a fix for many know issues. Having this information helps to establish the starting point investigation. MySQL only prints the server version when it starts, and by the moment a server crashes, its log may have grown significantly or even got rotated/truncated.

Here is one example of how a crash with stack trace will look like:

14:23:52 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.

Build ID: 55b4b87f230554777d28c6715557ee9538d80115
Server Version: 8.0.21-12-debug Source distribution

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
/usr/local/mysql/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x55) [0x55943894c280]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x2e0) [0x559437790768]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13f40) [0x7f9d413bcf40]
/lib/x86_64-linux-gnu/libc.so.6(__poll+0x49) [0x7f9d40858729]
/usr/local/mysql/bin/mysqld(Mysqld_socket_listener::listen_for_connection_event()+0x64) [0x55943777db6a]
/usr/local/mysql/bin/mysqld(Connection_acceptor<Mysqld_socket_listener>::connection_event_loop()+0x30) [0x55943737266e]
/usr/local/mysql/bin/mysqld(mysqld_main(int, char**)+0x30c6) [0x559437365de1]
/usr/local/mysql/bin/mysqld(main+0x20) [0x559437114005]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f9d4076db6b]
/usr/local/mysql/bin/mysqld(_start+0x2a) [0x559437113f2a]
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file using lib coredumper


Libcoredumper serves as an alternative for current –core-file functionality for generating memory dumps. In case of any crash of MySQL, a core dump is written and can be later processed /read via GDB to understand the circumstances of such a crash.
Users can enable it by adding the below variable to [mysqld] section of my.cnf:


Percona Server for MySQL versions starting from 5.7.31 and 8.0.31 include the library by default. Refer to below documentation pages for more details:




If you faced any issue or limitation on enabling core dumps before feel free to test new versions of Percona Server for MySQL/Percona XtraDB Cluster and use libcoredumper. Also, any feedback is very welcome on how we can improve the troubleshooting of bugs/crashes even further.


Community Contributors Invited to Update Percona Monitoring and Management Documentation

Percona Monitoring and Management Documentation

Percona Monitoring and Management DocumentationPercona is making changes in its documentation processes for Percona Monitoring and Management (PMM) to make it easier and more welcoming for community members to contribute to our documentation.

Percona Monitoring and Management (PMM) is a free, open-source database monitoring tool. It ‘mind-melds’ with your MySQL, MongoDB, or PostgreSQL servers and tells you what’s going on inside; what queries are draining resources, and why.

PMM’s technical documentation is good. But, with your help, we want to make it better. Here’s what we’ve got planned for it.

  • a move from reStructuredText to Markdown, making it easier for anyone to contribute;
  • a complete reorganization of sections, to make them more logical and consistent;
  • more ways to navigate the documentation (search, content maps, topic directories);
  • more examples and screenshots better suited to your use-cases;
  • regular ‘test-drives’ of our documentation, to make sure it’s technically correct and up-to-date;
  • more visual elements, to help you understand and navigate the content;
  • a (gradual) rewording and reformatting of everything, to make it easier (and, who knows? more fun) to read.

Why Change?

The PMM team writes great software, but we don’t use it the way you do.

We want PMM’s documentation to be as open as its code, and to encourage contributions to improve the quality and usability of the information. In effect, we want all our users to be PMM technical writers. (It’s one of the motivations for moving to Markdown: to encourage more community contributions.)

To see for yourselves, take a look at our PMM Documentation GitHub repository, or click the Edit this page link anywhere in the PMM2 documentation site to jump to the Markdown source file for that page. (But you can still use Jira to suggest changes or improvements.)

What Will Change?

The PMM technical documentation website is static HTML created by Sphinx and reStructuredText. These are great tools for technical writers, less so for developers and users.

We converted our documentation to Markdown because it’s a cleaner syntax, it’s easier to learn, and the files look good when seen in GitHub. There’s also a wider variety of tools for converting Markdown into HTML.

Behind the scenes, we’ll switch to using MkDocs to convert the Markdown files into HTML. At first, the new site will have the same look and feel, the same section names, and the same page URLs.

Rest assured, the content stays the same! (At least for now.)

There will be some subtle changes to the navigation bars—MkDocs treats sections differently—but only the most loyal, avid and eagle-eyed readers should notice.

We’ll try to preserve your favorite bookmarks, and help you reorient yourself to the new section layout when it comes. But some links will change—the web wasn’t written in stone.


The documentation for PMM 2 evolved from that for PMM 1. Its structure (the sections and their order) has grown organically over time, not always in the most logical way.

What’s we have now is not the best way to explain PMM 2 and its potential, nor will it accommodate the new exciting product features coming down the line.

We’re working on changes. (We’d love your help.)


Percona Distribution for PostgreSQL 13, Updates to Percona Distribution for MySQL: Release Roundup October 26, 2020

Percona Software Release Oct 26

Percona Software Release Oct 26It’s release roundup time here at Percona!

Our Release Roundups showcase the latest software updates, tools, and features to help you manage and deploy our software, with highlights and critical information, as well as links to the full release notes and direct links to the software or service itself.

Today’s post includes those releases and updates that have come out since October 13, 2020, including the release of Percona Distribution for PostgreSQL 13.0 with pg_stat_monitor, updates to both Percona Server for MySQL and Percona XtraDB Cluster variants of Percona Distribution for MySQL, a bug fix for Percona XtraDB Cluster, and new features for Percona Backup for MongoDB.

Percona Distribution for PostgreSQL 13.0

On October 16, 2020, we released Percona Distribution for PostgreSQL 13.0, a collection of tools to assist you in managing PostgreSQL. This release is based on the latest major version of PostgreSQL 13.0 and also includes pg_stat_monitor (Tech Preview Feature [1]), a new statistics collection extension for PostgreSQL.

Download Percona Distribution for PostgreSQL 13.0


Percona Distribution for MySQL (PS and PXC Variants)

We have updated both Percona Distribution for MySQL (PS variant) 8.0.21 and Percona Distribution for MySQL (PXC variant) 8.0.20. Percona Distribution for MySQL is a single solution with the best and most critical enterprise components from the MySQL open source community, designed and tested to work together.  It provides two deployment variants: one is based on Percona Server for MySQL and another one is based on Percona XtraDB Cluster. This enables you to choose the MySQL deployment approach that best meets your specific needs.

Download Percona Distribution for MySQL (PS variant) 8.0.21

Download Percona Distribution for MySQL (PXC variant) 8.0.20


Percona Server for MySQL 8.0.21-12

On October 13, 2020, Percona Server for MySQL 8.0.21-12 was released. It includes all the features available in MySQL 8.0.21 Community Edition in addition to enterprise-grade features developed by Percona. As well as bug fixes as listed in the release notes, this version offers several improvements, including making the default value of rocksdb_wal_recovery_mode compatible with InnoDB, blocking enable/disable redo log with lock tables for backup, and the introduction of crypt_schema 2 for better error checking in encryption threads.

Download Percona Server for MySQL 8.0.21-12


Percona XtraDB Cluster 5.6.49-28.42.3

Percona XtraDB Cluster 5.6.49-28.42.3 was released on October 22, 2020. It is a free, open source, enterprise-grade solution that includes the high availability and security features your business needs to meet customer expectations and business goals. Bug fix PXC-3456 allows specific characters in SST method names and SST request data.

Download Percona XtraDB Cluster 5.6.49-28.42.3


Percona XtraDB Cluster 5.7.31-31-45.3

On October 22, 2020, Percona XtraDB Cluster 5.7.31-31-45.3 was released, which fixed a bug that allows for specific characters in SST method names and SST request data.

Download Percona XtraDB Cluster 5.7.31-31-45.3


Percona XtraDB Cluster 8.0.20-11.3

Percona XtraDB Cluster 8.0.20-11.3 was released on October 22, 2020. This release fixes a bug to allow specific characters in SST method names and SST request data.

Download Percona XtraDB Cluster 8.0.20-11.3


Percona Server for MongoDB 3.6.20-9.0

Percona Server for MongoDB 3.6.20-9.0 was released on October 22, 2020. It is an enhanced, open source, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB 3.6.20 Community Edition, supporting MongoDB 3.6.20 protocols and drivers. In this release, we improved audit log performance and fixed several bugs, including when createBackup using AWS remote location fails with “EntityTooLarge” and LDAP authentication randomly fails with the “Bad parameter to an ldap routine” message in the log.

Download Percona Server for MongoDB 3.6.20-9.0


Percona Backup for MongoDB 1.3.2

October 14, 202 saw the release of Percona Backup for MongoDB 1.3.2. It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. A new feature is the addition of AWS KMS key encryption/decryption for S3 buckets and improvements include the use of s2 compression as default for pbm-speed-test instead of gzip. There are also several bug fixes.

Download Percona Backup for MongoDB 1.3.2


Percona Monitoring and Management 2.11.0

On October 14, 2020, we released Percona Monitoring and Management 2.11.0, a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance. This release introduces a technical preview of a new PostgreSQL extension ‘pg_stat_monitor’, as well as an update to Prometheus to v2.21.0 and Grafana plugin updates grafana-polystat-panel=1.2.2, grafana-piechart-panel=1.6.1.

Download Percona Monitoring and Management 2.11.0


Percona Monitoring and Management 2.11.1

October 19, 2020, saw the release of Percona Monitoring and Management 2.11.1. This release fixes a bug that contributed to high CPU usage after update to 2.11.0.

Download Percona Monitoring and Management 2.11.1


That’s it for this roundup, and be sure to follow us on Twitter to stay up-to-date on the most recent releases! Percona is a leader in providing best-of-breed enterprise-class support, consulting, managed services, training, and software for MySQL, MongoDB, PostgreSQL, MariaDB, and other open source databases in on-premises and cloud environments.

We understand that choosing open source software for your business can be a potential minefield. You need to select the best available options, which fully support and adapt to your changing needs. Choosing the right open source software can allow you access to enterprise-level features, without the associated costs.

In our white paper, we discuss the key features that make open source software attractive, and why Percona’s software might be the best option for your business.

Download: When is Percona Software the Right Choice?


MySQL New Releases and Percona XtraBackup Incompatibilities

MySQL Percona Backup Incompatibilities

MySQL Percona Backup IncompatibilitiesEarlier this week, Oracle released their Q4 releases series. As on the previous releases, backward compatibility has been broken with previous versions of the server. This time on both MySQL 5.7 and 8.0:

MySQL 5.7.32

While our QA team was performing an extensive test on it,  we found out this version introduced a new compression format version. This change breaks backward compatibility with older versions of MySQL, which is expected on the 8.0 series but is not on 5.7. As Percona XtraBackup (PXB) is based on MySQL code, it makes MySQL 5.7.32 incompatible with current versions of Percona XtraBackup 2.4.20 and prior.

The issue does not affect only Percona XtraBackup but also prevents users from downgrading the server from 5.7.32 to any lower version on the 5.7 series – More details at https://bugs.mysql.com/bug.php?id=101266.

In summary, if you have tables with compression flag as below:


The issue will manifest if a user using 5.7.32:

  • Creates a new compressed table.
  • Runs any ALTER TABLE  that uses the algorithm copy (table rebuild) on a compressed table.

At this moment, we advise users using compressed tables to hold the upgrade to 5.7.32.

We are currently working on making Percona XtraBackup 2.4.21 fully compatible with 5.7.32.

MySQL 8.0.22

Percona XtraBackup 8.0.14 (the latest version available) is not compatible with MySQL 8.0.22 due to disk format changes introduced in the 8.0.22 release.

WL#13782: InnoDB: Add dynamic config option to use fallocate() on Linux introduced a new redo log record MLOG_FILE_EXTEND which is written on the file extension and doesn’t depend on –innodb-extend-and-initialize option. Unfortunately this time, the redo log format version is not bumped up. Percona XtraBackup 8.0.14 during backup, cannot parse this new redo log record and so backup fails.

If by chance, MLOG_FILE_EXTEND is checkpointed, PXB during backup doesn’t see this new record. This leads to a misleading successful backup that cannot be prepared. Let’s see why.

This bug fix in 8.0.22, increased the DD version to 8022. PXB during prepare, de-serializes the SDI from IBD file to bootstrap dictionary. Due to the higher DD_VERSION in SDI, PXB 8.0.14 cannot deserialize the SDI and prepare fails.

At this moment, we advise all users to hold the upgrade to 8.0.22.

We are working on these incompatible issues, look out for an upcoming release of PXB release to take successful, consistent backups of 8.0.22


CVE-2020-26542: SimpleLDAP Authentication in Percona Server for MySQL, Percona Server for MongoDB



When using the SimpleLDAP authentication in conjunction with Microsoft’s Active Directory, Percona has discovered a flaw that would allow authentication to complete when passing a blank value for the account password, leading to access against the service integrated with which Active Directory is deployed at the level granted to the authenticating account.


Percona Server for MySQL

Percona Server for Mysql 8.x. < 8.0.21

Percona XtraDB Cluster

Percona XtraDB Cluster 8.x. <

Percona Server for MongoDB

Only the exact minor versions listed here are affected: 3.6.19-7.0, 4.0.18-11, 4.0.19-12, 4.0.20-13, 4.2.5-5, 4.2.6-6, 4.2.7-7, 4.2.8-8, 4.2.9-9, 4.4.0-1, 4.4.1-2

More Information



Release Notes


Percona Distribution for MySQL (PXC Variant) 8.0.20, Fixes For Security Vulnerability: Release Roundup October 13, 2020



Using Volume Snapshot/Clone in Kubernetes

Volume snapshot and clone Kubernetes

Volume snapshot and clone KubernetesOne of the most exciting storage-related features in Kubernetes is Volume snapshot and clone. It allows you to take a snapshot of data volume and later to clone into a new volume, which opens a variety of possibilities like instant backups or testing upgrades. This feature also brings Kubernetes deployments close to cloud providers, which allow you to get volume snapshots with one click.

Word of caution: for the database, it still might be required to apply fsfreeze and FLUSH TABLES WITH READ LOCK or



It is much easier in MySQL 8 now, because as with atomic DDL, MySQL 8 should provide crash-safe consistent snapshots without additional locking.

Let’s review how we can use this feature with Google Cloud Kubernetes Engine and Percona Kubernetes Operator for XtraDB Cluster.

First, the snapshot feature is still beta, so it is not available by default. You need to use GKE version 1.14 or later and you need to have the following enabled in your GKE: https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver#enabling_on_a_new_cluster.

It is done by enabling “Compute Engine persistent disk CSI Driver“.

Now we need to create a Cluster using storageClassName: standard-rwo for PersistentVolumeClaims. So the relevant part in the resource definition looks like this:

        storageClassName: standard-rwo
        accessModes: [ "ReadWriteOnce" ]
            storage: 11Gi

Let’s assume we have cluster1 running:

NAME                                               READY   STATUS    RESTARTS   AGE
cluster1-haproxy-0                                 2/2     Running   0          49m
cluster1-haproxy-1                                 2/2     Running   0          48m
cluster1-haproxy-2                                 2/2     Running   0          48m
cluster1-pxc-0                                     1/1     Running   0          50m
cluster1-pxc-1                                     1/1     Running   0          48m
cluster1-pxc-2                                     1/1     Running   0          47m
percona-xtradb-cluster-operator-79d786dcfb-btkw2   1/1     Running   0          5h34m

And we want to clone a cluster into a new cluster, provisioning with the same dataset. Of course, it can be done using backup into a new volume, but snapshot and clone allow for achieving this much easier. There are still some additional required steps, I will list them as a Cheat Sheet.

1. Create VolumeSnapshotClass (I am not sure why this one is not present by default)

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
        name: onesc
driver: pd.csi.storage.gke.io
deletionPolicy: Delete

2. Create snapshot

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
  name: snapshot-for-newcluster
  volumeSnapshotClassName: onesc
    persistentVolumeClaimName: datadir-cluster1-pxc-0

3. Clone into a new volume

Here I should note that we need to use the following as volume name convention used by Percona XtraDB Cluster Operator, it is:


Where CLUSTERNAME is the name used when we create clusters. So now we can clone snapshot into a volume:


Where newcluster is the name of the new cluster.

apiVersion: v1
kind: PersistentVolumeClaim
  name: datadir-newcluster-pxc-0
    name: snapshot-for-newcluster
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  storageClassName: standard-rwo
    - ReadWriteOnce
      storage: 11Gi

Important: the volume spec in storageClassName and accessModes and storage size should match the original volume.

After volume claim created, now we can start newcluster, however, there is still a caveat; we need to use:

forceUnsafeBootstrap: true

Because otherwise, Percona XtraDB Cluster will think the data from the snapshot was not after clean shutdown (which is true) and will refuse to start.

There is still some limitation to this approach, which you may find inconvenient: the volume can be cloned in only the same namespace, so it can’t be easily transferred from the PRODUCTION namespace into the QA namespace.

Though it still can be done but will require some extra steps and admin Kubernetes privileges, I will show how in the following blog posts.


Annotations Provide Context to Timelines in Percona Monitoring and Management

Annotations Percona Monitoring and Management

About the AuthorThis blog was written as a collaboration with my colleague Jiri Ctvrtka.  Jiri is a senior software developer from Brno, Czech Republic, and has been partnering with Percona for almost a year working on various components of our Percona Platform. He’s been programming in Go since 2015 and Jiri’s got a passion for simplicity, speed, and precision of data and has focused that passion on understanding impacts of changes and reimagined the Percona Monitoring and Management (PMM) Annotations functionality.

“Does anyone remember what caused this large spike last Thursday at 2:24 am?”

Annotations Percona Monitoring and ManagementWhat are Annotations?

Annotations are a way to provide context to timelines in Percona Monitoring and Management (PMM). For example, in bigger teams, it is a good way to inform others about an event, or important changes that may have occurred. It can contain any kind of information but we see it most commonly used for indicating there was an outage, maintenance window start/end, deployment of new code, security event, etc.  Annotations in PMM help provide quick explanations to peaks and valleys in graphs or indicate when something took place on the timeline and create correlations.


An example of annotations automatically added at the beginning and end of a benchmark test.

Every annotation can be shown/hidden by a simple toggle button presented in the filter options. So you don’t need to be worried about crowded charts of annotations. You can toggle on/off and zoom in or out around events to get better detail.

Filter contains a toggle button to turn on/off the visibility of annotations.

How Can I Add Annotations?

Annotations can simply be added by the pmm-admin option: annotate. So, let’s try it:

pmm-admin annotate “Deployed web version 2.4.6“

That command will place an annotation on every chart in every node and service…maybe more than you need. What if you only needed to add an annotation for a specific node or service to indicate a broad network outage? Or add specific annotations for specific nodes or services to indicate a software update was applied? This is all possible as pmm-admin provides four flags, which can be used for just this purpose.

--node = annotate the node the pmm-admin command was run on
--node-name = annotate node with specified name
--service = annotate all services running on the node the pmm-admin command was run on
--service-name = annotate service with specified name

All these flags can be combined together for options to annotate more nodes or services by just only one command. The order of flags doesn’t matter. Just imagine how many combinations you have!  Even better, imagine how easily this can be integrated into our CI/CD or deploy pipelines!

You can also add annotations via the API with curl commands:

curl 'http://admin:admin@localhost/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"tags": ARRAY OF TAGS,"text":"TEXT"}'

ARRAY OF TAGS = Names of node or service, like for node pmm-server and service pmm-server-mysql it will be “tags: [“pmm-server”, “pmm-server-mysql”]”

Some Examples for Better Understanding

Case 1: We have a DB node we need to take offline for maintenance and want to capture this on all its graphs to explain the gap in reporting data.

pmm-admin annotate "`date` - System is going down for Maintenance - RFC-2232-2020" --node-name=”db1002”

via API:

curl 'http://admin:admin@localhost/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"tags": ["db1002"],"text":"`date` - System is going down for Maintenance - RFC-2232-2020"}'

or if pmm-admin is running on this node then you don’t even need to know the name of this node and can set it as part of the shutdown routine.

pmm-admin annotate "`date` - System is being shut down/rebooted"

Case 2: We have node pmm-server and three services running on the current node (mysql, postgres, mongo). So, yeah, it’s simple right?

pmm-admin annotate “`date` - Apply configuration change - RFC-1009-2020“ –service-name=mysql
pmm-admin annotate “Restarting Postgres to apply security patch - “ –service-name=postgres
pmm-admin annotate “`date` - Service Disruption Reported via Support - SUP-239“ –service-name=mongo

via API:

curl 'http://admin:admin@localhost/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"tags": ["mysql", "postgresl", "mongo"],"text":"`date` - Apply configuration change - RFC-1009-2020"}'

Or you can do it in one command:

pmm-admin annotate “Services Recycled to pickup config changes“ –service

And that’s it!  All services found running on that node will be annotated.

Case 3: We have node “registrationDb” and many services running on the current node. What if we want to annotate that node and also every service running on this node by just one command? Again, no problem:

pmm-admin annotate “`date` - Security alerted to possible event“ –node-name=registrationDb --service

via API:

curl 'http://admin:admin@localhost/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"tags": ["registrationDb", "service1", "service2",...*],"text":"`date` - Security alerted to possible event"}'

* while the PMM admin command –service flag will add to all services, you need to add all services names of the current node to get the same result using the API but you can make an API call to get all services on a given node

That’s it, no matter how many services you are running on node registrationDb the annotations will be presented on all of them and on node graphs as well.

Case 4: We have 100 services running on the current node and also another node named pmm-server2 and we need to annotate all 100 services on the current node (but not the current node) along with node pmm-server2.  Simple:

pmm-admin annotate “`date` - Load Test Start“ –node-name=pmm-server2 --service

via API:

curl 'http://admin:admin@localhost/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"tags": ["pmm-server2", "service1", "service2",...*],"text":"`date` - Load Test - Increasing to 100 threads"}'

* while the PMM admin command –service flag will add to all services, you need to add all services names of the current node to get the same result using the API but you can make an API call to get all services on a given node

The current node will not be annotated, but every service on this node will be along with node pmm-server2.

Here’s a little guide to see many of the possible combinations of flags and what will result:

–node = current node
–node-name = node with name
–node –node-name = current node and node with name
–node –service-name = current node and service with name
–node –node-name –service-name = current node, node with name, and service with name
–node –service = current node and all services of current node
–node-name –service = all services of current node, node with name
–node –node-name –service = all services of current node, node with name, and current node
–service = all services of current node
–service-name = service with name
–service –service-name = all services of current node, service with name
–service –node-name = all services of current node and node with name
–service-name –node-name = service with name and node with name
–service –service-name –node-name = service with name, all services on current node, and node with name

So thanks to annotations, correlating events on your servers is now easier than ever.  We’d love to hear or even see how you’re using annotations to make your life easier, hopefully, we’ve given you some ideas to get started right away!

Download Percona Monitoring and Management Today


Introducing the Percona Enterprise Platform

Percona Enterprise Platform

Percona Enterprise PlatformThere is an on-going shift within many organizations where the focus placed on application engineers is expanding while at the same time infrastructure teams within the DBA landscape are being trimmed back. The result is application engineers with little to no database experience and DBAs who are completely overwhelmed – both looking for resources to complete the work expected of them. As we know, helping consolidate this work and empowering those responsible is not only critical but often paramount in the success of an organization.

Introducing the Percona Enterprise Platform

Our vision is to provide a truly open source platform that won’t lock you in, with a single pane of glass to easily manage your open source database infrastructure, and a self-service experience enabling fast and consistent open source database deployment. 

Our goal is to deliver a platform which provides the enterprise benefits our customers are looking for, including:

  • A single interface to deploy and manage your open source databases on-prem, in the cloud, or across hybrid and multi-cloud environments.
  • The ability to configure a database once and deploy it anywhere.
  • Critical database management operations such as backup, recovery, and patching.
  • Enhanced automation and advisory services, which allow you to find, eliminate, and prevent outages, security issues, and slowdowns.
  • A viable alternative platform to public cloud and large enterprise database vendor DBaaS offerings, enabling you to eliminate vendor lock-in.

The Percona Enterprise Platform Beta Program is Kicking Off This Year!

The beta program will officially launch before the end of 2020 and we’re hoping you’ll join us as this is your opportunity to help shape the future of our open-source database platform offering. We know you’re curious as to what that entails, so read on to learn more.

The beta program 101:
  • This beta is a year-long program with four phases
  • Each three-month phase will involve a different area of emphasis in terms of participant feedback
  • Each phase requires around 10 hours of self-paced activities during the three month period. Participants can commit to as many phases as they like.
  • Our initial focus will be on MySQL and MongoDB, with PostgreSQL added later in the program.
  • Beta participant activities will be facilitated through our Percona Forum, so go ahead and register today if you haven’t already!

Register now in order to hear more about the beta program as we get closer to kick-off later this year!

Register Now


Announcing Percona Distribution for PostgreSQL 13

Percona Distribution PostgreSQL

Percona Distribution PostgreSQLPercona is pleased to announce the release of Percona Distribution for PostgreSQL 13. This release is available for immediate download and includes all of the latest features of PostgreSQL 13 Core Distribution.

Reduce Resource Usage

This release includes enhancements to existing features that deliver a more streamlined and optimized environment, helping you to reduce overall resource usage. This includes minimizing bloat with parallel Vacuum, which helps the Vacuum process to run more quickly and efficiently, improving the use of storage and improving performance. Deduplication of B-Tree indexes also helps save storage space by reducing the amount of storage needed for indexes. It also uses resources more efficiently during query with additional opportunities to benefit from partition enhancements. This means that your query processes less data and frees up resources.

Improve Response Times

With enhancements that deliver more efficient use of indexes and smarter sorting capabilities, you can expect better overall performance and improved response times. This includes incremental sort which avoids sorting data that has already been sorted and delivering query results faster.  New partition wise joins break down a join between partitioned tables into joins between their partitions, resulting in smaller data volumes being processed more quickly. Improvements to GiST, SP-GiST, and GIN indexes, provide overall performance improvement and speed up query processing. There are also some new PostgreSQL commands and authentication changes.

Additionally, we are including a technical preview of pg_stat_monitor, a custom extension written by Percona.  This extension gathers and aggregates query performance data, enabling better and faster query analysis. It can be used alone, but its capabilities are best used when combined with the latest release of Percona Monitoring and Management. This enables you to easily analyze your PostgreSQL queries, using the pg_stat_monitor metrics, to quickly identify and remedy issues with scaling, bottlenecks, and potential outages.

Percona is also planning to extend our Managed Services offerings to include PostgreSQL early next year. This means that you will be able to have your MySQL, MongoDB, PostgreSQL, and Maria DB databases all managed by a single source. As always, we continue to provide support for these open-source database products and offer professional consulting and training services.

For more details on PostgreSQL 13, check out the release notes from PostgreSQL. To learn more about Percona Distribution for PostgreSQL, check out our release notes.

To provide feedback on the technical preview of pg_stat_monitor or its integration with Percona Monitoring and Management, visit the Percona Community Forum. Bugs should be submitted through Jira.

Download Percona Distribution for PostgreSQL

Download Percona Monitoring and Management


PMM 101: Troubleshooting MongoDB with Percona Monitoring and Management

Troubleshooting MongoDB with Percona Monitoring and Management

Troubleshooting MongoDB with Percona Monitoring and ManagementPercona Monitoring and Management (PMM) is an open-source tool developed by Percona that allows you to monitor and manage your MongoDB, MySQL, and PostgreSQL databases.  This blog will give you an overview of troubleshooting your MongoDB deployments with PMM.

Let’s start with a basic understanding of the architecture of PMM. PMM has two main architectural components:

  1. PMM Client – Client that lives on each database host in your environment.  Collects server metrics, system metrics, and database metrics used for Query Analytics
  2. PMM Server – Central part of PMM that the clients report all of their metric data to.   Also presents dashboards, graphs, and tables of that data in its web interface for visualization of your metric data.

For more details on the architecture of PMM, check out our docs.

Query Analytics

PMM Query Analytics (“QAN”) allows you to analyze MongoDB query performance over periods of time.  In the below screenshot you can see that the most longest-running query was against the testData collection.

Percona Monitoring and Management query analytics

If we drill deeper by clicking on the query in PMM we can see exactly what it was running. In this case, the query was searching in the testData collection of the mpg database looking for records where the value of x is 987544.

Percona Monitoring and Management

This is very helpful in determining what each query is doing, how much it is running, and which queries make up the bulk of your load.

The output is from db.currentOp(), and I agree it may not be clear at a glance what the application-side (or mongo shell) command was. This is a limitation of the MongoDB API in general – the drivers will send the request with perfect functional accuracy but it does not necessarily resemble what the user typed (or programmed).  But with an understanding of this, and focusing first on what the “command” field contains it is not too hard to picture a likely original format. For example the example above could have been sent by running “use mpg; db.testData.find({“x”: { “$lte”: …, “$gt”: … }).skip(0)” in the shell. The last “.skip(0)” is optional as it is 0 by default.

Additionally, you can see the full explain plan for your query just as you would by adding .explain() to your query.   In the below example we can see that the query did a full collection scan on the mpg.testData collection and we should think about adding an index to the ‘x’ field to improve the performance of this query.

Metrics Monitor

Metrics Monitor allows you to monitor, alert, and visualize different metrics related to your database overall, its internal metrics, and the systems they are running on.

Overall System Performance View

The first view that is helpful is your overall system performance view.   Here you can see at a high level, how much CPU and memory are being used, the amount of writes and reads from disk, network bandwidth, # of database connections, database queries per second, RAM, and the uptime for both the host and the database.   This view can often lead you to the problematic node(s) if you’re experiencing any issues and can also give you a high level of the overall health of your monitored environment.

Percona Monitoring and Management system overview

WiredTiger Metrics

Next, we’ll start digging into some of the database internal metrics that are helpful for troubleshooting MongoDB.  These metrics are mostly from the WiredTiger Storage Engine that is the default storage engine for MongoDB since MongoDB 3.0.  In addition to the metrics I cover, there are more documented here.

The WiredTiger storage engine uses tickets as a way to handle concurrency, The default is for WiredTiger to have 128 read and 128 write tickets.  PMM allows you to alert when your available tickets are getting low.  You can also correlate with other metrics as to why so many tickets are being utilized. The graph sample below shows a low-load situation – only ~1 ticket out of 128 was checked out at any time.

Percona Monitoring and Management wiredtiger

One of the metrics that could be causing you to use a large number of tickets is if your checkpoint time is high.   WiredTiger, by default, does a full checkpoint at least every 60 seconds, this is controlled by the WiredTiger parameter checkpoint=(wait=60)).  Checkpointing flushes all the dirty pages to disk. (By the way ‘dirty’ is not as bad as it sounds – it’s just a storage engine term meaning ‘not committed to disk yet’.)  High checkpointing times can lead to more tickets being in use.

Finally, we have WiredTiger Cache Activity metrics. WiredTiger Cache activity indicates the level of data that is being read into or written from the cache.  These metrics can help you baseline your normal cache activity, so you can notice if you have a large amount of data being read into the cache, perhaps from a poorly tuned query, or a lot of data being written from the cache.

WiredTiger Cache Activity

Database Metrics

PMM also has database metrics that are not WiredTiger specific.   Here we can see the uptime for the node, queries per second, latency, connections, and number of cursors.   These are higher-level metrics which can be indicative of a larger problem such as connection storms, storage latency, and excessive queries per second.  These can help you hone in on potential issues for your database.

Percona Monitoring and Management database metrics

Node Overview Metrics

System metrics can point you towards an issue at the O/S level that may or may not correlate to your database.  CPU, CPU saturation, core usage, DISK I/O, Swap Activity, and Network Traffic are some of the metrics that can help you find issues that may start at the O/S level or below.  Additional metrics to the below can be found in our documentation.

Node Overview Metrics


In this blog, we’ve discussed how PMM can help you troubleshoot your MongoDB deployment, whether you’re looking at the WiredTiger specific metrics, system-level metrics, or database level metrics PMM has you covered and can help you troubleshoot your MongoDB deployment.  Thanks for reading!

Additional Resources:

Download Percona Monitoring and Management

PMM for MongoDB Quick Start Guide

PMM Blog Topics

MongoDB Blog Topics

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com