Extend Metrics for Percona Monitoring and Management Without Modifying Code

PMM Extended Metrics

Percona Monitoring and Management (PMM) provides an excellent solution for system monitoring. Sometimes, though, you’ll have the need for a metric that’s not present in the list of node_exporter metrics out of the box. In this post, we introduce a simple method and show how to extend the list of available metrics without modifying the node_exporter code. It’s based on the textfile collector.

Enable the textfile collector in pmm-client

This collector is not enabled by default in the latest version of pmm-client. So, first let’s enable the textfile collector.

# pmm-admin rm linux:metrics
OK, removed system pmm-client-hostname from monitoring.
# pmm-admin add linux:metrics -- --collectors.enabled=diskstats,filefd,filesystem,loadavg,meminfo,netdev,netstat,stat,time,uname,vmstat,textfile"/tmp"
OK, now monitoring this system.
# pmm-admin ls
pmm-admin 1.13.0
PMM Server      |  
Client Name     | pmm-client-hostname
Client Address  |  
Service Manager | linux-upstart
-------------- -------------------- ----------- -------- ------------ --------
-------------- -------------------- ----------- -------- ------------ --------
linux:metrics  pmm-client-hostname  42000       YES      -

Notice that the whole list of default collectors has to be re-enabled. Also, don’t forget to specify the directory for reading files with the metrics (–”/tmp”). The exporter reads files with the extension .prom

Add a crontab task

The second step is to add a crontab task to collect metrics and place them into a file.

Here are the cron commands for collecting the number of running and stopping docker containers.

*/1 * * * *     root   echo -n "" > /tmp/docker_all.prom; /usr/bin/docker ps -a | sed -n '1!p'| /usr/bin/wc -l | sed -ne 's/^/node_docker_containers_total /p' >> /tmp/doc
*/1 * * * *     root   echo -n "" > /tmp/docker_running.prom; /usr/bin/docker ps | sed -n '1!p'| /usr/bin/wc -l | sed -ne 's/^/node_docker_containers_running_total /p' >>

The result of the commands is placed into the files 




and read by exporter.

Look - we got a new metric!

Adding the crontab tasks by using a script

Also, we have a few bash scripts that make it much easier to add crontab tasks.

The first one allows you to collect the logged-in users and the size of Innodb data files.

Modifying the cron job - a script

You may use the suggested names of files and metrics or set new ones.

The second script is more universal. It allows us to get the size of any directories or files. This script can be placed directly into a crontab task. You should just specify the list of monitored instances (e.g. /var/log /var/cache/apt /var/lib/mysql/ibdata1)

echo  "*/5 * * * * root bash  /root/ /var/log /var/cache/apt /var/lib/mysql/ibdata1"  > /etc/cron.d/object_size

So, I hope this has provided useful insight into how to set up the collection of new PMM metrics without the need to write code. Please feel free to use the scripts or configure commands similar to the ones provided above.

More resources you might enjoy

If you are new to PMM, there is a great demo site of the latest version, showing you those out of the box metrics. Or how about our free webinar on monitoring Amazon RDS with PMM?

The post Extend Metrics for Percona Monitoring and Management Without Modifying Code appeared first on Percona Database Performance Blog.


“How to monitor MySQL performance” with Percona Cloud Tools: June 25 webinar

We recently released a new version of Percona Cloud Tools with MySQL monitoring capabilities. Join me June 25 and learn the details about all of the great new features inside Percona Cloud Tools – which is now free in beta. The webinar is titled “Monitoring All (Yes, All!) MySQL Metrics with Percona Cloud Tools” and begins at 10 a.m. Pacific time.

In addition to MySQL metrics, Percona Cloud Tools also monitors OS performance-related stats. The new Percona-agent gathers metrics with fine granularity (up to once per second), so you are able to see any of these metrics updated real-time.

During the webinar I’ll explain how the new Percona-agent works and how to configure it. And I’ll demonstrate the standard dashboard with the most important MySQL metrics and how to read them to understand your MySQL performance.

Our goal with the new implementation was to make installation as easy as possible. Seriously it should not take so much effort as it has in the past to get visibility into your MySQL performance. We also wanted to provide as much visibility as possible.

Please take a moment and register now for the webinar. I also encourage you, if you haven’t already, to sign up for access to the free Percona Cloud Tools beta ahead of time. At the end of the next week’s webinar you’ll know how to use the Percona-agent and will be able to start monitoring MySQL in less than 15 minutes!

See you June 25 and in the meantime you can check out our previous related posts: “From zero to full visibility of MySQL in 3 minutes with Percona Cloud Tools” and “Introducing the 3-Minute MySQL Monitor.”

The post “How to monitor MySQL performance” with Percona Cloud Tools: June 25 webinar appeared first on MySQL Performance Blog.


From zero to full visibility of MySQL in 3 minutes with Percona Cloud Tools

First, I would like to invite you to my webinar, “Monitoring All (Yes, All!) MySQL Metrics with Percona Cloud Tools,” on Wednesday, June 25 at 10 a.m. Pacific Daylight Time, where I will talk on the
new features in Percona Cloud Tools, including monitoring capabilities.

In this post I’d like to show the cool and interesting things we’ve implemented in Percona Cloud Tools, including the recently released agent that Daniel also talks about here in this post.

Basically our agent allows users to collect ALL MySQL metrics plus important environment’s metrics, like CPU, memory, IO stats.

And when I talk all MySQL it is:

  • Metrics from SHOW GLOBAL STATUS (I counted 571 entries on my Percona Server 5.6 with TokuDB)
  • Metrics from INFORMATION_SCHEMA.INNODB_METRICS ( 214 entries)
  • Data from SHOW GLOBAL VARIABLES ( 522 entries).

So you see we collect more than 1000 points. We collect data every second, aggregate per minute and it becomes available with a 1-minute resolution, but with descriptive stats like min, max, avg, median, 5%-percentile, 95%-percentile of collected data. We found this represents data better than 1-sec points, which can be a quite noisy.

So for example this is how a chart with MySQL command counters looks like:
Image 12

If you do not have MySQL, in fact, you can just monitor your OS without MySQL if you need to.
For example CPU stats
Image 9

Please note you can see a value at any point in the past. We even can go to week range and see values several days ago
Image 10
You can choose a custom timerange back to hours, days, weeks, etc with zoom-in capabilities.

Why do we need avg, min, max stats? Let see Peter’s graph from a server with periodical stalls.

Averaging metric smooths the line, and really hides the problem, while with min we are able to see that throughput drops to zero, that means that during some seconds the server did not execute any queries (which is essentially stalls)

More about the agent.

Our percona-agent is open source. This is our way of sharing our Go knowledge, and also you can check that we do not do anything insane on your server (like bitcoin mining or black magic regression modeling math).

You can see source code here
and pre-compiled binaries are available from our website:

What is also interesting about our technology is that we use a permanent connection (based on WebSockets technology, so it looks like a connection to web browser) between an agent and our servers. This way
we support a bi-directional real-time communication between and an agent.

That means you can manage an agent and receive data updates at the same time. Pretty cool, yeah?

Our agent comes with “minimal efforts” in mind.

  • 1. There is minimal efforts to install agent. Basically it takes 3-minutes
    to download binaries, install them, and start seeing real-time updates of charts with MySQL metrics.
  • 2. Agent comes with self-update capabilities (not activated at this moment). Later you will need
    to worry if there is new version of agent is available, it will updated itself. We thought if Android Apps can do that, why can’t we?
  • 3. Minimal efforts for a maintenance: you do not need to install a dedicated server, configure and maintain database, care about its backups and availability. Basically no more hassle with Cacti configuration and managing a storage for it.

The goal of Percona Cloud Tools is to provide you with FULL visibility on what’s going on inside your system right now or at some point in the past, and to gain additional insights on your database server.

Our tools are useful not only if you have hundreds of database servers to manage, but pretty much for single installations, too. Well, of course, we always can run mysql -e "SHOW GLOBAL STATUS" , vmstat 10 ; iostat -dxm 10 manually when we need to troubleshoot something, but is it not useful to collect all this data automatically and be able to go to any point in the past?

You can register for the Beta right now. No invitation is needed, but it may take sometime for us to activate your account, we see a quite a demand right now and we need to prime our servers.

And what’s important, eventually our tools will require a payment, but we will always provide a free level, which will be useful enough for small accounts (this is not a bait-and-switch 30-days trial approach).

The post From zero to full visibility of MySQL in 3 minutes with Percona Cloud Tools appeared first on MySQL Performance Blog.

Powered by WordPress | Theme: Aeros 2.0 by