Personal blog of Yzmir Ramirez

You can show your support for open source software by participating in an event next March in Pasadena, California – The Southern California Linux Expo, or SCaLE. SCaLE is the largest community-run open source and free software conference in North America. Percona has sponsored this show for many years, provided presenters, and participated in the expo hall. Thousands of people attend each year, and their support helps keep open source software flourishing. The presentations have always come from the community and run the gambit from well-known individuals in the community to those less famous but still passionate about their subjects. And now, hopefully, you will participate!

What happens At SCaLE?

There will be several two to four-hour hands-on workshop sessions that provide attendees with practical experience implementing and using open technologies. Plus, hundreds of talks on subjects ranging from automating your home with open source software to zero downtime Kubernetes implementation. The one thing SCaLE and the open source community need to add to all this to make it better is you!

As you would expect, Percona is involved in several of the presentation tracks that feature database or Kubernetes topics. We are instrumental in helping launch a Data on Kubernetes track. And we will be back in the PostgreSQL and MySQL tracks as well.

First-time presenters welcome!

Submitting a talk for a first-timer can be nerve-racking. SCaLE is a great show for novice speakers, and everyone in the audience looking up to you wants you to succeed. They want to know what you did to solve a problem, learn what you want to teach, or hear your story. A talk on how you, as a newbie, learned to integrate CI/CD into your operational stack is as valuable as one using Common Table Expressions in MySQL. In your experience, is it better to back up servers with disk snapshots or a series of incremental backups? How does someone learn to program in JavaScript/Go/Rust? The SCaLE audience is very appreciative and always values new things or new approaches. And if you want someone to review your submission before you send it in, I am more than happy to do so.

The best way to prepare to submit a talk is to come up with a descriptive title about the subject. The title does not need to be cute or overly filled with jargon, but please keep it descriptive of the contents. Next, outline five points you want to cover in your talk. And from there, turn your five points into sentences. Double-check that the title and those sentences refer to the same topic. Congratulations, you have material for a submission.

More details on the show

Please head over to the SCaLE webpage at https://www.socallinuxexpo.org/scale/21x. The show runs March 14th through the 17th, 2024, at the Pasadena Convention Center. The expo hall is open Friday – Sunday. Talks and workshops are Thursday – Sunday. And the call for proposals is open Aug 1, 2023 – November 1, 2023. A pass for the expo hall and free events is $20, and a full pass that includes four days of presentations is $80.

What will be presented? Take a look at the schedule from this year for a guide on what will be going on next year. Hopefully, your presentations will be on the 2024 schedule (do not forget that November 1st deadline!).

Sponsorships are also available if you would like to showcase your project, group, or company.

And I look forward to seeing you in Pasadena.

Starting with PMM 1.13,  PMM uses Prometheus 2 for metrics storage, which tends to be heaviest resource consumer of CPU and RAM.  With Prometheus 2 Performance Improvements, PMM can scale to more than 1000 monitored nodes per instance in default configuration. In this blog post we will look into PMM scaling and capacity planning—how to estimate the resources required, and what drives resource consumption.

We have now tested PMM with up to 1000 nodes, using a virtualized system with 128GB of memory, 24 virtual cores, and SSD storage. We found PMM scales pretty linearly with the available memory and CPU cores, and we believe that a higher number of nodes could be supported with more powerful hardware.

What drives resource usage in PMM ?

Depending on your system configuration and workload, a single node can generate very different loads on the PMM server. The main factors that impact the performance of PMM are:

1. Number of samples (data points) injected into PMM per second
2. Number of distinct time series they belong to (cardinality)
3. Number of distinct query patterns your application uses
4. Number of queries you have on PMM, through the user interface on the API, and their complexity

These specifically can be impacted by:

• Software version – modern database software versions expose more metrics)
• Software configuration – some metrics are only exposed in certain configuration
• Workload – a large number of database objects and high concurrency will increase both the number of samples ingested and their cardinality.
• Exporter configuration – disabling collectors can reduce amount of data collectors
• Scrape frequency –  controlled by METRICS_RESOLUTION

All these factors together may impact resource requirements by a factor of ten or more, so do your own testing to be sure. However, the numbers in this article should serve as good general guidance as a start point for your research.

On the system supporting 1000 instances we observed the following performance:

As you can see, we have more than 2.000 scrapes/sec performed, providing almost two million samples/sec, and more than eight million active time series. These are the main numbers that define the load placed on Prometheus.

Capacity planning to scale PMM

Both CPU and memory are very important resources for PMM capacity planning. Memory is the more important as Prometheus 2 does not have good options for limiting memory consumption. If you do not have enough memory to handle your workload, then it will run out of memory and crash.

We recommend at least 2GB of memory for a production PMM Installation. A test installation with 1GB of memory is possible. However, it may not be able to monitor more than one or two nodes without running out of memory. With 2GB of memory you should be able to monitor at least five nodes without problem.

With powerful systems (8GB of more) you can have approximately eight systems per 1GB of memory, or about 15,000 samples ingested/sec per 1GB of memory.

To calculate the CPU usage resources required, allow for about 50 monitored systems per core (or 100K metrics/sec per CPU core).

One problem you’re likely to encounter if you’re running PMM with 100+ instances is the “Home Dashboard”. This becomes way too heavy with such a large number of servers. We plan to fix this issue in future releases of PMM, but for now you can work around it in two simple ways:

You can select the host, for example “pmm-server” in your home dashboard and save it, before adding a large amount of hosts to the system.

Or you can make some other dashboard of your choice and set it as the home dashboard.

Summary

• More than 1,000 monitored systems is possible per single PMM server
• Your specific workload and configuration may significantly change the resources required
• If deploying with 8GB or more, plan 50 systems per core, and eight systems per 1GB of RAM

The post Scaling Percona Monitoring and Management (PMM) appeared first on Percona Database Performance Blog.

This post is a followup to my previous article https://www.percona.com/blog/2018/08/29/scaling-io-bound-workloads-mysql-cloud/

In this instance, I want to show the data in different dimensions, primarily to answer questions around how throughput scales with increasing IOPS.

A recap: for the test I use Amazon instances and Amazon gp2 and io1 volumes. In addition to the original post, I also tested two gpl2 volumes combined in software RAID0. I did this for the following reason: Amazon cap the single gp2 volume throughput to 160MB/sec, and as we will see from the charts, this limits InnoDB performance.

Also, a reminder from the previous post: we can increase gp2 IOPS by increasing volume size (to the top limit 10000 IOPS), and for io1 we can increase IOPS by paying per additional IOPS.

Scaling with InnoDB

So for the first result, let’s see how InnoDB scales with increasing IOPS.

There are a few interesting observations here: InnoDB scales linearly with additional IOPS, but it faces a throughput limit that Amazon applies to volumes.

So besides considering IOPS, we should take into account the maximal throughout of volumes.

In the second chart we compare InnoDB performance vs the cost of volumes:

It’s interesting to see here the slope for gp2 volumes is steeper than for io1 volumes. This means we can get a bigger increase in InnoDB performance per dollar using gp2 volumes, but only until we reach the IOPS and throughput limits that are applied to gp2 volumes.

Scaling with MyRocks

And here’s the similar chart but for MyRocks:

Here we can also see that MyRocks scales linearly, showing identical results on gp2 and io1 volumes. This means that running on gp2 will be cheaper. Also, there is no plateau in throughput, as we saw for InnoDB, which means that MyRocks uses less IO throughput.

And the chart for the cost of running MyRocks:

This charts also shows that it is cheaper to run on gp2 volume but only while it provides enough IOPS. I assume that using two gp2 volumes would allow me to double the throughput. (I did not run the test for MyRocks using two volumes)

Conclusions

• Both MyRocks and InnoDB can scale (linearly) with additional IOPS on gp2 and io1 Amazon volumes.
• Take into account that IOPS is not the only factor to consider. There is also throughput limit, which affects InnoDB results, so for further scaling you might need to use multiple volumes.

The post Scaling IO-Bound Workloads for MySQL in the Cloud – part 2 appeared first on Percona Database Performance Blog.

Please join Percona’s CEO, Peter Zaitsev as he presents MySQL: Scaling and High Availability – Production Experience Over the Last Decade(s) on Tuesday, June 19th, 2018 at 7:00 AM PDT (UTC-7) / 10:00 AM EDT (UTC-4).

 

Percona is known as the MySQL performance experts. With over 4,000 customers, we’ve studied, mastered and executed many different ways of scaling applications. Percona can help ensure your application is highly available. Come learn from our playbook, and leave this talk knowing your MySQL database will run faster and more optimized than before.

Register Now

About Peter Zaitsev, CEO

Peter Zaitsev co-founded Percona and assumed the role of CEO in 2006. As one of the foremost experts on MySQL strategy and optimization, Peter leveraged both his technical vision and entrepreneurial skills to grow Percona from a two-person shop to one of the most respected open source companies in the business. With over 140 professionals in 30 plus countries, Peter’s venture now serves over 3000 customers – including the “who’s who” of internet giants, large enterprises and many exciting startups. Percona was named to the Inc. 5000 in 2013, 2014, 2015 and 2016.

Peter was an early employee at MySQL AB, eventually leading the company’s High Performance Group. A serial entrepreneur, Peter co-founded his first startup while attending Moscow State University where he majored in Computer Science. Peter is a co-author of High Performance MySQL: Optimization, Backups, and Replication, one of the most popular books on MySQL performance. Peter frequently speaks as an expert lecturer at MySQL and related conferences, and regularly posts on the Percona Database Performance Blog. He has also been tapped as a contributor to Fortune and DZone, and his recent ebook Practical MySQL Performance Optimization Volume 1 is one of percona.com’s most popular downloads. Peter lives in North Carolina with his wife and two children. In his spare time, Peter enjoys travel and spending time outdoors.

The post Webinar Tues 19/6: MySQL: Scaling and High Availability – Production Experience from the Last Decade(s) appeared first on Percona Database Performance Blog.

 In-flight VR entertainment and security guard drones were amongst our favorites from the 44 startups that launched at Y Combinator’s Summer 2016 Demo Day 1. After querying investors and our writers, here are TechCrunch’s picks for the 7 most promising companies from the first half of the batch, plus an honorable mention. Check back later for all the startups and our picks from… Read More