Feb
17
2021
--

The Most Important Skills for an SRE, DBRE, or DBA

Important Skills for an SRE DBRE or DBA

Important Skills for an SRE DBRE or DBAI have talked extensively about the DBA’s evolving role and how many DBA’s and operations professionals are now becoming SRE’s (site reliability engineers) or DBRE’s (database reliability engineers). Often, databases get blamed as the bottleneck for application slowdowns and issues, so DBAs have had to develop the skills needed to chase problems up and down the stack over the years. This full-stack approach to hunting out problems has resulted in many former DBAs and Sysadmins successfully taking on the role of an SRE/DBRE.

The question is, then, what are the most critical skills for this important role?

I personally have interviewed 1000’s of technical candidates over the last 10 years and have hired hundreds in various roles here at Percona. I often get asked the most critical skill for the next generation of technical engineers, SREs, or DBREs. The answer has been consistent for me over my career – I want engineers, SREs, etc., with good problem-solving skills and the ability to think outside the box. I am not looking for book knowledge or a detailed understanding of every concept; I want people who can see something new and…

  1. Be curious enough to ask “Why?” and want to know the answer.
  2. Will dig into the ambiguous and want to learn, and can learn the why.
  3. Can solve the issue, answer the question, and share that knowledge effectively.

From a technical perspective, while it is wonderful to have a great depth of knowledge, I generally am not looking for the experts’ expert. Rather, I look for people who are smart, passionate, and who learn quickly. I am not alone in this. I asked the question of those involved in hiring technical people here at Percona.

Peter Zaitsev (CEO) said the number one skill he is looking for is this: “Attitude and Ability to do independent research and find information to solve the problem at hand.” For many like Peter, having an encyclopedic knowledge of how things work or the right commands to use is secondary to solving problems never seen before. Many problems and issues that come up you cannot specifically train for. The unique nature of the workload, size, and way too many external factors often offer unique challenges to even the most experienced SRE. Peter added: “So many people now have this ‘I have not been trained on this’ attitude instead of doing some basic googling for the answer.” Indeed, there is a lot of information out there, and while searching for an answer quickly during an outage or performance event may seem like a no-brainer, more than half the people I have interviewed don’t even think about it. Thinking on your feet, reacting quickly, and restoring service can save companies millions of dollars of lost revenue and business in an outage.

Have open source expertise you want to share? Submit your talk for Percona Live ONLINE 2021!

Marco Tusa (MySQL Tech Lead) echoed Peter’s sentiment by saying that there are two important skills for him. One of these is the ability to learn what they don’t know. “This is because no matter what, often the best one on tech knowledge won’t know some important stuff. The will to learn is the key.Lenz Grimmer (Sr Director of Server Engineering) could not have agreed more, adding: “I’m seeking talent that is open-minded about acquiring new skills. So fast learners with the right sense of humility and the right attitude.

Teamwork Makes the Dream Work…

Attitude and humility are critical in building an effective team (especially in a remote team). This was Marco’s second trait he is looking for. Marco went on to add he is also looking strongly into their fit with the team and if they will be a team player. The “no jerks” or “no soloist prima donna” mottos are very important. You have to be willing to share what you learned and look for help from your teammates.

This is the same thing Jay Janssen (Director of IT) said when asked about the number one thing he looks for: “Humility comes to mind — smart and humble is a good combination. While kind of cliche, it’s generally true.” We are all looking to hire smart people, but smart people who are Jerks or flaunt how smart they are generally don’t operate well in a team environment. You want someone who is smart but does not make other people feel small or insignificant.

Sanja Bonic (Head of Percona’s Open Source Program Office) also values teamwork and makes sure she tries to understand how people handle positive and negative interactions as a team.  Sanja, who has previously led Engineering teams at OpenShift and now works with Percona’s community, asks people in interviews about “their best and worst experiences in teams they’ve previously worked with. This usually shows what people are paying attention to, and you’ll quickly get a hint of what people attribute value to.”

While you need people to work and learn independently, you equally also need them to function as a unit (or as a team). Remember to ensure the uptime, availability, and performance of the entire application spanning potentially hundreds or thousands of nodes, you need to use all the resources at your disposal when things go wrong, and having teammates who you trust, can help, and can augment your knowledge with is very important. You can’t do it all alone, so having the ability to “team-up” and work with others is a must.

The strength of the team is each individual member. The strength of each member is the team.” ~ Phil Jackson

Sharing is Caring…

The ability for smart people to effectively share their knowledge and have good meaningful conversations is also critical in this role. Vadim Tkachenko (CTO) said he is looking for “People who have a brain and can have a meaningful conversation.” He went on to say he is looking for people who “Can speak well about previous relevant experiences.” This ability to share goes a long way internally to increase the collaborative spirit within the team. But this is not merely about speaking a single language; rather, it’s being able to talk about the technologies and match your audience’s expectations (or teammates).

Tate Mcdaniel (DBA Manager) says this is the number one thing he looks for when hiring people. His approach, in his words – “I ask questions about contentious and complicated things, and I look for answers that explain the complexity in ways a layperson can understand while giving pros/cons judiciously.” Taking the complex, explaining it, and educating others is of critical importance.

It is why Peter, Vadim, Jay, Marco, Tate, Lenz, and myself all said we look online at what people have written, where they have talked at conferences, what code they may have written, and other traces of their public persona before interviewing someone.

When I asked Lenz Grimmer if he looked at a candidate’s online persona, he said: “Absolutely, that’s one of the beauties of hiring in the open-source ecosystem. A public track record of contributions in various forms tells me much more than a CV. Forum and mailing list contributions, YouTube videos, all of which help get a better understanding of the candidate.”

One Person is an Island…

I personally highly value people’s willingness to share their insights, knowledge, and sometimes struggles. This is especially critical in the open-source space. I mentioned that no one person could manage a complex environment alone. Training and educating team members and others in the community is critical. The willingness to share and educate via online blogs, articles, and technical talks is, in my opinion, essential to the SRE/DBRE community as a whole.

So what do we see as the must-have skills?

  1. Problem-solving skills, the ability to troubleshoot unique and challenging problems.
  2. The passion and desire to learn, research, and acquire skills quickly.
  3. Humility and the ability to be a “team player” – No jerks allowed!
  4. The ability and passion for sharing their knowledge and educating others.

What do you think? Did we miss any?

Feb
02
2021
--

Why SSPL is Bad For You, Part 2

Why SSPL is Bad

Why SSPL is BadAfter MongoDB switched to SSPL I wrote part one of this article, titled “Why MongoDB’s SSPL is Bad For You”. At the time it was a MongoDB phenomenon, though over the last 2+ years more companies (typically VC-funded unicorns or public companies) changed licenses to SSPL or similar. This culminated in Elastic changing their license to SSPL, which is a particularly concerning case, for a number of reasons.

So, before we jump to the main topic of this article, let me talk about why I see Elastic’s change as being much more concerning than MongoDB’s.

Elastic’s Promise, Unkept

Elastic already did some license soul searching in 2018. In that post, Shay Banon made the commitment “We will remain an open source company.”  You can also see that because Open and Non-Open code were mixed in the same GitHub repository, this explanation and promise was provided:

SSPL bad

There is an interesting Twitter thread on this, check it out.

As far as I know, MongoDB never gave a similar promise, and in general, they have been a “Reluctant Open Source Company” and quite open that releasing MongoDB as Open Source was their Freemium Marketing Strategy (source).

Changing From Permissive License

MongoDB was AGPL licensed to begin with… which is a very restrictive copyleft license. In theory, it should have protected MongoDB from having to compete with cloud hyper-scalers in offering DBaaS. In practice, though, there were a number of companies offering AGPL licensed MongoDB as part of their DBaaS and as rumor has it, it was not easily enforceable.

The fact that MongoDB was AGPL licensed to begin with means there never was a “level playing field” for MongoDB and other contributors. For example, at Percona we are building an extended version of MongoDB – Percona Server for MongoDB – and we can only distribute it first under the AGPL and then the SSPL license. MongoDB can sell it under a proprietary license (think MongoDB Enterprise) or any other Open Source license if they chose to.  This is important as a number of companies, including Google, do not permit the use of AGPL software.

Apache 2.0 is a very permissive license though, as one is able to embed such software in virtually any Proprietary (or Open Source) software, make it available as cloud, etc. This means the Elastic license change will affect many more users, and much more drastically, than in the case of MongoDB.

Contributors, Mislead

Elastic has a huge number of contributors, more than 1,500 if you believe this source. Elastic relied on its contributors far more heavily than MongoDB for speed of innovation. This is not a surprise, as due to the Apache 2.0 license used for core software, it made sense for many companies to make ElasticSearch better to meet their own needs.

Many of those contributions were done because the combined work could be used under the Apache 2.0 license, and because Shay was very vocal about Elastic’s commitment to sticking to the Apache 2.0 license forever.

I wonder how many of them would have contributed if they had known the only way they could use derived work was under the terms of the SSPL license… Probably not many.

You can argue, of course, that contributors still have access to Apache 2.0-licensed old software and they could fork it if they do not like what Elastic is doing. But, it is not practical for companies who don’t specialize in Elasticsearch development to do this.

There are companies that have big business tied to the need for Elastic to be truly open (but not Elastic’s definition of open).  As I expected and hoped, AWS is forking ElasticSearch, building on its work of Open Distro for Elasticsearch. And it is not alone. Logz stated they are doing the same, and while not promising to do it themselves, Aiven expressed hope that the community would come together to do so.

Dangerous Wordplay

The title of the blog post on the license change is “Doubling down on open, Part II.”  It does not say we’re moving to non-OSI license or “Source Available License,” as such licenses are often called, but actively uses the words “Open” and “Free” (as in Beer) through the post. These words are typically associated with Open Source Software, so it looks to me like intent to confuse the reader.

To be fair, it is not just Elastic. For ages, SQL was marketed as “Open Standard”. AWS talks about “Open Source Databases” while, of course, AWS offerings range from proprietary code database engine technology, such as DocumentDB and Aurora, to the Proprietary Management level.

I very much worry that the marketing efforts of companies with budgets much larger than those of most Open Source communities will erode the understanding of what Open Source was really intended to be about.

With all that in mind, let’s now think about why SSPL is bad for you.

SSPL is Bad for End Users

If you’re just using software, SSPL is bad for you because it restricts your choices. SSPL is usually chosen to give one vendor a monopoly on their SaaS/DBaaS deployment model, which means reduced choice and higher prices. The SaaS/DBaaS model is the fastest-growing usage model for a lot of software.

Do not believe vendors downplaying it by saying, “well, unless you’re going to sell it as SaaS it will not affect you.” It is likely to affect you directly, or even indirectly, if other vendors have to pay higher costs for their DBaaS, as they inevitably pass costs on to you as a customer.

SSPL stifles innovation by discouraging code contributors and creating increased vendor lock-in. While SSPL can be better for you than a conventional proprietary license, you should choose a  more liberally-licensed alternative if you have a choice.

SSPL is Bad for Code Contributors

If you’re contributing or planning to contribute code, SSPL is bad for you because you only get access to the combined work under the terms of SSPL. This benefits the vendor but places severe restrictions on how you can use the software yourself.

If a project’s license is changed to the SSPL after the fact, you are likely to find yourself a member of a quickly-shrinking contributor community, because fewer people will find it attractive to contribute.

SSPL is Bad for Vendors

MongoDB and Elastic will probably not agree with this, but in fact, they changed the license to be in a better position as a business. I believe this is short-term thinking though. The wheels of Open Source often move slowly, but inevitably. I would be very interested to look at MongoDB and Elastic 5-10 years from now. Will it still be a thriving technology that businesses build new applications with, or will it be a new Oracle, sucking more and more revenue from a shrinking user base of legacy deployments?

MongoDB and Elastic are probably betting on the Cloud making Open Source all but irrelevant. However, I believe we will see a new wave of Open Source Software, offering us usability of current Proprietary Public Clouds, but as a no-lock-in Open Source solution you can run everywhere. Similar to server hardware, Virtualization Hypervisors were commoditized and so will the Cloud be. With commoditized Cloud, we will be looking for truly Open Source Data Stores to have a complete stack, which will inevitably lead to their creation.

Looking for more wisdom? Take a look at the article Why Real Open Source is Better by Matt Yonkovit.

Feb
02
2021
--

Percona Software Feedback – Help Us, Help You!

Software Survey Percona

Software Survey PerconaHello, open source community!  

We need your help.  We want to make sure you are getting what you need from Percona software, so we have built a series of quick surveys – of six or fewer questions – each asking if you are using our software and if you would recommend it to others.  They won’t take more than a minute each unless you want to add a long comment, and you can answer one or all of them.

You do not need to put in an email address for these, but you can provide it if you choose to do so.  (We did add an optional box for you to provide it.)  If used, it will ONLY be used to enter into a drawing for a free Percona T-shirt.  We will be giving away 10 total Percona Shirts randomly to those who fill in the survey.  If you fill in all five mini-surveys, you will get five entries into the drawing!

Your feedback will help us keep our commitment to building and enhancing the open-source database ecosystem, so thank you in advance!

Matt

Jan
20
2021
--

Keeping Open Source Open, or, Why Open is Better

Keeping Open Source Open

Last week Elastic announced that they were “Doubling Down” on open source by changing their licensing to a non-open license – MongoDB’s Server Side Public License, or SSPL.  Let me clarify in my opinion this is not doubling down – unless, as our good friend @gabidavila highlighted, that maybe the thinking was a double negative makes a positive? VM Brasseur posted on her blog that she feels Elastic and Kibana are now a business risk for enterprises.  Peter Zaitsev has penned why he felt SSPL was bad for you before this announcement and then sat down with me to discuss his thoughts last week as well.

This is not the first and regrettably, this won’t be the last “open” company to run away from open source or the model which got them as a company to where they are today.

Why Do Some Companies Have Open Source Problems Today?

I personally have no direct problem with companies selling proprietary software. I have no problem with companies selling software with new licensing, and I have no problem with cloud services.  I simply feel that open is better. Open gives not only paying customers but the entire community a better and more viable product. It’s ok if you disagree with that.  It is ok if you as a company, consumer, or other entity, choose proprietary software or some other model, I won’t hate on you. What gets me angry though is when people wrap themselves in the veil of openness, ride the coattails of open software, pretending to be something they are not, and fool customers into getting locked in.  Companies that use open source as their gateway drug to get your applications completely dependent on paying them forever or risk massive costs migrating away.

Let Me Give You an “Outside of Tech” Example

My father in law worked on the line for GM for over 30 years. One of the things he really enjoyed when he was not working was golfing.  He traveled the state he lived in and tried all kinds of courses, but there was one, in particular, he really liked.  It was a resort in northern Michigan that had four golf courses with plans to add a couple of more.  He decided to plan his retirement by buying a lot on one of the soon to be built golf courses after painstaking research. He was going to build his dream house and spend his retirement golfing.  He was so thrilled about the plans he would drive me up and force me to trudge into the woods to show me his future spot.

A few years later as he was nearing retirement the resort announced that they would no longer be building the extra courses and would not invest in the infrastructure to build the homes/community where he had invested.  My father in law, who invested so much in this, was left with worthless land in the middle of nowhere.  Sure, he could retire somewhere else, but he invested here and now there is a terrible cost to try that again.

It’s similar to the current trend in open source.  You adopt open source because you have options, you can get things started easily, and if you love the product you can expand.  You do your research, you test, you weigh your options, finally, you commit.  You invest in building out the infrastructure using the open products, knowing you have options. Open source allows you to be agile, to contribute, to fix, and enhance.  Then, one day, the company says, “Sorry, we are now only sort of open and you have to play by the new rules.”

The Impact of Financing

So what the hell is happening? A few things actually.  The first is that open source companies scored some big exits and valuations over the last 15 years. Everyone loves a winner. The rush to emulate the success of those early companies and models was viewed by many entrepreneurs and investors as the new digital gold rush. Let’s look at this from an investor’s point of view for a second:

  • Free labor from the community = higher profit margins
  • Free advertising from the community = higher profit margins
  • Grassroots adoption and product evolution = better products and higher profit margins
  • Community-led support = better profit margins
  • People adopt, then they need help or features … so we can sell them a better more stable version (open core) = better profit margins

So investors poured in.  Many open source startups were started not because the founders believed in open source, but it was a quick and very popular way to get funding. They could potentially gain a huge audience and quickly monetize that audience. Don’t take my word for it, the CEO of MongoDB Inc admits it:

“[W]e didn’t open source it to get help from the community, to make the product better. We open sourced as a freemium strategy; to drive adoption.” – MongoDB CEO Dev Ittycheria

If you look at their publicly available Wall Street analyst briefings, they talk about their strategy around “Net Expansion” and their model is predicated on getting 20-25% expansion from their existing customer base.  Unless you spend more the model breaks.  The stock price continues to rise and the valuation of the company continues to grow despite not being able to be profitable.  In the stock market today growth – delivered by getting more users, beating earnings, squeezing your customer base, and locking you in – is rewarded with bigger paydays. Again let’s pull a quote, this time from a Wall Street analyst:

In Billy Duberstein’s article, If You Invested $1000 in MongoDB’s IPO, This Is How Much Money You’d Have Now, he wrote:

 “…the database is a particularly attractive product, even by enterprise-software standards, because it stores and organizes all or part of a large corporation’s most important data. If a company wanted to switch vendors, it would have to lift all of that data from the old database and insert it into a new one. That’s not only a huge pain; it’s also terribly risky, should any data get lost. Therefore, most companies tend to stick with their database vendor over time, even if that vendor raises prices. That’s how Oracle became such a tech powerhouse throughout the 1990s.”

Essentially, database companies are a good investment because, as well as storing your most important data, once your data is captured it’s painful and risky for users to switch. While this is great for investors, it is not always good news for enterprise customers or any consumers.

Making the Right Decisions for Your Customers

This brings in the debate around retention, expansion, and stickiness. Retention in a pure open source business is difficult. I won’t lie about it.  You have to continually up your game in support, services, features, and tooling.  It is hard!  Really hard!  Because in a pure open-source model, the user can always choose to self-support.  So your offering has to be so compelling that companies want to pay for it (or they view it as “insurance”).

This has led to a discussion that occurs at every open source company in the world.  How do we generate “stickiness.”  Stickiness is the term given to the efforts around keeping paying customers as paying customers.  Here’s an example: MySQL AB had a problem with the “stickiness” of MySQL Enterprise, so they developed the MySQL Enterprise Monitor.  People who used this tool loved it, so they kept paying for MySQL Enterprise. This approach is the right one, as it helps customers that need functionality or services without penalizing the community.

Open core is another form of “Stickiness” – If you want to use these features you have to pay. While stickiness is not always bad, if you have such an awesome product and service offering that people want to pay you, then great. However, oftentimes this stickiness is basically vendor lock-in. It’s a fine line between creating compelling and awesome features and locking your customers in, but that line is there.

Ask yourself, if you wanted to do this yourself without paying the licensing fees how hard would it be?  If you would have to rearchitect or redesign large portions of your application, or you need to reinstall everything, you are locked in. In other words, companies taking this approach do something unique that the community cannot replicate. If it would take additional time and resources it is a value-added service that makes you more efficient.  Using the MySQL Enterprise monitor example, you can monitor MySQL without this specific feature, it just was a value add on that made it easier and more efficient. It created stickiness for the product but without lock-in.

Companies start to be focused on shareholder value and increasing the stock price at all costs.  The mantra I have heard from CEOs is, “I work for the shareholders, I do what’s best for them.”  In many areas, this conflicts with open source communities and culture.

When I work on an open-source project I view it like I am part of the project or part of the team, even if I am not paid.  I am contributing because I love the product, I love the community, and I want to make a difference. As a user of open source, my motivations tend to be a bit different.  I want software that is evolving, that is tested by millions around the world. I want the option and freedom to deploy anywhere at any time. I want incremental improvement, and I only want to pay if I need to.

This is where there is a disconnect between shareholders and the community and users comes in.  If you’re looking at the priority list, shareholder value in many of these companies will come at the expense of the needs of the community or the users.  Peter Zaitsev founded Percona in 2006 because MySQL AB started that shift to shareholder first, and it deeply upset him.

When Circumstances Change

MongoDB started as a shareholder value first company, focused on the “freemium” model.  I think Elastic falls into a second category of companies.  What are some of the characteristics of this second category?

These companies start off as open.  They do build the company on free and open principles and they put users and community as a higher priority.  But, as they take funding or go public, the demands of revenue, profit, increased expansion by shareholders get louder and louder.  These companies start bringing in industry veterans from big companies that “know how software companies should be built.” Oftentimes these executives have little or no background in open source software development or community development.

Executive pedigree matters a lot in valuation and Wall Street acceptance, so coming from a multi-billion dollar company makes a huge difference.  Every hire a company makes changes culture. These companies end up not only having external shareholder pressure but also pressure from the executive management team to focus on increasing profits and revenue.

Keep in mind that this shareholder pressure is nothing new. This is how “Big” business tends to work.  In the 1970s and 1980s, American car companies sacrificed quality and safety for better margins. In the 90s and 00s, airlines reduced seat size, eliminated “amenities” like free baggage and magazines, and added charges for things that used to be included.  Sacrificing the benefits and features of the users or customers to make more money is nothing new.  The pressure to get more growth or revenue is always there. In fact today we can often see companies “Meet Earning Expectations”  or in some cases beat expectations – and still lose value.

As I outlined above, we are now in an environment where the expectation is to beat revenue estimates, continue to show massive growth, and focus on exceeding the expectations of your shareholders. Outside risk factors that could cause volatility or risk slowing growth need to be addressed. For many database companies, the cloud is both a massive risk as well as a massive opportunity to deliver shareholder value.  This is especially true for companies that are not profitable. Wall Street is banking on eventual profitability just like eventual consistency.  If you look at the margins and spending from both MongoDB and Elastic, they are losing money.  They are spending on sales and marketing to capture a larger market share which they hope will lead to expansion dollars and then profitability.

Why Is This Market Special?

Database vendors have been calling out cloud vendors for a couple of years.  The big complaint has been that cloud vendors take advantage of open source licensing by offering “Database as a Service” using their software without paying anything back.   The open nature of some licensing puts cloud providers completely within their rights to do this.  Similarly, however cloud providers do offer other open source software (like Linux) through their services without the same sort of level of blowback.

In the case of MongoDB and Elastic (and others I am sure), it is not a simple matter of database providers withering and dying due to the stranglehold of the cloud.  In fact, they probably make more money from the cloud being available than they would without it.  More than half the open source databases we support here at Percona are in the cloud. Coupled with the fact that both MongoDB and Elastic are growing their quarterly and yearly revenue, it hardly seems like they are on the ropes.  So why the angst?

A couple of theories:  first, the loss of “Potential Revenue” and seeing the money and revenue the other guys are making and you are not getting any of it.  Second, back to growth. Increasing shareholder value and getting to the magic break-even number is job #1.  If they can win or claim the business other vendors have, they can accelerate that growth. As mentioned, both companies are overspending for growth now, so the faster they can reach profitability the better.

In my opinion, the cloud is the convenient excuse here for trying to accelerate growth and increase shareholder value. “If the cloud played fair, our growth would be 40% instead of 30%! Think of how much more our stock would be worth!”

Let’s not lie about it.  Companies are switching licensing not for the good of the community or their users but for the good of their bottom line and in an effort to improve their true overlords:  Shareholders and Investors.  There is nothing wrong with a for-profit company focused on shareholders. Just don’t lie about it and pretend to be something you are not.  In my opinion, open source driven by an active community of contributors, users, and inventors leads to better quality, faster innovation, and more freedom.

The user base, the contributors, and the “community” as a whole got the most successful open source companies where they are today. Communities are built on trust and goodwill.  Open Source is about innovation, freedom, and collaboration.  These recent changes hurt all three.  They turn what were iconic open source companies into just another company focused on the shareholder over the users, customers, and community that got them to where they are.

I will leave you with this.  As a user or a paying customer of any product, but certainly, as an active member of a community, my position is built on trust.  I may not agree with all changes a community, project, or company makes but I want them to be open, honest, and transparent.  I also want companies to show me with consistent actions, and not tell me in empty words, how I will be impacted.  It is very hard to trust and build respect when you say one thing and do another.

Using Elastic as an example here – and thanks to Corey Quinn on his Twitter feed for pointing this out –  over a year ago Elastic changed their licensing to a more open core model.  At the time they told us this via Github release notes:

Github elastic

The last time they made changes that upset the community, they promised they would leave these products open.  A year later and the license changed.  This leaves those users and community members of the product wondering what will change next?  Can we trust that this is it?  Can we trust when they say really SSPL is “just as good”?  And that they won’t move the line of what is acceptable in the future?

I am going to stand up (as many of you are) for more innovation, more freedom of choice, and certainly more truly open software!

Jan
13
2021
--

Percona 2020 Recap: Great Content and Software Releases

Percona 2020 content and releases

Percona 2020 content and releasesThe Percona team provided the community with some excellent content and several new releases in 2020. I wanted to highlight some of your favorites (based on popularity) if you missed them.

First up is our most-read blog from last year, which ironically was published before 2020. Ananias Tsalouchidis’s blog on when you should use Aurora and when should you use RDS MYSQL continued to attract readers all year long. People don’t always understand the key differences between the two, so having a guide is great and timely for many.

What about the most read blogs or watched videos published in 2020?

PostgreSQL Takes Our Most-Read Spot of 2020

The Percona blog is known for its great in-depth MySQL coverage, but experts in the MongoDB and PostgreSQL space have also written some quality content over the last few years. It is exciting to see that the most popular blog published last year was outside of MySQL: Ibrar Ahmed’s deep dive into handling null values in PostgreSQL.

Interested in the top six PostgreSQL reads from 2020? Here they are:

We also had some fantastic conference talks this year you may want to check out. Here are the most-watched PostgreSQL videos of 2020:

Awesome PostgreSQL talks and blogs from the community:

Our Percona University Online posted its first PostgreSQL training last year; if you are looking for a deeper understanding of indexes (and who isn’t), check out our training, Deep Dive Into PostgreSQL Indexes.

MySQL is Still as Popular as Ever

Even though PostgreSQL took this year’s top spot, not too far behind was a great blog series by our CEO Peter Zaitsev on solving MySQL bottlenecks. His three-part series, 18 things you can do to remove MySQL Bottlenecks caused by high traffic, was not only highly read, but it also spawned one of the most-watched webinars of the year. Scalability and performance are critical to any application and can mean life or death for any application. A vital read and a great link to bookmark for when you have one of those odd performance issues you can not seem to find!

Interested in the top five MySQL reads from 2020? Here they are:

Interested in watching some outstanding MySQL sessions? Check out some of the most-watched MySQL sessions of 2020:

Awesome MySQL talks and blogs from the community:

Our Percona University Online posted its first MySQL training; if you are looking at how to upgrade to MySQL 8, it is worth watching. Check out the training, How to Upgrade to MySQL 8.0.

The Staying Power of MongoDB is Undeniable

MongoDB growth in 2020 was undeniable, which is why it’s no surprise that another one of our top blogs was on MongoDB. Percona most-read tech blog on MongoDB published in 2020 was Vinicius Grippa’s must-read work outlining the best practices for running MongoDB. If you are new or old to MongoDB, it is worth reading and double-checking to ensure you have MongoDB optimized.

Interested in the top five MongoDB reads from 2020? Here they are:

Interested in watching some MongoDB sessions? Check out some of the most-watched MongoDB sessions of 2020:

Awesome MongoDB talks and blogs from the community:

More Popular Blogs and Discussions

Sometimes topics cross databases and delve into general advice. Let’s look at some of the more popular talks and blogs that are not tied to a specific database.

If you like videos, you may want to check out these great Percona Live Sessions from last year:

Other Popular Blogs:

Finally, Some Great Percona Software Released This Year

Here is the list of interesting software changes and news on Percona software in 2020:

Percona Distributions for MongoDB and MySQL:

  • What are Percona distributions? We take the best components from the community and ensure they work together. This way, you know your backup, HA, monitoring, etc., will all work together seamlessly.

Percona XtraDB Cluster 8.0 (PXC) was released, with improved performance, scalability, and security. Long sought after features include:

  • Streaming replication to support larger transactions
  • More granular and improved logging and troubleshooting options
  • Multiple system tables help find out more about the state of the cluster state.
  • Percona XtraDB Cluster 8.0 now automatically launches the upgrade as needed (even for minor releases), avoiding manual intervention and simplifying operation in the cloud.

Percona Distribution for PostgreSQL 13. Version 13 of PostgreSQL was a leap forward, and our distribution was updated to support all the additional functionality. Better indexing, better performance, and better security! Sign me up!

Percona Monitoring And Management (PMM) jumped forward from 2.2 to 2.13 adding some very cool features like:

  • Alert manager integration and integrated alerting
  • A brand new Query Analyzer with awesome features to allow you to find problem queries quicker and more efficiently
  • Enhanced metrics for AWS RDS monitoring
  • Added support for External Exporters so you can monitor 3rd party and custom services through the installed PMM-agent
  • New security threat tool allows for alerts and visibility into the most common security issues
  • Support for group replication
  • Better MongoDB and PostgreSQL monitoring
  • Better support for larger environments (Monitor More Stuff Faster)
  • Plus a ton of misc small enhancements!

Percona Kubernetes Operator for Percona XtraDB Cluster continued to evolve with several new features helping users build their own DYI DBaaS:

  • Auto-Tuning MySQL Parameters
  • Integration with Percona Monitoring and Management
  • Full data encryption at rest
  • Support for Percona XtraDB Cluster 8.0
  • Support for the latest version of Open Shift and Amazon’s Elastic Container Service
  • Dual support for ProxySQL and HA Proxy
  • Automated minor upgrades
  • Clone backups to set up a new PXC cluster on a different Kubernetes cluster

Percona Kubernetes Operator for Percona Server for MongoDB added several features, including:

  • Support for Percona Server for MongoDB 4.4
  • Automated management of system users
  • Support for the latest version of Open Shift and Amazon’s Elastic Container Service
  • Automated minor upgrades

While 2020 was far from the best year for many of us and we are glad it is behind us, it did generate some good content that we can use in 2021 and going forward to help us better manage and run our databases. Thanks for reading and happy database tuning!

Jan
07
2021
--

TAM Enterprise Experiences – Data Encryption

TAM Enterprise Experiences – Data Encryption

TAM Enterprise Experiences – Data EncryptionIn previous TAM Enterprise Experiences posts, we have outlined typical aspects of utilizing MySQL in an Enterprise environment. One thing we have not yet covered is the topic of database encryption, both from the standpoint of business requirements as well as some of the more technical aspects of encryption.

In this post, we will cover:

  • Common enterprise compliance requirements
  • Types of MySQL encryption
  • Choosing the right encryption
  • Vault

Common Compliance Requirements

Beyond the obvious security concerns with sensitive data, most enterprise businesses also need to meet various compliance requirements, with the compliance requirement(s) dependent on the country the business is located in, the type of business, and the type of data being stored. Note that in all cases, the onus is on the business to protect the data based on these compliance requirements. Some of the more common enterprise compliance requirements are:

  • GDPR
    • Applies to businesses located within the EU.
    • 32(1) of the General Data Protection Regulation to implement appropriate technical and organizational measures to secure personal data. 
    • The GDPR deliberately does not define which specific technical and organizational measures are considered suitable in each case, in order to accommodate individual factors.
    • Source: https://gdpr-info.eu/issues/encryption/
  • HIPPA
    • Applies to the medical industry within the United States. 
    • The HIPAA encryption requirements for transmission security state that covered entities should “implement a mechanism to encrypt personal health information (PHI) whenever deemed appropriate.
    • Source: https://www.hipaajournal.com/hipaa-encryption-requirements/
  • PCI DSS
    • Applies to protect monetary transactions.
    • PCI encryption Requirement 3 of the Payment Card Industry’s Data Security Standard (PCI DSS) is to “protect stored cardholder data.” 
    • The public assumes merchants and financial institutions will protect data on payment cards to thwart theft and prevent unauthorized use.
    • Encryption of cardholder data with strong cryptography is an acceptable method of rendering the data unreadable in order to meet PCI DSS Requirement 3.4.
    • Source: https://www.pcisecuritystandards.org/pdfs/pci_fs_data_storage.pdf

Outside of compliance, there are of course other very critical reasons for an enterprise business to encrypt and protect data. A breach of security could result in a major negative business impact at best, and complete ruin at worst. Protecting business secrets from the competition, as well as an overall ethical and moral responsibility to protect information and data are other reasons that data security and encryption should always be taken seriously, regardless of business size or industry.

MySQL Encryption

There are several types of MySQL encryption:

  • Encryption At Rest
    • Full disk encryption
    • Encrypted database files
  • Encryption In Transit
    • TLS + Enforcement of SSL for TCP/IP user and replication accounts
    • Use of a UNIX SOCKET connection instead of the TCP/IP mysql connection
  • Encryption In Use
    • Applications encrypt data before storing it and decrypt it once retrieved.
    • The application takes responsibility for data security.

Choosing the Right Encryption

No matter the circumstances, at a bare minimum, encryption in-transit should be utilized to protect data in flight. All replication accounts and all user accounts should be using TLS + enforcement of SSL.

At some point in the future perhaps MySQL will mature to the point where in-use encryption won’t need to be handled by the application and the last bullet point can be dropped from the list. For now, however, the use of debuggers like strace can give access to the unencrypted data in memory on the MySQL server. Adding in this additional layer of application encryption can ensure that data in memory on the database server is encrypted and protected.

Encryption at the Volume/OS/Block Level

Strengths:

  • Simple to encrypt the volume or disk
  • MySQL isn’t aware of any change
  • The application isn’t aware of a change
  • Inexpensive

Weaknesses:

  • Doesn’t protect from insider threats
  • Centralized key storage and compliance can be problematic

Encryption at the Database Level

Strengths:

  • Protects from insider threats
  • Can encrypt across volumes you don’t control
  • Backups and restores are automatically encrypted
  • Lower overhead (3-5% performance hit)
  • DBA controlled
  • Centralized key storage
  • Compliance ready

Weaknesses:

  • Still vulnerable to in-memory attacks
  • More setup / complications
  • Loss of keys would be catastrophic

Encryption at the Application Level

Strengths:

  • Database servers are protected at all levels automatically since the data is unusable without decryption
  • Most flexible
  • Little to no overhead on databases

Weaknesses:

  • Many applications are not built with this in mind and are not easy to change
  • Full text and partial text search can be a problematic
  • Application shoulders all the responsibility for key security

What is Hashicorp’s Vault?

Hasicorp’s Vault is software for securely managing secrets. In this case, a secret is anything you want to tightly control access to, such as API Keys, passwords, or certificates. Vault is flexible with administration and can be controlled via a web GUI or the command line. There is also a strong API using curl with various ways to authenticate, and Hashicorp regularly pushes out updates to Vault keeping it up to date.

Vault Strengths:

  • One easily managed, centralized location for all keys
  • No backups of keyring file needed
  • Better security as the key is nowhere on the MySQL server itself
  • Powerful auditing capabilities

Conclusion

In a perfect world, your data would never be at risk and would always be protected. In reality, however, it is up to the data owners/administrators to protect their data using the methods outlined above.

Jan
07
2021
--

PostgreSQL: DB-Engines.com’s Database of the Year

PostgreSQL database of the year

For the third time in the last five years, PostgreSQL took home the Database of the Year award from DB-Engines. If you have not checked out DB-Engines before, they track databases’ popularity by looking at job openings, blog posts, search traffic, and more.

PostgreSQLCongratulations to the PostgreSQL community, its contributors, and its users! There is no doubt that over the last five years, PostgreSQL has not only provided a wonderfully active and welcoming community but has developed a growing install base based on its in-depth features, accessibility, and openness. Independently, our own Percona Open Source Data Management Software Survey and the Stack Overflow 2020 Developer Survey both showed growth in the PostgreSQL install base in the past year.

Why Are More People Choosing PostgreSQL?

A few of the more popular reasons companies are moving to PostgreSQL are:

  • PostgreSQL is fully open; there is no single entity behind it, making it a truly open source project.
  • Licensing makes this easy to put into place, embed, and modify as needed.
  • It has a well fleshed-out and complete stored procedure language.
  • There is an easier path than most for classic database developers and DBAs to learn and become productive.
  • There are lots of enterprise extensions and options available.
  • An extensive list of options for support, services, and DBaaS is available in the marketplace.

These make PostgreSQL a popular choice for legacy applications, companies leary of lockin, and Oracle or Microsoft SQL Server DBA teams.

PostgreSQL Continued Evolution

In 2020, PostgreSQL released its 13th version, which continued and enhanced its evolution toward a stable, secure, and performant database. Some of the highlights include:

A Wonderfully Welcoming Community

PostgreSQL is not just a great piece of software; it is an incredible and welcoming community. Percona started offering support, services, and software around PostgreSQL in 2018. Over the almost three years we have officially provided services, we have seen just how welcoming and remarkable the entire PostgreSQL community is. Thank you to all the contributors of code, content, and time in the community. You have helped make PostgreSQL even better! We would not be here today without you all.

Jan
05
2021
--

TAM Enterprise Experiences – Troubleshooting Methodology

TAM Troubleshooting Methodology

TAM Troubleshooting MethodologyOne of the great things about being a Technical Account Manager (TAM, for short) at Percona is having the opportunity to see a wide variety of issues across many clients, and having the space to identify the common threads that seem to bind many of these issues. While the challenges vary as widely as the troubleshooting methods themselves, being able to step back from a technical issue and look in from the outside at the whole picture has proven to be incredibly useful. No matter the complexity of the problem, troubleshooting is often best served by going back to the simplest of questions. A broader, more holistic view can allow honing in on the actual problem being presented, and in many cases, you may discover that the wrong questions are being asked, at least initially.

In this TAM Enterprise Experiences blog post, I’d like to outline a few helpful suggestions to help simplify troubleshooting and triage by utilizing a “big picture” mentality when working through a given issue.

What’s Your Problem?

The first question to ask is what specifically is the problem being addressed? Oftentimes this isn’t as obvious as you might first think. Sometimes the way the problem was described is misleading, and it is helpful to confirm and verify exactly what problem is being solved.

For instance, a common theme maybe for an application team to report to the database team that the database has become slow and sluggish, impacting the overall user experience. This seems straightforward, and at first, it seems the problem has already been identified. However, has the database actually slowed down based on historical metrics or has application traffic increased to the point where the existing architecture can no longer keep up?

Both of these options could potentially have the same symptoms while having drastically different causes and solutions. Accurately defining the problem will ensure you don’t end up on a wild goose chase and can get straight to work on addressing the actual issue.

First Time?

Once the problem has been defined, the next question you should ask is, “has this happened before?” As is often the case in this industry, many problems are recurring, so knowing if you have faced this issue in the past and how it was dealt with then is critical information to have.

Answering this question can often depend on documenting any institutional knowledge within your organization, to be sure that any issues that are resolved are well documented and defined for future recurrences. This is often done within a ticketing system (JIRA, ServiceNow, Zendesk), and while that does work I would still recommend writing more formal internal knowledge-base articles for any major issues or changes, and making sure that these articles are kept up to date and searchable.

It Was Fine Yesterday

When was the last time the system was behaving normally? If this is an ongoing issue, knowing when the problem started is critical in determining what changed that could have caused problems. If you’re unsure exactly when the issue happened, how can you determine if a recent code release might be causing the issue? How would determine if any recent schema or user changes happened immediately prior to causing the issue? Without precise timing, how would you determine if traffic patterns during the event are consistent with historical norms? Knowing when an issue occurs will let you examine the puzzle pieces on either side to get a better picture of what was happening at or near that time.

As you might imagine, answering this question accurately is made much simpler with good historical monitoring in place. With a tool such as Percona Monitoring and Management (PMM), it is often a simple matter to review the metrics and see visually the exact moment when things went wrong.

The Times, They Are a-Changing

Now that we know when the issue occurred, you should next ask, “what changed prior to this happening?” A very common occurrence is for a production code change or some other system or OS change to negatively impact the database, especially if robust change and usage testing weren’t done beforehand. Databases don’t exist in a vacuum, so a problem occurring out of thin air is not something that commonly occurs. In nearly all cases, you’ll find a change was made that either directly caused the issue, or at least contributed to it in some way, possibly exacerbating a less impactful problem to priority status.

Knowing what has changed will also be critical to assist in finding the root cause of an issue after it has been triaged, as often the change itself will be the root cause. If it is not the cause directly, knowing what changed will at least ensure that you aren’t looking in the wrong places and put you on track to quickly and confidently documenting the cause.

May I Have Another?

The next question that you may want to tackle is determining if the issue is repeatable. We now know what the issue is, what has been done previously, when the problem actually happened, and what changes were made prior to the issue.  Can you now identify the exact circumstances that led up to the issue, and detail/test this scenario so that a reproducible test case is possible? This can be important, as having a reproducible test case will make any metrics collection during the actual issue relevant, whereas if this is a one-time issue, gathering relevant metrics will be difficult if not impossible.

This is also critical if you determine you need to pull in additional assistance for a given problem. Perhaps you have vendor support, or you engage a third-party support partner for your database environment (go Percona!). In these cases, being able to tell the support partner how to reproduce the issue will allow them to begin troubleshooting directly, and avoid the back and forth walls of text attempting to describe the issue in a ticket without a reproducible test case.

Wrapping Up

While it may seem counterintuitive, it is very helpful to step back and ask these simple questions before you begin work on triaging an issue, regardless of the complexity. Perhaps you do already know the answer to all of these questions, and in that case, you’ve at least confirmed your suspicions before proceeding with no harm done. In many cases, however, stepping back to ask these simple questions can reframe an entire conversation highlighting what is actually relevant and what is just background noise, saving both time and effort.

Percona is a leading provider of unbiased open source database solutions that allow organizations to easily, securely, and affordably maintain business agility, minimize risks, and stay competitive. To put Percona’s expertise to work for you, please contact us.

Dec
15
2020
--

Matt’s Big, Bold, Open Source Predictions for 2021

2021 Open Source Predictions

2021 Open Source PredictionsCovid-19 has had a significant impact on most aspects of our lives in 2020. Although some companies saw unexpected growth due to offering in-demand tools or services, I think most of us will be happy to draw a line under 2020 and hope for a better 2021. 

In the spirit of 2021 optimism, I have been considering the current state of the open source database market and putting together my Big Bold Open Source Predictions for the next 12 months. 

Big Bold Prediction One:

database expertI believe reliance on classic operational roles such as sysadmins, database administrators (DBAs), and web admins will continue to shrink with the drive towards “easy hosting” and “as a service” models for running databases.  

We will continue to see the sysadmin, DBA, and web admin roles evolve into site reliability engineers (SREs) and specialists instead. We will also see architects, developers, and non-infrastructure experts continue to pick the backend stacks. This furthers what I call, the technology inheritance problem, where you have to run and support legacy tech stacks. While they might have originally been picked for good reasons, those reasons might not still apply today.

As these changes accelerate, the demand for specialized expertise on how to build, design, and architect databases will grow. The “application” database expert will be an in-demand role in the future. This will force businesses to fill the gap with consultants or on-staff expertise, or risk increased costs and lock-in over time.

Big Bold Prediction Two:

not so openOpen source software has ‘won!’ Most new software projects will include at least some open source software components, and the majority of new releases will take place under open source licenses. However, at the same time, there is a reduction in the level of openness taking place. Classic software infrastructure vendors will accelerate putting additional controls in place around how their open source projects can be used, but cloud providers will race to embrace a more open model. 

It’s weird that companies like Amazon are now releasing more open source, while traditional open source vendors like MongoDB, Elastic, Redis, and others are introducing more restrictive licenses and “as a service” offerings that actively lock people in. Many database companies’ newest and most innovative features are starting to be reserved as “DBaaS Exclusives”.

Ironically the push to retain market share by so-called “open source” vendors will actually drive people away from their platforms. Look back at Oracle in the ’90s and ’00s… Their introduction of price hikes, lock-ins, and other restrictive and expensive methods, forced people to look for alternatives. The good news here is that we may be on the verge of a new wave of disruptive innovation to deal with this trend.

Non-open “open source” sounds like an oxymoron, but it is really becoming a thing! At Percona we believe in true open source, not in versions that are crippled by design and act as a gateway drug to get you hooked, unable to escape, and filled with regret.

Big Bold Prediction Three:

open source trapConsumers and users will revolt against these open source shenanigans. Docker and Red Hat are good examples here – they have taken open source projects or products and then limited them in order to get more people to upgrade to the “Enterprise” version. This then in turn leads to potential lock-in.

Docker started playing games, locking people out of downloading images on Docker hub, and trying to generate a business model that forced people/vendors to pay for access. Red Hat has changed its approach to CentOS from a fixed release model with long term support, through to a streamed model with no such certainty.

Where will this lead in 2021? People will find alternatives. Kubernetes is depreciating support for Docker and vendors are stepping up with alternative catalogs. A fork of CentOS is already available with Rocky Linux. 

Big Bold Prediction Four:

cloud databaseDatabases will continue to move towards a run-anywhere-on-anything reality. Users want DBaaS type solutions, but want to avoid sole vendor lock-in. They are considering technologies like Kubernetes to manage this. 

Enterprise companies (Nutanix for instance) are looking to bridge database deployments across cloud providers and data centers. Open source companies like Red Hat are pushing the option of running your databases like you are already running your application, in a cloud-native way.  

Of course, all cloud providers are now digging into the multiverse, with Google Anthos, Azure Arc, and AWS EKS available anywhere. At AWS re:Invent 2020, they announced the decision to allow companies to sell professional services via AWS Marketplace. This signals that even AWS knows that users committing to a single provider is unrealistic and that users will still want advice, consulting, and support from other vendors.  

What makes this even more interesting is that database providers can offer “single database” and “any hosting provider” as a service offering. This means that more friction will materialize between the two groups. 

Eventually, some of the cloud providers will use their market cap and purchase some of the open-source vendors in the space. MongoDB as a Microsoft company? Elastic as an AWS company? Yep! It will happen, maybe not in 2021, but this consolidation is bound to occur as it becomes too tempting to acquire resources to create value.

Alongside all this, getting true open source DBaaS to market will be important. Users want the flexibility of running their databases in the cloud, but they will also want an easy route out of using a particular cloud service if they choose to move. For cloud providers, facilitating this open approach will allow them to meet the needs of their customers, and ensure the relationship is based on mutual respect.

Big Bold Prediction Five:

dataIncreasing innovation will come from the data management space. There is simply too much data, too many applications, and too much sprawl to not necessitate innovation in this space.  

We have seen rapid growth in the number of technologies developed to deal with the increase in databases and data technology. For instance, the growth in the troubleshooting and observability space over the last five years has been unprecedented. Over the next twelve months, this will continue to develop rapidly.

No company ever says; “I want to have less data.” Instead, technology teams will need to ensure they can get value out of their data more efficiently, to justify the cost of managing all that data over time. Optimizing this spend, and how data is stored, will go hand-in-hand.

So, these are my Big Bold Open Source Predictions for 2021! Do you think these are on track, or do you think the open source database market is going to move in another direction?

Please share your thoughts below and let us know what you think is in store over the next 12 months!

Nov
17
2020
--

Tame Black Friday Gremlins — Optimize Your Database for High Traffic Events

Optimize Your Database for High Traffic Events

Optimize Your Database for High Traffic EventsIt’s that time of year! The Halloween decorations have come down and the leaves have started to change and the Black Friday/Cyber Monday buying season is upon us!

For consumers, it can be a magical time of year, but for those of us that have worked in e-commerce or retail, it usually brings up…different emotions. It’s much like the Gremlins — cute and cuddly unless you break the RULES:

  1. Don’t expose them to sunlight,
  2. Don’t let them come in contact with water,
  3. NEVER feed them after midnight!

I love this analogy and how it parallels the difficulties that we experience in the database industry — especially this time of year. When things go well, it’s a great feeling. When things go wrong, they can spiral out of control in destructive and lasting ways.

Let’s put these fun examples to work and optimize your database!

Don’t Expose Your Database to “Sunlight”

One sure-fire way to make sure that your persistent data storage cannot do its job, and effectively kill it is to let it run out of storage. Before entering the high-traffic holiday selling season, make sure that you have ample storage space to make it all the way to the other side. This may sound basic, but so is not putting a cute, fuzzy pet in the sunlight — it’s much harder than you think!

Here are some great ways to ensure the storage needs for your database are met (most obvious to least obvious):

  1. If you are on a DBaaS such as Amazon RDS, leverage something like Amazon RDS Storage Auto Scaling
  2. In a cloud or elastic infrastructure:
    1. make sure network-attached storage is extensible on the fly, or
    2. properly tune the database mount point to be leveraging logical volume management or software raid to add additional volumes (capacity) on the fly.
  3. In an on-premise or pre-purchased infrastructure, make sure you are overprovisioned — even by end of season estimates — by ~25%.
  4. Put your logs somewhere else than the main drive. The database may not be happy about running out of log space, but logs can be deleted easily — data files cannot!

Don’t Let Your Database Come in “Contact With Water”

We don’t want to feed or allow simple issues to multiply. Actions we take to get out of a bind in the near term can cause problems that require more attention in the future — just like when you put water on a Gremlin, it will multiply!

What are some of these scenarios?

  1. Not having a documented plan of action can cause confusion and chaos if something doesn’t go quite right. Having a plan documented and distributed will keep things from getting overly complicated when issues occur.
  2. Throwing hardware at a problem. Unless you know how it will actually fix an issue, it could be like throwing gasoline on a fire and throw your stack into disarray with blocked and unblocked queries. It also mandates database tuning to be effective.
  3. Understanding (or misunderstanding) how users behave when or if the database slows down:
    1. Do users click to retry five times in five seconds causing even more load?
    2. Is there a way to divert attention to retry later?
    3. Can your application(s) ignore retries within a certain time frame?
  4. Not having just a few sources of truth, with as much availability as possible:
    1. Have at least one failover candidate
    2. Have off-server transaction storage (can you rebuild in a disaster?)
    3. If you have the two above, then delayed replicas are your friend!

Never “Feed” Your Database After “Midnight”

What’s the one thing that can ensure that all heck breaks loose on Black Friday? CHANGE is the food here, and typically, BLACK FRIDAY is the midnight.

Have you ever felt like there is just one thing that you missed and want to get off your backlog? It could be a schema change, a data type change, or an application change from an adjacent team. The ‘no feeding’ rule is parallel to CODE FREEZE in production.

Most companies see this freeze start at the beginning of November when the most stable prod is the one that is already out there, not the one that you have to make stable after a new release:

  1. Change Management is your friend; change that needs to happen should still have a way to happen.
  2. Observability is also your friend; know in absolute terms what is happening to your database and stack so you don’t throw a wrench in it (Percona Monitoring and Management can help).
  3. Educate business stakeholders on the release or change process BEFORE the event, not DURING the event.
  4. Don’t be afraid to “turn it off” when absolute chaos is happening. Small downtime is better than an unusable site over a longer period of time.

Conclusion

Black Friday, Cyber Monday, and the Holidays can be the most wonderful time of the year — and now that we’ve covered the rules, some of the “Gremlins” can stay small and fuzzy and your business won’t get wrecked by pesky database issues or outages.

How Percona Can Help

Percona experts optimize your database performance with open source database support, highly-rated training, managed services, and professional services.

Contact Us to Tame Your Database Gremlins!

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com