Nov
11
2019
--

Prepare Your Databases for High Traffic on Black Friday

Prepare Your Databases For High Traffic

Prepare Your Databases For High TrafficIt’s November, so we all know what that means; it’s peak shopping season, and no date is bigger than Black Friday. But how will your database handle all that new, relentless traffic? Not only does your database have to handle traffic without slowing down, but web servers can sometimes see such sudden traffic as an attack; meaning your site(s) could go down completely.

Every year there are news stories about websites that were unresponsive or disappeared completely right at the exact moment when potential shoppers were coming online. Don’t let this be you! To ensure your databases are prepared for a high traffic event, download our Database Disaster Prevention Checklist and read what we advise you do before, during, and after the big day. Let’s get started.

What You Need To Know About High Traffic Events

  1. Your users expect instant response and immediate feedback from your application. If you don’t give it to them, your competition will.
  2. Database slowdowns for you can be perceived as downtime for your customers, and they will lose confidence in your ability.
  3. A few seconds of downtime costs you not only direct revenue but also lost future business. Use our Cost of Database Downtime Calculator to see just how much downtime could cost you.
  4. When you failover a slow system, the slowness follows to the new system. If your slowness alleviates, it may be because your customers went somewhere else when you were down.
  5. Time is finite; you can’t get more of it. You have to make the most of it. It can be much more valuable than money.

Before the Event

  1. Setup monitoring and tooling before the event. If you can’t see what’s going on, how can you measure your success? Percona Monitoring and Management can help with that.
  2. Load test your applications and test how you will scale under normal and peak loads. The best time to find bottlenecks is before the event, as during is costly and impactful.
  3. Test failover and understand how quickly you can recover. The worst time to find out your failover doesn’t work is during the busiest day of the year.
  4. Put a code and configuration freeze in place in advance of the event. It’s really hard to ensure performance if the application is growing and evolving.
  5. Get a second opinion, double-check, and don’t assume. A large portion of your business is tied to this event. Trust but verify things are ready; it’s worth it. Most big outages are caused by easily overlooked things.
  6. Check your backups. Make sure you have reliable and consistent backups of your databases. This will make it easier to restore a crashed database. Percona XtraBackup for MySQL can help.

During the Event

  1. Realize that failing over to another system is the absolute last resort.  Failing over a busy system moves the traffic to a new server. Additionally, most systems are slower when traffic is added as it takes time to warm up the cache.
  2. Have the right people standing by to monitor, tweak, and fix issues before they get out of hand. Too many issues are prolonged by waiting to get the right people in the room to fix issues. Many larger companies make this an all-hands event.
  3. Don’t lose sight of the goal! Getting back up and running and allowing customers access to the site is your goal, not developing a permanent fix in the heat of the moment. Time to make an impact is finite, but equally important people under pressure are more prone to make mistakes.  An easier temporary fix to get you through the day can work in your favor, as long as you don’t forget to make the permanent one eventually.
  4. Don’t make the problem worse. Some activities can cause cascading slowness… know the impact of the changes before you make them. We get a lot of calls from people who had good intentions to try and fix a minor issue but made a much bigger problem. The road to hell is paved with good intentions.
  5. Collect and store the data needed to analyze and improve for the next event. Often when things don’t go right, issues end up being transient and difficult to understand after the fact. Get the data you need, when things are going well or not.

After the Event

  1. Analyze and understand your traffic and usage. Use this data to plan, enhance, and tweak your strategy for future events.
  2. Don’t leave the quick fixes in place and just forget about them, these could escalate at the worst times. Take the time and expense to fix them during slow periods.
  3. Learn from your mistakes and build a plan to mitigate problems and risks in the future.
  4. Update your systems to the latest builds and security fixes. Take advantage of the slower load and catch up after a freeze/blackout period around the event.
  5. Don’t fall into complacency. Congrats that you survived this year, but each application and user base is a living, breathing entity, and what worked last year may not work this year. You have to analyze, plan, and review on a regular basis.

Conclusion

Now you’re better prepared for the database high-traffic days coming your way soon. For more information, learn how to ensure peak database performance for your event and download our Database Disaster Prevention Checklist, or contact us today. Percona’s experts can maximize your application performance with our open source database support, managed services or consulting for MySQL, MariaDB, MongoDB, PostgreSQL in on-premises and cloud environments.

Database Disaster Prevention Checklist

Dec
21
2017
--

Three P’s of a Successful Black Friday: Percona, Pepper Media Holding, and PMM

Successful Black Friday

As we close out the holiday season, let’s look at some data that tells us how to guarantee a successful Black Friday (from a database perspective).

There are certain peak times of the year where companies worldwide hold their breath in the hope that their databases do not become overloaded or unresponsive. A large percentage of yearly profits are achieved in a matter of hours during peak events. It is critical that the database environment remains online and responsive. According to a recent survey, users will not wait more than 2.5 seconds for a site to load before navigating elsewhere. Percona has partnered with many clients over the years to ensure success during these critical events. Our goal is always to provide our clients with the most responsive, stable open-source database environments in order to meet their business needs.

First Stop: Germany

In this blog post, we are going to take a closer look at what happened during Black Friday for a high-demand, high-traffic, business-critical application. Pepper Media Holding runs global deals sites where users post and vote on top deals on products in real-time. To give you a better idea of what the user sees, there is a screenshot below from their Germany mydealz.de branch of Pepper Media Holding.Successful Black Friday

As you can imagine, Black Friday results in a huge spike in traffic and user contribution. In order to ensure success during these crucial times, Pepper Media Holding utilizes Percona’s fully managed service offering. Percona’s Managed Services team has become an extension of Pepper Media Holding’s team by helping plan, prepare, and implement MySQL best-practices across their entire database environment.

Pepper Media Holding and Percona thought it would be interesting to reflect on Black Friday 2017 and how we worked together to flourish under huge spikes in query volume and user connections.

Below is a graph of MySQL query volume for Germany servers supporting the mydealz.de front-end. This graph is taken from Percona’s Managed Service Team’s installation of Percona Monitoring and Management (PMM), which they use to monitor Pepper Media’s environment.

As to be expected, MySQL query volume peaked shortly before and during midnight local time. It also spiked early in the morning as users were waking up. The traffic waned throughout the day. The most interesting data point is the spike from 5 AM to 9 AM which saw an 800% increase from the post-midnight dip. The sustained two-day traffic surge was on average a 200% increase when compared to normal, day-to-day query traffic hitting the database.

For more statistics on how the mydealz.de fared from a front-end and user perspective, visit Pepper Media Holding’s newsroom where Pepper Media has given a breakdown of various statistics related to website traffic during Black Friday.

Next Stop: United Kingdom

Another popular Pepper Media Holding branch is in the United Kingdom – better known as HotUKDeals. HotUKDeals hosts user-aggregated and voted-on deals for UK users. This is the busiest Pepper Media Holding database environment on average. Below is a screenshot of the user interface.

The below graphs are from our Managed Service Team’s Percona Monitoring and Management installation and representative of the UK servers supporting the HotUKDeals website traffic.

The first graph we are taking a look at is MySQL Replication Delay. As you can see, the initial midnight wave of Black Friday deals caused a negligible replica delay. The Percona Monitoring and Management MySQL Replication Delay graph is based on seconds_behind_master which is an integer value only. This means the delay is somewhere between 0 and 1 most of the time. Only once did it go between 1 and 2 over the entire course of Black Friday traffic.

The below graphs highlight the MySQL Traffic seen on the UK servers during the Black Friday traffic spike. One interesting note with this graph is the gradual lead-up to the midnight Black Friday spike. It looks like Black Friday is overstepping its boundaries into Gray Thursday. The traffic spikes here mimic the ones we saw in Germany. There’s an initial spike at midnight on Black Friday and then another spike as shoppers are waking up for their day. The UK servers saw a 361% spike in traffic the morning of Black Friday.

MySQL connections also saw an expected and significant spike during this time. Neglecting to consider max_connections system parameter during an event rush might result in “ERROR 1040 (00000): Too many connections.” However, our CEO, Peter Zaitsev, cautions against absent-mindedly setting this parameter at an unreachable level just to avoid this error. In a blog post, he explained best-practices for this scenario.

The MySQL query graph below shows a 400% spike in MySQL queries during the peak Black Friday morning traffic rush. The average number of queries hitting the database over this two day period is significantly higher than normal – approximately 183%.

Conclusion

Percona reported no emergencies during the Black Friday period for its Managed Service customers – including Pepper Media Holding. We saw similarly high traffic spikes among our customers during this 2017 Black Friday season. I hope that this run-down of a few PMM graphs taken during Pepper Media Holding’s Black Friday traffic period was informative and interesting. Special thanks to Pepper Media Holding for working with us to create this blog post.

Note: Check out our Pepper Media case study on how Percona helps them manage their database environment.

If you would like to further explore the graphs and statistics that Percona Monitoring and Management has to offer, we have a live demo available at https://pmmdemo.percona.com. To discuss how Percona Managed Services can help your database thrive during event-based traffic spikes (and all year round), please call us at +1-888-316-9775 (USA), +44 203 608 6727 (Europe), or have us contact you.

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com