The Criticality of a Kubernetes Operator for Databases

Importance of Kubernetes Operators for Databases

Importance of Kubernetes Operators for DatabasesAs a Solutions Engineer at Percona, one of my responsibilities is to support our customers as they investigate new and emerging technologies. This affords me the opportunity to speak to many current and new customers who partner with Percona. The topic of Kubernetes is becoming more popular as companies are investigating and adopting this technology. The issue most companies are encountering is architecting a stateful database that doesn’t fall victim to an environment tuned for ephemeral workloads. This obviously introduces a level of complexity as to how to run a stateful database in an inherently stateless world, as databases are not natively designed for that.

To make your life easier, as a part of the Percona Cloud-Native Autonomous Database Initiative, our engineering teams have built two Kubernetes Operators: Percona Kubernetes Operator for Percona XtraDB Cluster and Percona Kubernetes Operator for Percona Server for MongoDB, which allows for Kubernetes Pods to be destroyed, moved, or created with no impact to the application. To see an overview of Kubernetes, you can read this previous blog of mine Introduction to Percona Kubernetes Operator for Percona XtraDB Cluster that covers this topic. It’s common for companies new to Kubernetes to attempt to run their databases in Kubernetes the same way they would in a traditional environment. But, this is not advised as it introduces the possibility of data loss and it is not recommended for production workloads. Why is this dangerous and how has Percona solved this?

Appropriate Workloads for Kubernetes

Kubernetes is not the answer for everyone. It’s even not the answer for most people. Do not be misled into thinking that moving a database into Kubernetes is going to solve any of your problems. Before you consider moving your database into Kubernetes, ensure the rest of your application is cloud-native and can be used with Kubernetes. Moving your database to Kubernetes should happen after you have started both elastic vertical and horizontal scale and need to orchestrate it to control costs.

As more companies are moving to Kubernetes something has to happen to the legacy workloads. Oftentimes we see a lift and shift mentality into Kubernetes, which can be dangerous or cause more work than expected. We have seen two primary ideal use cases for moving database workloads to Kubernetes: Microservices and Unified Abstraction Layer.

Monolithic, large datasets can prohibit some of Kubernetes’ strong points: self-healing and availability. This can be an issue due to the time it takes to physically transmit data to a new Pod instance as it joins the database cluster. If your dataset is too large, this process is slow due to physical limitations and prohibits performance and the availability of your database. Microservices are a great fit due to the relatively smaller datasets, which allows Kubernetes automation to work well with the dataset size.

Companies looking to take full advantage of cloud-native applications and databases can be a really good fit for Kubernetes as well. If you truly want the ability to deploy and run your databases anywhere utilizing the concept of a Unified Abstraction Layer, Kubernetes is a great option. You can move your databases to anywhere that is running Kubernetes and know it will work.

We talked about large unsharded datasets and the limitations Kubernetes presents when handling them, but we should mention a few more workloads better suited for traditional platforms. Applications with a throughput sensitivity may not do well on Kubernetes, or they may not be cost-effective to do so. Kubernetes is fundamentally designed for container orchestration and is not designed to handle highly performant databases that require low latency. This may be possible to achieve, but at what cost? This applies to highly performant distributed applications as well. Lowest latency across all nodes is not a core tenant of Kubernetes, so ensure you have planned and tested against this before you move everything over to Kubernetes.

Pods Are Cattle, Not Pets

If you’re not familiar with Pets vs Cattle, it’s a DevOps concept that differentiates deployment methodologies of unique servers that require attention when issues arise (pets) versus the ability to replace a server with a copy if issues arise (cattle). Due to the nature of how Kubernetes operates, Pods can be destroyed, spun up, and moved at any time due to factors outside of the application’s control, much like how cattle are treated. Kubernetes uses a scheduler, which by design, can destroy and recreate Pods to meet the configuration needs of your Kubernetes Cluster. This is great for stateless applications as any failure in the application will result in a Pod containing the application being destroyed and recreated, eliminating the need for human interaction, and greatly speeding up the process to a resolution. This isn’t ideal for databases as you don’t want your database to suddenly stop working, halt the application, and introduce the potential for lost or corrupted data. One of the tools Kubernetes can utilize to help combat this is called Stateful Sets. These help by keeping a Pod’s identity assigned to it as it is destroyed and re-created. This helps facilitate stateful workloads, but how does this come into play with high availability and utilizing the automation aspects of Kubernetes?

Databases Are Pets, Not Cattle

Databases by design need to keep their identity, information, and most importantly their data safe and accessible at all times. They are the backbones of the application as they are the source of truth an application relies on for normal processing. Any errors in their operations will quickly stop an application from functioning. They are important, to say the least. How can we safely run databases in Kubernetes and still ensure we have highly available database deployments? By using Stateful Sets and Persistent Volumes we can maintain data integrity, but we need an additional set of hands to take on database administrator tasks such as ensuring failover happens, database members are recovered, and re-join the highly available architecture, along with other technology-specific functions. Fortunately, Kubernetes is extensible and has Operators, which aims to automate the key task of a human operator who is managing a service or set of services.

Automation, Automation, Automation

We know the complexities of running a database (safely) in Kubernetes and some of the concepts used to help bridge the gaps between automation and traditional human functions. With the help of Percona’s Kubernetes Operators, we can safely run databases the way they were intended to run. Percona’s Kubernetes Operators are able to automate tasks that are usually done by a database administrator such as:

  • Fully automated deployments with strict consistency and no single point of failure
  • Automated scaling with the ability to change the size parameter to add or remove members of a Cluster or Replica-Set
  • Fully automated backups and restores
  • Fully automated self-healing by automatically recovering from the failure of a single Cluster or Replica-Set member.
  • Automatically manage system users on password rotation
  • Simplified updates

Always Use a Kubernetes Operator

With the complexities of running a highly available database environment and the inherent dangers introduced by using the dynamic Kubernetes environment, an Operator should always be used when deploying databases in Kubernetes. Fortunately, Percona has already solved this by providing Percona Kubernetes Operator for Percona XtraDB Cluster and Percona Kubernetes Operator for Percona Server for MongoDB. Percona provides full support for databases running in Kubernetes with the Percona Operators. If you are interested in learning more or obtaining support or professional services to maximize your database deployments, please reach out to us.


Webinar October 12: Percona Server for MongoDB – A Solid and Compatible Alternative to MongoDB

Percona Server MongoDB Italian Webinar

Percona Server MongoDB Italian WebinarThis webinar will be delivered in Italian.

MongoDB is the most used document-oriented database but some of the more requested features are available only in the Enterprise version. Percona Server for MongoDB (PSMDB) is an alternative to MongoDB with its open source, solid, and 100% compatible server. In this webinar, in Italian, you will learn about PSMDB and our Italian financial customer, Fabrick, will discuss its experience with Percona Support Services.

Please join Corrado Pandiani, Sr Consultant, and Maurizio La Plagia Head, of IT PFM & Smart Banking at Fabrick, on Monday, October 12, 2020, at 9 am EDT/ 3 pm CET for the webinar “Percona Server for MongoDB: A Solid and Compatible Alternative to MongoDB“.

Register for Webinar

If you can’t attend, sign up anyway and we’ll send you the slides and recording afterward.


MongoDB 101: 5 Configuration Options That Impact Security (And How to Set Them)

MongoDB Security

MongoDB SecurityAs with any database platform, MongoDB security is of paramount importance to keeping your data safe.  MongoDB and other data platforms like Redis and Elasticsearch are often in the news for data breaches because of misconfigured settings in the database.  So how do you keep you and your company’s data from being compromised and from becoming another statistic?

We’ll show you five configuration options, as well as others that are required to go along with them, for your MongoDB deployment that will help keep your data secure while allowing use by users and applications with least-privileged access using modern authentication methods, keeping your data encrypted on disk and over the wire, and to see who is accessing your data as well. These configuration options are across the following areas in security: authentication, authorization, encryption, and auditing.

MongoDB Security Overview

MongoDB security is composed of four main areas of focus, authentication (who), authorization(what), encryption (how), and auditing (when).  This section is intended to give you a high-level overview of the different security focus areas for MongoDB.

Authentication is how you identify yourself to MongoDB.  There are many ways to authenticate oneself to a MongoDB database, from standard username and password using the SCRAM (Salted Challenge Response Authentication Mechanism) protocol, certificate-based authentication to tying into an identity management solution such as LDAP (Lightweight Directory Access Protocol), Active Directory and Kerberos.

Authorization is how MongoDB determines what you, as an authenticated user, can do.  MongoDB supports authorization using the RBAC (Role-Based Access Control) method.  MongoDB lets you create roles which are groupings of privileges that any user granted that role can do.  MongoDB comes with a comprehensive set of built-in roles as well as giving you the flexibility to create your own custom roles.  Common roles like read-only and write are there of course, but also ones useful for monitoring any node, backup and restore, and user administration. Additionally, MongoDB also supports LDAP authorization which allows you to sync LDAP groups with roles to simplify management.

Encryption is how your data can be encrypted while in flight (Transport) and while on disk (At Rest).   Transport encryption keeps your data encrypted while it is sent to and from your application to MongoDB.  MongoDB supports TLS/SSL encryption for data in-flight using x.509 Certificates, and here’s an example of setting up Transport EncryptionEncryption at Rest keeps your data safe from an external party who might get a copy of your data files as they’ll be completely unreadable in their encrypted form.

Auditing shows you when users connected, when privileges were changed, various admin events, users attempt something they shouldn’t, etc. based on filter criteria you can set.  This is helpful in compliance situations where you have to be able to show what who was on the database at what time, what privileges they had, when privileges were changed, etc.

Tip: Don’t confuse auditing as a way to track users’ activities in real-time, but rather think of it as a way to create a tamper-proof, append-only log file that you can go back to that shows what was happening and by whom during a specific time.

Tip:  Auditing is an expensive operation and will impact performance, be sure that you’re getting value from it and your IT Compliance team is able to actively use it, before setting it up.

Configuration Options

So while knowing the important areas of MongoDB Security is great, how do we ensure they are enabled or set up correctly?  And which ones are the most important?  We’ll now go through 5 configuration options that will help you secure your MongoDB environment!  We’ll also list some required configuration options that will work in conjunction with our 5 most important configuration options to keep your data safe.  We’ll break these configuration options into their security focus areas.

MongoDB uses a configuration file in the YAML file format.   The configuration file is usually found in the following locations, depending on your Operating System:


  • On Linux, a default /etc/mongod.conf configuration file is included when using a package manager to install MongoDB.
  • On Windows, a default <install directory>/bin/mongod.cfg configuration file is included during the installation.
  • On macOS, a default /usr/local/etc/mongod.conf configuration file is included when installing from MongoDB’s official Homebrew tap.


Our first configuration option, security.authorization, is perhaps the most important for enabling security on your MongoDB Deployment.  This configuration option not only enforces MongoDB using authentication so that a user must at least login using credentials but it also simultaneously engages role-based access control which limits what a user can do.

  authorization: enabled


Tip:  If you set this configuration option up before creating a user in MongoDB, you could use the localhost exception in order to create your first user.  This is especially helpful in cases of automation or other situations where you want to have all your configuration options configured only once and then come in and add users.

There are several other authentication configuration options that are required for your MongoDB deployment:

  • security.clusterAuthMode – The authentication mode used between replica set or sharded cluster nodes to authenticate.  This will typically be either keyFile or x509.  Acceptable values are:
    • keyFile – default, only accepts keyfiles
    • x509 – uses only x509 certificates for cluster authentication
    • sendKeyFile – only used when transitioning from keyFile to x509 certificate authentication.   Accepts keyFiles and x509 certificates
    • sendX509 – only used when transitioning from x509 certificate authentication to keyFile authentication.   Accepts x509 certificates and keyFiles
  • security.keyFile – sets the destination of the keyFile if using keyFile based authentication.  Note that the user MongoDB is running as must have read-only or read/write level permissions on the keyfile, with no permissions granted to other users.


The security.authorization configuration option that enabled authentication is also the most important configuration option for setting up authorization since it also gives us roles that allow us to authorize users to have specific permissions.   While there are some authorization configuration options for the inbuilt authorization system, most of the options are useful for integrating LDAP or other external authentication mechanisms with your roles.  Read more about setting up LDAP Authorization, as well as a great blog post discussing how to set it up.


Transport Encryption ensures that your data is encrypted between your application and the database, it also can be used to encrypt data between members of your replica set and sharded cluster.  The most important configuration option here is net.tls.mode. This configuration option is new in MongoDB 4.2, previous to MongoDB 4.2, this configuration option is named net.ssl.mode.   This configuration option decides how strictly you want to enforce TLS encryption.   The options for this configuration option are:

  • Disabled – signifies that there is no encryption whatsoever.
  • requireTLS – signifies that all traffic, regardless of origin, is encrypted.  requireTLS is the most secure setting for this configuration option.
  • allowTLS – signifies that there is no encryption going on between members of the replica set or sharded cluster, but the DB server will accept both encrypted and non-encrypted traffic from the application hosts.  Only used for transitioning between disabled to requireTLS in a rolling restart fashion.
  • preferTLS – signifies that there is encryption going on between members of the replica set or sharded cluster and that the DB server will accept both encrypted and non-encrypted traffic from the application hosts. Only used for transitioning between disabled to requireTLS in a rolling restart fashion.
    mode: requireTL


Additional required configuration options for transport encryption are:

  • net.tls.certificateKeyFile – location of the .pem file with the certificate and it’s key to be used for application connections.  Note that the user MongoDB is running as must have read permissions on this file.
  • net.tls.clusterFile – location of the .pem file used for transport encryption between replica set or cluster members.   Note that the user MongoDB is running as must have read permissions on this file.
  • net.tls.CAFile – location of the .pem file with the root certificate chain from the Certificate Authority.  Note that the user MongoDB is running as must have read permissions on this file.

Data at Rest Encryption ensures that your data can’t be read by someone who steals your database’s data files unless they also steal the key.  This prevents someone from reading your MongoDB data files at the file system level.  Here the most important configuration option is security.enableEncryption.

  enableEncryption: true

Additional required configuration options for Data At Rest Encryption are:

  • security.encryptionCipherMode – form of encryption to use, options are AES256-CBC and AES256-GCM

Percona Server for MongoDB Specific Configuration Options:

Percona Server for MongoDB has integration with HashiCorp Vault for secret management for your Data at Rest Encryption. Read the documentation for Vault and Using Vault to Store the Master Key for Data at Rest Encryption on Percona Server for MongoDB.

Important configuration options for the Vault Integration are:

  • security.vault.serverName – server name that your vault server is on
  • Security.vault.port – port for vault connectivity
  • security.vault.tokenFile – location of file with vault token
  • Security.vault.secret – location for secrets, since these are set up per node, this should have a distinguishing characteristic such as node name in it
  • security.vault.serverCAFile – location of CAFile (Certificate Authority) on your local mongodb node
  • security.vault.rotateMasterKey – only used to rotate the master key

MongoDB Enterprise Specific Data At Rest Encryption Configuration Options:

Currently, MongoDB Enterprise does not have Vault Integration for Encryption at rest except in MongoDB Atlas.  MongoDB Enterprise does support the KMIP protocol and you can integrate MongoDB with any Key Management tool that utilizes the KMIP protocol.  Documentation can be found here.

Important configuration options to support Key Management through the KMIP protocol are:

  • security.kmip.serverName – server name where your Key Management tool resides
  • security.kmip.port – port for your key management tool
  • security.kmip.serverCAfile – path on your MongoDB hosts of a CA file (Certificate Authority) for secure connection to your Key Management Tool
  • security.kmip.clientCertificateFile – path to the client certificate used for authentication to your Key Management tool
  • security.kmip.rotateMasterKey – only used to rotate the master key


Auditing allows IT Security Compliance teams to track and log activities that are run against the MongoDB database.  There are several important auditing configuration options for MongoDB,  auditLog.filter is the most important as it decides what exactly you are setting up in your auditing log.

  filter:  { <field1>: <expression1>, ... }


For example, if we only wanted to have an audit log entry created every time someone created or removed a collection, we would set the auditLog.Filter as such:

  filter: { atype: { $in: [ "createCollection", "dropCollection" ] } }


If we wanted to audit everyone with a specific role, we could set the auditFilter as such:

  filter:{ roles: { role: "readWrite", db: "appdb" } }


Additional required configuration options for auditing are:

  • auditLog.destination – whether the audit log will be written to a file, to the console, or to the syslog
  • auditLog.path – if outputting to a file, the destination directory, and file name of the audit log.   Note that the user MongoDB runs as must have read and write permissions to this directory.
  • auditLog.format – the format the audit log is output to, options are JSON and BSON, with JSON being the more commonly used format.

Finally, while auditing is important to track and log activity in your database, including accessing PII or other sensitive data, you don’t want to expose PII in your auditing or other log files.  To accomplish this you must set up log redaction on your MongoDB Replica Set or Sharded Cluster.   The important configuration option for log redaction is security.redactClientLogData.   Acceptable values for this configuration option are true and false.

  redactClientLogData: true


In this blog post, we’ve gone over five important MongoDB configuration options to ensure you have a more secure MongoDB deployment as well as some other configuration options that help the five keep your data secure.  We hope that these configuration options will help you build more secure MongoDB deployments and avoid being a statistic of a data breach.   Thanks for reading!

Additional Resources:

MongoDB Top Five Security Concerns

Securing MongoDB Webinar

Our latest resource, Using Open Source Software to Ensure the Security of Your MongoDB Database, documents how to deploy a secure, enterprise-grade, MongoDB deployment without worrying about license fees, giving organizations the flexibility to deploy consistent models across their entire infrastructure.

Download “Using Open Source Software to Ensure the Security of Your MongoDB Database”


Percona Server for MongoDB 4.2 vs 4.4 in Python TPCC Benchmark

Percona Server for MongoDB 4.2 vs. 4.4

Percona Server for MongoDB 4.2 vs. 4.4Following my previous blogs on py-tpcc benchmark for MongoDB, Evaluating the Python TPCC MongoDB Benchmark and Evaluating MongoDB Under Python TPCC 1000W Workload, and the recent release of Percona Server for MongoDB 4.4, I wanted to evaluate 4.2 vs 4.4 in similar scenarios.

Hardware Specs

For the client and server, I will use identical bare metal servers, connected via a 10Gb network.

The node specification:

# Percona Toolkit System Summary Report ######################
        Date | 2020-09-14 16:52:46 UTC (local TZ: EDT -0400)
    Hostname | node3
      System | Supermicro; SYS-2028TP-HC0TR; v0123456789 (Other)
    Platform | Linux
     Release | Ubuntu 20.04.1 LTS (focal)
      Kernel | 5.4.0-42-generic
Architecture | CPU = 64-bit, OS = 64-bit
# Processor ##################################################
  Processors | physical = 2, cores = 28, virtual = 56, hyperthreading = yes
      Models | 56xIntel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
      Caches | 56x35840 KB
# Memory #####################################################
       Total | 251.8G
  Swappiness | 0
 DirtyPolicy | 80, 5
 DirtyStatus | 0, 0

The drive I used for the storage in this benchmark is a Samsung SM863 SATA SSD.

MongoDB Topology

For MongoDB I used:

  • Single node instance without limiting cache size. As the bare metal server has 250GB of RAM, MongoDB should allocate 125GB of memory for WiredTiger cache and the rest will be used for OS cache. This should produce more CPU bound workload.
  • Single node instance with limited cache size. For WiredTiger cache I will set a limit 25GB, and to limit OS cache I will limit the memory available to a mongodb instance to 50GB, as described in Using Cgroups to Limit MySQL and MongoDB memory usage.
  • However I did not use cgroups in this case, but I rather used Docker to run different versions and set limits.

The script to start Percona Server for MongoDB in docker with memory limits:

> bash startserver.sh 4.4 1
=== script startserver.sh ===
docker run -d --name db$2 -m 50g  \
          -v /mnt/data/psmdb$2-$1:/data/db \
          --net=host \
          percona/percona-server-mongodb:$1 --replSet "rs$2" --port $(( 27016 + $2 )) \
          --logpath /data/db/server1.log --slowms=10000 --wiredTigerCacheSizeGB=25 

sleep 10

mongo mongodb://$(( 27016 + $2 )) --eval "rs.initiate( { _id : 'rs$2',  members: [      { _id: 0, host: '$(( 27016 + $2 ))' }   ] })"

MongoDB Versions:

Benchmark Results

Unlimited Memory

The results are in New Order Transactions per Minute (NOTPM), and more is better:

Clients 4.2 4.4
10 541.31 691.89
30 999.89 1105.88
50 1048.50 1171.35
70 1095.72 1335.90
90 1184.38 1433.09
110 1210.18 1521.56
130 1231.38 1575.23
150 1245.31 1680.81
170 1224.13 1668.33
190 1300.11 1641.45
210 1240.86 1619.58
230 1220.89 1575.57
250 1237.86 1545.01

MongoDB 4.4 Unlimited Memory

Limited Memory, 50GB in Total and 25GB for Cache

The results are in New Order Transactions per Minute (NOTPM), and more is better:

Clients 4.2 4.4
10 351.45 377.29
30 483.88 447.22
50 535.34 522.59
70 576.30 574.14
90 604.49 582.10
110 618.59 542.11
130 593.31 386.33
150 386.67 301.75
170 265.91 298.80
190 259.56 301.38
210 254.57 301.88
230 249.47 299.15
250 251.03 300.00

MongoDB 4.2 Limited Memory


Actually I wanted to perform more benchmarks on 4.4 vs 4.2, but some interesting behavior in 4.4 made me reconsider my plans and I’ve gotten distracted trying to understand the issue, and I share this in the post MongoDB 4.4 Performance Regression: Overwhelmed by Memory.

Besides that, in my tests, 4.4 outperformed 4.2 in case of unlimited memory, but I want to consider a variation of throughput during the benchmark so we are working on a py-tpcc version that would report data with 1-sec resolution. Also, I want to re-evaluate how 4.4 would perform in a long-running benchmark, as the current length of the benchmark is 900 sec.

In the case with limited memory, 4.4 did identically or worse than 4.2 with concurrency over 100 clients.

Both versions did not handle the increased number of clients well, showing worse results with 150 clients compared to 10 clients.

Learn more about the history of Oracle, the growth of MongoDB, and what really qualifies software as open source. If you are a DBA, or an executive looking to adopt or renew with MongoDB, this is a must-read!

Download “Is MongoDB the New Oracle?”


Q & A on Webinar “Percona Server for MongoDB Data-at-Rest Encryption”

Percona Server for MongoDB Data-at-Rest Encryption

Percona Server for MongoDB Data-at-Rest EncryptionWe really enjoyed presenting Percona Server for MongoDB’s data-at-rest encryption functionality. The webinar Percona Server for MongoDB Data-at-Rest Encryption was recorded and can be viewed here at any time. We had several great questions, which we would like to address with everyone and help further elaborate on the answers given during the webinar.

Q: Where can you keep encryption keys for Percona Server for MongoDB?

A: You can store them locally on the server where mongod process runs or separately, in an external secret manager. Percona Server for MongoDB supports integration with Hashicorp Vault. From the security perspective, it is recommended to store the encryption key outside of the server that it secures. We have a blog post that describes how to configure Vault to work with Percona Server for MongoDB.


Q: Does Percona Monitoring and Management (PMM) support slow queries for Percona Server for MongoDB?

A: Yes, PMM supports query analytics for MongoDB (both Percona Server for MongoDB and MongoDB Community Edition) since version 3.2. The Query Analytics dashboard shows how queries are executed and where they spend their time. It helps you analyze database queries over time, optimize database performance, and find and remedy the source of problems.


Q: Is data-at-rest encryption supported for any versions before Percona Server for MongoDB 4.0?

A: Yes, Percona Server for MongoDB supports Wired Tiger data-at-rest encryption since version 3.6.


Q: Is Hashicorp Vault supported in Percona Server for MongoDB 3.6?

A: Yes, Hashicorp Vault integration was introduced in version 4.0.10 of Percona Server for MongoDB and then backported to 3.6.13.


Q: Do we have any documents for how to implement data-at-rest encryption for Percona Server for MongoDB?

A: This documentation shows how to enable data-at-rest encryption for Percona Server for MongoDB and this blog post here shows how to configure Hashicorp Vault to work with Percona Server for MongoDB.


Q: Do you need a separate master key for each member in a replica set or sharded cluster?

A: No, it’s not mandatory, however, from the security perspective it’s strongly recommended to have a separate master key for each node in your replica set or shared cluster.


Q: How would you recover your database if you lose your master key?

A: If you lose your master key, your only option to recover your database is from backup. Logical backups can be used in such cases as their encryption is handled independently from data-at-rest encryption.


Q: When you enable data-at-rest encryption, is the admin database also encrypted?

A: Yes, all databases are encrypted when encryption is enabled, effectively the MongoDB admin database is encrypted too.


Q: What are the CPU overhead implications when data-at-rest encryption is enabled?

A: The exact overhead differs use case by use case (e.g. how much data fits the memory when you read it, what datatypes do you use), but overall estimations are between 5%-10% CPU overhead with the data-at-rest encryption enabled


Q: If you have to remove all the data to enable data-at-rest encryption, does that mean that each secondary in production will then need full initial sync after restarting mongod with the new security features?

A: Yes. You would want to do this on all the secondaries first, step down the primary, and then do this on the previous primary which is now secondary.


Q: If I convert a node to Percona Server for MongoDB, can I convert it back to MongoDB Community Edition if I find a bug?

A: Yes, migration is easy in both ways, you’re not forced to stay on Percona Server for MongoDB. If you find a bug, however, we encourage you to file the bug report to our Jira, seek help on our forums, or consider getting a Support subscription if you’re environment is mission-critical. Percona Server for MongoDB is entirely based on MongoDB Community Edition codebase, thus, it’s very likely that you’ll find the same bugs in MongoDB Community Edition.


Q: How long does master key rotation take with respect to data size?

A: Data size isn’t related to the master key rotation procedure time. In accordance with the envelope encryption model, the key rotation procedure re-encrypts a keystore (the place where all database-specific keys are stored) not the dataset itself.


Q: Does Percona Server for MongoDB support AWS KMS?

A: At this time no. When such a feature is available, there will be an announcement.


Q: Does Percona Server for MongoDB support client-side encryption?

A: Yes. Client-side encryption is entirely a client (application) responsibility. If the application encrypts the data prior to storing it in Percona Server for MongoDB, there’s nothing that Percona Server for MongoDB does or does not support.


Q & A on Webinar “Converting MongoDB to Percona Server for MongoDB”

Converting MongoDB to Percona Server for MongoDB

Converting MongoDB to Percona Server for MongoDBWe had great attendance, questions, and feedback from our “Converting MongoDB to Percona Server for MongoDB” webinar which was recorded and can be viewed here. Without further ado, here are your questions and their responses.

Q: What is the recommended path to go from MongoDB Community 3.4 (no Percona) to Percona for MongoDB 4.2?

We would recommend upgrading on Community version from 3.4 -> 3.6 -> 4.0 -> 4.2 with testing and validation at each major version milestone. Once you are on Community 4.2, you can upgrade to Percona Server for MongoDB 4.2. Because PSMDB is a true drop-in replacement you can really do this at any point, but I would do it separately from a major version upgrade.

Q: How does migration work when replica is configured to use “requireSSL”?

Migration works exactly the same as the steps described in the webinar. SSL connections are part of MongoDB Community, so they function identically in Percona Server for MongoDB.

Q: What about migrating encrypted instances to Percona? I guess we need to do initial sync from scratch.

Yes, that is correct. Because encryption affects the way data is stored, full syncs are required. Nodes can be added to an existing enterprise replica set for a close to zero downtime migration.

Q: Do you have any benchmark details with encryption version of Percona?

We have not published any benchmarks for this. However, by default, Percona Server for MongoDB uses the AES256-CBC cipher mode. If you want to use the AES256-GCM cipher mode, then use the encryptionCipherMode parameter to change it. In general, CBC and GCM cipher modes work differently. CBC is faster and GCM is safer (compared to each other). I found some interesting discussion and benchmark here: https://kazoo.ga/quick-benchmark-cbc-vs-gcm/.

Q: Is there any difference in storage of the data between MongoDB vs Percona?

No. They both utilize the same storage engines. WiredTiger is the default. If you’re interested in exploring MMAP, I suggest this blog post MongoDB Engines: MMAPV1 Vs WiredTiger.

Q: Can you migrate from Enterprise Advanced to Percona?


Q: Are MongoDB dumps compatible?

Yes. This is another migration option if the database can tolerate downtime. The methods in our webinar are designed to limit downtime as much as possible.

Q: How does PMM integrate with the mongo replica set? How do you configure backups? Is it in PMM or a separate process?

Yes, PMM has dashboards and metrics for MongoDB replica sets. Our backup solutions are managed outside of Percona Monitoring and Management at this time, so it would be considered a separate process. Our backups are designed to be run on the MongoDB nodes themselves, so no additional middleware is required for their functionality. PMM should be installed on a separate server as it can potentially be storing, ingesting, and displaying a significant amount of data depending upon the size of your MongoDB deployment. View a free demo of Percona Monitoring and Management.

Q: Do MongoDB drivers working with MongoDB Community will work with Percona MongoDB Server?

Yes, MongoDB drivers are compatible with Percona Server for MongoDB. Percona designs its software to be fully compatible with upstream. From an application or connectivity perspective, they are identical.

Q: What would be the situation for a major binary upgrade, 3.4 Percona binary to 4.2 Percona binary?

We would recommend upgrading from 3.4 -> 3.6 -> 4.0 – 4.2 with testing and validation between each major version upgrade. Be sure to research compatibility changes, test, validate, and have tested rollback plans.

Q: Which tools we can use for backup in Percona? Same as Mongo community edition or different?

Percona Server for MongoDB supports all backup methods supported by the Community edition – for example, mongodump. We also have our own backup tool called Percona Backup for MongoDB that supports cluster-wide consistent backups of Sharded Clusters and ReplicaSets.

Thanks to everyone who attended. If you have any feedback or ideas on future MongoDB workshops, please let us know.

Learn more about the history of Oracle, the growth of MongoDB, and what really qualifies software as open source. If you are a DBA, or an executive looking to adopt or renew with MongoDB, this is a must-read!

Download “Is MongoDB the New Oracle?”


Securing MongoDB: Top Five Security Concerns

securing mongodb

I think most of the time hackers behind the attacks do it just for fun, because they can and because it’s very simple”, — says Diachenko. Source: “Meowing” attack completely destroyed more than 1000 databases

securing mongodbThese are the words of Bob Diachenko, one of the most respected cybersecurity researchers in relation to the last “Meowing” attack, which destroyed between 1000-4000 databases around the world.

Whether it’s for fun or for money, this is not the first reported attack, and it won’t be the last (of 2020). With an increasing number of databases in the cloud, the question is now not a matter of IF it will happen again, but when.

Because of its NoSQL origin and document architecture design, MongoDB is much more flexible and scalable than SQL databases such as MySQL. As a result, typically much more data is stored in MongoDB than in traditional SQL databases. MongoDB databases commonly exceed a terabyte of data. The large amount of data exposed in a single database makes breaches involving MongoDB much more devastating.

The good news is that most of the actions you should take to avoid this are simple to execute. If you are using the open-source Percona Distribution for MongoDB, you also have extra features such as LDAP authentication support, which is present in the Enterprise and Atlas MongoDB versions.

What does MongoDB offer to mitigate security threats? Let’s explore the top five measures that we can take when securing MongoDB.

Authentication in MongoDB

Most breaches involving MongoDB occur because of a deadly combination of authentication disabled and MongoDB opened to the internet. MongoDB provides support for authentication on a per-database level. Users exist in the context of a single logical database. However, MongoDB does not support items like password complexity, age-based rotation, and centralization and identification of user roles versus service functions.

Thankfully, LDAP can fill many of these gaps. Many connectors allow the use of Windows Active Directory (AD) systems to talk with LDAP.

LDAP support is available in MongoDB Enterprise, but not in MongoDB Community Edition. However, it is available in other open-source versions of MongoDB, such as Percona Server for MongoDB.

Note: LDAP in the Percona Distribution for MongoDB will work with MongoDB Compass.

In the following links, we describe how to use LDAP:

Percona Server for MongoDB LDAP Enhancements: User-to-DN Mapping

Percona Server for MongoDB Authentication Using Active Directory

Authorization in MongoDB

Enabling access control on a MongoDB deployment enforces authentication, requiring users to identify themselves. When accessing a MongoDB deployment that has access control enabled, users can only perform the actions determined by their roles. Replica sets and sharded clusters also require internal authentication between members when access control is enabled. It is essential to follow the principle of least privilege. No one should have more permissions than they need to do their job, and even a DBA should log in with a non-elevated account.

MongoDB grants access to data and commands through role-based authorization and include built-in roles that provide the different levels of access commonly needed in a database system. Additionally, it is possible to create user-defined roles.

To create a role in MongoDB and add it to a user:

role : 'write_foo2_Collection',
privileges : [ {resource : {db : "percona", collection : "foo2"}, actions : ["insert","remove"]}
roles : ["read"]

db.updateUser('client_read', roles : ['write_foo2_Collection'])

Transport Encryption in MongoDB

MongoDB has support for using transport encryption between the client and the nodes, and between the nodes in the cluster. Encrypting traffic ensures that no one can “sniff” sensitive data on the network. For example, tools like Wireshark or Tcpdump can easily capture unencrypted sensitive data such as usernames and passwords.

MongoDB supports X.509 certificate authentication for use with a secure TLS/SSL connection. Members can use X.509 certificates to verify their membership in the replica set and shards.

It is necessary to create certificates on all nodes and have a certificate authority (CA) that signs them. As using a CA can be costly, it is also possible to use self-signed certificates. Using a public CA is not necessary inside a private infrastructure.

Here are the detailed steps to create the certificates and configure Mongo:

MongoDB: Deploy a Replica Set with Transport Encryption

Data Encryption in MongoDB

One of the most severe problems with MongoDB was that data files didn’t have encryption at rest. Since version 3.6.8, Percona Server for MongoDB has offered at rest encryption for the MongoDB Community Edition. In upstream MongoDB software, data encryption at rest is available in MongoDB Enterprise version only.

The example below shows how to activate WiredTiger encryption for data at rest in Percona Server for MongoDB. First, it is necessary to edit the encryption options in mongod.conf:

# Encryption variables in mongod.conf shell
[root@app ~]# grep security -A2 /etc/mongod.conf
  enableEncryption: true
  encryptionKeyFile: /data/key/mongodb.key

By default, Percona Server for MongoDB uses the AES256-CBC cipher mode. It is necessary to create the key with OpenSSL as below:

# Create Encryption KeyShell
[root@app ~]# mkdir /data/key
[root@app ~]# openssl rand -base64 32 > /data/key/mongodb.key
[root@app ~]# chmod 600 /data/key/mongodb.key

Now start Percona Server for MongoDB:

[root@app ~]# systemctl start mongod

To check whether encryption is successfully enabled in the database, use the command below:

# Security outputShell
mongo > db.serverCmdLineOpts().parsed.security
{ "enableEncryption" : true, "encryptionKeyFile" : "/data/key/mongodb.key" }

PS: Note that for existing nodes, it is necessary to perform a new initial sync to encrypt the data.

Auditing in MongoDB

Auditing is not designed to mitigate a security threat but helps when investigating unauthorized access or tracking data access and modification. The general database auditing concept is about tracking the use of database records and authority. When we audit a database, each operation on the data can be monitored and logged to an audit trail. This includes information about which database object or data record was touched, which account performed the action, and when the activity occurred.

MongoDB Atlas offers audit logging natively, as does MongoDB Enterprise Edition and Percona Server for MongoDB. To enable the audit log in Percona Server for MongoDB in the command line or the config file, add these entries in the command line:

mongod --dbpath /var/lib/mongodb --auditDestination file --auditFormat BSON --auditPath /var/lib/mongodb/auditLog.bson

Or in the MongoDB configuration file:

   destination: file
   format: BSON
   path: /var/lib/mongodb/auditLog.bson


It is vital that you secure your database before the deployment phase. The five mentioned measures above can be automated using tools like Ansible/Puppet, and are a good start to secure your database. A MongoDB database with these five security measures taken would not be affected by the Meow attack.

For further information on database security, please watch our recent webinar from Akira Kurogane, MongoDB Lead, Percona – On-Demand Webinar: Securing MongoDB.

You can also access additional expert guidance and thought-leadership via our website.

Learn more about the history of Oracle, the growth of MongoDB, and what really qualifies software as open source. If you are a DBA, or an executive looking to adopt or renew with MongoDB, this is a must-read!

Download “Is MongoDB the New Oracle?”


Setting up MongoDB with Member x509 auth and SSL + easy-rsa

MongoDB Member with x509 auth

MongoDB Member with x509 authHi everyone! This is one of the most requested subjects to our support team and I’d like to share the steps as a tutorial blog post. Today, we will set up internal authentication using x.509 certificates as well as enabling TSL/SSL.

If using authentication in MongoDB, there are two ways to configure intra-cluster authentication:

  • Using a Key File
  • Using x509 certs

Key files are very straight forward; just create a random text file and share it with all the members in the replicaset/sharding. However, this is not the most secure way and for this reason, it is very common to use certificates instead.

It is perfectly possible to have self-signed certificates, but in this blog, we will use easy-rsa to make real certificates signed by one certificate authority. By the documentation, easy-rsa is a CLI utility to build and manage a PKI CA. In laymen’s terms, this means to create a root certificate authority, and request and sign certificates, including sub-CAs and certificate revocation lists (CRL). This project is hosted on GitHub on https://github.com/OpenVPN/easy-rsa and we are going to use release 2.x for this tutorial.

We will use Percona Server for MongoDB v3.6 – which is currently one of the most used versions – but this works for any MongoDB version starting at 3.2. The steps as to how to create a user will be omitted in this blog. We are considering the primary is configured with authentication and the first user was already created.


  1. Download and configure easy-rsa:
    yum install git -y
    mkdir /usr/share/easy-rsa
    git clone -b release/2.x https://github.com/OpenVPN/easy-rsa.git
    cp easy-rsa/easy-rsa/2.0/* /usr/share/easy-rsa
    cd  /usr/share/easy-rsa
  2. Edit the source files with information about your company:
    cd /usr/share/easy-rsa
    nano vars
    # These are the default values for fields
    # which will be placed in the certificate.
    # Don't leave any of these fields blank.
    export KEY_COUNTRY="US" <your data>
    export KEY_PROVINCE="NC" <your data>
    export KEY_CITY="DURHAM" <your data>
    export KEY_ORG="Percona" <your data>
    export KEY_EMAIL="me@percona.com" <your data>
    export KEY_OU="MongoDB" <your data>
    #You may need to add the following variable:
    #Bug: https://bugs.launchpad.net/serverguide/+bug/1504676
    export KEY_ALTNAMES=""
  3. Load the variables with the source command:
    source ./vars
  4.  Edit the openssl-1.0.0.cnf file commenting the keys right after [ usr_cert ]
    #keyUsage = digitalSignature

    More info here on Extended Key Usage

  5. Now everything is prepared to create our CA file. Let’s create the CA and the members’ certificates:
    cd /usr/share/easy-rsa
    # this command will clean all the data in the ./keys folder
    # It will generate a key for the CA as well as a certificate
    # It will generate a key for the server as well as a certificate
    ./built-key <server_name>
  6. We suggest keeping the default values for the CA and informing the FQN or the hostname in the certificates. (It will be validated by MongoDB.)This is the expected output:
    Generating a 2048 bit RSA private key
    writing new private key to 'server_name.key'
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    Country Name (2 letter code) [US]:
    State or Province Name (full name) [NC]:
    Locality Name (eg, city) [Durham]:
    Organization Name (eg, company) [Percona]:
    Organizational Unit Name (eg, section) [MongoDB]:
    Common Name (eg, your name or your server's hostname) [server_name]:
    Name [EasyRSA]:
    Email Address [percona@percona.com]:
    Please enter the following 'extra' attributes
    to be sent with your certificate request
    A challenge password []:
    An optional company name []:
    Using configuration from /usr/share/easy-rsa/openssl-1.0.0.cnf
    Check that the request matches the signature
    Signature ok
    Certificate is to be certified until Mar  9 11:29:40 2028 GMT (3650 days)
    Sign the certificate? [y/n]:y
    1 out of 1 certificate requests certified, commit? [y/n]y
    Write out database with 1 new entries
    Data Base Updated
  7. After creating all the certificates, we need to combine the keys and its certificate in order to create the .pem file.
    cd keys
    -rw-r--r-- 1 root root 1,7K Mar  9 09:35 ca.crt
    -rw------- 1 root root 1,7K Mar  9 09:35 ca.key
    -rw-r--r-- 1 root root 4,1K Mar 12 08:29 server_name.crt
    -rw-r--r-- 1 root root 1,1K Mar 12 08:29 server_name.csr
    -rw------- 1 root root 1,7K Mar 12 08:29 server_name.key
    # combining .key and .crt into a single file.
    cat server_name.key server_name.crt > server_name.pem

    Repeat this process to all the server keys.

  8. Now that we have the server .pem files prepared we need to edit the mongod.conf, considering the keys were moved to /var/lib/mongodb/ 
    security.clusterAuthMode : x509
    security.authorization : enabled 
      port: 27017
      bindIp: <ip_number>
        mode: requireSSL
        PEMKeyFile: /var/lib/mongodb/server_name.pem
        CAFile: /var/lib/mongodb/ca.crt
  9. Once the changes are made, the services must be started and the members should start normally.
  10. It is now time to configure the clients, as otherwise, no one will be able to log in to this environment. Again we need to edit the openssl-1.0.0.cnf removing the comments. Clients need to have those keys in the certificate.
    cd /usr/share/easy-rsa 
    keyUsage = digitalSignature
  11. After editing the file, create the client file, it is as simple as creating a new key:
    cd /usr/share/easy-rsa 
    ./build-key <client_name>

    There is a caveat here, the Organization Unit must be different than MongoDB. I recommend calling as a MongoDBClient, and once the files are created repeat the process of linking the client_name.crt and the client_name.key file in a single file and using it to log in to the environment.

    ./build-key client_name
    Country Name (2 letter code) [US]:
    State or Province Name (full name) [NC]:
    Locality Name (eg, city) [DURHAM]:
    Organization Name (eg, company) [Percona]:
    Organizational Unit Name (eg, section) [MongoDB]:MongoDBClient
    Common Name (eg, your name or your server's hostname) [client_name]:
    cd keys
    cat client_name.key client_name.crt > client_name.pem
  12. Connecting to the database is simple; we need to specify the ca file along with the certificate the client is connecting.
    Please be aware you’ll need to connect to the server local IP instead of localhost, and you may need to edit the /etc/hosts in order to force the databases and clients to resolve the hostnames.

    mongo --ssl --host server_name --sslCAFile /usr/share/easy-rsa/keys/ca.crt  \
       --sslPEMKeyFile /usr/share/easy-rsa/keys/client_name.pem --port 27017 \
       -u <user> -p --authenticationDatabase admin

With these described steps you should be able to enable SSL + member authentication in your environment. Please feel free to give us feedback here or tweet to @AdamoTonete or @Percona on Twitter!


Percona Server for MongoDB 4.0.12-6 Now Available

Percona Server for MongoDB

Percona Server for MongoDBPercona announces the release of Percona Server for MongoDB 4.0.12-6 on September 12, 2019. Download the latest version from the Percona website or the Percona software repositories.

Percona Server for MongoDB is an enhanced, open source, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB 4.0 Community Edition. It supports MongoDB 4.0 protocols and drivers.

Percona Server for MongoDB extends the functionality of the MongoDB 4.0 Community Edition by including the Percona Memory Engine storage engine, encrypted WiredTiger storage engineaudit loggingexternal authenticationhot backups, and enhanced query profilingPercona Server for MongoDB requires no changes to MongoDB applications or code.

Percona Server for MongoDB 4.0.12-6 enables uploading hot backups to Amazon S3 or compatible storage service, such as MinIO. In this release, this feature has the EXPERIMENTAL status. For more information, see the documentation of Percona Server for MongoDB.

This release includes all features of MongoDB 4.0 Community Edition. Most notable among these are:

Note that the MMAPv1 storage engine is deprecated in MongoDB 4.0 Community Edition.

Percona Server for MongoDB 4.0.12-6 is based on MongoDB 4.0.12.

New Features

  • PSMDB-371: Hot backup streaming to a remote destination

Bugs Fixed

  • PSMDB-396: Per database encryption keys were not removed on database drop
  • PSMDB-408: In some cases, the encryption keydb instance was not destroyed on the instance shutdown
  • PSMDB-412: PSMDB was not pulling the ECDHE ciphers from OpenSSL due to upstream bug SERVER-38141.

Known Issues

  • PSMDB-441: Connectivity errors when making backups to GCP on Debian 9 (stretch).

The Percona Server for MongoDB 4.0.12-6 release notes are available in the official documentation.


Announcing the Launch of Percona Server for MongoDB 4.2.0-1

Percona Server for MongoDB

Percona Server for MongoDB

We are pleased to announce the launch of Percona Server for MongoDB 4.2.0-1 on September 9, 2019. Our engineers have been hard at work to bring this to you less than a month after the launch of MongoDB Community Edition 4.2.

Percona Server for MongoDB 4.2.0-1 includes all of the new features of the latest Community version, as well as providing our users with Enterprise-level enhancements for free.

Download the latest version from the Percona website or the Percona software repositories.

Percona Server for MongoDB is an enhanced, open source, and highly-scalable database that acts as a fully-compatible, drop-in replacement for MongoDB 4.2 Community Edition. It supports MongoDB 4.2 protocols and drivers.

Percona Server for MongoDB extends the functionality of the MongoDB 4.2 Community Edition by including the Percona Memory Engine storage engine, encrypted WiredTiger storage engineaudit loggingexternal LDAP authenticationhot backups, and enhanced query profilingPercona Server for MongoDB requires no changes to MongoDB applications or code.

Percona Server for MongoDB 4.2.0-1 includes all of the features of MongoDB 4.2 Community Edition. The most notable among these are:

Improving its security, MongoDB 4.2 Community Edition resolves the following CVEs: CVE-2019-2389 and CVE-2019-2386.

MongoDB 4.2 Community Edition removes the MMAPv1 storage engine along with MMAPv1 specific configuration options, parameters and commands. It also removes a number of commands and methods not related to MMAPv1.

Percona Server for MongoDB 4.2.0-1 is based on MongoDB 4.2.0.

There are a number of benefits to downloading the latest version of our software, including the impressive new features referenced above, which make it easier to run and manage your MongoDB database and help to keep your data safe.

For further information on the application of these new features, you can also visit our recent blogs which dig into the pros and cons of each in detail:

We believe that MongoDB 4.2 Community Edition is a great improvement on previous versions, with some long-awaited enhancements, including features which were previously only available in the Enterprise version.

Percona Server for MongoDB is a free and open source drop-in for MongoDB Community Edition and includes all of the new features of 4.2, and more! This makes it an attractive proposition for businesses looking to have a robust database software solution in place and make the most of their IT budget.

The Percona Server for MongoDB 4.2.0-1 release notes are available in the official documentation.

Download our product flyer for more information on Percona’s commitment to MongoDB.

Please contact us if you would like any assistance with your MongoDB database set-up, or if you would like to discuss any of the 4.2 features in more detail.

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com