There are various ways to backup and restore Percona Server for MongoDB clusters when you run them on Kubernetes. Percona Operator for MongoDB utilizes Percona Backup for MongoDB (PBM) to take physical and logical backups, continuously upload oplogs to object storage, and maintain the backup lifecycle. Cloud providers and various storage solutions provide the capability […]
08
2024
Unlocking the Power of Cloud Snapshots: Backup and Restore Your MongoDB Clusters on Kubernetes
15
2023
Automating Physical Backups of MongoDB on Kubernetes
We at Percona talk a lot about how Kubernetes Operators automate the deployment and management of databases. Operators seamlessly handle lots of Kubernetes primitives and database configuration bits and pieces, all to remove toil from operation teams and provide a self-service experience for developers.
Today we want to take you backstage and show what is really happening under the hood. We will review technical decisions and obstacles we faced when implementing physical backup in Percona Operator for MongoDB version 1.14.0. The feature is now in technical preview.
The why
Percona Server for MongoDB can handle petabytes of data. The users of our Operators want to host huge datasets in Kubernetes too. However, using a logical restore recovery method takes a lot of time and can lead to SLA breaches.
Our Operator uses Percona Backup for MongoDB (PBM) as a tool to backup and restore databases. We have been leveraging logical backups for quite some time. Physical backups in PBM were introduced a few months ago, and, with those, we saw significant improvement in recovery time (read more in this blog post, Physical Backup Support in Percona Backup for MongoDB):
So if we want to reduce the Recovery Time Objective (RTO) for big data sets, we must provide physical backups and restores in the Operator.
The how
Basics
When you enable backups in your cluster, the Operator adds a sidecar container to each replset pod (including the Config Server pods if sharding is enabled) to run pbm-agent. These agents listen for PBM commands and perform backups and restores on the pods. The Operator opens connections to the database to send PBM commands to its collections or to track the status of PBM operations.
Backups
Enabling physical backups was fairly straightforward. The Operator already knew how to start a logical backup, so we just passed a single field to the PBM command if the requested backup was physical.
Unlike logical backups, PBM needs direct access to mongod’s data directory, and we mounted the persistent volume claim (PVC) on PBM sidecar containers. With these two simple changes, the Operator could perform physical backups. But whether you know it from theory or from experience (ouch!), a backup is useless if you cannot restore it.
Restores
Enabling physical restores wasn’t easy. PBM had two major limitations when it came to physical restores:
1. PBM stops the mongod process during the restore.
2. It runs a temporary mongod process after the backup files are copied to the data directory.
What makes physical restores a complex feature is dealing with the implications of these two constraints.
PBM kills the mongod process
PBM kills the mongod process in the container, which means the operator can’t query PBM collections from the database during the restore to track their status. This forced us to use the PBM CLI tool to control the PBM operations during the physical restore instead of opening a connection to the database.
Note: We are now considering using the same approach for every PBM operation in the Operator. Let us know what you think.
Another side effect of PBM killing mongod was restarting the mongod container. The mongod process runs as PID 1 in the container, and when you kill it, the container restarts. But the PBM agent needs to keep working after mongod is killed to complete the physical restore. For this, we have created a special entrypoint to be used during the restore. This special entrypoint starts mongod and just sleeps after mongod has finished.
Temporary mongod started by PBM
When you start a physical restore, PBM does approximately the following:
1. Kills the mongod process and wipes the data directory.
2. Copies the backup files to the data directory.
3. Starts a temporary mongod process listening on a random port to perform operations on oplog.
4. Kills the mongod and PBM agent processes.
For a complete walkthrough, see Physical Backup Support in Percona Backup for MongoDB.
From the Operator’s point of view, the problem with this temporary mongod process is its version. We support MongoDB 4.4, MongoDB 5.0, and now MongoDB 6.0 in the Operator, and the temporary mongod needs to be the same version as the MongoDB running in the cluster. This means we either include mongod binary in PBM docker images and create a separate image for each version combination or find another solution.
Well… we found it: We copy the PBM binaries into the mongod container before the restore. This way, the PBM agent can kill and start the mongod process as often as it wants, and it’s guaranteed to be the same version with cluster.
Tying it all together
By now, it may seem like I have thrown random solutions to each problem. PBM, CLI, special entrypoint, copying PBM binaries… How does it all work?
When you start a physical restore, the Operator patches the StatefulSet of each replset to:
1. Switch the mongod container entrypoint to our new special entrypoint: It’ll start pbm-agent in the background and then start the mongod process. The script itself will run as PID 1, and after mongod is killed, it’ll sleep forever.
2. Inject a new init container: This init container will copy the PBM and pbm-agent binaries into the mongod container.
3. Remove the PBM sidecar: We must ensure that only one pbm-agent is running in each pod.
Until all StatefulSets (except mongos) are updated, the PerconaServerMongoDBRestore object is in a “waiting” state. Once they’re all updated, the operator starts the restore, and the restore object goes into the requested state, then the running state, and finally, the ready or error state.
Conclusion
Operators simplify the deployment and management of applications on Kubernetes. Physical backups and restores are quite a popular feature for MongoDB clusters. Restoring from physical backup with Percona Backup for MongoDB has multiple manual steps. This article shows what kind of complexity is hidden behind this seemingly simple step.
23
2023
Streaming MongoDB Backups Directly to S3
If you ever had to make a quick ad-hoc backup of your MongoDB databases, but there was not enough disk space on the local disk to do so, this blog post may provide some handy tips to save you from headaches.
It is a common practice that before a backup can be stored in the cloud or on a dedicated backup server, it has to be prepared first locally and later copied to the destination.
Fortunately, there are ways to skip the local storage entirely and stream MongoDB backups directly to the destination. At the same time, the common goal is to save both the network bandwidth and storage space (cost savings!) while not overloading the CPU capacity on the production database. Therefore, applying on-the-fly compression is essential.
In this article, I will show some simple examples to help you quickly do the job.
Prerequisites for streaming MongoDB backups
You will need an account for one of the providers offering object storage compatible with Amazon S3. I used Wasabi in my tests as it offers very easy registration for a trial and takes just a few minutes to get started if you want to test the service.
A second need is a tool allowing you to manage the data from a Linux command line. The two most popular ones — s3cmd and AWS — are sufficient, and I will show examples using both.
Installation and setup will depend on your OS and the S3 provider specifics. Please refer to the documentation below to proceed, as I will not cover the installation details here.
* https://s3tools.org/s3cmd
* https://docs.aws.amazon.com/cli/index.html
Backup tools
Two main tools are provided with the MongoDB packages, and both do a logical backup.
- mongodump – backups in a form of binary JSON files (BSON)
- mongoexport – backups in a form of regular JSON files
Compression tool
We all know gzip or bzip2 are installed by default on almost every Linux distro. However, I find zstd way more efficient, so I’ll use it in the examples.
Examples
I believe real-case examples are best if you wish to test something similar, so here they are.
Mongodump & s3cmd – Single database backup
- Let’s create a bucket dedicated to MongoDB data backups:
$ s3cmd mb s3://mbackups Bucket 's3://mbackups/' created
- Now, do a simple dump of one example database using the −−archive option, which changes the behavior from storing collections data in separate files on disk, to streaming the whole backup to standard output (STDOUT) using common archive format. At the same time, the stream gets compressed on the fly and sent to the S3 destination.
- Note the below command does not create a consistent backup with regards to ongoing writes as it does not contain the oplog.
$ mongodump --db=db2 --archive| zstd | s3cmd put - s3://mbackups/$(date +%Y-%m-%d.%H-%M)/db2.zst 2023-02-07T19:33:58.138+0100 writing db2.products to archive on stdout 2023-02-07T19:33:58.140+0100 writing db2.people to archive on stdout 2023-02-07T19:33:59.364+0100 done dumping db2.people (50474 documents) 2023-02-07T19:33:59.977+0100 done dumping db2.products (516784 documents) upload: '<stdin>' -> 's3://mbackups/2023-02-07.19-33/db2.zst' [part 1 of -, 15MB] [1 of 1] 15728640 of 15728640 100% in 1s 8.72 MB/s done upload: '<stdin>' -> 's3://mbackups/2023-02-07.19-33/db2.zst' [part 2 of -, 1491KB] [1 of 1] 1527495 of 1527495 100% in 0s 4.63 MB/s done
- After the backup is done, let’s verify its presence in S3:
$ s3cmd ls -H s3://mbackups/2023-02-07.19-33/ 2023-02-07 18:34 16M s3://mbackups/2023-02-07.19-33/db2.zst
Mongorestore & s3cmd – Database restore directly from S3
The below mongorestore command uses archive option as well, which allows us to stream the backup directly to it:
$ s3cmd get --no-progress s3://mbackups/2023-02-07.20-14/db2.zst - |zstd -d | mongorestore --archive --drop 2023-02-08T00:42:41.434+0100 preparing collections to restore from 2023-02-08T00:42:41.480+0100 reading metadata for db2.people from archive on stdin 2023-02-08T00:42:41.480+0100 reading metadata for db2.products from archive on stdin 2023-02-08T00:42:41.481+0100 dropping collection db2.people before restoring 2023-02-08T00:42:41.502+0100 restoring db2.people from archive on stdin 2023-02-08T00:42:42.130+0100 dropping collection db2.products before restoring 2023-02-08T00:42:42.151+0100 restoring db2.products from archive on stdin 2023-02-08T00:42:43.217+0100 db2.people 16.0MB 2023-02-08T00:42:43.217+0100 db2.products 12.1MB 2023-02-08T00:42:43.217+0100 2023-02-08T00:42:43.654+0100 db2.people 18.7MB 2023-02-08T00:42:43.654+0100 finished restoring db2.people (50474 documents, 0 failures) 2023-02-08T00:42:46.218+0100 db2.products 46.3MB 2023-02-08T00:42:48.758+0100 db2.products 76.0MB 2023-02-08T00:42:48.758+0100 finished restoring db2.products (516784 documents, 0 failures) 2023-02-08T00:42:48.758+0100 no indexes to restore for collection db2.products 2023-02-08T00:42:48.758+0100 no indexes to restore for collection db2.people 2023-02-08T00:42:48.758+0100 567258 document(s) restored successfully. 0 document(s) failed to restore.
Mongodump & s3cmd – Full backup
The below command provides a consistent point-in-time snapshot thanks to oplog option:
$ mongodump --port 3502 --oplog --archive | zstd | s3cmd put - s3://mbackups/$(date +%Y-%m-%d.%H-%M)/full_dump.zst 2023-02-13T00:05:54.080+0100 writing admin.system.users to archive on stdout 2023-02-13T00:05:54.083+0100 done dumping admin.system.users (1 document) 2023-02-13T00:05:54.084+0100 writing admin.system.version to archive on stdout 2023-02-13T00:05:54.085+0100 done dumping admin.system.version (2 documents) 2023-02-13T00:05:54.087+0100 writing db1.products to archive on stdout 2023-02-13T00:05:54.087+0100 writing db2.products to archive on stdout 2023-02-13T00:05:55.260+0100 done dumping db2.products (284000 documents) upload: '<stdin>' -> 's3://mbackups/2023-02-13.00-05/full_dump.zst' [part 1 of -, 15MB] [1 of 1] 2023-02-13T00:05:57.068+0100 [####################....] db1.products 435644/516784 (84.3%) 15728640 of 15728640 100% in 1s 9.63 MB/s done 2023-02-13T00:05:57.711+0100 [########################] db1.products 516784/516784 (100.0%) 2023-02-13T00:05:57.722+0100 done dumping db1.products (516784 documents) 2023-02-13T00:05:57.723+0100 writing captured oplog to 2023-02-13T00:05:58.416+0100 dumped 136001 oplog entries upload: '<stdin>' -> 's3://mbackups/2023-02-13.00-05/full_dump.zst' [part 2 of -, 8MB] [1 of 1] 8433337 of 8433337 100% in 0s 10.80 MB/s done $ s3cmd ls -H s3://mbackups/2023-02-13.00-05/full_dump.zst 2023-02-12 23:05 23M s3://mbackups/2023-02-13.00-05/full_dump.zst
Mongodump & s3cmd – Full backup restore
By analogy, mongorestore is using the oplogReplay option to apply the log contained in the archived stream:
$ s3cmd get --no-progress s3://mbackups/2023-02-13.00-05/full_dump.zst - | zstd -d | mongorestore --port 3502 --archive --oplogReplay 2023-02-13T00:07:25.977+0100 preparing collections to restore from 2023-02-13T00:07:25.977+0100 don't know what to do with subdirectory "db1", skipping... 2023-02-13T00:07:25.977+0100 don't know what to do with subdirectory "db2", skipping... 2023-02-13T00:07:25.977+0100 don't know what to do with subdirectory "", skipping... 2023-02-13T00:07:25.977+0100 don't know what to do with subdirectory "admin", skipping... 2023-02-13T00:07:25.988+0100 reading metadata for db1.products from archive on stdin 2023-02-13T00:07:25.988+0100 reading metadata for db2.products from archive on stdin 2023-02-13T00:07:26.006+0100 restoring db2.products from archive on stdin 2023-02-13T00:07:27.651+0100 db2.products 11.0MB 2023-02-13T00:07:28.429+0100 restoring db1.products from archive on stdin 2023-02-13T00:07:30.651+0100 db2.products 16.0MB 2023-02-13T00:07:30.652+0100 db1.products 14.4MB 2023-02-13T00:07:30.652+0100 2023-02-13T00:07:33.652+0100 db2.products 32.0MB 2023-02-13T00:07:33.652+0100 db1.products 18.0MB 2023-02-13T00:07:33.652+0100 2023-02-13T00:07:36.651+0100 db2.products 37.8MB 2023-02-13T00:07:36.652+0100 db1.products 32.0MB 2023-02-13T00:07:36.652+0100 2023-02-13T00:07:37.168+0100 db2.products 41.5MB 2023-02-13T00:07:37.168+0100 finished restoring db2.products (284000 documents, 0 failures) 2023-02-13T00:07:39.651+0100 db1.products 49.3MB 2023-02-13T00:07:42.651+0100 db1.products 68.8MB 2023-02-13T00:07:43.870+0100 db1.products 76.0MB 2023-02-13T00:07:43.870+0100 finished restoring db1.products (516784 documents, 0 failures) 2023-02-13T00:07:43.871+0100 restoring users from archive on stdin 2023-02-13T00:07:43.913+0100 replaying oplog 2023-02-13T00:07:45.651+0100 oplog 2.14MB 2023-02-13T00:07:48.651+0100 oplog 5.68MB 2023-02-13T00:07:51.651+0100 oplog 9.34MB 2023-02-13T00:07:54.651+0100 oplog 13.0MB 2023-02-13T00:07:57.651+0100 oplog 16.7MB 2023-02-13T00:08:00.651+0100 oplog 19.7MB 2023-02-13T00:08:03.651+0100 oplog 22.7MB 2023-02-13T00:08:06.651+0100 oplog 25.3MB 2023-02-13T00:08:09.651+0100 oplog 28.1MB 2023-02-13T00:08:12.651+0100 oplog 30.8MB 2023-02-13T00:08:15.651+0100 oplog 33.6MB 2023-02-13T00:08:18.651+0100 oplog 36.4MB 2023-02-13T00:08:21.651+0100 oplog 39.1MB 2023-02-13T00:08:24.651+0100 oplog 41.9MB 2023-02-13T00:08:27.651+0100 oplog 44.7MB 2023-02-13T00:08:30.651+0100 oplog 47.5MB 2023-02-13T00:08:33.651+0100 oplog 50.2MB 2023-02-13T00:08:36.651+0100 oplog 53.0MB 2023-02-13T00:08:38.026+0100 applied 136001 oplog entries 2023-02-13T00:08:38.026+0100 oplog 54.2MB 2023-02-13T00:08:38.026+0100 no indexes to restore for collection db1.products 2023-02-13T00:08:38.026+0100 no indexes to restore for collection db2.products 2023-02-13T00:08:38.026+0100 800784 document(s) restored successfully. 0 document(s) failed to restore.
Mongoexport – Export all collections from a given database, compress, and save directly to S3
Another example is using the tool to create regular JSON dumps; this is also not a consistent backup if writes are ongoing.
$ ts=$(date +%Y-%m-%d.%H-%M) $ mydb="db2" $ mycolls=$(mongo --quiet $mydb --eval "db.getCollectionNames().join('n')") $ for i in $mycolls; do mongoexport -d $mydb -c $i |zstd| s3cmd put - s3://mbackups/$ts/$mydb/$i.json.zst; done 2023-02-07T19:30:37.163+0100 connected to: mongodb://localhost/ 2023-02-07T19:30:38.164+0100 [#######.................] db2.people 16000/50474 (31.7%) 2023-02-07T19:30:39.164+0100 [######################..] db2.people 48000/50474 (95.1%) 2023-02-07T19:30:39.166+0100 [########################] db2.people 50474/50474 (100.0%) 2023-02-07T19:30:39.166+0100 exported 50474 records upload: '<stdin>' -> 's3://mbackups/2023-02-07.19-30/db2/people.json.zst' [part 1 of -, 4MB] [1 of 1] 4264922 of 4264922 100% in 0s 5.71 MB/s done 2023-02-07T19:30:40.015+0100 connected to: mongodb://localhost/ 2023-02-07T19:30:41.016+0100 [##......................] db2.products 48000/516784 (9.3%) 2023-02-07T19:30:42.016+0100 [######..................] db2.products 136000/516784 (26.3%) 2023-02-07T19:30:43.016+0100 [##########..............] db2.products 224000/516784 (43.3%) 2023-02-07T19:30:44.016+0100 [##############..........] db2.products 312000/516784 (60.4%) 2023-02-07T19:30:45.016+0100 [##################......] db2.products 408000/516784 (78.9%) 2023-02-07T19:30:46.016+0100 [#######################.] db2.products 496000/516784 (96.0%) 2023-02-07T19:30:46.202+0100 [########################] db2.products 516784/516784 (100.0%) 2023-02-07T19:30:46.202+0100 exported 516784 records upload: '<stdin>' -> 's3://mbackups/2023-02-07.19-30/db2/products.json.zst' [part 1 of -, 11MB] [1 of 1] 12162655 of 12162655 100% in 1s 10.53 MB/s done $ s3cmd ls -H s3://mbackups/$ts/$mydb/ 2023-02-07 18:30 4M s3://mbackups/2023-02-07.19-30/db2/people.json.zst 2023-02-07 18:30 11M s3://mbackups/2023-02-07.19-30/db2/products.json.zst
Mongoimport & s3cmd – Import single collection under a different name
$ s3cmd get --no-progress s3://mbackups/2023-02-08.00-49/db2/people.json.zst - | zstd -d | mongoimport -d db2 -c people_copy 2023-02-08T00:53:48.355+0100 connected to: mongodb://localhost/ 2023-02-08T00:53:50.446+0100 50474 document(s) imported successfully. 0 document(s) failed to import.
Mongodump & AWS S3 – Backup database
$ mongodump --db=db2 --archive | zstd | aws s3 cp - s3://mbackups/backup1/db2.zst 2023-02-08T11:34:46.834+0100 writing db2.people to archive on stdout 2023-02-08T11:34:46.837+0100 writing db2.products to archive on stdout 2023-02-08T11:34:47.379+0100 done dumping db2.people (50474 documents) 2023-02-08T11:34:47.911+0100 done dumping db2.products (516784 documents) $ aws s3 ls --human-readable mbackups/backup1/ 2023-02-08 11:34:50 16.5 MiB db2.zst
Mongorestore & AWS S3 – Restore database
$ aws s3 cp s3://mbackups/backup1/db2.zst - | zstd -d | mongorestore --archive --drop 2023-02-08T11:37:08.358+0100 preparing collections to restore from 2023-02-08T11:37:08.364+0100 reading metadata for db2.people from archive on stdin 2023-02-08T11:37:08.364+0100 reading metadata for db2.products from archive on stdin 2023-02-08T11:37:08.365+0100 dropping collection db2.people before restoring 2023-02-08T11:37:08.462+0100 restoring db2.people from archive on stdin 2023-02-08T11:37:09.100+0100 dropping collection db2.products before restoring 2023-02-08T11:37:09.122+0100 restoring db2.products from archive on stdin 2023-02-08T11:37:10.288+0100 db2.people 16.0MB 2023-02-08T11:37:10.288+0100 db2.products 13.8MB 2023-02-08T11:37:10.288+0100 2023-02-08T11:37:10.607+0100 db2.people 18.7MB 2023-02-08T11:37:10.607+0100 finished restoring db2.people (50474 documents, 0 failures) 2023-02-08T11:37:13.288+0100 db2.products 47.8MB 2023-02-08T11:37:15.666+0100 db2.products 76.0MB 2023-02-08T11:37:15.666+0100 finished restoring db2.products (516784 documents, 0 failures) 2023-02-08T11:37:15.666+0100 no indexes to restore for collection db2.products 2023-02-08T11:37:15.666+0100 no indexes to restore for collection db2.people 2023-02-08T11:37:15.666+0100 567258 document(s) restored successfully. 0 document(s) failed to restore.
In the above examples, I used both mongodump/mongorestore and mongoexport/mongoimport tools to backup and recover your MongoDB data directly to and from the S3 object storage type, while doing it the streaming and compressed way. Therefore, these methods are simple, fast, and resource-friendly. I hope what I used will be useful when you are looking for options to use in your backup scripts or ad-hoc backup tasks.
Additional tools
Here, I would like to mention that there are other free and open source backup solutions you may try, including Percona Backup for MongoDB (PBM), which now offers both logical and physical backups:
- https://docs.percona.com/percona-backup-mongodb/backup-types.html
- https://www.percona.com/blog/physical-backup-support-in-percona-backup-for-mongodb/
With the Percona Server for MongoDB variant, you may also stream hot physical backups directly to S3 storage:
It is as easy as this:
mongo > db.runCommand({createBackup: 1, s3: {bucket: "mbackups", path: "my_physical_dump1", endpoint: "s3.eu-central-2.wasabisys.com"}}) { "ok" : 1 } $ s3cmd du -H s3://mbackups/my_physical_dump1/ 138M 26 objects s3://mbackups/my_physical_dump1/
For a sharded cluster, you should use PBM rather for consistent backups.
Btw, don’t forget to check out the MongoDB best backup practices!
Percona Distribution for MongoDB is a freely available MongoDB database alternative, giving you a single solution that combines the best and most important enterprise components from the open source community, designed and tested to work together.
08
2022
AWS RDS Backups: What’s the True Cost?
You have your database instance deployed with AWS and you are using AWS RDS for MySQL. All work smoothly in terms of satisfying queries for your application and delivering reliable uptime and performance. Now you need to take care of your backup strategy. Business is defined to have this retention policy:
- 7 daily full backups
- 4 weekly backups
- 12 monthly backups
Plus the ability to do point-in-time recovery (PITR) for the last 24 hours since the last full backup was taken.
The cloud vendor solution
This is a piece of cake. The daily backups: just set the backup retention period to six days. Done. This also has the plus that PITR is already covered since RDS uploads the transaction logs to S3 every five minutes and stores them in the parquet format, making it smaller than the regular text file. All amazing, right?
Now, the weekly and monthly retention policies are more tricky. But no problem since there is a backup service in AWS called…..AWS Backup. Without going into too much detail, you can create a backup plan for your RDS instance with the desired frequency that suits your retention policies, copy backups between regions for business continuity compliance, etc.
All this comes with a price tag. Let’s do some numbers using the AWS calculator, starting with the snapshots:
According to the documentation “There is no additional charge for backup storage up to 100% of your total database storage for a region.” In other words, this means that one daily snapshot is free. What about the remaining six?
For this example, I will assume the following: a single MySQL RDS instance using m6g.large (the smallest graviton type which is cheaper), Single-AZ, Reserved instance with three years term and paying it upfront. Now, for the disk: a gp2 disk (cheaper than an io1) with a storage capacity of 500GB and I will assume that only half is used.
From the AWS calculator, we have that the monthly cost of the additional backup storage (250GB x 5 days) is $118,75 USD per month or $1,425 USD annually. But that number is not static. Your data most likely will grow. How much will it cost when my data is no longer 250GB but 400GB? The monthly storage cost for the backups could increase to $218 USD, or $2,622 USD annually.
For the PITR, RDS uploads transaction logs to S3 every five minutes. It is not clear if this is charged or not. What it is clear is that in order to restore a DB to a specific point in time, one needs to create a completely new DB instance; you cannot do a PITR over the existing DB, so more costs.
If you are required to distribute your backups for disaster recovery plans or just for business continuity, moving snapshots to a different region comes with a cost depending on the destination region. For example: move to US-EAST-2 (Ohio) comes at a cost of $0.01 per GB. Moving the most recent full daily backup (400GB in our case) will cost $4 USD, which annually means $1,460 USD.
So far we have only discussed the daily backups made by the RDS snapshot feature. And it sums up to $4,082 USD yearly for only 400GB.
Moving on, above we mentioned that the weekly and monthly backups can be taken care of by AWS Backups service. We are not gonna deep dive into these costs primarily because the AWS calculator does not provide a means to use it for that service. However, it is documented that the storage pricing is $0.095 per GB-Month. Now, assuming the unrealistic scenario where your data size is static at 400GB, at the end of the first 12 months the cost will be $6,400 USD. Adding that to the cost of the snapshots, the grand total is:
$10,482 USD annually
And that is for just one RDS instance, with a fairly small data size (400GB), and is not assuming the cost of the instance itself, which depends on the type of instance and its size. Multiply that for the number of clusters in your environment!
Now, to provide some context of how much money that is, know that having an RDS Graviton MySQL instance of size db.m6g.4xlarge costs a bit more than $6000 and a db.m6g.2xlarge is at around $3000 a year. In other words: the annual cost of your backups is similar to having ON a whole year a server with 8 vCPU and 32GiB of memory AND a server with 32 vCPU with 128GiB of memory.
There are additional costs that we haven’t mentioned like if you want to move your backups to cold storage. Currently, the AWS Backup service doesn’t support cold storage for RDS backups, meaning that you need to take care of that by yourself and pay for the cost of that (additional to the storage cost). And if you want to store your data encrypted with your own KMS key, it comes with an additional cost.
Managing your backups like an expert
Percona is prepared to manage the complete lifecycle of your backups for a fraction of the cost and with additional features and capabilities, like PITR at the item level (you don’t need to restore the whole backup to get just the needed table or the needed rows) and can do it over the same instance, with encrypted backups for the same price, cold storage as part of the lifecycle of your data, and many more features.
Percona Managed Services is a solution from Percona to consider!
06
2021
MyDumper 0.10.7 is Now Available
The new MyDumper 0.10.7 version, which includes many new features and bug fixes, is now available. You can download the code from here.
For this release, we have added several features like WHERE support that is required for partial backups. We also added CHECKSUM for tables which help to speed up the restore of large tables to take advantage of fast index creation, and more.
New Features:
- Adding metadata file per table that contains the number of rows #353
- Adding –where support #347 #223
- Option to automatically disable/enable REDO_LOG #305 #334
- Adding wsrep_sync_wait support #327
- Adding fast index creation functionality #286
- Adding ORDER BY Primary Key functionality #227
- Added support for dumping checksums #141 #194
- Dump strings using single quote instead of double quotes #191
- Specify the number of snapshots #118
Bug Fixes:
- Fixed the create database section when creating with –source-db enabled #213
- Escaping quotes on detect_generated_fields as it caused segfault #349
- [usingfastindex] Indexes on AUTO_INCREMENT column should not be selected for fast index creation #322
- Fixed checksum compression #355
- Fixed as constraint was ignored #351
- Fixed int declaration to comply with C99 #346
Documentation:
- Added s to install #360
- Added libatomic1 dependency reference on ubuntu #358 #359
- Release signatures #28
Refactoring:
Won’t-fix:
07
2021
MyDumper 0.10.5 is Now Available
The new MyDumper 0.10.5 version, which includes many new features and bug fixes, is now available. You can download the code from here.
For this release, we focused on fixing some old issues and testing old pull requests to get higher quality code. On releases 0.10.1, 0.10.3, and 0.10.5, we released the packages compiled against MySQL 5.7 libraries, but from now on, we are also compiling against MySQL 8 libraries for testing purposes, not releasing, as we think that more people in the community will start compiling against the latest version, and we should be prepared.
New Features:
- Password obfuscation #312
- Using dynamic parameter for SET NAMES #154
- Refactor logging and enable –logfile in myloader #233
- Adding purge-mode option (NONE/TRUNCATE/DROP/DELETE) to decide what is the preferable mode #91 #25
- Avoid sending COMMIT when commit_count equals 1 #234
- Check if directory exists #294
Bug Fixes:
- Adding homebrew build support #313
- Removing MyISAM dependency in temporary tables for VIEWS #261
- Fix warnings in sphinx document generation #287
- Fix endless loop when processlist couldn’t be checked #295
- Fix issues when daemon is used on glist #317
Documentation:
- Correcting ubuntu/debian packages dependencies #310
- Provide better CentOS 7 build instructions #232
- –defaults-file usage for section mydumper and client #306
- INSERT IGNORE documentation added #195
Refactoring:
- Adding function new_table_job #315
- Adding IF NOT EXISTS to SHOW CREATE DATABASE #314
- Update FindMySQL.cmake #149
21
2021
Back From a Long Sleep, MyDumper Lives!
MySQL databases keep getting larger and larger. And the larger the databases get, the harder it is to backup and restore them. MyDumper has changed the way that we perform logical backups to enable you to restore tables or objects from large databases. Over the years it has evolved into a tool that we use at Percona to back up petabytes of data every day. It has several features, but the most important one, from my point of view, is how it speeds up the entire process of export and import.
Until the beginning of this year, the latest release was from 2018; yes, more than two years without any release. However, we started 2021 with release v0.10.1 in January, with all the merges up to that point and we committed ourselves to release every two months… and we delivered! Release v0.10.3 was released in March with some old pull requests that have been sleeping for a long time. The next release is planned to be in May, with some of the newest features.
Just to clarify, mydumper/myloader are not officially-supported Percona products. They are open source, community-managed tools for handling logical backups and restores with all flavors of MySQL.
What Has Changed?
The principal maintainer remains Max Bubenick, and I’ve been helping out with reviewing issues and pull requests to give better support to the community.
Better planning means that it is not just released on time; we also need to decide what is the new feature that we are going to be packaging in the next release, and the level of quality.
Register for Percona Live ONLINE
A Virtual Event about Open Source Databases
The releases were in 2021, but this effort started in 2020 and I had been working on the release repository as it wasn’t being maintained.
What’s Next?
Exciting times! There are three features that I would like to mention, not because the others are not important, but rather, because they will speed up the import stage.
- Fast index creation is finally arriving! This was one of the requested features that even mysqldump implemented.
- Not long ago I realized that two requests can be merged into one, and the community was asking about CSV export and LOAD DATA support.
- Finally, this request had been waiting for a long time – Is it possible for MyDumper to stream backups? We found a way and we are going to be working on v0.10.9.
I was able to measure the speed-up of the first two and we could get up to 40% on large tables with multiple secondary indexes. The latest one is not implemented yet, but taking into account that the import will be started alongside the export, we can expect a huge reduction in the timing.
Conclusion
We still have a lot of opportunities to make MyDumper a major league player. Feel free to download it, play with it, and if you want to contribute, we need your help writing code, testing, or asking for new features.
18
2020
MongoDB Database Backup: Best Practices & Expert Tips
This blog was originally published in September 2020 and was updated in May 2023.
In today’s data-driven world, losing critical data can be catastrophic for any organization. As a MongoDB user, it’s crucial to ensure that your data is safe and secure in the event of a disaster or system failure. That’s why it’s essential to implement the best practices and strategies for MongoDB database backups.
Why are MongoDB database backups important?
Regular database backups are essential to protect against data loss caused by system failures, human errors, natural disasters, or cyber-attacks. In the absence of a proper backup strategy, the data can be lost forever, leading to significant financial and reputational damage.
For organizations that rely on data to operate, database backups are critical for business continuity. With a robust backup and recovery plan in place, companies can restore their systems and data quickly and minimize downtime, which is essential to maintain customer trust and avoid business disruption.
In this blog, we will be discussing different MongoDB database backup strategies and their use cases, along with pros and cons and a few other useful tips.
What are the two different types of MongoDB database backups?
Generally, there are two types of backups used with databases technologies like MongoDB:
- Logical backups
- Physical backups
Additionally, when working with logical backups, we have the option of taking incremental backups as well, where we capture the deltas or incremental data changes made between full backups to minimize the amount of data loss in case of any disaster.
We will be discussing these two backup options, how to proceed with them, and which one suits better depending on requirements and environment setup.
Also, we will take a look at our open-source backup utility custom-built to help avoid costs and proprietary software – Percona Backup for MongoDB or PBM. PBM is a fully-supported community backup tool capable of performing cluster-side consistent backups in MongoDB for replica sets and sharded clusters.
Logical backups
These are the types of backups where data is dumped from the databases into the backup files. A logical backup with MongoDB means you’ll be dumping the data into a BSON-formatted file.
During the logical backups, using client API, the data gets read from the server and returned back to the same API, which will be serialized and written into respective “.bson”, “.json” or “.csv” backup files on disk depending upon the type of backup utilities used.
MongoDB offers the below utility to take the logical backups
Mongodump: Takes dump/backup of the databases into “.bson” format, which can be later restored by replaying the same logical statements captured in dump files back to the databases.
mongodump --host=mongodb1.example.net --port=27017 --username=user --authenticationDatabase=admin --db=demo --collection=events --out=/opt/backup/mongodump-2011-10-24
Note: If we don’t specify the DB name or Collection name explicitly in the above “mongodump” syntax then the backup will be taken for the entire database or collections, respectively. If “authorization” is enabled, then we must specify the “authenticationDatabase”
Also, you should use “–oplog” to take the incremental data while the backup is still running. We can specify “–oplog” with mongodump. Keep in mind that it won’t work with –db and –collection since it will only work for entire databases backups
mongodump --host=mongodb1.example.net --port=27017 --username=user --authenticationDatabase=admin --oplog --out=/opt/backup/mongodump-2011-10-24
Pros of logical backups
- It can take the backup at a more granular level, like a specific database or a collection which will be helpful during restoration.
- It does not require you to halt writes against a specific node where you will be running the backup. Hence, the node would still be available for other operations.
Cons of logical backups
- As it reads all data, it can be slow and will require disk reads too for databases that are larger than the RAM available for the WT cache—the WT cache pressure increases, which slows down the performance.
- It doesn’t capture the index data in the metadata backup file. Thus while restoring, all the indexes have to be built again for each collection after the collection has been reinserted. This will be done serially in one pass through the collection after the inserts have finished, which can add a lot of time for big collection restores.
- The speed of backup also depends on allocated IOPS and type of storage since lots of read/writes would be happening during this process.
- Logical backups such as mongodump are, in general, very time-consuming for large systems.
Best practice tip: It is always advisable to use secondary servers for backups to avoid unnecessary performance degradation on the PRIMARY node.
As we have different types of environment setups, we should approach each one of them like below.
- Replica set: Always preferred to run on secondaries.
- Shard clusters: Take a backup of config server replicaset and each shard individually using the secondary nodes of them.
Since we are discussing distributed database systems like sharded clusters, we should also keep in mind that we want to have consistency in our backups at a point in time. ( Replica sets backups using mongodump are generally consistent using “–oplog” )
Let’s discuss this scenario where the application is still writing data and cannot be stopped for business reasons. Even if we take a backup of the config server and each shard separately, the backups of each shard will finish at different times because of data volume, data distribution, load, etc. Hence, while restoring, some inconsistencies might occur for the same reason.
Now comes the restoration part when dealing with logical backups. Same as for backups, MongoDB provides the below utilities for restoration purposes.
Mongorestore: Restores dump files created by “mongodump”. Index recreation will take place only after the data is restored, which causes the use of additional memory resources and time.
mongorestore --host=mongodb1.example.net --port=27017 --username=user --password --authenticationDatabase=admin --db=demo --collection=events /opt/backup/mongodump-2011-10-24/events.bson
For the restore of the incremental dump, we can add –oplogReplay in the above syntax to replay the oplog entries as well.
Best practice tip: The “–oplogReplay” can’t be used with –db and –collection flag as it will only work while restoring all the databases.
Learn how to restore a MongoDB logical backup
Percona Backup for MongoDB
It is a distributed, low-impact solution for achieving consistent backups of MongoDB sharded clusters and replica sets. Percona Backup for MongoDB helps overcome the issues around consistency while taking backups of sharded clusters. Percona Backup for MongoDB is an uncomplicated command-line tool by design that lends itself well to backing up larger data sets. PBM uses the faster “s2” library and parallelized threads to improve speed and performance if extra threads are available as resources.
Some main advantages of PBM include the following:
- Enables Backups with replica set and sharded cluster consistency via oplog capture
- Provides Distributed transaction consistency with MongoDB 4.2+
- Back up anywhere – to the cloud (use any S3-compatible storage) or on-premise with a locally-mounted remote file system
- It allows you to choose which compression algorithms to use. In some internal experiments, the “s2” library with snappy compression running parallelized with multiple threads was significantly faster than regular gzip. Caveat: Good as long as you have the additional resources available for running the parallel threads.
- Records backup progress logging. If you would like to see the speed of the backup (upload MB/s rate), you can look at the pbm-agent node’s logs to see the current progress. If you have a large backup, you can track backup progress in pbm-agent logs. A line is appended every minute, showing bytes copied vs. total size for the current collection.
- PBM Allows for Point-in-Time Recoveries – restoring a database up to a specific moment. P-I-T-R’s restore data from a backup and then replay all actions that happened to the data up to the specified moment from oplog slices.
- PITR’s help you prevent data loss during a disaster such as crashed database, accidental data deletion or drop of tables, and unwanted updates of multiple fields instead of a single one.
- PBM is optimized to allow backups and has minimal impact on your production performance.
Best practice tip: Use PBM to time huge backup sets. Many people don’t realize how long it takes to back up very large data sets. And they are generally very surprised at how long it takes to restore them! Especially if going into or out of storage types that may throttle bandwidth/network traffic.
Best practice tip: When running PBM from an unsupervised script, we recommend using a replica set connection string. A direct or stand-alone style connection string will fail if that MongoDB host happens to be unavailable or down temporarily.
When a PBM backup is triggered, it tails and captures the oplog from config server replica set and all the shards while the backup is still running, thus providing consistency once the backup is completed.
It has a feature of taking incremental backups as well, apart from complete database backup with “PITR” parameter enabled. It does all this by running “pbm-agent” on the DB (“mongod”) nodes of the cluster and is responsible for the backups and restore purposes.
As we can see below, the “pbm list” command shows the complete list of backups in the backup snapshots section and the incremental backups in the “PITR” section.
Below is the sample output:
$ pbm list Backup snapshots: 2020-09-10T12:19:10Z 2020-09-14T10:44:44Z 2020-09-14T14:26:20Z 2020-09-17T16:46:59Z PITR <on>: 2020-09-14T14:26:40 - 2020-09-16T17:27:26 2020-09-17T16:47:20 - 2020-09-17T16:57:55
If you have a large backup, you can track backup progress in pbm-agent logs. Let’s take a look at the output of “pbm-agent” as well while it is taking the backup.
Aug 19 08:46:51 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 08:46:51 Got command backup [{backup {2020-08-19T08:46:50Z s2} { } { 0} 1597826810}] Aug 19 08:47:07 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 08:47:07 [INFO] backup/2020-08-19T08:46:50Z: backup started Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.891+0000 writing admin.system.users to archive on stdout Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.895+0000 done dumping admin.system.users (2 documents) Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.895+0000 writing admin.system.roles to archive on stdout Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.904+0000 done dumping admin.system.roles (1 document) Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.904+0000 writing admin.system.version to archive on stdout Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.914+0000 done dumping admin.system.version (5 documents) Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.914+0000 writing testmongo.col to archive on stdout Aug 19 08:47:09 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:09.942+0000 writing test.collC to archive on stdout Aug 19 08:47:13 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:13.499+0000 done dumping test.collC (1146923 documents) Aug 19 08:47:13 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:13.500+0000 writing test.collA to archive on stdout Aug 19 08:47:27 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:27.964+0000 done dumping test.collA (389616 documents) Aug 19 08:47:27 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:27.965+0000 writing test.collG to archive on stdout Aug 19 08:47:54 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:54.891+0000 done dumping testmongo.col (13280501 documents) Aug 19 08:47:54 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T08:47:54.896+0000 writing test.collF to archive on stdout Aug 19 08:48:09 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 08:48:09 [........................] test.collG 1533/195563 (0.8%) Aug 19 08:48:09 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 08:48:09 [####################....] test.collF 116432/134747 (86.4%) Aug 19 10:01:09 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 10:01:09 [#######################.] test.collG 195209/195563 (99.8%) Aug 19 10:01:17 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 10:01:17 [########################] test.collG 195563/195563 (100.0%) Aug 19 10:01:17 ip-172-30-2-122 pbm-agent[24331]: 2020-08-19T10:01:17.650+0000 done dumping test.collG (195563 documents) Aug 19 10:01:20 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 10:01:20 [INFO] backup/2020-08-19T08:46:50Z: mongodump finished, waiting for the oplog Aug 19 10:11:04 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 10:11:04 [INFO] backup/2020-08-19T08:46:50Z: backup finished Aug 19 10:11:05 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 10:11:05 [INFO] pitr: streaming started from 2020-08-19 08:47:09 +0000 UTC / 1597826829 Aug 19 10:29:37 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 10:29:37 [INFO] pitr: created chunk 2020-08-19T08:47:09 - 2020-08-19T10:20:59. Next chunk creation scheduled to begin at ~2020-08-19T10:31:05 Aug 19 10:39:34 ip-172-30-2-122 pbm-agent[24331]: 2020/08/19 10:39:34 [INFO] pitr: created chunk 2020-08-19T10:20:59 - 2020-08-19T10:30:59. Next chunk creation scheduled to begin at ~2020-08-19T10:41:05
The last three lines of the above output mean that the full backup is completed, and the incremental backup is started with a sleep interval of 10 minutes. This is an example of the Backup Progress Logging mentioned above.
We will be discussing more about Percona Backup for MongoDB in an upcoming blog post. Until then, you can find more details on the Percona Backup for MongoDB Documentation page on our website.
Physical/filesystem backups
It involves snapshotting or copying the underlying MongoDB data files (–dbPath) at a point in time and allowing the database to cleanly recover using the state captured in the snapshotted files. They are instrumental in backing up large databases quickly, especially when used with filesystem snapshots, such as LVM snapshots, or block storage volume snapshots.
There are several general methods to take the filesystem level backup, also known as physical backups.
- Manually Copying the entire data files (using Rsync → Depends on N/W bandwidth)
- LVM based snapshots
- Cloud-based disk snapshots (AWS / GCP / Azure or any other cloud provider)
- Percona Server for MongoDB also includes an integrated open-source Hot Backup system that creates a physical data backup on a running server without notable performance and operating degradation. You can find more information about Percona Server for MongoDB Hot Backup here.
We’ll be discussing all these above options, but first, let’s look at the Pros and Cons of Physical Backups over Logical backups.
Pros of physical backups
- They are at least as fast as, and usually faster than, logical backups.
- Can be easily copied over or shared with remote servers or attached NAS.
- Recommended for large datasets because of speed and reliability
- Can be convenient while building new nodes within the same cluster or new cluster
Cons of physical backups
- It is not possible to restore on a less granular level, such as a specific DB or Collection restore
- Incremental backups cannot be achieved yet
- A dedicated node is recommended for backup (it might be a hidden one) as it requires halting writes or shutting down “mongod” cleanly prior to the snapshot against the node to achieve consistency.
Below is the backup time consumption comparison for the same dataset:
DB Size: 267.6GB
Index Size: <1MB (since it was only on _id for testing)
demo:PRIMARY> db.runCommand({dbStats: 1, scale: 1024*1024*1024}) { "db" : "test", "collections" : 1, "views" : 0, "objects" : 137029, "avgObjSize" : 2097192, "dataSize" : 267.6398703530431, "storageSize" : 13.073314666748047, "numExtents" : 0, "indexes" : 1, "indexSize" : 0.0011749267578125, "scaleFactor" : 1073741824, "fsUsedSize" : 16.939781188964844, "fsTotalSize" : 49.98826217651367, "ok" : 1, ... } demo:PRIMARY>
=============================
- Percona Server for MongoDB’s hot backup:
Syntax:
> use admin switched to db admin > db.runCommand({createBackup: 1, backupDir: "/my/backup/data/path"}) { "ok" : 1 }
Best practice tip: The backup path “backupDir” should be absolute. It also supports storing the backups on filesystem and AWS S3 buckets.
[root@ip-172-31-37-92 tmp]# time mongo < hot.js Percona Server for MongoDB shell version v4.2.8-8 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("c9860482-7bae-4aae-b0e7-5d61f8547559") } Percona Server for MongoDB server version: v4.2.8-8 switched to db admin { "ok" : 1, ... } bye real 3m51.773s user 0m0.067s sys 0m0.026s [root@ip-172-31-37-92 tmp]# ls hot hot.js mongodb-27017.sock nohup.out systemd-private-b8f44077314a49899d0a31f99b31ed7a-chronyd.service-Qh7dpD tmux-0 [root@ip-172-31-37-92 tmp]# du -sch hot 15G hot 15G total
Notice the time taken by “Percona Hot Backup” was just 4 minutes approx.
This is very helpful when rebuilding a node or spinning up new instances/clusters with the same dataset. The best part is it doesn’t compromise performance with locking of writes or other performance hits.
Best practice tip: It is recommended to run it against the secondaries.
- Filesystem snapshot:
The approx time taken for the snapshot to be completed was only four minutes.
[root@ip-172-31-37-92 ~]# aws ec2 describe-snapshots --query "sort_by(Snapshots, &StartTime)[-1].{SnapshotId:SnapshotId,StartTime:StartTime}" { "SnapshotId": "snap-0f4403bc0fa0f2e9c", "StartTime": "2020-08-26T12:26:32.783Z" }
[root@ip-172-31-37-92 ~]# aws ec2 describe-snapshots > --snapshot-ids snap-0f4403bc0fa0f2e9c { "Snapshots": [ { "Description": "This is my snapshot backup", "Encrypted": false, "OwnerId": "021086068589", "Progress": "100%", "SnapshotId": "snap-0f4403bc0fa0f2e9c", "StartTime": "2020-08-26T12:26:32.783Z", "State": "completed", "VolumeId": "vol-0def857c44080a556", "VolumeSize": 50 } ] }
3. Mongodump:
[root@ip-172-31-37-92 ~]# time nohup mongodump -d test -c collG -o /mongodump/ & [1] 44298 [root@ip-172-31-37-92 ~]# sed -n '1p;$p' nohup.out 2020-08-26T12:36:20.842+0000 writing test.collG to /mongodump/test/collG.bson 2020-08-26T12:51:08.832+0000 [####....................] test.collG 27353/137029 (20.0%)
Results: As you can see from this quick example using the same dataset – both the file system level snapshot and Percona Server for MongoDB Hot Backup methods took only 3-5 minutes. However, “mongodump” took almost 15 minutes for just 20% of the dump to complete. Hence the speed to back up the data with mongodump is definitely very slow when compared to the other two options discussed. That is where the s2 compression and the parallelized threads of Percona Backup for MongoDB can help.
Learn more about physical backup support in Percona Backup for MongoDB
Key factors to consider when choosing a MongoDB database backup solution
In this section below, we will discuss the key factors to consider when choosing a MongoDB database backup solution.
Scalability
To ensure the longevity of a MongoDB database, a backup solution must be created with the database’s growth in mind. MongoDB is a flexible NoSQL database that can expand horizontally by incorporating additional servers or shards, as well as vertically by increasing the resources available on existing servers.
Furthermore, an effective MongoDB backup solution should incorporate scalable storage alternatives, such as cloud storage or distributed file systems. These storage solutions offer the ability to expand storage capacity without requiring significant alterations to your existing backup infrastructure.
Performance
MongoDB backup solutions can have a significant impact on database performance, particularly when you are backing up large databases or using them during peak usage hours. Here are some of the things to consider when choosing a backup solution to minimize its impact on your MongoDB database performance:
- The type of backup solution: Full backups are time-consuming and resource-intensive, while incremental backups only save changes since the last backup and are typically faster and less resource-intensive.
- Storage destination: Backups stored on the same disk as the database can impact read and write operations, while backups stored on a remote server can increase network traffic and cause latency.
- Database size: The larger the database, the longer it will need to backup and restore.
- Frequency of backups: Frequent backups consume more resources, while infrequent backups increase the risk of data loss. It is important to balance data protection and database performance to achieve optimal results.
- Backup schedule: To minimize any effect on database users, schedule backups during off-peak hours.
- Compression and security: Although compression and encryption can reduce the backup size and improve security, they may also impact database performance. Compression necessitates additional CPU resources, while encryption requires additional I/O resources, both of which can potentially affect database performance.
Security
Backing up your MongoDB database is critical to safeguard your data from unauthorized access, damage, or theft. Here are some ways in which a MongoDB backup solution can help:
- Disaster recovery: In case of a natural disaster or a hacker, a backup solution helps you recover your data. Regularly backing up your MongoDB database ensures that you can restore your data to a previous state if it gets lost or corrupted.
- Data encryption: Sensitive data can be kept secure with data encryption at rest and in transit via a backup solution.
- Access control: A good backup solution lets you regulate data access and set up encryption and authentication protocols to ensure only authorized users have access.
- Version control: Keeping track of different versions of your data is easier with a backup solution, enabling you to roll back to a previous version (or compare versions over time).
- Offsite backup: Offsite backups protects data from physical theft or damage. It can also help you comply with any regulations requiring offsite backup storage.
Free your applications with Percona Distribution for MongoDB
Choosing a MongoDB backup solution
The best method for taking the backups depends on multiple factors like the type of infrastructure, environment, resources available, dataset size, load, etc. However, consistency and complexity also play a major role while taking backups of distributed database systems
In general, for smaller instances, simple logical backups via mongodump are fine. As you reach somewhat larger database sizes above around 100G, use backup methods like Percona Backup for MongoDB that include incremental backups and capture the oplogs in order to be able to perform Point-in-Time Recoveries and minimize potential data loss.
PBM allows you to backup to anywhere – in the cloud or on-prem, can handle your larger backups, and it is optimized to have minimal impact on your production performance. PBM is also faster due to the use of the “s2” compression method and using parallelized threads. Finally, PBM can overcome consistency issues often seen with replica set and sharded clusters by capturing the changes in the oplog.
For very large systems, aka once you reach around the 1TB+ range, you should look to utilize physical file system-level snapshot backups. One available tool for that is open-source – Percona Server for MongoDB has the integrated Hot Backup functionality built-in for the default WiredTiger storage engine and takes around the same time as other physical snapshots.
01
2019
Setting World-Writable File Permissions Prior to Preparing the Backup Can Break It
It’s bad practice to provide world-writable access to critical files in Linux, though we’ve seen time and time again that this is done to conveniently share files with other users, applications, or services. But with Xtrabackup, preparing backups could go wrong if the backup configuration has world-writable file permissions.
Say you performed a backup on a MySQL instance configured with data-at-rest encryption using the keyring plugin. On the backup directory, the generated backup-my.cnf contains these instructions to load this plugin that will be used by Xtrabackup while preparing the backup:
backup-my.cnf
[mysqld] innodb_checksum_algorithm=crc32 innodb_log_checksum_algorithm=strict_crc32 innodb_data_file_path=ibdata1:12M:autoextend innodb_log_files_in_group=2 innodb_log_file_size=1073741824 innodb_fast_checksum=false innodb_page_size=16384 innodb_log_block_size=512 innodb_undo_directory=./ innodb_undo_tablespaces=0 server_id=0 redo_log_version=1 plugin_load=keyring_file.so server_uuid=00005726-0000-0000-0000-000000005726 master_key_id=1
Perhaps you wanted to share the backup with another user, but made a mistake of making the directory and its contents world-writable: chmod -R 777 /backup/mysql
When that user prepares the backup, the corresponding output will show that Xtrabackup ignored reading backup-my.cnf and so it doesn’t know that it has to load the keyring plugin to decrypt the .ibd files:
~$ xtrabackup --prepare --keyring-file-data=/backup/mysql/keyring --target-dir=/backup/mysql xtrabackup: [Warning] World-writable config file '/backup/mysql/backup-my.cnf' is ignored. xtrabackup: recognized server arguments: xtrabackup: [Warning] World-writable config file '/backup/mysql/backup-my.cnf' is ignored. xtrabackup: recognized client arguments: --prepare=1 --target-dir=/backup/mysql xtrabackup version 2.4.14 based on MySQL server 5.7.19 Linux (x86_64) (revision id: ef675d4) xtrabackup: cd to /backup/mysql/ xtrabackup: This target seems to be not prepared yet. InnoDB: Number of pools: 1 xtrabackup: xtrabackup_logfile detected: size=215089152, start_lsn=(3094928949) xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = . xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend xtrabackup: innodb_log_group_home_dir = . xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 215089152 xtrabackup: [Warning] World-writable config file './backup-my.cnf' is ignored. xtrabackup: using the following InnoDB configuration for recovery: xtrabackup: innodb_data_home_dir = . xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend xtrabackup: innodb_log_group_home_dir = . xtrabackup: innodb_log_files_in_group = 1 xtrabackup: innodb_log_file_size = 215089152 xtrabackup: Starting InnoDB instance for recovery. xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) InnoDB: PUNCH HOLE support available InnoDB: Mutexes and rw_locks use GCC atomic builtins InnoDB: Uses event mutexes InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier InnoDB: Compressed tables use zlib 1.2.8 InnoDB: Number of pools: 1 InnoDB: Using CPU crc32 instructions InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M InnoDB: Completed initialization of buffer pool InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). InnoDB: Highest supported file format is Barracuda. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest2.ibd can't be decrypted. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest1.ibd can't be decrypted. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest4.ibd can't be decrypted. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest3.ibd can't be decrypted. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest5.ibd can't be decrypted. InnoDB: Log scan progressed past the checkpoint lsn 3094928949 ** redacted ** InnoDB: Doing recovery: scanned up to log sequence number 3097681408 (1%) InnoDB: Doing recovery: scanned up to log sequence number 3102924288 (4%) InnoDB: Doing recovery: scanned up to log sequence number 3108167168 (6%) InnoDB: Doing recovery: scanned up to log sequence number 3113410048 (9%) InnoDB: Doing recovery: scanned up to log sequence number 3118652928 (12%) InnoDB: Doing recovery: scanned up to log sequence number 3123895808 (15%) InnoDB: Doing recovery: scanned up to log sequence number 3129138688 (17%) InnoDB: Doing recovery: scanned up to log sequence number 3134381568 (20%) InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 ** redacted ** InnoDB: Doing recovery: scanned up to log sequence number 3265453568 (89%) InnoDB: Doing recovery: scanned up to log sequence number 3270696448 (91%) InnoDB: Doing recovery: scanned up to log sequence number 3275939328 (94%) InnoDB: Doing recovery: scanned up to log sequence number 3281182208 (97%) InnoDB: Doing recovery: scanned up to log sequence number 3286158358 (100%) InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: xtrabackup: Last MySQL binlog file position 568369058, file name mysql-bin.000004 InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest1.ibd can't be decrypted. InnoDB: Removing missing table `sbtest/sbtest1` from InnoDB data dictionary. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest2.ibd can't be decrypted. InnoDB: Removing missing table `sbtest/sbtest2` from InnoDB data dictionary. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest3.ibd can't be decrypted. InnoDB: Removing missing table `sbtest/sbtest3` from InnoDB data dictionary. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest4.ibd can't be decrypted. InnoDB: Removing missing table `sbtest/sbtest4` from InnoDB data dictionary. InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. InnoDB: Encryption information in datafile: ./sbtest/sbtest5.ibd can't be decrypted. InnoDB: Removing missing table `sbtest/sbtest5` from InnoDB data dictionary. InnoDB: Creating shared tablespace for temporary tables InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... InnoDB: File './ibtmp1' size is now 12 MB. InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active. InnoDB: 32 non-redo rollback segment(s) are active. InnoDB: page_cleaner: 1000ms intended loop took 6627ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.) InnoDB: 5.7.19 started; log sequence number 3286158358 InnoDB: xtrabackup: Last MySQL binlog file position 568369058, file name mysql-bin.000004
Even if you fix the permissions on backup-my.cnf, if you try to prepare the same backup again, Xtrabackup will warn you that it has already prepared the backup.
~$ xtrabackup --prepare --keyring-file-data=/backup/mysql/keyring --target-dir=/backup/mysql xtrabackup: recognized server arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksum_algorithm=strict_crc32 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=1073741824 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=./ --innodb_undo_tablespaces=0 --server-id=0 --redo-log-version=1 xtrabackup: recognized client arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksum_algorithm=strict_crc32 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=1073741824 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=./ --innodb_undo_tablespaces=0 --server-id=0 --redo-log-version=1 --prepare=1 --target-dir=/backup/mysql xtrabackup version 2.4.14 based on MySQL server 5.7.19 Linux (x86_64) (revision id: ef675d4) xtrabackup: cd to /backup/mysql/ xtrabackup: This target seems to be already prepared. InnoDB: Number of pools: 1
This means that changes made while the backup was taking place will not be applied and what you have restored is an inconsistent, potentially corrupt backup. You need to perform a full backup again and make sure that you do not place world/other writable permissions on the backup this around so that you will not face the same issue.
17
2019
Percona Backup for MongoDB 0.5.0 Early Release is Now Available
Percona is pleased to announce the early release of our latest software product Percona Backup for MongoDB 0.5.0 on June 17, 2019. The GA version is scheduled to be released later in 2019.
Percona Backup for MongoDB is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a single replica set), and for restoring those backups to a specific point in time. Percona Backup for MongoDB uses a distributed client/server architecture to perform backup/restore actions. The project was inspired by (and intends to replace) the Percona-Lab/mongodb_consistent_backup tool.
Percona Backup for MongoDB supports Percona Server for MongoDB or MongoDB Community Server version 3.6 or higher with MongoDB replication enabled. Binaries for the supported platforms as well as the tarball with source code are available from the Percona Backup for MongoDB download page. For more information about Percona Backup for MongoDB and the installation steps, see the documentation.
Percona Backup for MongoDB 0.5.0 features the following:
- Enables storing backup metadata on Amazon Simple Storage Service storages.
- The API of Percona Backup for MongoDB introduces HTTP basic authentication to prevent an unauthorized user from running backups or restoring data if they manage to access the API port.
- To optimize the usage of network resources, the pbm-agent on
mongos
is not needed anymore andbackup coordinator
automatically establishes connection to the appropriatemongos
instance. - The output of
pbmctl list nodes
now includes the replica set name and informs the backup status of the node.
Percona doesn’t recommend this release for production as its API and configuration fields are still likely to change. It only features a basic API level security. Please report any bugs you encounter in our bug tracking system.
New Features and Improvements
- PBM-93: Support storage of backup metadata on AWS S3.
- PBM-99:
pbm-agent
is deprecated onmongos
- PBM-105: Log a warning if a Primary node-type is used for a backup
- PBM-122: Include the replica set name to the output of
pmbctl list nodes
- PBM-130: Add HTTP Basic Authentication to gRPC servers (API and RPC)
- PBM-139: Support listing backup status in the output of
pmbctl list nodes
- PBM-170: Enable setting the ‘stopOnError’ attribute in
mongorestore
to ensure consistency of the data being restored.
Percona Backup for MongoDB, a free open source back-up solution, will enable you to manage your own back-ups without third party involvement or costly licenses. Percona Backup for MongoDB is distributed under the terms of Apache License v2.0.
We look forward to sharing the GA version of Percona Backup for MongoDB later this year and appreciate any feedback that you might have on our 0.5.0 version in the meantime.