Jan
07
2022
--

Configure wiredTiger cacheSize Inside Percona Distribution for MongoDB Kubernetes Operator

wiredTiger cacheSize Inside Percona Distribution for MongoDB Kubernetes Operator

wiredTiger cacheSize Inside Percona Distribution for MongoDB Kubernetes OperatorNowadays we are seeing a lot of customers starting to use our Percona Distribution for MongoDB Kubernetes Operator. The Percona Kubernetes Operators are based on best practices for the configuration of a Percona Server for MongoDB replica set or the sharded cluster. The main component in MongoDB is the wiredTiger cache which helps to define the cache used by this engine and we can set it based on our load.

In this blog post, we will see how to define the resources’ memory and set the wiredTiger cache for the shard replicaset to improve the performance of the sharded cluster.

The Necessity of WT cache

The parameter storage.wiredTiger.engineConfig.cacheSizeGB limits the size of the WiredTiger internal cache. The operating system will use the available free memory for filesystem cache, which allows the compressed MongoDB data files to stay in memory. In addition, the operating system will use any free RAM to buffer file system blocks and file system cache. To accommodate the additional consumers of RAM, you may have to set WiredTiger’s internal cache size properly.

Starting from MongoDB 3.4, the default WiredTiger internal cache size is the larger of either:

50% of (RAM - 1 GB), or 256 MB.

For example, on a system with a total of 4GB of RAM the WiredTiger cache will use 1.5GB of RAM (0.5 * (4 GB – 1 GB) = 1.5 GB). Conversely, a system with a total of 1.25 GB of RAM will allocate 256 MB to the WiredTiger cache because that is more than half of the total RAM minus one gigabyte (0.5 * (1.25 GB – 1 GB) = 128 MB < 256 MB).

WT cacheSize in Kubernetes Operator

The mongodb wiredTiger cacheSize can be tune with the parameter storage.wiredTiger.engineConfig.cacheSizeRatio and its default value is 0.5. As explained above, if the system allocated memory limit is too low, then the WT cache is set to 256M or calculated as per the formula.

Prior to PSMDB operator 1.9.0, the cacheSizeRatio can be tuned under the sharding section of the cr.yaml file. This is deprecated from v1.9.0+ and unavailable from v1.12.0+. So you have to use the cacheSizeRatio parameter available under replsets configuration instead. The main thing that you will need to check here before changing the cacheSize is to make sure that the resources’ memory limit allocated is also available as per your cacheSize’s requirement. i.e the below section limiting the memory:

     resources:
       limits:
         cpu: "300m"
         memory: "0.5G"
       requests:
         cpu: "300m"
         memory: "0.5G"

 

https://github.com/percona/percona-server-mongodb-operator/blob/main/pkg/psmdb/container.go#L307

From the source code that calculates the mongod.storage.wiredTiger.engineConfig.cacheSizeRatio:

// In normal situations WiredTiger does this default-sizing correctly but under Docker
// containers WiredTiger fails to detect the memory limit of the Docker container. We
// explicitly set the WiredTiger cache size to fix this.
//
// https://docs.mongodb.com/manual/reference/configuration-options/#storage.wiredTiger.engineConfig.cacheSizeGB//

func getWiredTigerCacheSizeGB(resourceList corev1.ResourceList, cacheRatio float64, subtract1GB bool) float64 {
 maxMemory := resourceList[corev1.ResourceMemory]
 var size float64
 if subtract1GB {
  size = math.Floor(cacheRatio * float64(maxMemory.Value()-gigaByte))
 } else {
  size = math.Floor(cacheRatio * float64(maxMemory.Value()))
 }
 sizeGB := size / float64(gigaByte)
 if sizeGB < minWiredTigerCacheSizeGB {
  sizeGB = minWiredTigerCacheSizeGB
 }
 return sizeGB
}

 

Changing the cacheSizeRatio

Here for the test, we deployed the PSMDB operator on GCP. You can refer here for the steps – https://www.percona.com/doc/kubernetes-operator-for-psmongodb/gke.html. With the latest operator v1.11.0, the sharded cluster has been started with a shard and a config server replicaSets along with mongos pods.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-cluster-name-cfg-0 2/2 Running 0 4m9s
my-cluster-name-cfg-1 2/2 Running 0 2m55s
my-cluster-name-cfg-2 2/2 Running 1 111s
my-cluster-name-mongos-758f9fb44-d4hnh 1/1 Running 0 99s
my-cluster-name-mongos-758f9fb44-d5wfm 1/1 Running 0 99s
my-cluster-name-mongos-758f9fb44-wmvkx 1/1 Running 0 99s
my-cluster-name-rs0-0 2/2 Running 0 4m7s
my-cluster-name-rs0-1 2/2 Running 0 2m55s
my-cluster-name-rs0-2 2/2 Running 0 117s
percona-server-mongodb-operator-58c459565b-fc6k8 1/1 Running 0 5m45s

Now login into the shard and check the default memory allocated to the container and to the mongod instance. In below, the memory size available is 15G, but the memory limit to use in this container is 476MB only:

rs0:PRIMARY> db.hostInfo()
{
"system" : {
"currentTime" : ISODate("2021-12-30T07:16:59.441Z"),
"hostname" : "my-cluster-name-rs0-0",
"cpuAddrSize" : 64,
"memSizeMB" : NumberLong(15006),
"memLimitMB" : NumberLong(476),
"numCores" : 4,
"cpuArch" : "x86_64",
"numaEnabled" : false
},
"os" : {
"type" : "Linux",
"name" : "Red Hat Enterprise Linux release 8.4 (Ootpa)",
"version" : "Kernel 5.4.144+"
},
"extra" : {
"versionString" : "Linux version 5.4.144+ (builder@7d732a1aec13) (Chromium OS 12.0_pre408248_p20201125-r7 clang version 12.0.0 (/var/tmp/portage/sys-devel/llvm-12.0_pre408248_p20201125-r7/work/llvm-12.0_pre408248_p20201125/clang f402e682d0ef5598eeffc9a21a691b03e602ff58)) #1 SMP Sat Sep 25 09:56:01 PDT 2021",
"libcVersion" : "2.28",
"kernelVersion" : "5.4.144+",
"cpuFrequencyMHz" : "2000.164",
"cpuFeatures" : "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities",
"pageSize" : NumberLong(4096),
"numPages" : 3841723,
"maxOpenFiles" : 1048576,
"physicalCores" : 2,
"mountInfo" : [
..
..

 

The cachesize in MB of wiredTiger engine allocated in Shard is as follows:

rs0:PRIMARY> db.serverStatus().wiredTiger.cache["maximum bytes configured"]/1024/1024
256

The cache size of 256MB is too low for the real environment. So let’s see how to tune the memory limit and also the cacheSize of WT engine. You can use the parameter called cacheSizeRatio to mention the WT cache ratio (out of 1) and memlimit to mention the memory allocated to the container. To do this, edit the cr.yaml file under deploy directory in the operator to change the settings. From the PSMDB operator v1.9.0, editing cacheSizeRatio parameter under mongod section is deprecated. So for the WT cache limit, use the cacheSizeRatio parameter under the section “replsets” and to set memory, use the memlimit parameter. Setting 3G for the container and 80% of the memory calculations.

deploy/cr.yaml:58

46 configuration: |
47 # operationProfiling:
48 # mode: slowOp
49 # systemLog:
50 # verbosity: 1
51 storage:
52 engine: wiredTiger
53 # inMemory:
54 # engineConfig:
55 # inMemorySizeRatio: 0.9
56 wiredTiger:
57 engineConfig:
58 cacheSizeRatio: 0.8

 

deploy/cr.yaml:229-232:

226 resources:
227 limits:
228 cpu: "300m"
229 memory: "3G"
230 requests:
231 cpu: "300m"
232 memory: "3G"

 

Apply the new cr.yaml

# kubectl appli -f deploy/cr.yaml
perconaservermongodb.psmdb.percona.com/my-cluster-name configured

The shard pods are re-allocated and you can check the progress as follows:

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-cluster-name-cfg-0 2/2 Running 0 36m
my-cluster-name-cfg-1 2/2 Running 0 35m
my-cluster-name-cfg-2 2/2 Running 1 34m
my-cluster-name-mongos-758f9fb44-d4hnh 1/1 Running 0 34m
my-cluster-name-mongos-758f9fb44-d5wfm 1/1 Running 0 34m
my-cluster-name-mongos-758f9fb44-wmvkx 1/1 Running 0 34m
my-cluster-name-rs0-0 0/2 Init:0/1 0 13s
my-cluster-name-rs0-1 2/2 Running 0 60s
my-cluster-name-rs0-2 2/2 Running 0 8m33s
percona-server-mongodb-operator-58c459565b-fc6k8 1/1 Running 0 38m

Now check the new settings of WT cache as follows:

rs0:PRIMARY> db.hostInfo().system
{
"currentTime" : ISODate("2021-12-30T08:37:38.790Z"),
"hostname" : "my-cluster-name-rs0-1",
"cpuAddrSize" : 64,
"memSizeMB" : NumberLong(15006),
"memLimitMB" : NumberLong(2861),
"numCores" : 4,
"cpuArch" : "x86_64",
"numaEnabled" : false
}
rs0:PRIMARY> 
rs0:PRIMARY> 
rs0:PRIMARY> db.serverStatus().wiredTiger.cache["maximum bytes configured"]/1024/1024
1474

Here, the memory calculation for WT is done roughly as follows (Memory limit should be more than 1G, else 256MB is allocated by default:
(Memory limit – 1G) * cacheSizeRatio

(2861 - 1) *0.8 = 1467

 

NOTE:

Till PSMDB operator v1.10.0, the operator takes the change of cacheSizeRatio only if the resources.limit.cpu is also set. This is a bug and it got fixed in v1.11.0 – refer https://jira.percona.com/browse/K8SPSMDB-603 . So if you’re in an older version, don’t be surprised and you have to make sure the resources.limit.cpu is set as well.

https://github.com/percona/percona-server-mongodb-operator/blob/v1.10.0/pkg/psmdb/container.go#L194

if limit, ok := resources.Limits[corev1.ResourceCPU]; ok && !limit.IsZero() {
args = append(args, fmt.Sprintf(
"--wiredTigerCacheSizeGB=%.2f",
getWiredTigerCacheSizeGB(resources.Limits, replset.Storage.WiredTiger.EngineConfig.CacheSizeRatio, true),
))
}

From v1.11.0:
https://github.com/percona/percona-server-mongodb-operator/blob/v1.11.0/pkg/psmdb/container.go#L194

if limit, ok := resources.Limits[corev1.ResourceMemory]; ok && !limit.IsZero() {
    args = append(args, fmt.Sprintf(
       "--wiredTigerCacheSizeGB=%.2f",
       getWiredTigerCacheSizeGB(resources.Limits, replset.Storage.WiredTiger.EngineConfig.CacheSizeRatio, true),
))
}

 

Conclusion

So based on the application load, you will need to set the cacheSize of WT for better performance. You can use the above methods to tune the cache size for the shard replicaset in the PSMDB operator.

Reference Links :

https://www.percona.com/doc/kubernetes-operator-for-psmongodb/operator.html

https://www.percona.com/doc/kubernetes-operator-for-psmongodb/gke.html

https://www.percona.com/doc/kubernetes-operator-for-psmongodb/operator.html#mongod-storage-wiredtiger-engineconfig-cachesizeratio

MongoDB 101: How to Tune Your MongoDB Configuration After Upgrading to More Memory

Jan
03
2022
--

MongoDB – Converting Replica Set to Standalone

Converting Replica Set to Standalone MongoDB

Converting Replica Set to Standalone MongoDBOne of the reasons for using MongoDB is its simplicity in scaling its structure,  either as Replicaset(scale-up) or as Sharded Cluster(scale-out).

Although a bit tricky in Sharded environments, the reverse process is also feasible.

From the possibilities, you can:

So far, so good, but if for some reason that mongo project which started as a Replica Set does not require such structure and now a Standalone server meets the requirements; 

  • Can you shrink from Replica Set to a Standalone server? 
    • If possible, what is the procedure to achieve this configuration?

This blog post will walk through the necessary steps to convert a Replica Set into a Standalone server.

Observations:

Before starting the process, it is important to mention that it’s not advised to run Standalone configuration on production servers as your entire availability will only rely on a single point.

Another observation is this activity requires downtime, that’s because you will need to remove the replication parameter, which can not be at runtime.

Last note; to avoid any data change during the procedure, it’s recommended to stop writing from the application. You will need to adjust the connection string at the end of the process for the new configuration.

How To:

The testing environment is a standard Replica Set configuration with three nodes, as seen at the following replication status.

[
	{
		"_id" : 0,
		"name" : "node1:27017",
		"stateStr" : "PRIMARY"
	},
	{
		"_id" : 1,
		"name" : "node2:27017",
		"stateStr" : "SECONDARY",
	},
	{
		"_id" : 2,
		"name" : "node3:27017",
		"stateStr" : "SECONDARY",
]

  • The PRIMARY is our reliable data source; thus, it will be the Standalone server at the end of the process.

It is not needed to trigger any election process unless you have a preferred node to be the Standalone. If that’s your case and that node is not the PRIMARY yet, please make sure to make PRIMARY before starting the procedure.

That said, let’s get our hands dirty!

1) Removing the secondaries:

  • From the PRIMARY node, let’s remove node 2 and 3:
rs0:PRIMARY> rs.remove("node2:27017")
{
	"ok" : 1,
	"$clusterTime" : {
		"clusterTime" : Timestamp(1640109346, 1),
		"signature" : {
			"hash" : BinData(0,"663Y5u3GkZkQ0Ci7EU8IYUldVIM="),
			"keyId" : NumberLong("7044208418021703684")
		}
	},
	"operationTime" : Timestamp(1640109346, 1)
}

rs0:PRIMARY> rs.remove("node3:27017")
{
	"ok" : 1,
	"$clusterTime" : {
		"clusterTime" : Timestamp(1640109380, 1),
		"signature" : {
			"hash" : BinData(0,"RReAzLZmHWfzVcdnLoGJ/uz04Vo="),
			"keyId" : NumberLong("7044208418021703684")
		}
	},
	"operationTime" : Timestamp(1640109380, 1)
}

Please note: Although my Replica Set comprises three nodes, you must let only the PRIMARY on replication at this step. If you have more nodes, the process is the same.

Then, the expected status should be:

rs0:PRIMARY> rs.status().members
[
	{
		"_id" : 0,
		"name" : "node1:27017",
		"health" : 1,
		"state" : 1,
		"stateStr" : "PRIMARY",
		"uptime" : 1988,
		"optime" : {
			"ts" : Timestamp(1640109560, 1),
			"t" : NumberLong(1)
		},
		"optimeDate" : ISODate("2021-12-21T17:59:20Z"),
		"syncSourceHost" : "",
		"syncSourceId" : -1,
		"infoMessage" : "",
		"electionTime" : Timestamp(1640107580, 2),
		"electionDate" : ISODate("2021-12-21T17:26:20Z"),
		"configVersion" : 5,
		"configTerm" : 1,
		"self" : true,
		"lastHeartbeatMessage" : ""
	}
]

 

2) Changing the configuration:

  • Even though node2 and node3 are no longer part of the replication, let’s stop the mongod process. Keeping their data safe, as it can be used in a recovery scenario if necessary:
node2-shell> systemctl stop mongod
node3-shell> systemctl stop mongod

  • Then, you can remove the replication parameter from PRIMARY and restart it, but first! In the configuration file:
    • Please remove or comment the lines:
#replication:
#  replSetName: "rs0"

If you start MongoDB via the command line, please remove the option  --replSet.

  • Once changed, you can restart the mongod on PRIMARY:
node1-shell> systemctl restart mongod

3) Removing replication objects:

  • Once connected to former PRIMARY, now Standalone, MongoDB will warn you with the following message:
2021-12-21T18:23:52.056+00:00: Document(s) exist in 'system.replset', but started without --replSet. Database contents may appear inconsistent with the writes that were visible when this node was running as part of a replica set. Restart with --replSet unless you are doing maintenance and no other clients are connected. The TTL collection monitor will not start because of this. For more info see http://dochub.mongodb.org/core/ttlcollections

 

That is because the node was working under a Replica Set configuration, and although it’s a Standalone now, there is the old structure existing.

For further detail, that structure resides under the local database, which holds all necessary data for replication, and for the database itself:

mongo> use local
switched to db local

mongo> show collections
oplog.rs
replset.election
replset.minvalid
startup_log
system.replset

 

To remove that warning, you should remove the old Replica Set object on local.system.replset.

local.system.replset holds the replica set’s configuration object as its single document. To view the object’s configuration information, issue rs.conf() from mongosh. You can also query this collection directly.

However, it’s important to mention the local.system.replset is a system object, and there is no built-in role that you can use to remove objects –  Unless you create a custom role that grants anyAction on anyResource to the user, Or you grant the internal role __system to a user.

?

Please note, both of that options gives access to every resource in the system and is intended for internal use. Do not use this permissions, other than in exceptional circumstances.  

For this process, we will create a new user, temporary, with __system privilege in which we can drop later after the action.

3. a) Creating the temporary user:

mongo> db.getSiblingDB("admin").createUser({user: "dba_system", "pwd": "sekret","roles" : [{ "db" : "admin", "role" : "__system" }]});

Confirmation output:

Successfully added user:{
   "user":"dba_system",
   "roles":[
      {
         "db":"admin",
         "role":"__system"
      }
   ]
}

3. b) Removing the object:
  1. First; Connect with the created user.
  2. Then:
mongo> use local;
mongo> db.system.replset.remove({})

Confirmation output:

WriteResult({ "nRemoved" : 1 })

 

3. c) Dropping the temporary user:
  1. Disconnect from that system user.
  2. Then:
mongo> db.dropUser("dba_system")

 

4) Restart the service:

shell> systemctl restart mongod

 

Once reconnected, you will not see that warning again, and the server is ready!

 

5) Connection advice:

  • Please make sure to adjust the connection string from the app and clients, passing only the Standalone server.

 

Conclusion

The conversion process, in general, is simple and should not face problems. But is essential to highlight that we strongly recommend you test and run this procedure in a lower environment. And if you plan to use it on production, please bear in mind the said before.

And if you have any questions, feel free to contact us in the comment section below!

 

See you!

 

Dec
30
2021
--

MongoDB Config Server Upgrade From SCCC to CSRS ReplicaSet

MongoDB Config Server Upgrade

MongoDB Config Server UpgradeRecently, I got some of our customers doing an upgrade to v4.x from v3.0 (Yeah, still could see older MongoDB versions and we suggest everyone do the upgrade to stay with the current GA!). There were some pain points in the upgrade process, and especially before migrating to v3.4, you will need to change from SCCC config setup to CSRS replicaSet setup. Interestingly, I did some tests in that process and would like to share them here. Even though this topic is pretty outdated, this would be beneficial to the people out there who still seek to migrate from the older version and are stuck at this config server upgrade.

In this blog post, we will see how to migrate a mirrored config server configuration (SCCC) to the replicaSet configuration (CSRS) along with the storage engine from MMAP to wiredTiger change. From MongoDB 3.2.4, the replicaSet configuration is supported for config servers too and the support of SCCC (Sync Cluster Connection Configuration) configuration is removed from MongoDB 3.4. So it is important to make sure the config servers are configured with the replicaSet before upgrading to MongoDB 3.4. Also, the config servers must run on wiredTiger when configured as replicaSet. There are two ways to do this – one is that when you want to replace existing config servers and another one is to migrate to the new set of servers/ports. Let’s see them in the following topics.

Migrate From SCCC to CSRS on the Different Hosts/Ports:

This section gives a brief about migrating the existing mirrored hosts/ports to different hosts or ports. It is always recommended to backup your config server before your steps and get downtime to avoid any writes on this crucial step. The test shared cluster setup runs on Percona Server for MongoDB (PSMDB) v3.2.22 with SCCC set up on the following ports:

47080 – mongos
47087, 47088, 47089 – mongoc – config servers with mmapv1 engine (sccc)
47081, 47082, 47083 – shard01 replicaset
47084, 47085, 47086 – shard02 replicaset

New Config servers to migrate (for testing purposes, using localhost but different ports to migrate):

47090, 47091, 47092 – new mongoc (csrs)

Now stop the balancer:

mongos> sh.stopBalancer()
Waiting for active hosts...
Waiting for the balancer lock...
Waiting again for active hosts after balancer is off...
WriteResult({ "nMatched" : 0, "nUpserted" : 1, "nModified" : 0, "_id" : "balancer" })
mongos> 
mongos> sh.getBalancerState()
false

 

Initiate the replicaSet in one of the config servers (here using config db running on 47087). Use this member as a base for migrating from SCCC to CSRS:

$ mongo32 localhost:47087/admin
MongoDB shell version: 3.2.21-2-g105acca0d4
connecting to: localhost:47087/admin
configsvr> 
configsvr> rs.initiate({ _id: "configRepl", configsvr: true, version:1, members: [ {_id:0, host: "localhost:47087"} ] } )
{ "ok" : 1 }

 

Start the new config DB instances using the same replicaset name used in the above initiate command:

$ /home/balaguru/mongodb/mongodb-linux-x86_64-3.2.21-2-g105acca0d4/bin/mongod --dbpath /home/balaguru/mongodb/testshard32/data/configrepl/rs1/db/ --logpath /home/balaguru/mongodb/testshard32/data/configrepl/rs1/mongod.log --port 47090 --fork --configsvr --storageEngine wiredTiger --wiredTigerCacheSizeGB 1 --replSet configRepl
about to fork child process, waiting until server is ready for connections.
forked process: 476923
child process started successfully, parent exiting

 

$ /home/balaguru/mongodb/mongodb-linux-x86_64-3.2.21-2-g105acca0d4/bin/mongod --dbpath /home/balaguru/mongodb/testshard32/data/configrepl/rs2/db/ --logpath /home/balaguru/mongodb/testshard32/data/configrepl/rs2/mongod.log --port 47091 --fork --configsvr --storageEngine wiredTiger --wiredTigerCacheSizeGB 1 --replSet configRepl
about to fork child process, waiting until server is ready for connections.
forked process: 478193
child process started successfully, parent exiting

 

$ /home/balaguru/mongodb/mongodb-linux-x86_64-3.2.21-2-g105acca0d4/bin/mongod --dbpath /home/balaguru/mongodb/testshard32/data/configrepl/rs3/db/ --logpath /home/balaguru/mongodb/testshard32/data/configrepl/rs3/mongod.log --port 47092 --fork --configsvr --storageEngine wiredTiger --wiredTigerCacheSizeGB 1 --replSet configRepl
about to fork child process, waiting until server is ready for connections.
forked process: 478254
child process started successfully, parent exiting

 

Now restart 47087 with the option –replSet and –configsvrMode which allow this mirrored config DB to be in replicaset and have MMAP engine.

$ /home/balaguru/mongodb/mongodb-linux-x86_64-3.2.21-2-g105acca0d4/bin/mongod --dbpath /home/balaguru/mongodb/testshard32/data/config/db --logpath /home/balaguru/mongodb/testshard32/data/config/mongod.log --port 47087 --fork --configsvr --storageEngine mmapv1 --configsvrMode sccc --replSet configRepl
about to fork child process, waiting until server is ready for connections.
forked process: 477682
child process started successfully, parent exiting

 

Add the new config DB instances to this replicaSet (set priority/votes 0 to avoid election):

configRepl:PRIMARY> rs.add({host: "localhost:47090", priority:0, votes:0})
{ "ok" : 1 }

configRepl:PRIMARY> rs.add({host: "localhost:47091", priority:0, votes:0})
{ "ok" : 1 }

configRepl:PRIMARY> rs.add({host: "localhost:47092", priority:0, votes:0})
{ "ok" : 1 }

configRepl:PRIMARY> rs.status()
{
        "set" : "configRepl",
        "date" : ISODate("2021-11-23T08:11:13.065Z"),
        "myState" : 1,
        "term" : NumberLong(1),
        "configsvr" : true,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "localhost:47087",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 154,
                        "optime" : {
                                "ts" : Timestamp(1637655072, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2021-11-23T08:11:12Z"),
                        "electionTime" : Timestamp(1637654919, 1),
                        "electionDate" : ISODate("2021-11-23T08:08:39Z"),
                        "configVersion" : 4,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "localhost:47090",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 69,
                        "optime" : {
                                "ts" : Timestamp(1637655069, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2021-11-23T08:11:09Z"),
                        "lastHeartbeat" : ISODate("2021-11-23T08:11:11.608Z"),
                       "lastHeartbeatRecv" : ISODate("2021-11-23T08:11:11.611Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "localhost:47087",
                        "configVersion" : 4
                },
                {
                        "_id" : 2,
                        "name" : "localhost:47091",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 6,
                        "optime" : {
                                "ts" : Timestamp(1637655069, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2021-11-23T08:11:09Z"),
                        "lastHeartbeat" : ISODate("2021-11-23T08:11:11.608Z"),
                        "lastHeartbeatRecv" : ISODate("2021-11-23T08:11:09.609Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "localhost:47090",
                        "configVersion" : 4
                },
                {
                        "_id" : 3,
                        "name" : "localhost:47092",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 3,
                        "optime" : {
                                "ts" : Timestamp(1637655069, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2021-11-23T08:11:09Z"),
                        "lastHeartbeat" : ISODate("2021-11-23T08:11:11.608Z"),
                        "lastHeartbeatRecv" : ISODate("2021-11-23T08:11:11.620Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "localhost:47091",
                        "configVersion" : 4
                }
        ],
        "ok" : 1
}

 

Shut down one of the other mirrored config servers (either 47088 or 47089). If one of the mirrored DB is down in between the migration, then the sharded cluster goes into read-only mode, i.e there will be a failure when issuing DDL (mentioned in the below example). So it is advised to do the activity when there is downtime to avoid any unexpected error from the application side.

// try to create new db and collection as follows to test when 47088 is down
mongos> use vinodh  
switched to db vinodh
mongos> db.testCreateColl.insert({id:1})
WriteResult({
        "writeError" : {
                "code" : 13663,
                "errmsg" : "unable to target write op for collection vinodh.testCreateColl :: caused by :: Location13663: exception creating distributed lock vinodh/balaguru-UbuntuPC:47080:1637654366:1487823871 :: caused by :: SyncClusterConnection::update prepare failed:  localhost:47088 (127.0.0.1) failed:9001 socket exception [CONNECT_ERROR] server [couldn't connect to server localhost:47088, connection attempt failed] "
        }
})

 

Once the replicaSet is set, change the priority and votes of other nodes to participate in the election:

configRepl:PRIMARY> cfg.members[1].host
localhost:47090
configRepl:PRIMARY> cfg.members[1].priority
0
configRepl:PRIMARY> cfg.members[1].priority = 1
1
configRepl:PRIMARY> cfg.members[1].votes 
0
configRepl:PRIMARY> cfg.members[1].votes = 1
1
configRepl:PRIMARY> cfg.members[2].priority = 1
1
configRepl:PRIMARY> cfg.members[3].priority = 1
1
configRepl:PRIMARY> cfg.members[2].votes = 1
1
configRepl:PRIMARY> cfg.members[3].votes = 1
1
configRepl:PRIMARY> rs.reconfig(cfg)
{ "ok" : 1 }

 

Now change the PRIMARY role to the new member and make 47087 as SECONDARY:

configRepl:PRIMARY> rs.stepDown()
2021-11-23T13:58:37.885+0530 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host 'localhost:47087'  :
DB.prototype.runCommand@src/mongo/shell/db.js:135:1
DB.prototype.adminCommand@src/mongo/shell/db.js:152:1
rs.stepDown@src/mongo/shell/utils.js:1202:12
@(shell):1:1

2021-11-23T13:58:37.887+0530 I NETWORK  [thread1] trying reconnect to localhost:47087 (127.0.0.1) failed
2021-11-23T13:58:37.887+0530 I NETWORK  [thread1] reconnect localhost:47087 (127.0.0.1) ok
configRepl:SECONDARY>

 

Restart the old config member 47087 without –configsvrMode option. Here 47087 will go into REMOVED which is totally fine as it still has MMAPV2 engine + running without the configsvrMode option. Then you can restart mongos with new –-configdb option mentioning only the new config servers (–configdb “configRepl/localhost:47090,localhost:47091,localhost:47092” ):

$ /home/balaguru/mongodb/mongodb-linux-x86_64-3.2.21-2-g105acca0d4/bin/mongos --logpath /home/balaguru/mongodb/testshard32/data/mongos.log --port 47080 --configdb "configRepl/localhost:47090,localhost:47091,localhost:47092" --fork

 

Now 47087 is ready to be removed from the replicaSet finally:

configRepl:PRIMARY> rs.remove("localhost:47087")
{ "ok" : 1 }

You can then check the shard status and the data through mongos. Once everything looks good, stop the other mirrored config DBs on 47088 and 47089 ports followed by enabling the balancer.

 

Migrate From SCCC to CSRS on the Same Hosts:

This section gives a brief about replacing the existing mirrored hosts/port. This has steps until enabling the replicaSet in one of the mirrored config servers. For this testing case, the shared cluster setup runs with PSMDB v3.2.22 with SCCC set up on the following ports:

27100 – mongos
27107, 27108, 27109 – mongoc – config servers with mmapv1 engine
27101, 27102, 27103 – shard01 replicaset
27104, 27105, 27106 – shard02 replicaset

Here using the instance running on port 27107 to start the migration. As said in the above method,

  • Initiate the replicaSet
  • Restart the instance with –-replSet and –-configsvrMode options

The replicaSet status on 27107 instance.

csReplSet:PRIMARY> rs.status()
{
 "set" : "csReplSet",
 "date" : ISODate("2021-10-07T09:04:20.266Z"),
 "myState" : 1,
 "term" : NumberLong(1),
 "configsvr" : true,
 "heartbeatIntervalMillis" : NumberLong(2000),
 "members" : [
  {
   "_id" : 0,
   "name" : "localhost:27107",
   "health" : 1,
   "state" : 1,
   "stateStr" : "PRIMARY",
   "uptime" : 9,
   "optime" : {
    "ts" : Timestamp(1633597452, 1),
    "t" : NumberLong(1)
   },
   "optimeDate" : ISODate("2021-10-07T09:04:12Z"),
   "infoMessage" : "could not find member to sync from",
   "electionTime" : Timestamp(1633597451, 1),
   "electionDate" : ISODate("2021-10-07T09:04:11Z"),
   "configVersion" : 1,
   "self" : true
  }
 ],
 "ok" : 1
}

 

Then stop the second config 27108 instance and clear the dbpath files to remove MMAPv2 files. Start it with the replicaset + WT engine options like below:

$ rm -rf data/config2/db/*

$ /home/vinodh.krishnaswamy/mongo/mongodb-linux-x86_64-3.2.22/bin/mongod --dbpath /home/vinodh.krishnaswamy/mongo/testshard32/data/config2/db --logpath /home/vinodh.krishnaswamy/mongo/testshard32/data/config2/mongod.log --port 27108 --fork --configsvr  --wiredTigerCacheSizeGB 1  --replSet csReplSet
about to fork child process, waiting until server is ready for connections.
forked process: 42341
child process started successfully, parent exiting

At this point, as said earlier let me remind you of the point of the sharded cluster going into a read-only mode when an existing mirrored instance is down or not reachable. So don’t be surprised if you get any warning about DDL errors while in this situation.

Now add this 27108 to the replicaset from PRIMARY (27107) as follows with priority:0 and votes:0 to avoid this server taking part in the election:

$ mongo32 localhost:27107
MongoDB shell version: 3.2.22
connecting to: localhost:27107/test
csReplSet:PRIMARY> rs.add({host: "localhost:27108", priority: 0, votes: 0 })
{ "ok" : 1 }

 

Then repeat the same for the third config server – 27109. Remove DB files + restart it with –replSet and WT options for 27109 followed by adding it to the replicaSet from PRIMARY:

csReplSet:PRIMARY> rs.add({host: "localhost:27109", priority:0 , votes: 0})
{ "ok" : 1 }

 

Once they are in sync and the replicaSet looks healthy, you can adjust the priority and votes for the added members(27108, 27109) as follows to allow them to participate in the election.

csReplSet:PRIMARY> cfg = rs.conf()
{
 "_id" : "csReplSet",
 "version" : 3,
 "configsvr" : true,
 "protocolVersion" : NumberLong(1),
 "members" : [
  {
   "_id" : 0,
   "host" : "localhost:27107",
   "arbiterOnly" : false,
   "buildIndexes" : true,
   "hidden" : false,
   "priority" : 1,
   "tags" : {
   },
   "slaveDelay" : NumberLong(0),
   "votes" : 1
  },
  {
   "_id" : 1,
   "host" : "localhost:27108",
   "arbiterOnly" : false,
   "buildIndexes" : true,
   "hidden" : false,
   "priority" : 0,
   "tags" : {
   },
   "slaveDelay" : NumberLong(0),
  "votes" : 0
  },
  {
   "_id" : 2,
   "host" : "localhost:27109",
   "arbiterOnly" : false,
   "buildIndexes" : true,
   "hidden" : false,
   "priority" : 0,
   "tags" : {
   },
   "slaveDelay" : NumberLong(0),
   "votes" : 0
  }
 ],

 "settings" : {
  "chainingAllowed" : true,
  "heartbeatIntervalMillis" : 2000,
  "heartbeatTimeoutSecs" : 10,
  "electionTimeoutMillis" : 10000,
  "getLastErrorModes" : {
  },
  "getLastErrorDefaults" : {
   "w" : 1,
   "wtimeout" : 0
  },
  "replicaSetId" : ObjectId("615eb78fdb1ed4092c166786")
 }
}

csReplSet:PRIMARY> cfg.members[1].priority = 1
1
csReplSet:PRIMARY> cfg.members[2].priority = 1
1
csReplSet:PRIMARY> cfg.members[1].votes = 1
1
csReplSet:PRIMARY> cfg.members[2].votes = 1
1
csReplSet:PRIMARY> cfg.members[2].votes
1
csReplSet:PRIMARY> rs.reconfig(cfg)
{ "ok" : 1 }

 

Then go to the first config server 27107 and step down it for another member to become PRIMARY. So that you can make it down and change the storage engine to WT in it like above as well + without configsvrMode option. When it joins back, it does the initial sync from the syscSource of the replicaSet.

csReplSet:PRIMARY> rs.stepDown()
2021-10-07T05:14:26.617-0400 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host 'localhost:27107'  :
DB.prototype.runCommand@src/mongo/shell/db.js:135:1
DB.prototype.adminCommand@src/mongo/shell/db.js:153:16
rs.stepDown@src/mongo/shell/utils.js:1202:12
@(shell):1:1

2021-10-07T05:14:26.619-0400 I NETWORK  [thread1] trying reconnect to localhost:27107 (127.0.0.1) failed
2021-10-07T05:14:26.619-0400 I NETWORK  [thread1] reconnect localhost:27107 (127.0.0.1) ok
csReplSet:SECONDARY>

 

To remind you that if you forget to change the storage engine to WT and without —configsvrMode option, it will go into REMOVED state, so please make note of it. (Here it is started purposefully started to show as an example.)

$ mongo32 localhost:27107
MongoDB shell version: 3.2.22
connecting to: localhost:27107/test
csReplSet:REMOVED>

 

Now, bringing down the DB instance 27017 and clearing DB files + starting back with WT and replSet options (remove configsvrMode option this time)

$ rm -rf data/config/db/*

$ /home/vinodh.krishnaswamy/mongo/mongodb-linux-x86_64-3.2.22/bin/mongod --dbpath /home/vinodh.krishnaswamy/mongo/testshard32/data/config/db --logpath /home/vinodh.krishnaswamy/mongo/testshard32/data/config/mongod.log --port 27107 --fork --configsvr --wiredTigerCacheSizeGB 1 --replSet csReplSet
about to fork child process, waiting until server is ready for connections.
forked process: 35202
child process started successfully, parent exiting

 

Then restart the mongos instances with the below –configDB option for the new config to take effect:

mongos <options> –configDB csReplSet/localhost:27107,localhost:27108,localhost:27109

From the mongos logfile, you will see the below message if the config server replicaset was connected successfully:

2021-10-07T05:48:25.679-0400 I NETWORK  [conn6] Successfully connected to csReplSet/localhost:27107,localhost:27108,localhost:27109 (1 connections now open to csReplSet/localhost:27107,localhost:27108,localhost:27109 with a 0 second timeout)

Conclusion

We at Percona don’t recommend anyone to have EOL versions running as they don’t get any bugs fixed or you cannot use the recently added features. Hope this blog helps some who still looking to migrate from older versions.

Dec
30
2021
--

Percona Monitoring and Management 2 Test Drive Using VirtualBox and SSH Tunnels

PMM using VirtualBox and SSH

PMM using VirtualBox and SSHPercona Monitoring and Management 2 (PMM2) is the database monitoring suite assembled and developed by Percona. It is based on standard open source components and custom-made software integrations. PMM2 helps you reduce complexity, optimize performance, and improve the security of your business-critical database environments, no matter where they are located or deployed.

This blog post will describe a method to test PMM2 using your laptop’s VirtualBox, ssh tunnels, and without installing any agents on the database servers. This is a testing and evaluation environment, not intended for production. If you want to perform a full-fledged test of PMM2, we recommend an environment as similar as possible to your final production setup: enterprise virtualization, docker containers, or AWS. We assume that your laptop doesn’t have direct connectivity to the databases, this is why we use ssh tunnels.

PMM2 architecture consists of 2+1 components:PMM2 High Level Architecture
Two components run on your infrastructure: PMM Agents and PMM Server. The agents gather the metrics at the database and operating system levels. PMM Server takes care of processing, storing, grouping, and displaying these metrics. It can also perform additional operations like capturing serverless databases metrics, backups, and sending alerts (the last two features are in technical preview as of this writing). The other component to complete the formula is the Percona Platform, which adds more features to PMM, from advisors to DBaaS. Disclaimer: Percona Platform is in preview release with limited functionality – suitable for early adopters, development, and testing. Besides the extended features added to PMM, the Percona Platform brings together distributions of MySQL, PostgreSQL, and MongoDB including a range of open-source tools for data backup, availability, and management. You can learn more about the Platform here.

To make setup easier, PMM2 Server can be run either as a docker container or importing an OVA image, executed using VMWare VSphere, VirtualBox, or any other hypervisor. If you run your infrastructure in AWS, you can deploy PMM from the AWS Marketplace.

To run the agents, you need a Linux box. We recommend running the agents and the database on the same node. PMM can also gather the metrics using a direct connection to a server-less database or running an operating system that does not support the agent.

Often, installing the agents is a stopper for some DBAs who would like to test PMM2. Also, while containers are frequent in large organizations, we find virtualization and containers relegated to development and quality assurance environments. These environments usually don’t have direct access to production databases.

TCP Forwarding Across SSH Connections

AllowTcpForwarding is the ssh daemon configuration option that allows forwarding TCP ports across the ssh connection. At first sight, this may seem a security risk, but as the ssh documentation states: “disabling TCP forwarding does not improve security unless users are also denied shell access, as they can always install their forwarders.”

If your system administrators do not allow TCP forwarding, other options available to accomplish the same results are socat or netcat. But we will not cover them here.

If your laptop has direct access to the databases, you can skip all the ssh tunnels and use the direct access method described later in this post.

Install PMM 2 Ova

Download the Open Virtualization Format compatible image from https://www.percona.com/downloads/pmm2/ or use the command line:

$ wget https://www.percona.com/downloads/pmm2/2.25.0/ova/pmm-server-2.25.0.ova

You can import the OVA file using the UI, with the import option from the file menu, or using the command line:

$ VBoxManage import pmm-server-2.25.0.ova --vsys 0 --vmname "PMM Testing"
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /Users/pep/Downloads/pmm-server-2.25.0.ova...
OK.
Disks:
vmdisk1 42949672960 -1 http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized PMM2-Server-2021-12-13-1012-disk001.vmdk -1 -1
vmdisk2 429496729600 -1 http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized PMM2-Server-2021-12-13-1012-disk002.vmdk -1 -1

Virtual system 0:
0: Suggested OS type: "RedHat_64"
(change with "--vsys 0 --ostype "; use "list ostypes" to list all possible values)
1: VM name specified with --vmname: "PMM Testing"
2: Suggested VM group "/"
(change with "--vsys 0 --group ")
3: Suggested VM settings file name "/Users/Pep/VirtualBox VMs/PMM2-Server-2021-12-13-1012/PMM2-Server-2021-12-13-1012.vbox"
(change with "--vsys 0 --settingsfile ")
4: Suggested VM base folder "/Users/Pep/VirtualBox VMs"
(change with "--vsys 0 --basefolder ")
5: Product (ignored): Percona Monitoring and Management
6: Vendor (ignored): Percona
7: Version (ignored): 2021-12-13
8: ProductUrl (ignored): https://www.percona.com/software/database-tools/percona-monitoring-and-management
9: VendorUrl (ignored): https://www.percona.com
10: Description "Percona Monitoring and Management (PMM) is an open-source platform for managing and monitoring MySQL and MongoDB performance"
(change with "--vsys 0 --description ")
11: Number of CPUs: 1
(change with "--vsys 0 --cpus ")
12: Guest memory: 4096 MB
(change with "--vsys 0 --memory ")
13: Network adapter: orig NAT, config 3, extra slot=0;type=NAT
14: SCSI controller, type LsiLogic
(change with "--vsys 0 --unit 14 --scsitype {BusLogic|LsiLogic}";
disable with "--vsys 0 --unit 14 --ignore")
15: Hard disk image: source image=PMM2-Server-2021-12-13-1012-disk001.vmdk, target path=PMM2-Server-2021-12-13-1012-disk001.vmdk, controller=14;channel=0
(change target path with "--vsys 0 --unit 15 --disk path";
disable with "--vsys 0 --unit 15 --ignore")
16: Hard disk image: source image=PMM2-Server-2021-12-13-1012-disk002.vmdk, target path=PMM2-Server-2021-12-13-1012-disk002.vmdk, controller=14;channel=1
(change target path with "--vsys 0 --unit 16 --disk path";
disable with "--vsys 0 --unit 16 --ignore")
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully imported the appliance.

Once the machine is imported, we will connect it to a host-only network. This network restricts network traffic only between the host and the virtual machines. But first, let’s find a suitable network:

$ VBoxManage list hostonlyifs
Name: vboxnet0
GUID: 786f6276-656e-4074-8000-0a0027000000
DHCP: Disabled
IPAddress: 192.168.56.1
NetworkMask: 255.255.255.0
IPV6Address:
IPV6NetworkMaskPrefixLength: 0
HardwareAddress: 0a:00:27:00:00:00
MediumType: Ethernet
Wireless: No
Status: Up
VBoxNetworkName: HostInterfaceNetworking-vboxnet0

Select the first one that has Status up, write down the name and IP address. Then make sure that there is a DHCP server assigned to that interface:

$ VBoxManage list dhcpservers
NetworkName: HostInterfaceNetworking-vboxnet0
Dhcpd IP: 192.168.56.100
LowerIPAddress: 192.168.56.101
UpperIPAddress: 192.168.56.254
NetworkMask: 255.255.255.0
Enabled: Yes
Global Configuration:
minLeaseTime: default
defaultLeaseTime: default
maxLeaseTime: default
Forced options: None
Suppressed opts.: None
1/legacy: 255.255.255.0
Groups: None
Individual Configs: None

Now we will assign two network interfaces to our PMM virtual machine. One is allocated to the internal network, and the other uses NAT to connect to the internet and, for example, check for upgrades.

$ VBoxManage modifyvm "PMM Testing" --nic1 hostonly --hostonlyadapter1 vboxnet0
$ VBoxManage modifyvm "PMM Testing" --nic2 natnetwork

Once networking is configured, we may start the virtual machine.

$ VBoxManage startvm "PMM Testing"

The next step is to retrieve the IP address assigned to our PMM box. First, we will obtain the MAC address of the network card we recently added:

$ VBoxManage showvminfo "PMM Testing" | grep -i vboxnet0
NIC 1: MAC: 08002772600D, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none

Using the retrieved MAC address we can look for the DHCP leases:

$ VBoxManage dhcpserver findlease --interface=vboxnet0 --mac-address=08002772600D
IP Address: 192.168.56.112
MAC Address: 08:00:27:72:60:0d
State: acked
Issued: 2021-12-21T22:15:54Z (1640124954)
Expire: 2021-12-21T22:25:54Z (1640125554)
TTL: 600 sec, currently 444 sec left

This is the IP address we will use to access the PMM server. Open a browser to connect to https://192.168.56.112 with the default credentials: admin/admin.

PMM2 Login window
The following step configures the tunnels to connect to the databases we monitor.

Set Up the SSH Tunnels

This is the topology of our network:

Private network topology
We will open two ssh connections per server we want to access from PMM. Open a terminal session and execute the following command, replacing it with the username you normally use to connect to your jump host:

$ ssh -L 192.168.56.1:3306:10.0.0.1:3306 @192.168.55.100

This creates a tunnel that connects the MySQL Server port 3306 with our local internal address in the same port. If you want to connect to more than one MySQL instance, you must use different ports. To open the tunnel for the MongoDB server, use the following command:

$ ssh -L 192.168.56.1:27017:10.0.0.2:27017 @192.168.55.100

Test the tunnel connectivity to the MySQL host using netcat:

$ nc -z 192.168.56.1 3306
Connection to 192.168.56.1 port 3306 [tcp/mysql] succeeded!

And also test the connectivity to the MongoDB host:

$ nc -z 192.168.56.1 27017
Connection to 192.168.56.1 port 27017 [tcp/*] succeeded!

This is the topology of our network including the ssh tunnels.
SSH tunnels

Configure Accounts

Follow the PMM documentation and create a MySQL account (or use an already existing account) with the required privileges:

CREATE USER 'pmm'@'10.0.0.100' IDENTIFIED BY '' WITH MAX_USER_CONNECTIONS 10;
GRANT SELECT, PROCESS, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'10.0.0.100';

Note that we need to use the internal IP address for the jump host. If you don’t know the IP address, use the wildcard ‘%’.

Add also the credentials for MongoDB, run this in a Mongo session:

db.getSiblingDB("admin").createRole({
role: "explainRole",
privileges: [{
resource: {
db: "",
collection: ""
},
actions: [
"listIndexes",
"listCollections",
"dbStats",
"dbHash",
"collStats",
"find"
]
}],
roles:[]
})

db.getSiblingDB("admin").createUser({
user: "pmm_mongodb",
pwd: "",
roles: [
{ role: "explainRole", db: "admin" },
{ role: "clusterMonitor", db: "admin" },
{ role: "read", db: "local" }
]
})

Add the Services to PMM

We can’t install the agents because we don’t have access to our PMM testing environment from the database servers. Instead, we will configure both services as remote instances. Go to the “Configuration” menu , select “PMM Inventory” , then “Add instance” . Then choose MySQL Add a remote instance.
Complete the following fields:
Hostname: 192.168.56.1 (This is the internal Host-Only VirtualBox address)
Service name: MySQL8
Port: 3306
Username: pmm
Password: <password>

And press the button. It will check the connectivity and, if everything is correct, the MySQL service will be added to the inventory. If there is an error, double-check that the ssh connection is still open and you entered the correct credentials. Make sure that the host that you specified to create the MySQL user is correct.

We will use a similar process for MongoDB:

These are the fields you have to complete with the correct information:
Hostname: 192.168.56.1 (Again, the internal Host-Only VirtualBox address)
Service name: MongoDB
Port: 27017
Username: pmm_mongodb
Password: <password>

And press the button. It will check the connectivity and, if everything is correct, the MongoDB service will be added to the inventory. If there is an error, double-check again that the ssh connection is open and you entered the correct credentials. You can use also the MongoDB client application to check access.

Once you added both services, you just need to wait for a few minutes to give time to collect data and start testing PMM2.

PMM2 Query Analyzer

Dec
21
2021
--

Quick Guide on Azure Blob Storage Support for Percona Distribution for MongoDB Operator

Azure Blob Percona MongoDB Operator

If you have ever used backups with Percona Distribution for MongoDB Operator, you should already know that backed-up data is stored outside the Kubernetes cluster – on Amazon S3 or any S3-compatible storage. Storage types not compatible with the S3 protocol were supported indirectly in the case of an existing S3 wrapper/gateway. A good example of such a solution is running MinIO Gateway on Azure Kubernetes Service to store backups on Azure Blob Storage.

Starting with Operator version 1.11, it is now possible to use Azure Blob Storage for backups directly:

Backups on Azure Blob Storage

The following steps will allow you to configure it.

1. Get Azure Blob Storage Credentials

As with most other S3-compatible storage types, the first thing to do is to obtain credentials the Operator will use to access Azure Blob storage.

If you are new to Azure, these two tutorials will help you to configure your storage:

When you have a container to store your backups, getting credentials to access it involve the following steps:

  1. Go to your storage account settings,
  2. Open the “Access keys” section,
  3. Copy and save both the account name and account key as shown on a screenshot below:

Azure credentials

2. Create a Secret with Credentials

The Operator will use a Kubernetes Secrets Object to obtain the needed credentials. Create this Secret with credentials using the following command:

$ kubectl create secret generic azure-secret \
 --from-literal=AZURE_STORAGE_ACCOUNT_NAME=<your-storage-account-name> \
 --from-literal=AZURE_STORAGE_ACCOUNT_KEY=<your-storage-key>

3. Setup spec.backup Section in your deploy/cr.yaml file

As usual, backups are configured via the same-name section in the deploy/cr.yaml configuration file.

Make sure that backups are enabled (backup.enable key set to true), and add the following lines to the backup.storages subsection (use the proper name of your container): 

azure-blob:
  type: azure
  azure:
    container: <your-container-name>
    prefix: psmdb
    credentialsSecret: azure-secret

If you want to schedule a regular backup, add the following lines to the backup.tasks subsection:

tasks:
  - name: weekly
    enabled: true
    schedule: "0 0 * * 0"
    compressionType: gzip
    storageName: azure-blob

The backup schedule is specified in crontab format (the above example runs backups at 00:00 on Sunday). If you know nothing about cron schedule expressions, you can use this online generator.

The full backup section in our example will look like this:

backup:
  enabled: true
  restartOnFailure: true
  image: percona/percona-server-mongodb-operator:1.10.0-backup
  storages:
    azure-blob:
      type: azure
      azure:
        container: <your container name>
        prefix: psmdb
        credentialsSecret: azure-secret
  tasks:
    - name: weekly
      enabled: true
      schedule: "0 0 * * 0"
      compressionType: gzip
      storageName: azure-blob

You can find more information on backup options in the official backups documentation and the backups section of the Custom Resource options reference.

Percona Distribution for MongoDB is a freely available MongoDB database alternative, giving you a single solution that combines the best and most important enterprise components from the open source community, designed and tested to work together.

Download Percona Distribution for MongoDB Today!

Dec
21
2021
--

Data at Rest Encryption Support in Percona Distribution for MongoDB Operator

Data at Rest Encryption Support in Percona Distribution for MongoDB Operator

Data at Rest Encryption Support in Percona Distribution for MongoDB OperatorAs we all know, security is very important these days and we read about many data leaks. Security has many aspects, but one of the most important is securing data since it is a vital asset to companies. When we speak about data, it can be encrypted at rest (transparent data encryption – TDE, full disk encryption, column/field-level encryption) or in transit (TLS).

What we will concentrate on in this blog post is data at rest encryption (specifically TDE) and how it is currently supported in Percona Distribution for MongoDB Operator, but also what the limitations are and the features coming in some of the next releases.

TDE basically means that any data which is not actively used is encrypted at the storage engine level (WiredTiger in this case), but this does not include logs or data which is replicated.

TDE in Percona Distribution for MongoDB Operator

TDE in Operator is based on options that Percona Server for MongoDB (PSMDB) supports and which were developed to be mostly the same or similar as in MongoDB Enterprise edition. The differences are that PSMDB doesn’t support KMIP or Amazon AWS key management services, but instead offers the ability to store the master key inside HashiCorp Vault.

The Operator currently doesn’t support storing keys in HashiCorp Vault as PSMDB does, and the master key is stored in the Kubernetes secret and mounted locally in database pods, but I will mention this more in the limitations section and future plans.

Options for data at rest encryption support in the Operator are only a few, and the defaults in the Operator are:

  • security.enableEncryption: true
  • security.encryptionCipherMode: AES256-CBC
  • security.encryptionKeySecret: optional (needs to be a 32 character string encoded in base64

You can read about the options here, but as you can see in Operator, encryption is enabled by default and if you don’t specify some custom security.encryptionKeySecret the Operator will create one for you.

Limitations

Dynamically Enabling/Disabling

We cannot simply dynamically change the option to enable or disable encryption. If we try to do that, the Operator will try to restart MongoDB pods with new options and the pod start will fail. The reason is that MongoDB will just start with the new option on the old data directory, and it will not be able to read/write the data.

One of the ways this can be overcome is by creating a logical backup and then restoring on a cluster that has the desired option enabled. The second option would be to create a second cluster with the desired option and do a cross-cluster replication and then switch the main cluster to the new one.

Key Rotation

One of the most important things with encryption is the periodic rotation of the keys, but at this moment with the Operator, this is not so easy. This is basically the same issue as above, but if we try to just update the secret and restart the cluster the effect will be the same – MongoDB will not be able to read/write the data.

It can be overcome with the same options as above, but it will be made really easy with the ability to store the keys in the Vault. If you are interested in this functionality you can track the progress in this Jira ticket.

Storing the Key in the Vault

Currently, the Operator supports only storing the master key as a secret which is presented to PSMDB as a local file. This is not a recommended setup for production and is very limiting.

PSMDB has integration for storing the keys in HashiCorp Vault key management which is much more secure and also has the ability to rotate the master key. Basically, how it works is that PSMDB is restarted with the option “rotateMasterKey: true” and then it just generates a new key in the Vault and re-encrypts the specific database encryption keys (whole data is not re-encrypted).

Support for this is definitely one of the features in the roadmap and it will be a huge deal for data at rest encryption support in the Operator so stay tuned for upcoming changes. The request for implementing support for HashiCorp Vault integration can be tracked here.

Conclusion

As you can see, data at rest encryption in our Kubernetes Operator is supported but currently only at the most basic level. Our Percona Distribution for MySQL Operator already supports integration with HashiCorp Vault and since we like to keep the feature parity between our different operators, this functionality will be soon available in our MongoDB operator as well.

The Percona Kubernetes Operators automate the creation, alteration, or deletion of members in your Percona Distribution for MySQL, MongoDB, or PostgreSQL environment.

Learn More About Percona Kubernetes Operators

Nov
04
2021
--

Externally Sourced Configuration File Values in MongoDB

Externally Sourced Configuration File Values in MongoDB

Externally Sourced Configuration File Values in MongoDBSince version 4.2, MongoDB provides a new interesting feature called Externally Sourced Configuration File Values. It allows administrators to specify that particular variables will have their values loaded from an external resource.

The external resources that are supported, at this time, are:

  • REST APIs, which will cause MongoDB to perform HTTP GET requests to a given URL
  • Processes, which will cause MongoDB to run a specified binary and use the returned value

This feature has a lot of use cases, and providing the x509 certificate password without having it stored in the file is one of them. It can easily achieve it with the following configuration:

net:
  tls:
  …
    certificateKeyFile: "/etc/tls/mongod.pem"
    certificateKeyFilePassword:
      __exec: "/usr/bin/getCertificatePassword"
      type: "string"

or by using REST API:

net:
  tls:
  …
    certificateKeyFile: "/etc/tls/mongod.pem"
    certificateKeyFilePassword:
      __rest: "https://apiserver.invalid/getCertificatePassword"
      type: "string"

Within the scope of Externally Sourced Configuration File Values, MongoDB also introduces a unique feature, allowing users to use only the expected value. For instance, if someone wants to protect from the external resource mistakenly returning a wrong value, the user can provide a SHA256-HMAC digest of the expected value. By using that, the value is known to an administrator, can be used by MongoDB, and it’s still not stored in the configuration file.

Let’s consider the trivial example of:

…
 port:
    __exec: "/usr/local/bin/getPort"
    type: "string"
    trim: "whitespace"
    digest: c50bdabc0c9268b72c483eba6bc9a449c486bafbda077759aff214f1437dc152
    digest_key: 0123456789abcdef

The /usr/local/bin/getPort is a simple script that returns a single integer:

#!/bin/bash
echo 21010

The digest key was chosen randomly, and the digest itself is sha256-hmac of literal 21010:

$ echo -ne 21010 | openssl dgst -sha256 -mac hmac -macopt hexkey:0123456789abcdef
(stdin)= c50bdabc0c9268b72c483eba6bc9a449c486bafbda077759aff214f1437dc152

After starting it, MongoDB works fine, and it uses 21010/tcpAs defined in the provider script:

# mongod -f /etc/mongod.conf --configExpand "rest,exec"
{"t":{"$date":"2021-10-21T12:55:18.767Z"},"s":"I", "c":"CONTROL", "id":23318, "ctx":"main","msg":"Processing config expansion","attr":{"expansion":"__exec","node":"net.port"}}

# netstat -nlput | grep -i 21010
tcp 0 0 127.0.0.1:21010 0.0.0.0:* LISTEN 1463/mongod

After changing the returned value to 201011, but leaving the expected digest unchanged, MongoDB refuses to start and returns an error:

# mongod -f /etc/mongod.conf --configExpand "rest,exec"
{"t":{"$date":"2021-10-21T12:55:51.458Z"},"s":"I", "c":"CONTROL", "id":23318, "ctx":"main","msg":"Processing config expansion","attr":{"expansion":"__exec","node":"net.port"}}
SHA256HMAC of config expansion KQG2ytlnfaqo8gU1UG/q1eTf2fEdPUJYXIernXF8Qj8= does not match expected digest: xQvavAySaLcsSD66a8mkScSGuvvaB3dZr/IU8UN9wVI=
try 'mongod --help' for more information

As the expected and received value digest changed, MongoDB decided not to trust the received value. This could be used for passwords and tokens in real-life scenarios as the data won’t be stored in the config file, but MongoDB will still accept only a single expected value. This provides integrity of the critical configuration data but still allows MongoDB to be more flexible in fetching data from remote endpoints.

Complete the 2021 Percona Open Source Data Management Software Survey

Have Your Say!

Oct
20
2021
--

MongoDB 5.0 Time Series Collections

MongoDB 5.0 Time Series Collections

MongoDB 5.0 Time Series CollectionsIn a previous article, I tested a new feature of MongoDB 5.0: resharding. Today, I take a look at another new feature: the Time Series collections.

The Time Series collection is an astonishing new feature available in MongoDB 5.0. Based on the first tests I have done, the Time Series support provides comparable performance to the index usage on regular collections but saves a lot of disk and memory space. Aggregation pipelines, which are common queries you can run on time series data, can get even more benefit.

Let’s start the tests.

What is a Time Series Database?

Generally speaking, a Time Series database is a specialized database designed for efficiently storing data generated from a continuous stream of values associated with a timestamp. The typical use case is when you need to store data coming from sensory equipment that transmits data points at fixed intervals, but now they are used in support of a much wider range of applications.

Typical use cases are:

  • IoT data
  • Monitoring web services, applications, and infrastructure
  • Sales forecasting
  • Understanding financial trends
  • Processing self-driving car data or other physical devices

A Time Series specialized database utilizes compression algorithms to minimize the space requirement and also provides access paths to dig more efficiently into the data. This improves the performance of retrieving data based on time range filters and aggregating data. They are more efficient than using a common relational database.

Usually, the values of a Time Series shouldn’t change once recorded, they are defined as INSERT only, also known as immutable data points. Once the data is stored the update operation is really uncommon.

Another characteristic of Time Series is that every item should have a single value (a single temperature, a stock price, and so on).

Popular Time Series databases are InfluxDB, Prometheus, Graphite. There are also many others. VictoriaMetrics in particular is a popular fork of Prometheus and is used in our Percona Monitoring and Management software.

The New Time Series Collections in MongoDB 5.0

MongoDB, as well as relational databases, has been widely used for years for storing temperature data from sensors, stock prices, and any other kind of unchanging data over time. MongoDB version 5.0 promises that this can be done more efficiently, so let’s take a look at how it works.

A Time Series collection appears as a regular collection and the operations you can do are exactly the same: insert, update, find, delete, aggregate. The main difference is behind the curtain. MongoDB stores data into an optimized storage format on insert. Compared to a normal collection, a Time Series is smaller and provides more query efficiency.

MongoDB treats Time Series collections as writable non-materialized views. The data is stored more efficiently, saving disk space, and an automatically created internal index orders the data by time. By default, the data is compressed using the zstd algorithm instead of snappy. The new compression provides a higher ratio, less CPU requirements, and it is well suited for time series data where there are few variations from one document to the next one. You can eventually change the compression algorithm, but it is not really recommended.

A Time Series collection is not implicitly created when you insert a document, the same as regular collections. You must create it explicitly.

Let’s do some tests.

Create a Time Series Collection for Storing Stock Prices

We need to use the createCollection() method, providing some parameters.

[direct: mongos] timeseries> db.createCollection( 
"stockPrice1week", { 
  timeseries: { 
    timeField: "timestamp", 
    metaField: "metadata", 
    granularity: "minutes" 
  }, 
  expireAfterSeconds: 604800   
  }
)
{ ok: 1 }

The name of the collection is stockPrice1week and the only required parameter is timeField. The other parameters are optional.

timeField: the name of the field where the date is stored. This will be automatically indexed and used for retrieving data.

metaField: the field containing the metadata. It can be a simple scalar value or a more complex JSON object. It’s optional. It cannot be the _id or the same as the timeField. For example, the metadata for a temperature sensor could be the code of the sensor, the type, the location, and so on.

granularity: possible values are seconds, minutes, and hours. If not set, it defaults to seconds. If you specify the closest match between two consecutive values this will help MongoDB to store data more efficiently and improve the query performance.

expireAfterSeconds: you can automatically delete documents after the specified time, the same as TTL index. If not specified the documents will not expire.

Let’s insert some random data for three stocks: Apple, Orange, and Banana. Data is collected once per minute.

[direct: mongos] timeseries> var stockPriceDate = ISODate("2021-10-13T00:00:00.000Z")

[direct: mongos] timeseries> var priceApple = 100

[direct: mongos] timeseries> var priceOrange = 50

[direct: mongos] timeseries> var priceBanana = 80

[direct: mongos] timeseries> for (i = 1; i < 100000; i++) { 
  priceApple = priceApple + Math.random(); 
  priceOrange = priceOrange + Math.random(); 
  priceBanana = priceBanana + Math.random(); 
  db.stockPrice1week.insert({ "timestamp": stockPriceDate, "metadata": { "stockName": "Apple", "currency": "Dollar" }, "stockPrice": priceApple }); 
  db.stockPrice1week.insert({ "timestamp": stockPriceDate, "metadata": { "stockName": "Orange", "currency": "Dollar" }, "stockPrice": priceOrange }); 
  db.stockPrice1week.insert({ "timestamp": stockPriceDate, "metadata": { "stockName": "Banana", "currency": "Euro" }, "stockPrice": priceBanana }); 
  stockPriceDate = new Date(stockPriceDate.getTime() + 1000 * 60); 
}

We can query to check the inserted documents:

[direct: mongos] timeseries> db.stockPrice1week.find().limit(3)
[
  {
    _id: ObjectId("6166df318f32e5d3ed304fc5"),
    timestamp: ISODate("2021-10-13T00:00:00.000Z"),
    metadata: { stockName: 'Apple', currency: 'Dollar' },
    stockPrice: 100.6547271930824
  }, 
  {
    _id: ObjectId("6166df318f32e5d3ed304fc6"),
    timestamp: ISODate("2021-10-13T00:00:00.000Z"),
    metadata: { stockName: 'Orange', currency: 'Dollar' },
    stockPrice: 50.51709117468818
  },  
  {
    _id: ObjectId("6166df318f32e5d3ed304fc7"),
    timestamp: ISODate("2021-10-13T00:00:00.000Z"),
    metadata: { stockName: 'Banana', currency: 'Euro' },
    stockPrice: 80.17611551979255
  }
]

Check the Collection Size

Now, let’s create a regular collection having the same exact data.

[direct: mongos] timeseries> db.stockPrice1week.find().forEach(function (doc) {
  db.stockPrice1week_regular.insertOne(doc);
})


Let’s check the total size of the two collections.

[direct: mongos] timeseries> db.stockPrice1week.stats().totalSize
5357568
[direct: mongos] timeseries> db.stockPrice1week_regular.stats().totalSize
21934080

As expected, the Time Series collection is four times smaller than the regular one. Also, consider the regular collection doesn’t have any secondary index at the moment.

Query the Collections

Let’s run a simple query to find out the stock values for a specific timestamp. We test the query on both collections.

[direct: mongos] timeseries> db.stockPrice1week.find( { "timestamp": ISODate("2021-10-23T12:00:00.000Z") } )
[
  { 
    _id: ObjectId("6166dfc68f32e5d3ed3100f5"),
    timestamp: ISODate("2021-10-23T12:00:00.000Z"),
    metadata: { stockName: 'Apple', currency: 'Dollar' },
    stockPrice: 7636.864548363888
  },
  {
    _id: ObjectId("6166dfc68f32e5d3ed3100f6"),
    timestamp: ISODate("2021-10-23T12:00:00.000Z"),
    metadata: { stockName: 'Orange', currency: 'Dollar' },
    stockPrice: 7607.03756525094
  },
  {
    _id: ObjectId("6166dfc68f32e5d3ed3100f7"),
    timestamp: ISODate("2021-10-23T12:00:00.000Z"),
    metadata: { stockName: 'Banana', currency: 'Euro' },
    stockPrice: 7614.360031277444  
  }
]
[direct: mongos] timeseries> db.stockPrice1week_regular.find( { "timestamp": ISODate("2021-10-23T12:00:00.000Z") } )
[
  {
    _id: ObjectId("6166dfc68f32e5d3ed3100f5"),
    timestamp: ISODate("2021-10-23T12:00:00.000Z"),
    metadata: { stockName: 'Apple', currency: 'Dollar' },
    stockPrice: 7636.864548363888
  }, 
  {
    _id: ObjectId("6166dfc68f32e5d3ed3100f6"),
    timestamp: ISODate("2021-10-23T12:00:00.000Z"),
    metadata: { stockName: 'Orange', currency: 'Dollar' },
    stockPrice: 7607.03756525094
  },
  {
    _id: ObjectId("6166dfc68f32e5d3ed3100f7"),
    timestamp: ISODate("2021-10-23T12:00:00.000Z"),
    metadata: { stockName: 'Banana', currency: 'Euro' },
    stockPrice: 7614.360031277444
  }
]

We’ve got the same result, but what is important here is looking at the explain() to see the execution plan. Here is the explain() of the regular collection.

[direct: mongos] timeseries> db.stockPrice1week_regular.find( { "timestamp": ISODate("2021-10-23T12:00:00.000Z") } ).explain("executionStats")
{
...
winningPlan: {
  stage: 'COLLSCAN',
  filter: {
    timestamp: { '$eq': ISODate("2021-10-23T12:00:00.000Z") }
  },
  direction: 'forward'
...
...
executionSuccess: true,
nReturned: 3,
executionTimeMillis: 200,
totalKeysExamined: 0,
totalDocsExamined: 299997,
...
...

We didn’t create any secondary index, so the winning plan is a COLLSCAN, all documents must be examined. The query takes 200 milliseconds.

The following is the explain() of the Time Series collection instead.

[direct: mongos] timeseries> db.stockPrice1week.find( { "timestamp": ISODate("2021-10-23T12:00:00.000Z") } ).explain("executionStats")
{
...
...
executionStats: {
  executionSuccess: true,
  nReturned: 3,
  executionTimeMillis: 2,
  totalKeysExamined: 0,
  totalDocsExamined: 8,
  executionStages: {
  stage: 'COLLSCAN',
    filter: {
      '$and': [
        {
          _id: { '$lte': ObjectId("6173f940ffffffffffffffff") }
        },
        {
          _id: { '$gte': ObjectId("6172a7c00000000000000000") }
        },
      {
      'control.max.timestamp': {
        '$_internalExprGte': ISODate("2021-10-23T12:00:00.000Z")
      }
    },
    {
      'control.min.timestamp': {
        '$_internalExprLte': ISODate("2021-10-23T12:00:00.000Z")
      }
    }
  ]
},
...
...

Surprisingly it is a COLLSCAN, but with different numbers. The number of documents examined is now only eight and execution time is two milliseconds.

As already mentioned, the Time Series is a non-materialized view. It works as an abstraction layer. The actual data is stored into another system collection (system.buckets.stockPrice1week) where documents are saved in a slightly different format. It’s not the goal of this article to dig into the internals, just keep in mind the different storage format permits mongod to fetch only a few buckets of data instead of reading everything, even if it is flagged as a COLLSCAN. That’s amazing.

What Happens if I Create an Index on the Regular Collection?

Let’s try.

[direct: mongos] timeseries> db.stockPrice1week_regular.createIndex( { "timestamp": 1 } )
timestamp_1
[direct: mongos] timeseries> db.stockPrice1week_regular.getIndexes()
[
  { v: 2, key: { _id: 1 }, name: '_id_' },
  { v: 2, key: { timestamp: 1 }, name: 'timestamp_1' }
]
[direct: mongos] timeseries> db.stockPrice1week_regular.find({"timestamp": ISODate("2021-10-23T12:00:00.000Z")}).explain("executionStats")
{
...
...
winningPlan: {
  stage: 'FETCH',
  inputStage: {
  stage: 'IXSCAN',
  keyPattern: { timestamp: 1 },
  indexName: 'timestamp_1',
...
...
executionStats: {
  nReturned: 3,
  executionTimeMillis: 2,
  totalKeysExamined: 3,
  totalDocsExamined: 3,
...

Now the winning plan is an IXSCAN, the new index is used. Only three keys examined, three docs examined, and three docs returned. The query takes two milliseconds.

So, it is as fast as the Time Series collection. There is not such a big difference; the order of magnitude is the same.

Also notice the same performance comes at the cost of having a larger collection at the end because we have created a secondary index.

[direct: mongos] timeseries> db.stockPrice1week_regular.stats().totalSize
25251840
[direct: mongos] timeseries> db.stockPrice1week.stats().totalSize
5357568

For getting a comparable execution time, now the regular collection is five times larger than the Time Series.

A Query with a Time Range Filter

Let’s test a different query looking for a range of timestamps. The following are the explain() outputs.

[direct: mongos] timeseries> db.stockPrice1week_regular.find( { "timestamp": { $gte: ISODate("2021-10-20T00:00:00Z"), $lt: ISODate("2021-10-20T23:59:59Z") } } ).explain("executionStats")
{
...
winningPlan: {
  stage: 'FETCH',
  inputStage: {
    stage: 'IXSCAN',
    keyPattern: { timestamp: 1 },
...
executionStats: {
  nReturned: 4320,
  executionTimeMillis: 7,
  totalKeysExamined: 4320,
  totalDocsExamined: 4320,
...

 

[direct: mongos] timeseries> db.stockPrice1week.find( { "timestamp": { $gte: ISODate("2021-10-20T00:00:00Z"), $lt: ISODate("2021-10-20T23:59:59Z") } } ).explain("executionStats")
{
...
...
winningPlan: {
  stage: 'COLLSCAN',
  filter: {
    '$and': [
    {
      _id: { '$lt': ObjectId("6170ad7f0000000000000000") }
    },
    {
      _id: { '$gte': ObjectId("616e0a800000000000000000") }
    },
    {
      'control.max.timestamp': {
        '$_internalExprGte': ISODate("2021-10-20T00:00:00.000Z")
      }
    },
    {
      'control.min.timestamp': {
        '$_internalExprLt': ISODate("2021-10-20T23:59:59.000Z")
      }
    }
  ]
},
...
...
executionStats: {
  executionSuccess: true,
  nReturned: 6,
  executionTimeMillis: 6,
  totalKeysExamined: 0,
  totalDocsExamined: 11,
...

The same as before. The execution time is basically the same for both queries. The main problem remains the size of the regular collection that is significantly larger.

Only six documents are apparently returned by the Time Series, but it’s not. If you execute the query for real you’ll get 4320 documents. The six documents mentioned by explain() refer to the documents that must be returned by the real collection below the non-materialized view.

Aggregation Test

On our Time Series data, we would like to do some aggregation. This is a typical task: calculate averages over a period, find min and max values, and other kinds of statistics.

Let’s suppose we need to calculate the average stock price on a daily basis. We can use the following aggregation pipeline for example:

db.stockPrice1week.aggregate([
{
  $project: {
    date: {
      $dateToParts: { date: "$timestamp" }
    },
    stockPrice: 1
  }
},
{
  $group: {
    _id: {
      date: {
        year: "$date.year",
        month: "$date.month",
        day: "$date.day"
      }
    },
    avgPrice: { $avg: "$stockPrice" }
  }
}
])
[
{
_id: { date: { year: 2021, month: 12, day: 4 } },
avgPrice: 37939.782043249594
},
{
_id: { date: { year: 2021, month: 11, day: 22 } },
avgPrice: 29289.700949196136
},
{
_id: { date: { year: 2021, month: 10, day: 27 } },
avgPrice: 10531.347070537977
},
...
...

As usual, let’s have a look at the explain() of the aggregate against the two collections, just focusing on execution time and documents examined.

[direct: mongos] timeseries> db.stockPrice1week.explain("executionStats").aggregate([ { $project: { date: { $dateToParts: { date: "$timestamp" } }, stockPrice: 1 } }, { $group: { _id: { date: { year: "$date.year", month: "$date.month", day: "$date.day" } }, avgPrice: { $avg: "$stockPrice" } } }])
{
...
executionStats: {
  executionSuccess: true,
  nReturned: 300,
  executionTimeMillis: 615,
  totalKeysExamined: 0,
  totalDocsExamined: 300,
  executionStages: {
  stage: 'COLLSCAN',
...

[direct: mongos] timeseries> db.stockPrice1week_regular.explain("executionStats").aggregate([ { $project: { date: { $dateToParts: { date: "$timestamp" } }, stockPrice: 1 } }, { $group: { _id: { date: { year: "$date.year", month: "$date.month", day: "$date.day" } }, avgPrice: { $avg: "$stockPrice" } } }])
{
...
executionStats: {
  executionSuccess: true,
  nReturned: 299997,
  executionTimeMillis: 1022,
  totalKeysExamined: 0,
  totalDocsExamined: 299997,
  executionStages: {
    stage: 'PROJECTION_DEFAULT',
...

The aggregation pipeline runs 40 percent faster with the Time Series collection. This should be more relevant the larger the collection is.

Conclusion

MongoDB 5.0 is an interesting new version of the most popular document-based database, and new features like Time Series collections and resharding are amazing.

Anyway, due to many core changes to WiredTiger and the core server introduced to facilitate new features, MongoDB 5.0.x is still unstable. We do not recommend using it for production environments.

Check the documentation of Percona Server for MongoDB 5.0.3-2 (Release Candidate).

Complete the 2021 Percona Open Source Data Management Software Survey

Have Your Say!

Oct
19
2021
--

How to Build Percona Server for MongoDB for Various Operating Systems

How to Build Percona Server for MongoDB Operating Systems

How to Build Percona Server for MongoDB Operating SystemsFollowing this series of blog posts started by Evgeniy Patlan:

we’ll show you how to build Percona Server for MongoDB for various operating systems¹ using Docker on your local Linux machine/build server. In this very case, we’ll build packages of Percona Server for MongoDB 4.4.9-10 version for Centos 8 and Debian 11 (bullseye).

This can be useful when you need to test your changes to the code for different RPMs/DEBs based platforms and make sure that all works as expected in different environments. In our case, this approach is used for building Percona Server for MongoDB packages/binary tarballs for all supported OSes.

Prepare Build Environment

  • Make sure that you have at least 60GB of free disk space
  • Create a “build folder” – the folder where all the build actions will be performed, in our case “/mnt/psmdb-44/test
  • Make sure that you have installed the package which provides docker and docker service is up and running

Obtain Build Script of Needed Version²

You need to download the build script of the needed version to the “/mnt/psmdb-44” folder:

cd /mnt/psmdb-44/
wget https://raw.githubusercontent.com/percona/percona-server-mongodb/psmdb-4.4.9-10/percona-packaging/scripts/psmdb_builder.sh -O psmdb_builder.sh

Create Percona Server for MongoDB Source Tarball

  • Please note that for the creation of source tarball, we use the oldest supported OS, in this case, it is Centos 7.
docker run -ti -u root -v /mnt/psmdb-44:/mnt/psmdb-44 centos:7 sh -c '
set -o xtrace
cd /mnt/psmdb-44
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --install_deps=1
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --repo=https://github.com/percona/percona-server-mongodb.git \
--branch=release-4.4.9-10 --psm_ver=4.4.9 --psm_release=10 --mongo_tools_tag=100.4.1 --jemalloc_tag=psmdb-3.2.11-3.1 --get_sources=1
'

  • Check that source tarball has been created:
$ ls -la /mnt/psmdb-44/source_tarball/
total 88292
drwxr-xr-x. 2 root root     4096 Oct  1 10:58 .
drwxr-xr-x. 5 root root     4096 Oct  1 10:58 ..
-rw-r--r--. 1 root root 90398894 Oct  1 10:58 percona-server-mongodb-4.4.9-10.tar.gz

Build Percona Server for MongoDB Generic Source RPM/DEB:

Please note that for building generic source RPM/DEB, we still use the oldest supported RPM/DEB-based OS, in this case, Centos 7/ Ubuntu Xenial(16.04).

  • Build source RPM:
docker run -ti -u root -v /mnt/psmdb-44:/mnt/psmdb-44 centos:7 sh -c '
set -o xtrace
cd /mnt/psmdb-44
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --install_deps=1
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --repo=https://github.com/percona/percona-server-mongodb.git \
--branch=release-4.4.9-10 --psm_ver=4.4.9 --psm_release=10 --mongo_tools_tag=100.4.1 --jemalloc_tag=psmdb-3.2.11-3.1 --build_src_rpm=1
'

  • Build source DEB:
docker run -ti -u root -v /mnt/psmdb-44:/mnt/psmdb-44 ubuntu:xenial sh -c '
set -o xtrace
cd /mnt/psmdb-44
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --install_deps=1
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --repo=https://github.com/percona/percona-server-mongodb.git \
--branch=release-4.4.9-10 --psm_ver=4.4.9 --psm_release=10 --mongo_tools_tag=100.4.1 --jemalloc_tag=psmdb-3.2.11-3.1 --build_src_deb=1
'

  • Check that both SRPM and Source DEB have been created:
$ ls -la /mnt/psmdb-44/srpm/
total 87480
drwxr-xr-x. 2 root root     4096 Oct  1 11:35 .
drwxr-xr-x. 6 root root     4096 Oct  1 11:35 ..
-rw-r--r--. 1 root root 89570312 Oct  1 11:35 percona-server-mongodb-4.4.9-10.generic.src.rpm

$ ls -la /mnt/psmdb-44/source_deb/
total 88312
drwxr-xr-x. 2 root root     4096 Oct  1 11:45 .
drwxr-xr-x. 7 root root     4096 Oct  1 11:45 ..
-rw-r--r--. 1 root root    10724 Oct  1 11:45 percona-server-mongodb_4.4.9-10.debian.tar.xz
-rw-r--r--. 1 root root     1528 Oct  1 11:45 percona-server-mongodb_4.4.9-10.dsc
-rw-r--r--. 1 root root     2075 Oct  1 11:45 percona-server-mongodb_4.4.9-10_source.changes
-rw-r--r--. 1 root root 90398894 Oct  1 11:45 percona-server-mongodb_4.4.9.orig.tar.gz

Build Percona Server for MongoDB RPMs/DEBs:

  • Build RPMs:
docker run -ti -u root -v /mnt/psmdb-44:/mnt/psmdb-44 centos:8 sh -c '
set -o xtrace
cd /mnt/psmdb-44
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --install_deps=1
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --repo=https://github.com/percona/percona-server-mongodb.git \
--branch=release-4.4.9-10 --psm_ver=4.4.9 --psm_release=10 --mongo_tools_tag=100.4.1 --jemalloc_tag=psmdb-3.2.11-3.1 --build_rpm=1
'

  • Build DEBs:
docker run -ti -u root -v /mnt/psmdb-44:/mnt/psmdb-44 debian:bullseye sh -c '
set -o xtrace
cd /mnt/psmdb-44
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --install_deps=1
bash -x ./psmdb_builder.sh --builddir=/mnt/psmdb-44/test --repo=https://github.com/percona/percona-server-mongodb.git \
--branch=release-4.4.9-10 --psm_ver=4.4.9 --psm_release=10 --mongo_tools_tag=100.4.1 --jemalloc_tag=psmdb-3.2.11-3.1 --build_deb=1
'

  • Check that RPMs for Centos 8 and DEBs for Debian 11 have been created:
$  ls -la /mnt/psmdb-44/rpm/
total 1538692
drwxr-xr-x. 2 root root      4096 Oct  1 13:19 .
drwxr-xr-x. 9 root root      4096 Oct  1 13:19 ..
-rw-r--r--. 1 root root      8380 Oct  1 13:19 percona-server-mongodb-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root  19603132 Oct  1 13:19 percona-server-mongodb-debugsource-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root  16199100 Oct  1 13:19 percona-server-mongodb-mongos-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root 382301668 Oct  1 13:19 percona-server-mongodb-mongos-debuginfo-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root  37794568 Oct  1 13:19 percona-server-mongodb-server-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root 829718252 Oct  1 13:19 percona-server-mongodb-server-debuginfo-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root  13310328 Oct  1 13:19 percona-server-mongodb-shell-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root 218625728 Oct  1 13:19 percona-server-mongodb-shell-debuginfo-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root  30823056 Oct  1 13:19 percona-server-mongodb-tools-4.4.9-10.el8.x86_64.rpm
-rw-r--r--. 1 root root  27196024 Oct  1 13:19 percona-server-mongodb-tools-debuginfo-4.4.9-10.el8.x86_64.rpm

$  ls -la /mnt/psmdb-44/deb/
total 2335288
drwxr-xr-x. 2 root root       4096 Oct  1 13:16 .
drwxr-xr-x. 9 root root       4096 Oct  1 13:16 ..
-rw-r--r--. 1 root root 2301998432 Oct  1 13:16 percona-server-mongodb-dbg_4.4.9-10.bullseye_amd64.deb
-rw-r--r--. 1 root root   14872728 Oct  1 13:16 percona-server-mongodb-mongos_4.4.9-10.bullseye_amd64.deb
-rw-r--r--. 1 root root   35356944 Oct  1 13:16 percona-server-mongodb-server_4.4.9-10.bullseye_amd64.deb
-rw-r--r--. 1 root root   12274928 Oct  1 13:16 percona-server-mongodb-shell_4.4.9-10.bullseye_amd64.deb
-rw-r--r--. 1 root root   26784020 Oct  1 13:16 percona-server-mongodb-tools_4.4.9-10.bullseye_amd64.deb
-rw-r--r--. 1 root root      18548 Oct  4 13:16 percona-server-mongodb_4.4.9-10.bullseye_amd64.deb

Now, the packages are ready to be installed for testing/working on Centos 8 and Debian 11.

As you can see from the above, the process of building packages for various operating systems is quite easy and doesn’t require lots of physical/virtual machines. All you need is the build script and Docker.

Also, as you may have noticed, all the build commands are similar to each other except the last passed argument, which defines the action that should be performed. Such an approach allows us to unify the build process and make it scripted so that the last argument can be passed as a parameter to the script. Surely all the rest arguments can and should also be passed as parameters in case you are going to automate the build process.

¹ Supported operating systems(version psmdb-4.4.9-10):

  • Centos 7
  • Centos 8
  • Ubuntu Xenial(16.04)
  • Ubuntu Bionic(18.04)
  • Ubuntu Focal(20.04)
  • Debian Stretch(9)
  • Debian Buster(10)
  • Debian Bullseye(11)

² In order to build Percona Server for MongoDB of another version, you need to use the build script of the proper version. For example, it is needed to build Percona Server for MongoDB of 4.2.7-7 version:

Complete the 2021 Percona Open Source Data Management Software Survey

Have Your Say!

Oct
13
2021
--

Migrating MongoDB to Kubernetes

Migrating MongoDB to Kubernetes

Migrating MongoDB to KubernetesThis blog post is the last in the series of articles on migrating databases to Kubernetes with Percona Operators. Two previous posts can be found here:

As you might have guessed already, this time we are going to cover the migration of MongoDB to Kubernetes. In the 1.10.0 release of Percona Distribution for MongoDB Operator, we have introduced a new feature (in tech preview) that enables users to execute such migrations through regular MongoDB replication capabilities. We have already shown before how it can be used to provide cross-regional disaster recovery for MongoDB, we encourage you to read it.

The Goal

There are two ways to migrate the database:

  1. Take the backup and restore it.
    – This option is the simplest one, but unfortunately comes with downtime. The bigger the database, the longer the recovery time is.
  2. Replicate the data to the new site and switch the application once replicas are in sync.
    – This allows the user to perform the migration and switch the application with either zero or little downtime.

This blog post is a walkthrough on how to migrate MongoDB replica set to Kubernetes with replication capabilities. 

MongoDB replica set to Kubernetes

  1. We have a MongoDB cluster somewhere (the Source). It can be on-prem or some virtual machine. For demo purposes, I’m going to use a standalone replica set node. The migration procedure of a replica set with multiple nodes or sharded cluster is almost identical.
  2. We have a Kubernetes cluster with Percona Operator (the Target). The operator deploys 3 standalone MongoDB nodes in unmanaged mode (we will talk about it below).
  3. Each node is exposed so that the nodes on the Source can reach them.
  4. We are going to replicate the data to Target nodes by adding them into the replica set.

As always all blog post scripts and configuration files are available publicly in this Github repository.

Prerequisites

  • MongoDB cluster – either on-prem or VM. It is important to be able to configure mongod to some extent and add external nodes to the replica set.
  • Kubernetes cluster for the Target.
  • kubectl to deploy and manage the Operator and database on Kubernetes.

Prepare the Source

This section explains what preparations must be made on the Source to set up the replication.

Expose

All nodes in the replica set must form a mesh and be able to reach each other. The communication between the nodes can go through the public internet or some VPN. For demonstration purposes, we are going to expose the Source to the public internet by editing mongod.conf:

net:
  bindIp: <PUBLIC IP>

If you have multiple replica sets – you need to expose all nodes of each of them, including config servers.

TLS

We take security seriously at Percona, and this is why by default our Operator deploys MongoDB clusters with encryption enabled. I have prepared a script that generates self-signed certificates and keys with the openssl tool. If you already have Certificate Authority (CA) in use in your organization, generate the certificates and sign them by your CA.

The list of alternative names can be found either in this document or in this openssl configuration file. Note DNS.20 entry:

DNS.20      = *.mongo.spron.in

I’m going to use this wildcard entry to set up the replication between the nodes. The script also generates an

ssl-secret.yaml

file, which we are going to use on the Target side.

You need to upload CA and certificate with a private key to every Source replica set node and then define it in the

mongod.conf

:

# network interfaces
net:
  ...
  tls:
    mode: preferTLS
    CAFile: /path/to/ca.pem
    certificateKeyFile: /path/to/mongod.pem

security:
  clusterAuthMode: x509
  authorization: enabled

Note that I also set

clusterAuthMode

to x509. It enforces the use of x509 authentication. Test it carefully on a non-production environment first as it might break your existing replication.

Create System Users

Our Operator needs system users to manage the cluster and perform health checks. Usernames and passwords for system users should be the same on the Source and the Target. This script is going to generate the

user-secret.yaml

to use on the Target and mongo shell code to add the users on the Source (it is an example, do not use it in production).

Connect to the primary node on the Source and execute mongo shell commands generated by the script.

Prepare the Target

Apply Users and TLS secrets

System users’ credentials and TLS certificates must be similar on both sides. The scripts we used above generate Secret object manifests to use on the Target. Apply them:

$ kubectl apply -f ssl-secret.yaml
$ kubectl apply -f user-secret.yam

Deploy the Operator and MongoDB Nodes

Please follow one of the installation guides to deploy the Operator. Usually, it is one step operation through

kubectl

:

$ kubectl apply -f operator.yaml

MongoDB nodes are deployed with a custom resource manifest – cr.yaml. There are the following important configuration items in it:

spec:
  unmanaged: true

This flag instructs Operator to deploy the nodes in unmanaged mode, meaning they are not configured to form the cluster. Also, the Operator does not generate TLS certificates and system users.

spec:
…
  updateStrategy: Never

Disable the Smart Update feature as the cluster is unmanaged.

spec:
…
  secrets:
    users: my-new-cluster-secrets
    ssl: my-custom-ssl
    sslInternal: my-custom-ssl-internal

This section points to the Secret objects that we created in the previous step.

spec:
…
  replsets:
  - name: rs0
    size: 3
    expose:
      enabled: true
      exposeType: LoadBalancer

Remember, that nodes need to be exposed and reachable. To do this we create a service per Pod. In our case, it is a

LoadBalancer

object, but it can be any other Service type.

spec:
...
  backup:
    enabled: false

If the cluster and nodes are unmanaged, the Operator should not be taking backups. 

Deploy unmanaged nodes with the following command:

$ kubectl apply -f cr.yaml

Once nodes are up and running, also check the services. We will need the IP-addresses of new replicas to add them later to the replica set on the Source.

$ kubectl get services
NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)           AGE
…
my-new-cluster-rs0-0    LoadBalancer   10.3.252.134   35.223.104.224   27017:31332/TCP   2m11s
my-new-cluster- rs0-1   LoadBalancer   10.3.244.166   34.134.210.223   27017:32647/TCP   81s
my-new-cluster-rs0-2    LoadBalancer   10.3.248.119   34.135.233.58    27017:32534/TCP   45s

Configure Domains

X509 authentication is strict and requires that the certificate’s common name or alternative name match the domain name of the node. As you remember we had wildcard

*.mongo.spron.in

included in our certificate. It can be any domain that you use, but make sure a certificate is issued for this domain.

I’m going to create A-records to point to public IP-addresses of MongoDB nodes:

k8s-1.mongo.spron-in -> 35.223.104.224
k8s-2.mongo.spron.in -> 34.134.210.223
k8s-3.mongo.spron-in -> 34.135.233.58

Replicate the Data to the Target

It is time to add our nodes in the Kubernetes cluster to the replica set. Login into the mongo shell on the Source and execute the following:

rs.add({ host: "k8s-1.mongo.spron.in", priority: 0, votes: 0} )
rs.add({ host: "k8s-2.mongo.spron.in", priority: 0, votes: 0} )
rs.add({ host: "k8s-3.mongo.spron.in", priority: 0, votes: 0} )

If everything is done correctly these nodes are going to be added as secondaries. You can check the status with the

rs.status()

command.

Cutover

Check that newly added node are synchronized. The more data you have, the longer the synchronization process is going to take. To understand if nodes are synchronized you should compare the values of

optime

and

optimeDate

of the Primary node with the values for the Secondary node in

rs.status()

output:

{
        "_id" : 0,
        "name" : "147.182.213.59:27017",
        "stateStr" : "PRIMARY",
...
        "optime" : {
                "ts" : Timestamp(1633697030, 1),
                "t" : NumberLong(2)
        },
        "optimeDate" : ISODate("2021-10-08T12:43:50Z"),
...
},
{
        "_id" : 1,
        "name" : "k8s-1.mongo.spron.in:27017",
        "stateStr" : "SECONDARY",
...
        "optime" : {
                "ts" : Timestamp(1633697030, 1),
                "t" : NumberLong(2)
        },
        "optimeDurable" : {
                "ts" : Timestamp(1633697030, 1),
                "t" : NumberLong(2)
        },
        "optimeDate" : ISODate("2021-10-08T12:43:50Z"),
...
},

When nodes are synchronized, we are ready to perform the cutover. Please ensure that your application is configured properly to minimize downtime during the cutover.

The cutover is going to have two steps:

  1. One of the nodes on the Target becomes the primary.
  2. Operator starts managing the cluster and nodes on the Source are no longer present in the replica set.

Switch the Primary

Connect with mongo shell to the primary on the Source side and make one of the nodes on the Target primary. It can be done by changing the replica set configuration:

cfg = rs.config()
cfg.members[1].priority = 2
cfg.members[1].votes = 1
rs.reconfig(cfg)

We enable voting and set priority to two on one of the nodes in the Kubernetes cluster. Member id can be different for you, so please look carefully into the output of

rs.config()

command.

Start Managing the Cluster

Once the primary is running in Kubernetes, we are going to tell the Operator to start managing the cluster. Change

spec.unmanaged

to

false

 in the Custom Resource with patch command:

$ kubectl patch psmdb my-cluster-name --type=merge -p '{"spec":{"unmanaged": true}}'

You can also do this by changing

cr.yaml

and applying it. This is it, now you have the cluster in Kubernetes which is managed by the Operator. 

Conclusion

You truly start to appreciate Operators once you get used to them. When I was writing this blog post I found it extremely annoying to deploy and configure a single MongoDB node on a Linux box; I don’t want to think about the cluster. Operators abstract Kubernetes primitives and database configuration and provide you with a fully operational database service instead of a bunch of nodes. Migration of MongoDB to Kubernetes is a challenging task, but it is much simpler with Operator. And once you are on Kubernetes, Operator takes care of all day-2 operations as well.

We encourage you to try out our operator. See our GitHub repository and check out the documentation.

Found a bug or have a feature idea? Feel free to submit it in JIRA.

For general questions please raise the topic in the community forum

You are a developer and looking to contribute? Please read our CONTRIBUTING.md and send the Pull Request.

Percona Distribution for MongoDB Operator

The Percona Distribution for MongoDB Operator simplifies running Percona Server for MongoDB on Kubernetes and provides automation for day-1 and day-2 operations. It’s based on the Kubernetes API and enables highly available environments. Regardless of where it is used, the Operator creates a member that is identical to other members created with the same Operator. This provides an assured level of stability to easily build test environments or deploy a repeatable, consistent database environment that meets Percona expert-recommended best practices.

Complete the 2021 Percona Open Source Data Management Software Survey

Have Your Say!

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com