Personal blog of Yzmir Ramirez

Managing database users within complex CICD pipelines and GitOps workflows has long been a challenge for MongoDB deployments. With Percona Operator for MongoDB 1.17, we introduce a new feature, currently in technical preview, that streamlines this process. Now, you can create the database users you need directly within the operator, eliminating the need to wait […]

 Businesses are gathering more and more information than they ever have before, whether it’s a manufacturer or a bank, and to Timescale CEO Ajay Kulkarni the existing database software soon won’t provide the horsepower they need. That’s why he and CTO Mike Freedman started Timescale, a new open source time series database software. The idea is that businesses increasingly… Read More

Introduction

In this blog, we’ll review MongoDB 3.2 elections and how they work, as well as what is really new and different in the election protocol.

MongoDB 3.2 revamped its election protocol for increased stability! Exciting times, with smarter and faster elections are here! With this latest release, you will find that replication (and the election protocol) have been improved. Some of the changes include:

• The addition of

  electionTimeoutMS

• WriteConcern

    now implies “j:true”

  • Old j:true meant just the primary node
  • New j:true means all involved nodes must ACK the journal
  • j:true means your journal MS will be thirded, and synchronization occurs every 10ms (MMAP) or 33ms (WiredTiger) by default
• Optime in rs.status now an Object, not a Timestamp

You’ll need to enable the Election Protocol when upgrading MongoDB from an earlier version, while new replSets get it enabled by default.

Election Protocol: what is an election?

Mongo uses a consensus protocol. This means that all nodes must agree who is the most current when handing:

• Hardware failure
• Network split
• Time shifts

New updates allow for faster elections using an (term) electionId to prevent timeout between separate voting rounds. This guarantees there aren’t double (and conflicting) votes while also reducing the time to wait to know a vote completed.

How does it do it?

Elections now have “term” or “vote” identifiers (ID). Terms are used to separate voting rounds. Every vote attempt increments the ID. The ID incrementation prevents a node from double voting in the same term, and makes it easier for nodes to know if a re-vote is needed where before it could be up to 5 minutes!

The protocol timeouts have some new features and behaviors:

• Now configurable
• Randomness added to each node
• Less chance all node timeout at the same time

Normal election process

Below I’m going to walk you through a typical replica set operation. The configuration looks like the following:

In this topology:

• There are three members
• All of them are heartbeating to each other
• There is no arbiter, so you get full high availability (HA)

The following diagram provides a more detailed picture of the interactions:

Notice how replication pulls from the primary to each secondary from the primary – the secondary does all the work. A heartbeat is still shared by all the nodes.

Now let’s see what happens when our primary crashes. It just did!

Nodes will still try to heartbeat to it until two have failed in a short period.

After the failure, things happen quickly.

1. Secondaries give up on heartbeats
2. They then vote with each other on who is newest in oplog
3. If they have > 50% of total voting population they select a new winner

A new Primary is selected, and the heartbeat system is cleaned up.

Replication now gets restarted. If the fatal node comes back online, it’s treated as a secondary once it “catches up” via the oplog.

Stepdown Election Process

The stepdown election process is the same as above, with the following caveats:

• It’s MUCH faster, as the existing primary “starts” an election
• There is less chance of the old primary not having data replicated
• It kills writes while doing election
• The election process doesn’t wait for heartbeat timeouts

Generally speaking, you should always try to use the stepdown election process. Timeouts are for crashes and failures, not general use.