Personal blog of Yzmir Ramirez

Announcement

I will be speaking at Percona Live 2023 about serverless PostgreSQL. Join us at this event if you are interested!

Introduction

Recently, Percona introduced Percona Builds for Neon (Introducing Percona Builds for Serverless PostgreSQL), which makes it easy to install and experiment with serverless PostgreSQL. And I followed it with how you can run easy experimentations with Neon using Docker (Using Docker To Deploy Neon Serverless PostgreSQL).

Before getting into more details about Neon functionality, we need to introduce two important concepts Neon operates with: Tenants and Data Branches.

Tenants

Multi-tenancy is a software architecture pattern where a single instance of a software application serves multiple tenants, allowing them to share resources like storage, processing power, and memory while maintaining separate, secure access to their respective data.

In the context of databases, a tenant’s data can be logically separated from other tenants’ data in the same database so that each tenant can only access and manage their own data. This is particularly useful in cloud-based applications, where multiple customers or organizations share the same underlying infrastructure to reduce costs and improve scalability.

The key aspect of multi-tenancy is the isolation of tenant data, ensuring that each tenant’s data remains secure, private, and separate from other tenants’ data.

In the case of Neon, the single pageserver already comes with multi-tenant functionality, and at the very least, you must have one tenant to create PostgreSQL compute node instances.

The requirement is that each tenant will operate with its own compute node.

Compute Nodes can be created/destroyed on demand; they are not required if there is no current need to access data for a given tenant.

Data branches

Neon enables you to seamlessly create branches of your Postgres database, enhancing your development workflow. With each code deployment, you can establish a separate branch for your testing environments. This way, developers and testers can obtain a data copy for testing or experimentation without impacting the main data.

Utilizing the “copy on write” method, branches are virtually cost-free. Neon refers to data branches as “timelines,” and these terms are often used interchangeably.

You can think (at least I do) of timelines as GitHub branches. Just keep in mind that data branches can’t be merged back.

In the example above:

• Timeline 1 is branched from the Main timeline after two commits.
• Timeline 2 is branched from the Main timeline after four commits.
• Timeline 3 is branched from Timeline 1 after two more commits to Timeline 1.

And to expand on compute node requirements: each timeline requires a separate compute node to operate (and you can shut down the compute node if you do not need to work with the timeline right now).

There is a restriction to be aware of: each tenant/timeline can have only one compute node.

Schematically it looks like this:

I will show the practical work with tenants/timelines in the following blog post, so subscribe for all updates.

Interested in serverless PostgreSQL?

You are welcome to experiment with Neon using our Docker images.

For feedback and questions, please use our Forums: (Latest Percona Labs/Percona Build for Serverless PostgreSQL topics – Percona Community Forum)

Follow our blog for more information about setups, performance, and unique features of serverless PostgreSQL.

If you want to talk with us about builds for serverless PostgreSQL or stay in touch for future updates – leave your contact details below.

Contact form

Also, if you are interested in possible Support for serverless PostgreSQL, fill out the Contact form, and I will get in touch with you.

Announcement

I will be speaking at Percona Live 2023 about serverless PostgreSQL. Join us at this event if you are interested!

Introduction

Recently, Percona introduced Percona Builds for Neon (Introducing Percona Builds for Serverless PostgreSQL), which makes it easy to install and experiment with serverless PostgreSQL. But now, there’s an even more convenient way to explore the capabilities of serverless PostgreSQL — Docker images. In this article, we’ll explore the benefits of using Docker images for experimenting with serverless PostgreSQL and show you how to get started with them.

Architecture

To understand more about deployment procedures, we need to look a little more at Neon architecture.

There is a section in our Documentation (Introduction to Serverless PostgreSQL) and a short overview of the primary components:

• Page Server
  • The storage server with the primary goal of storing all data pages and WAL records
• Safe Keeper
  • A component to store WAL records in memory (to reduce latency). Because of the critical importance of these components, the recommendation is to deploy multiple of them. They will connect in Paxos group to guarantee data consistency.
  • Later the WAL records will be shipped to Pageserver, where it will use them to update data pages
• Compute Nodes
  • The component to take and handle user queries. Basically, you can view this as a PostgreSQL instance but without a storage layer
• Storage Broker

Storage Broker is a coordination component between WAL Service and Pageserver.

The Pageserver listens for GetPage@LSN requests from the Compute Nodes and responds with pages from the repository.

Deployment with Docker

We published a Docker image with all components in Docker Hub:

perconalab/neon Tags | Docker Hub

And the source code is located in our build repository:

Percona-Lab/serverless-postgresql-build (github.com)

Deployment steps

For my experiments, I follow these steps. I prefer to test a distributed deployment where each component is placed on different servers or virtual machines, that’s why I do not put it into docker-compose.

The assumption is that I use a reachable server with IP address 172.16.0.9. Unfortunately, I have to use physical IP addresses for connectivity in multiple servers docker deployment.

1. Deploy storage broker

docker run -d -t --name storagebroker --net=host
--entrypoint "storage_broker"
perconalab/neon:latest -l 0.0.0.0:50051

2. Deploy safekeeper (or several of them for redundancy)

docker run -d -t --name safekeeper1 --net=host
--entrypoint "safekeeper"
perconalab/neon:latest
--id=1 -D /data --broker-endpoint=http://172.16.0.9:50051
-l 172.16.0.9:5454 --listen-http=0.0.0.0:7676

3. Deploy pageserver

docker run -d -t --name pageserver --net=host
--entrypoint "pageserver"
perconalab/neon:latest
-D /data -c "id=1" -c "broker_endpoint='http://172.16.0.9:50051'"
-c "listen_pg_addr='0.0.0.0:6400'" -c "listen_http_addr='0.0.0.0:9898'"
-c "pg_distrib_dir='/opt/neondatabase-neon/pg_install'"

4. Most interesting part – deploy compute nodes (components to handle client requests).

The most interesting part is deploying compute nodes, which are the components that handle client requests. Before we dive in, I need to introduce some new concepts that I’ll describe in more detail in future blog posts, so bear with me for now.

Concept 1: Tenants – Neon has the capability to serve multiple tenants, all located on the same pageserver but visible as separate PostgreSQL instances to clients.

Concept 2: Timelines – Each tenant can have multiple timelines and the ability to branch the current state into a new timeline. At a minimum, a tenant must have one timeline.

Now, let’s get back to deploying compute nodes.

Deploying compute node with creating new tenant and timeline:

docker run -d -t --name compute
--entrypoint "/compute.sh"
-p55432:55432 -e PAGESERVER=172.16.0.9
-e SAFEKEEPERS=172.16.0.9:5454 perconalab/neon:latest

This will create a lightweight PostgreSQL instance (compute node), accessible by port 55432. And you can connect to the instance with a normal PSQL client as:

psql -p55432 -h 127.0.0.1 -U cloud_admin postgres

Deploying compute node with existing tenant and timeline:

Assume we already have a tenant and timeline, and we want to attach a lightweight PostgreSQL instance (compute node):

To get timeline and tenant from the previous start, you can find them in docker logs (

docker logs compute

) identified as:

{
"name": "neon.timeline_id",
"value": "4b4541ad75370114cd7956e457cc875f",
"vartype": "string"
},
{
"name": "neon.tenant_id",
"value": "6c92c037a54c0e3a005cdd4a69d6e997",
"vartype": "string"
},

docker run -d -t --name compute1
--entrypoint "/compute.sh" -p55433:55432
-e PAGESERVER=172.16.0.9 -e SAFEKEEPERS=172.16.0.9:5454
-e TENANT=51021f53054316c6533d371c9d7e273c -e TIMELINE=e08a6f1526b3ad6249a7b08fc5585e0b
perconalab/neon:latest

Deploying compute node with branching from existing tenant and timeline:

This is the most exciting capability: we can fork (branch) existing data into a new timeline (more in the following blog posts):

docker run -d -t --name compute3
--entrypoint "/compute.sh" -p55435:55432
-e PAGESERVER=172.16.0.9
-e SAFEKEEPERS=172.16.0.9:5454
-e TENANT=6c92c037a54c0e3a005cdd4a69d6e997 -e TIMELINE=4b4541ad75370114cd7956e457cc875f
-e "CREATE_BRANCH=1" perconalab/neon:latest

Interested?

You are welcome to experiment with Neon using our Docker images.

For feedback and questions, please use our Forums: (Latest Percona Labs/Percona Build for Serverless PostgreSQL topics – Percona Community Forum)

Follow our blog for more information about setups, performance, and unique features of Serverless PostgreSQL.

If you would like to talk with us about builds for Serverless PostgreSQL or stay in touch for future updates – leave your contact details below.

Contact form

Also, if you are interested in possible Support for Serverless PostgreSQL, fill out the Contact form, and I will get in touch with you.

Recently, “serverless” has become a buzzword, and for good reason. The goal is to simplify the provisioning and management of database capacity. One approach is to separate compute and storage to allow for independent scaling.

We are thrilled to announce our collaboration with Neon (Neon – Serverless, Fault-Tolerant, Branchable Postgres) to provide a Serverless PostgreSQL that you can control and manage yourself.

So, what is Neon? It’s an open source alternative to AWS Aurora Postgres that utilizes a serverless architecture. By separating storage and compute, Neon replaces the PostgreSQL storage layer with data nodes, and compute nodes are distributed across a cluster of nodes.

Architecture

A Neon installation consists of compute nodes and the Neon storage engine. Compute nodes are stateless PostgreSQL nodes backed by the Neon storage engine.

One of the benefits of separating storage from compute nodes is the ability to scale them independently. For instance, if we run out of storage space, we can easily add additional storage nodes without affecting the PostgreSQL instances.

On the other hand, if we reach the limit of our compute nodes, we can simply add more compute nodes separately. This is because the compute nodes are stateless in this architecture, meaning they can be scaled without affecting the system’s performance.

By separating the storage from compute nodes, we’re able to optimize our resources and easily scale up as needed without disrupting the overall system. This approach allows for more efficient use of resources and improved performance.

Branches

Neon uses copy-on-write snapshots on Storage Nodes, which allows a very cheap way to branch data.

Have you ever wanted the ability to fork your data set and work on it independently, similar to using Git branches? This can now be achieved with ease using Serverless PostgreSQL. Simply fork your data, conduct experiments, and have peace of mind knowing that your primary data set remains unaffected.

Please subscribe to our newsletter for more blog articles and documentation on how to use Branches!

Builds

Percona provides binary builds for Serverless PostgreSQL based on the Neon. This is the only place where you can get pre-built binaries and help with how to use them.

At this point, the binaries are EXPERIMENTAL and ONLY FOR TESTING purposes. Percona does not provide official support for the build at this moment.

Binary releases location

The binaries for releases are hosted on the GitHub release page:

Releases · Percona-Lab/neon (github.com)

Website location: https://percona.community/labs/serverless-postgresql/

Documentation: https://percona.community/labs/serverless-postgresql/docs/

Interested?

For feedback and questions, please use our Forums.

Follow our blog for more information about setups, performance, and unique features of Serverless PostgreSQL.

If you would like to talk with us about builds for Serverless PostgreSQL, are interested in support, or want to stay in touch for future updates, leave your contact details in the form below.

Contact form