Nov
22
2022
--

Troubleshooting Percona Operators With troubleshoot.sh

Troubleshooting Percona Operators

Troubleshooting Percona OperatorsPercona loves and embraces Kubernetes. Percona Operators have found their way into the hearts and minds of developers and operations teams. And with growing adoption, we get a lot of valuable feedback from the community and our support organization. Two of the most common questions that we hear are:

  1. What are the best practices to run Operators?
  2. Where do I look if there is a problem? In other words, how do I troubleshoot the issues?

In this blog post, we will experiment with troubleshoot.sh, an open source tool developed by Replicated, and see if it can help with our problems. 

Prepare

Installation

Troubleshoot.sh is a collection of kubectl plugins. Installation is dead simple and described in the documentation.

Concepts

There are two basic use cases:

  1. Preflight – check if your cluster is ready and compliant to run an application, so it should cover the best practices question
  2. Support bundle – check if the application is running and behaving appropriately, it clearly helps with troubleshooting

Three functional concepts in the tool that you should know about:

  1. Collectors – where you can define what information you want to collect. A lot of basic information is collected about the cluster by default. 
  2. Redact – if there is sensitive information, you can redact it from the bundle. For example, AWS credentials are somewhere in the logs of your application.
  3. Analyze – act on collected information and provide actionable insights. 

 

Preflight

Checks are defined through YAML files. I have created an example to check if the Kubernetes cluster is ready to run Percona Operator for MySQL (based on Percona XtraDB Cluster). This checks the following:

  1. Kubernetes version and flavor – as we only test our releases on specific versions and providers
  2. Default storage class – it is possible to use local storage, but by default, our Operator uses Persistent Volume Claims
  3. Node size – we check if nodes have at least 1 GB of RAM, otherwise, Percona XtraDB Cluster might fail to start

Run the check with the following command:

kubectl preflight https://raw.githubusercontent.com/spron-in/blog-data/master/percona-troubleshoot/pxc-op-1.11-preflight.yaml

It is just an example. Preflight checks are very flexible and you can read more in the documentation.

Support bundle

Support bundles can be used by support organizations or by users to troubleshoot issues with applications on Kubernetes. Once you run the command, it also creates a tar.gz archive with various data points that can be analyzed later or shared with experts.

Similarly to Preflight, support bundle checks are defined in YAML files. See the example that I created here and run it with the following command:

kubectl support-bundle https://raw.githubusercontent.com/spron-in/blog-data/master/percona-troubleshoot/pxc-op-1.11-support.yaml

Check cluster state

First, we are going to check the presence of Custom Resource Definition and the health of the cluster. In my YAML the name of the cluster is hard coded to minimal-cluster (it can be deployed from our repo).

We are using Custom Resource fields to verify the statuses of Percona XtraDB Cluster, HAProxy, and ProxySQL. This is the example checking HAProxy:

 - yamlCompare:
        checkName: Percona Operator for MySQL - HAProxy
        fileName: cluster-resources/custom-resources/perconaxtradbclusters.pxc.percona.com/default.yaml
        path: "[0].status.haproxy.status"
        value: ready
        outcomes:
          - fail:
              when: "false"
              message: HAProxy is not ready
          - pass:
              when: "true"
              message: HAProxy is ready

perconaxtradbclusters.pxc.percona.com/default.yaml

has all

pxc

Custom Resources in the default namespace. I know I have only one there, so I just parse the whole YAML with the

yamlCompare

method.

Verify Pods statuses

If you see that previous checks failed, it is time to check the Pods. In the current implementation, you can get the statuses of the Pods in various namespaces. 

- clusterPodStatuses:
        name: unhealthy
        namespaces:
          - default
        outcomes:
…
          - fail:
              when: "== Pending"
              message: Pod {{ .Namespace }}/{{ .Name }} is in a Pending state with status of {{ .Status.Reason }}.

Parse logs for known issues

Parsing logs of the application can be tedious. But what if you can predefine all known errors and quickly learn about the issue without going into your central logging system? Troubleshoot.sh can do it as well.

First, we define the collectors. We will get the logs from the Operator and Percona XtraDB Cluster itself:

collectors:
    - logs:
        selector:
          - app.kubernetes.io/instance=minimal-cluster
          - app.kubernetes.io/component=pxc
        namespace: default
        name: pxc/container/logs
        limits:
          maxAge: 24h
    - logs:
        selector:
          - app.kubernetes.io/instance=percona-xtradb-cluster-operator
          - app.kubernetes.io/component=operator
        namespace: default
        name: pxc-operator/container/logs
        limits:
          maxAge: 24h

Now the logs are in our support bundle. We can analyze them manually or catch some known messages with a regular expression analyzer:

    - textAnalyze:
        checkName: Failed to update the password
        fileName: pxc-operator/container/logs/percona-xtradb-cluster-operator-.*.log
        ignoreIfNoFiles: true
        regex: 'Error 1396: Operation ALTER USER failed for'
        outcomes:
          - pass:
              when: "false"
              message: "No failures on password change"
          - fail:
              when: "true"
              message: "There was a failure to change the system user password. For more details: https://docs.percona.com/percona-operator-for-mysql/pxc/users.html"

In the example we are checking for a message in the Operator log, that indicates that the system user password change failed.

Check for security best practices

Except for dealing with operational issues, troubleshoot.sh can help with compliance checks. You should have some security guardrails in the code before you deploy to Kubernetes, but if something slipped you will have a second line of defense. 

In our example we check for weak passwords and if the statefulset matches high availability best practices. 

Collecting and sharing passwords is not recommended, but still can be a valuable addition to your security policies. To capture secrets you need to implicitly define a collector:

spec:
  collectors:
    - secret:
        namespace: default
        name: internal-minimal-cluster
        includeValue: true
        key: root

Replay support bundle with sbctl

sbctl is an awesome addition to troubleshoot.sh. It allows users and support engineers to examine support bundles as if it was regular interaction with the Kubernetes cluster. See the demo below:

What’s next

The tool is new and there are certain limitations. To list a few:

  1. For now, all checks are hard coded. It calls out for some templating mechanism or simple generator of YAML files.
  2. It is not possible to add sophisticated logic into analyzers. For example, to link one analyzer to another. 
  3. It would be great to apply targeted filtering. For example, do not gather the information for all Custom Resources, but only one. It can be useful for targeted troubleshooting. 

We are going to work with the community and see how Percona can contribute to this tool, as it can help our users to standardize the troubleshooting of our products. Please let us know about your use cases or stories around debugging Percona products on Kubernetes in the comments.

The Percona Kubernetes Operators automate the creation, alteration, or deletion of members in your Percona Distribution for MySQL, MongoDB, or PostgreSQL environment.

Learn More About Percona Kubernetes Operators

Nov
02
2022
--

First PostgreSQL Conference Ever: What You’ve Missed at PGConf.EU 2022

PostgreSQL PGConf.EU 2022

I just became part of the PostgreSQL ecosystem and was really curious to get in touch with the community. What are the best ways to interact with a community? Mailing lists for sure, forums, chat channels, or if possible an in-person conference. So I checked the events site of PostgreSQL and found an event nearby — PGConf.EU in Berlin. After reaching out internally, I was told this is one of the biggest or even the biggest PostgreSQL conferences in the world. So it seems I’ve made a good decision.

I was worried and also a little scared about going to a conference after almost three years of not attending any conferences and not seeing too many people. It took me almost six hours including a delay of over two hours (thank you for traveling with Deutsche Bahn!) to make it to Berlin. I was really impressed and surprised when I finally made it to the Berlin Marriott Hotel. What a nice venue for a conference.

View inside the hotel from the 9th floor down to the first floor

The check-in already had a great surprise waiting for me. “Sorry, Mr. Wagner but we’re fully booked and instead of your regular room we can only give you an executive suite”. What a bummer, but I took it nevertheless. I went to the room and almost immediately fell asleep, excited for the next days to come.

Let the show begin

The first day of the conference started with way too much food due to a really great breakfast. After that, I was waiting for the first keynote and opening speaker to come. The room was really crowded and I was wondering what “sold-out” actually meant for a PostgreSQL conference. Magnus Hagander and Vik Fearing did a great job in starting, organizing, and hosting the event. They quickly unveiled that 604+ people had shown up for the conference — what an impressive number.

Picture showing the number 604 attendees

The welcome and opening took about 45 minutes and directly afterward we had the first conference talk of the week: “Efficient Graph Analysis with SQL/PGQ”. It was a super interesting but also a hard topic for me, as this presentation was super technical. I had a hard time following parts of the presentation as I completely missed terms or background information. I definitely plan to look at the slides again and get my feet wet with this topic.

During the first coffee break of the day, I was able to wander around and started to explore the conference venue/booths and I was also able to spot my hungry colleague Umair. Can you spot him in the picture below?

Picture of the lunch break area

The second half of the morning started for me with “Changelog vs Two-dimensional time – what works better?” from Hettie Dombrovskaya. It was a very inspiring presentation as you could feel the passion and deep knowledge of Hettie in every sentence. She basically tries to implement “Time travel”, as she said. She explained bitemporal functions and also the difference between effective time vs asserted time which finally lead to time regions — a small travel in time.

From there I went to “PostgreSQL at GitLab.com”. Given that I’m a huge fan of the communication and transparency policies of GitLab, I had very high expectations about this presentation and I wouldn’t be disappointed. Alexander Sosna and Jose Finotto presented the journey of GitLab using PostgreSQL and also some limitations they’ve hit and how to overcome them. It was interesting to see that a simple use-case split between CI and the rest of the data actually led to such dramatically improved statistics and usability.

Leaving the lunch break behind, I had the first chance to see Hans-Jürgen Schönig in action during his presentation about “Performance tips you have never seen before”. He’s the CEO of CYBERTEC and a very passionate person on- and off-stage. He really engaged with the crowd and was able to get a lot of people smiling whenever he gave some real-world examples while presenting his tips.

Jan Birk presented “Administering large scale PostgreSQL installations” and explained how the Danish Ministry of Education handles 400 clusters divided into 100 products. He really went down the route and explained how they use different tools and extensions such as pgbouncer, pgBadger, barman, patroni, and many more.

As Microsoft had a booth at the conference and was always talking about Citus, I was curious about what they do and joined “Distributed Postgres: How to build a multi-tenant SaaS app with Citus”. It was interesting to see how Charles Feddersen presented the capabilities of Citus on how to scale out as needed and also how resharding works along those lines. Starting small and scaling up whenever you need it is a very common use case nowadays. I’m happy that Citus was open-sourced and can now be developed also by the community. 

I basically stayed in the sponsor’s room, as I was also interested to learn what EDB has to offer when they talk about “EDB Postgres Distributed – Evolution of Replication and High Availability”. Stacy Scoggins presented the solution of EDB as an answer to HA for PostgreSQL. They’ve developed a very nice failover solution into PostgreSQL, which makes it almost transparent. The only cost factor that I’m seeing is the downside of vendor locking, as the code isn’t open. 

Social event

The first day closed up with a social event that took place at the “Alte Münze” — basically a former mint in Berlin_mitte on the banks of the Spree. The venue was nice and a lot of people made it to the event. Underlined by some music, hand food, and drinks, you were able to engage with all parts of the community and I was able to make new friends inside the PostgreSQL community. After a very long day, I finally made it back to the hotel after midnight, waiting for the next two days to come. 

The day after the social event

The second day of the conference started a bit slower. You could tell by looking into people’s faces and eyes during the breakfast and the first few sessions of the day, that for some the social event had taken a little longer — so basically it was an overall success.

Gregory Stark talked about “The Future of Observability in Postgres” and how he envisions standardizing those. He presented his ideas that are needed for the PostgreSQL core to make that happen. Having an open standard, that could easily be used by modern tools across the PostgreSQL ecosystem, sounds like a really great idea to me. 

Being new in the community, I had to join the talk from Melih Mutlu about “Takeaways from the First 6 Months of Hacking on Postgres”. He explained his journey and obstacles getting his first patch landed in the community. He also touched on the current development practices and tools the community is using – I personally chimed in here by asking the GitHub vs Mailing List question, which directly led to some controversial discussion. I’ll follow up on that topic with the community via the mailing list. My personal mantra here is “To improve is to change; to be perfect is to change often” by Winston Churchill.

Photo of the lunch and coffee break area

After the break and some coffee later it was time to learn more about “Neon, cloud-native storage backend for PostgreSQL” by Heikki Linnakangas. He explained the architecture and functionality of the serverless open source alternative to AWS Aurora for Postgres. It separates storage and compute and also distributes the data across a cluster of nodes. Coming with a strong storage background, also in distributed storage, this was a really interesting talk to me and I will definitely follow up with their ongoing work.

I mentioned in the first part that I’m not yet used to all the terms and functionality of PostgreSQL, so I took the chance to listen to Karen Jex on “Everything You Wanted to Know About Databases as a Developer but Were Too Afraid to Ask Your DBA”. This session was really helpful for everyone who is new to databases or PostgreSQL in general. Karen did a great job in going through the various important basic methodologies and functions. I personally took a lot out of this presentation and the slide deck might be helpful to more beginners out there.

Encryption is important and as I just saw a mail thread on TDE (Transparent Data Encryption) on the hackers mailing list, I was curious to learn what Stephen Frost had to say about “On the Road to TDE”. Stephen presented the various different ways to encrypt your data as well as what would work best for Postgres — spoiler: TDE. This is an ongoing effort for over three years already but the broad community is interested in encryption, so this definitely needs to get some hands working on it, to finally build it into Postgres. Let’s see if and how Percona can support this effort, as data encryption is nowadays super important to almost everyone.

Robert Haas presented “How PostgreSQL Resists Automated Management (And How to Fix This)”. He showed some ways external software would like to manage PostgreSQL but fails and what’s the causing factor for it. He also went into his future vision of how this scenario could be improved and implemented into PostgreSQL.

Keynotes

Now Berlin was united again (the room was split up into two meeting rooms during regular sessions) and we were able to listen to the keynotes from the main sponsors of the event. Hans-Jürgen Schönig, Stephen Frost, Marc Linster, and Chris Engelbert did a great job in representing their companies but also the community.

The presentation that stuck out the most to me was the one from Marc about “Accelerating Postgres’ success in the modern enterprise”. He prepared slides that underlay the overall success of PostgreSQL and where this is coming from. Also, the undeniable facts of StackOverflow, where Postgres finally reached the top of the mountain as being the most important database to developers.

The day concluded with Umair and I stepping out for dinner — ok just burgers in the mall next to the hotel — and spending some time getting to know each other better and discussing what we’ve seen and heard at the conference. We both agree that the conference was going great and that the community is highly engaged and further growing.

Also, some sightseeing couldn’t be missed overall before calling it a day.

Umair and Kai in front of the Berlin Gate

Time flies by — The last day of the conference

The last day of the conference started with Robert Haas again talking about “Moving pg_basebackup Forward”. You were able to feel his passion for this extension in every sentence. He showed us the newest features that made it into PostgreSQL 15 and, as such, the significant improvements to pg_basebackup. Two of which are the server-side compression and bucket targets. We also talked about parallel backup — and why this isn’t a good idea most of the time — and incremental backup. There is still a lot of work ahead but the agenda looks very promising.

You might have noticed in the second half that I was quite interested in the new Neon project, so I couldn’t resist joining “Why we build Neon” which was presented by the CEO Nikita Shamgunov. He very openly showed how they’ve built up the current company and what happened within the last year. Pitch decks to investors as well as the founding numbers were presented openly to the audience. Why does this all matter and why is serverless important? — Nikita was able to give good insights into this new startup. 

One of the gold nuggets of the conference has been this talk from Claire Giordano on “How to make your Postgres blog posts reach a ton more people”. She basically presented, in 45 minutes, a condensed version of her fourth training session in an interactive and interesting way to the audience. If there would be only one thing for you to remember from the talk, it’s “Empathy”, Claire said.

Slide showing empathy for your readers

If you want to know how to write great blog posts and reach more people, you have to take a look at her slides

During the social event, I was talking to Harald already about high availability, and I didn’t know he had a talk about that exact topic on Friday. So this talk was a must-have after our conversation on Tuesday. Harald talked about “reasonable availability” and why HA isn’t always a good idea and sometimes even leads to more downtime and outages compared to a simple single-node system. 

Before we reached the end of the conference, it was time for the lightning talks. 

Flip chart showing the list of lightning talks

I was only able to be present for the first half of the session, as I had to catch my train back home, but those lightning talks are usually known to be really good. Given the fact that people have to condense their message into five minutes, they leave out all the blur and filler words in between, so you directly learn about the important things people want to talk about.

Simon Riggs started by announcing that he has retired from actively contributing code to the community moving forward but he still wants to support the community wherever and whenever he can. He received very long applause and a standing ovation, and you could feel the thankfulness from the crowd for everything he’s done over the years for the project and the community.

I’m sad that I missed the talk from Umair about open source licenses but I learned it was a distilled version of these slides. So if you want to learn more about this topic, take a look at the slides or reach out directly to @pg_umair on Twitter. 

Conclusion

PGConf Europe in Berlin was a great conference, with the chance to learn a lot, meet new people, and make friends. If you ever consider joining a PostgreSQL conference, PGConf Europe should be put on the top of your list. I personally saw that the community is welcoming and open and everyone who is interested is encouraged to participate. As this was my first PostgreSQL conference — ever — this is just the beginning of my journey. I’ll be back…

May
16
2022
--

Running Rocket.Chat with Percona Server for MongoDB on Kubernetes

Running Rocket.Chat with Percona Server for MongoDB on Kubernetes

Our goal is to have a Rocket.Chat deployment which uses highly available Percona Server for MongoDB cluster as the backend database and it all runs on Kubernetes. To get there, we will do the following:

  • Start a Google Kubernetes Engine (GKE) cluster across multiple availability zones. It can be any other Kubernetes flavor or service, but I rely on multi-AZ capability in this blog post.
  • Deploy Percona Operator for MongoDB and database cluster with it
  • Deploy Rocket.Chat with specific affinity rules
    • Rocket.Chat will be exposed via a load balancer

Rocket.Chat will be exposed via a load balancer

Percona Operator for MongoDB, compared to other solutions, is not only the most feature-rich but also comes with various management capabilities for your MongoDB clusters – backups, scaling (including sharding), zero-downtime upgrades, and many more. There are no hidden costs and it is truly open source.

This blog post is a walkthrough of running a production-grade deployment of Rocket.Chat with Percona Operator for MongoDB.

Rock’n’Roll

All YAML manifests that I use in this blog post can be found in this repository.

Deploy Kubernetes Cluster

The following command deploys GKE cluster named

percona-rocket

in 3 availability zones:

gcloud container clusters create --zone us-central1-a --node-locations us-central1-a,us-central1-b,us-central1-c percona-rocket --cluster-version 1.21 --machine-type n1-standard-4 --preemptible --num-nodes=3

Read more about this in the documentation.

Deploy MongoDB

I’m going to use helm to deploy the Operator and the cluster.

Add helm repository:

helm repo add percona https://percona.github.io/percona-helm-charts/

Install the Operator into the percona namespace:

helm install psmdb-operator percona/psmdb-operator --create-namespace --namespace percona

Deploy the cluster of Percona Server for MongoDB nodes:

helm install my-db percona/psmdb-db -f psmdb-values.yaml -n percona

Replica set nodes are going to be distributed across availability zones. To get there, I altered the affinity keys in the corresponding sections of psmdb-values.yaml:

antiAffinityTopologyKey: "topology.kubernetes.io/zone"

Prepare MongoDB

For Rocket.Chat to connect to our database cluster, we need to create the users. By default, clusters provisioned with our Operator have

userAdmin

user, its password is set in

psmdb-values.yaml

:

MONGODB_USER_ADMIN_PASSWORD: userAdmin123456

For production-grade systems, do not forget to change this password or create dedicated secrets to provision those. Read more about user management in our documentation.

Spin up a client Pod to connect to the database:

kubectl run -i --rm --tty percona-client1 --image=percona/percona-server-mongodb:4.4.10-11 --restart=Never -- bash -il

Connect to the database with

userAdmin

:

[mongodb@percona-client1 /]$ mongo "mongodb://userAdmin:userAdmin123456@my-db-psmdb-db-rs0-0.percona/admin?replicaSet=rs0"

We are going to create the following:

  • rocketchat

    database

  • rocketChat

    user to store data and connect to the database

  • oplogger

    user to provide access to oplog for rocket chat

    • Rocket.Chat uses Meteor Oplog tailing to improve performance. It is optional.
use rocketchat
db.createUser({
  user: "rocketChat",
  pwd: passwordPrompt(),
  roles: [
    { role: "readWrite", db: "rocketchat" }
  ]
})

use admin
db.createUser({
  user: "oplogger",
  pwd: passwordPrompt(),
  roles: [
    { role: "read", db: "local" }
  ]
})

Deploy Rocket.Chat

I will use helm here to maintain the same approach. 

helm install -f rocket-values.yaml my-rocketchat rocketchat/rocketchat --version 3.0.0

You can find rocket-values.yaml in the same repository. Please make sure you set the correct passwords in the corresponding YAML fields.

As you can see, I also do the following:

  • Line 11: expose Rocket.Chat through
    LoadBalancer

    service type

  • Line 13-14: set number of replicas of Rocket.Chat Pods. We want three – one per each availability zone.
  • Line 16-23: set affinity to distribute Pods across availability zones

Load Balancer will be created with a public IP address:

$ kubectl get service my-rocketchat-rocketchat
NAME                       TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
my-rocketchat-rocketchat   LoadBalancer   10.32.17.26   34.68.238.56   80:32548/TCP   12m

You should now be able to connect to

34.68.238.56

and enjoy your highly available Rocket.Chat installation.

Rocket.Chat installation

Clean Up

Uninstall all helm charts to remove MongoDB cluster, the Operator, and Rocket.Chat:

helm uninstall my-rocketchat
helm uninstall my-db -n percona
helm uninstall psmdb-operator -n percona

Things to Consider

Ingress

Instead of exposing Rocket.Chat through a load balancer, you may also try ingress. By doing so, you can integrate it with cert-manager and have a valid TLS certificate for your chat server.

Mongos

It is also possible to run a sharded MongoDB cluster with Percona Operator. If you do so, Rocket.Chat will connect to mongos Service, instead of the replica set nodes. But you will still need to connect to the replica set directly to get oplogs.

Conclusion

We encourage you to try out Percona Operator for MongoDB with Rocket.Chat and let us know on our community forum your results.

There is always room for improvement and a time to find a better way. Please let us know if you face any issues with contributing your ideas to Percona products. You can do that on the Community Forum or JIRA. Read more about contribution guidelines for Percona Operator for MongoDB in CONTRIBUTING.md.

Percona Operator for MongoDB contains everything you need to quickly and consistently deploy and scale Percona Server for MongoDB instances into a Kubernetes cluster on-premises or in the cloud. The Operator enables you to improve time to market with the ability to quickly deploy standardized and repeatable database environments. Deploy your database with a consistent and idempotent result no matter where they are used.

Nov
16
2021
--

Percona Sponsors Conferences and Supports Community

Percona Sponsors Conferences

We love supporting communities and open source. One of the things we are thrilled about at Percona is hybrid and in-person conferences coming back this fall. Local events are a perfect opportunity to meet and reinforce you, community members, and open source lovers, talk about databases and remote work, and maybe even hug!

Especially for this conference season, Matt Yonkovit built an arcade controller – a gadget to manage database workloads in real-time and play with them in Percona Monitoring and Management. Have you ever thought about managing your database workload with the help of selectors? Now all visitors of our booth can try to literally manipulate the database with their hands! That was really a hit. It consists of several buttons and switches which allow you to run a vacuum, stop all the workload, and do lots of other stuff. If you want to see it in action, watch the video of the live-streamed PMM meetup from OSS (Open Source Summit) in Seattle with the demo. And for those who decide to replicate it or build their own version, Matt shared the code on his GitHub.

Percona Sponsors Conferences

Traditionally, we sponsor the OSDN Conf Kyiv in September where Percona staff members from Ukraine actively participate and organize. It is a one-day, non-commercial, volunteer-organized event centered on free and open source software. Richard Stallman was their special guest this year with the talk “Free Software and the GNU General Public License.”

Also, it was really exciting for our team to meet again in person at All Things Open in Raleigh and celebrate Percona’s 15 years with everyone! Approximately 1000 people were on-site and 2500 people were online. All of them were able to get to know our experts and play with Matt’s PMM arcade controller. Also, Percona speakers gave four talks with good attendance.

One of the virtual events we were happy to support was Nerdearla 2021, the most important IT conference in the Latin American region. Though it was lots of fun virtually, we hope that in the future we will be able to meet in person again!

So, what is up next? At the end of November, we are flying to Las Vegas to the breathtaking AWS re:Invent. It promises to become a notable event of the year celebrating 10 years of the conference. If you are going to be there, stop by the Percona booth and see us!

Right after re:Invent, our team will be in New York at the non–profit, community-run PGConf NYC. We will be Platinum sponsors this time, and Percona Postgres experts will give five talks on Postgres topics. Preparations for this event are already in full swing!

We are thinking of the next year, too. In January, we will welcome you to the booth in Silicon Valley at the Postgres Conference. Come and enjoy this event with us. We are looking forward to it!

Want us to speak at your event? Organize a conference or a local meetup in any part of the globe and need speakers? Contact us at community-team@percona.com and we will work with you on the support we can provide.

And we are hiring!  The Community Team is actively looking for the Technical Evangelists to join our team and work with open source communities all over the world. Watch a short video to get an idea of the work of the team and how we encourage open source growth.

Percona Sponsors Conferences and Supports Community

Sep
01
2021
--

Congratulating Marcelo Altmann on his Promotion to Oracle ACE!

Marcelo Altmann promotion to Oracle ACE

Marcelo Altmann promotion to Oracle ACEWe’re excited to share that Marcelo Altmann from the Percona Server Engineering Team has just been promoted from Oracle ACE Associate to Oracle ACE.

Congratulations!

The Oracle ACE Program recognizes and rewards community members for their technical contributions to the Oracle community.

Marcelo initially joined Percona as a senior support engineer in our global services organization, where he helped customers with running their MySQL-based environments. In early 2020, he joined our Server Engineering team and has been actively involved in the development of Percona Server for MySQL and Percona XtraBackup since then.

Marcelo’s contributions to the MySQL ecosystem are countless – he’s an active blogger, he regularly submits bug reports to the MySQL team, organizes local and virtual meetups, and also contributes patches and bug fixes.

Congratulations again, Marcelo, we’re proud to have you on our team!

Feb
09
2021
--

MongoDB Sharding & Replication 101 – The Community’s Questions Answered!

MongoDB 101 Sharding and Replication

MongoDB 101 Sharding and ReplicationA few weeks ago I sat down and talked about tuning MongoDB memory settings with Mike Grayson, one of our MongoDB Database Engineers here at Percona. If you have not seen the video I would recommend it.  Since then I have heard from people in the community about other topics they would like to see.  I was able to do a written interview with Mike today (with some great extra details from Vinodh),  asking them to provide us a MongoDB Replication and Sharding 101 overview based on the questions I heard from the community.  Thank you, Mike and Vinodh, for writing up answers to our questions!  Also, stay on the lookout for a video chat I did with our MongoDB Software product owner Akira, asking him to explain sharding and replication at a high level.

In MongoDB what is the difference between sharding and replication?

Mike Grayson: Sharding is the act of partitioning your collections so that parts of your data are dispersed among multiple servers called shards.  Replication, or Replica Sets in MongoDB parlance, is how MongoDB achieves high availability, Replica Sets are a Primary, and 0 to n amount of secondaries which have read-only copies of the data and possibly an arbiter, which is a non-data bearing vote only member.  Upon Primary failures, one of the available secondaries will be elected as the new Primary.

Vinodh Krishnaswamy added: Don’t forget sharded clusters have three components – shards (where original data resides), config servers (holds information about data like which data resides which shard), and mongos (a router – the application connects to the cluster via mongos).  For high availability, you would need to implement both the shards and the config servers as replica sets. 

When should you start to look at MongoDB sharding?

Mike Grayson:  Typically you should look at sharding when either your working set has outpaced the available resources on your machine  (think vCPU or RAM) and scaling up either isn’t possible or isn’t feasible, perhaps due to cost OR when your data set is growing too large (Some people say that you should start looking at around 1-2TB, but this maybe too large and you should start thinking about sharding strategies in the 200GB-500GB range).

Vinodh Krishnaswamy’s tip:  Think early on the growth and potential size of your databases and plan out your sharding strategy.  

Can you shard MongoDB without replicating?  When would you do this?

Mike Grayson:  Can you? Yes. Should you?  No.  Sharding without replicating doesn’t enable the oplog which means backups aren’t possible for your sharded cluster.  Why might someone do this?   Think log data that’s easily reproduced or other data that’s not that important such as transient data.   Generally speaking, those reasons are few and far between, and setting up a replica set is still a best practice in those situations.

Are there downsides to sharding MongoDB too early?

Mike Grayson:  Cost, the amount of infrastructure and resources needed to support a bare-bones production level sharded cluster is at least twice that of a 3 node replica set.   However, if you know your data set is going to grow, spinning up a sharded cluster of 1 shard puts you in a great spot to scale out quickly and easily.

Vinodh Krishnaswamy added: One another drawback of sharding too late however is the extra potential load when balancing.  Migrating the data chunks from old shards to the new ones is an expensive operation.  When you have TBs of data, a lot of chunks will need to be migrated and might affect the performance and stability of your regular workload.  To mitigate this you will need to set a window to balance the shards and to allow chunk migrations only to occur at a certain period of the day!

Are there a minimum number of MongoDB nodes you need to shard?

Mike Grayson:    2 is the absolute minimum, 1 node with a mongos (query router used to route queries to the correct shard)/config server and another node for the shard.   This would be able to support a 1 shard cluster.  For production quality, you’d want a minimum of 7 nodes, 1 mongos on its own, 3 config servers and a 3 shard replica set nodes for a 1 shard cluster.

If you have a small 3 node MongoDB setup can you have both sharding and replication?

Mike Grayson:  Yes, if it’s not expected to support production load you could have 3 nodes that support a sharded cluster that utilizes replica sets.   They would be replica sets of 1 member though OR you could stack multiple mongod nodes on a single host, which is generally not a good idea in Production, but possible in test/dev type environments

Vinodh Krishnaswamy added:  I did a webinar a little over a year ago where I showed and walked through how to design your sharding topology.  If people would like a deeper dive I would encourage readers to check it out:  

Recorded Webinar: MongoDB Sharded Cluster & HA – How to Design your Topology

When happens if you get the MongoDB shard key wrong?

Mike Grayson: Getting the shard key wrong can lead to unbalanced data across shards or “hot” shards where all new data goes to the same shard.  In MongoDB 4.4, they’ve added refinable shard keys which makes this slightly less painful than previous as you can add (but not remove) fields from your shard key, but choosing your shard key should still be something you plan and test extensively before moving to production.

What is the most common issue people run into when sharding MongoDB?

Mike Grayson: Poor Shard Keys leading to data imbalance, whether it be through jumbo chunks (a chunk is a piece of your data that MongoDB can move around from shard to shard, jumbo chunks are too big to move and get stuck on a shard without intervention), hot shards where new data is all being written to one shard and the other shards struggle to keep up.

Vinodh Krishnaswamy: Not using the shard key fields in their query which will lead to broadcasting the requests to all shard for the data and then merging the data together to provide the result.  This is a much more expensive operation than when you query with the shard key.   With queries including shard key fields, the requests are redirected to only the shards where the requested data resides

Thank you, Mike and Vinodh for taking the time to write up answers to these questions!  

Jan
26
2019
--

Has the fight over privacy changed at all in 2019?

Few issues divide the tech community quite like privacy. Much of Silicon Valley’s wealth has been built on data-driven advertising platforms, and yet, there remain constant concerns about the invasiveness of those platforms.

Such concerns have intensified in just the last few weeks as France’s privacy regulator placed a record fine on Google under Europe’s General Data Protection Regulation (GDPR) rules which the company now plans to appeal. Yet with global platform usage and service sales continuing to tick up, we asked a panel of eight privacy experts: “Has anything fundamentally changed around privacy in tech in 2019? What is the state of privacy and has the outlook changed?” 

This week’s participants include:

TechCrunch is experimenting with new content forms. Consider this a recurring venue for debate, where leading experts – with a diverse range of vantage points and opinions – provide us with thoughts on some of the biggest issues currently in tech, startups and venture. If you have any feedback, please reach out: Arman.Tabatabai@techcrunch.com.


Thoughts & Responses:


Albert Gidari

Albert Gidari is the Consulting Director of Privacy at the Stanford Center for Internet and Society. He was a partner for over 20 years at Perkins Coie LLP, achieving a top-ranking in privacy law by Chambers, before retiring to consult with CIS on its privacy program. He negotiated the first-ever “privacy by design” consent decree with the Federal Trade Commission. A recognized expert on electronic surveillance law, he brought the first public lawsuit before the Foreign Intelligence Surveillance Court, seeking the right of providers to disclose the volume of national security demands received and the number of affected user accounts, ultimately resulting in greater public disclosure of such requests.

There is no doubt that the privacy environment changed in 2018 with the passage of California’s Consumer Privacy Act (CCPA), implementation of the European Union’s General Data Protection Regulation (GDPR), and new privacy laws enacted around the globe.

“While privacy regulation seeks to make tech companies betters stewards of the data they collect and their practices more transparent, in the end, it is a deception to think that users will have more “privacy.””

For one thing, large tech companies have grown huge privacy compliance organizations to meet their new regulatory obligations. For another, the major platforms now are lobbying for passage of a federal privacy law in the U.S. This is not surprising after a year of privacy miscues, breaches and negative privacy news. But does all of this mean a fundamental change is in store for privacy? I think not.

The fundamental model sustaining the Internet is based upon the exchange of user data for free service. As long as advertising dollars drive the growth of the Internet, regulation simply will tinker around the edges, setting sideboards to dictate the terms of the exchange. The tech companies may be more accountable for how they handle data and to whom they disclose it, but the fact is that data will continue to be collected from all manner of people, places and things.

Indeed, if the past year has shown anything it is that two rules are fundamental: (1) everything that can be connected to the Internet will be connected; and (2) everything that can be collected, will be collected, analyzed, used and monetized. It is inexorable.

While privacy regulation seeks to make tech companies betters stewards of the data they collect and their practices more transparent, in the end, it is a deception to think that users will have more “privacy.” No one even knows what “more privacy” means. If it means that users will have more control over the data they share, that is laudable but not achievable in a world where people have no idea how many times or with whom they have shared their information already. Can you name all the places over your lifetime where you provided your SSN and other identifying information? And given that the largest data collector (and likely least secure) is government, what does control really mean?

All this is not to say that privacy regulation is futile. But it is to recognize that nothing proposed today will result in a fundamental shift in privacy policy or provide a panacea of consumer protection. Better privacy hygiene and more accountability on the part of tech companies is a good thing, but it doesn’t solve the privacy paradox that those same users who want more privacy broadly share their information with others who are less trustworthy on social media (ask Jeff Bezos), or that the government hoovers up data at rate that makes tech companies look like pikers (visit a smart city near you).

Many years ago, I used to practice environmental law. I watched companies strive to comply with new laws intended to control pollution by creating compliance infrastructures and teams aimed at preventing, detecting and deterring violations. Today, I see the same thing at the large tech companies – hundreds of employees have been hired to do “privacy” compliance. The language is the same too: cradle to grave privacy documentation of data flows for a product or service; audits and assessments of privacy practices; data mapping; sustainable privacy practices. In short, privacy has become corporatized and industrialized.

True, we have cleaner air and cleaner water as a result of environmental law, but we also have made it lawful and built businesses around acceptable levels of pollution. Companies still lawfully dump arsenic in the water and belch volatile organic compounds in the air. And we still get environmental catastrophes. So don’t expect today’s “Clean Privacy Law” to eliminate data breaches or profiling or abuses.

The privacy world is complicated and few people truly understand the number and variety of companies involved in data collection and processing, and none of them are in Congress. The power to fundamentally change the privacy equation is in the hands of the people who use the technology (or choose not to) and in the hands of those who design it, and maybe that’s where it should be.


Gabriel Weinberg

Gabriel Weinberg is the Founder and CEO of privacy-focused search engine DuckDuckGo.

Coming into 2019, interest in privacy solutions is truly mainstream. There are signs of this everywhere (media, politics, books, etc.) and also in DuckDuckGo’s growth, which has never been faster. With solid majorities now seeking out private alternatives and other ways to be tracked less online, we expect governments to continue to step up their regulatory scrutiny and for privacy companies like DuckDuckGo to continue to help more people take back their privacy.

“Consumers don’t necessarily feel they have anything to hide – but they just don’t want corporations to profit off their personal information, or be manipulated, or unfairly treated through misuse of that information.”

We’re also seeing companies take action beyond mere regulatory compliance, reflecting this new majority will of the people and its tangible effect on the market. Just this month we’ve seen Apple’s Tim Cook call for stronger privacy regulation and the New York Times report strong ad revenue in Europe after stopping the use of ad exchanges and behavioral targeting.

At its core, this groundswell is driven by the negative effects that stem from the surveillance business model. The percentage of people who have noticed ads following them around the Internet, or who have had their data exposed in a breach, or who have had a family member or friend experience some kind of credit card fraud or identity theft issue, reached a boiling point in 2018. On top of that, people learned of the extent to which the big platforms like Google and Facebook that collect the most data are used to propagate misinformation, discrimination, and polarization. Consumers don’t necessarily feel they have anything to hide – but they just don’t want corporations to profit off their personal information, or be manipulated, or unfairly treated through misuse of that information. Fortunately, there are alternatives to the surveillance business model and more companies are setting a new standard of trust online by showcasing alternative models.


Melika Carroll

Melika Carroll is Senior Vice President, Global Government Affairs at Internet Association, which represents over 45 of the world’s leading internet companies, including Google, Facebook, Amazon, Twitter, Uber, Airbnb and others.

We support a modern, national privacy law that provides people meaningful control over the data they provide to companies so they can make the most informed choices about how that data is used, seen, and shared.

“Any national privacy framework should provide the same protections for people’s data across industries, regardless of whether it is gathered offline or online.”

Internet companies believe all Americans should have the ability to access, correct, delete, and download the data they provide to companies.

Americans will benefit most from a federal approach to privacy – as opposed to a patchwork of state laws – that protects their privacy regardless of where they live. If someone in New York is video chatting with their grandmother in Florida, they should both benefit from the same privacy protections.

It’s also important to consider that all companies – both online and offline – use and collect data. Any national privacy framework should provide the same protections for people’s data across industries, regardless of whether it is gathered offline or online.

Two other important pieces of any federal privacy law include user expectations and the context in which data is shared with third parties. Expectations may vary based on a person’s relationship with a company, the service they expect to receive, and the sensitivity of the data they’re sharing. For example, you expect a car rental company to be able to track the location of the rented vehicle that doesn’t get returned. You don’t expect the car rental company to track your real-time location and sell that data to the highest bidder. Additionally, the same piece of data can have different sensitivities depending on the context in which it’s used or shared. For example, your name on a business card may not be as sensitive as your name on the sign in sheet at an addiction support group meeting.

This is a unique time in Washington as there is bipartisan support in both chambers of Congress as well as in the administration for a federal privacy law. Our industry is committed to working with policymakers and other stakeholders to find an American approach to privacy that protects individuals’ privacy and allows companies to innovate and develop products people love.


Johnny Ryan

Dr. Johnny Ryan FRHistS is Chief Policy & Industry Relations Officer at Brave. His previous roles include Head of Ecosystem at PageFair, and Chief Innovation Officer of The Irish Times. He has a PhD from the University of Cambridge, and is a Fellow of the Royal Historical Society.

Tech companies will probably have to adapt to two privacy trends.

“As lawmakers and regulators in Europe and in the United States start to think of “purpose specification” as a tool for anti-trust enforcement, tech giants should beware.”

First, the GDPR is emerging as a de facto international standard.

In the coming years, the application of GDPR-like laws for commercial use of consumers’ personal data in the EU, Britain (post-EU), Japan, India, Brazil, South Korea, Malaysia, Argentina, and China will bring more than half of global GDP under a similar standard.

Whether this emerging standard helps or harms United States firms will be determined by whether the United States enacts and actively enforces robust federal privacy laws. Unless there is a federal GDPR-like law in the United States, there may be a degree of friction and the potential of isolation for United States companies.

However, there is an opportunity in this trend. The United States can assume the global lead by doing two things. First, enact a federal law that borrows from the GDPR, including a comprehensive definition of “personal data”, and robust “purpose specification”. Second, invest in world-leading regulation that pursues test cases, and defines practical standards. Cutting edge enforcement of common principles-based standards is de facto leadership.

Second, privacy and antitrust law are moving closer to each other, and might squeeze big tech companies very tightly indeed.

Big tech companies “cross-use” user data from one part of their business to prop up others. The result is that a company can leverage all the personal information accumulated from its users in one line of business, and for one purpose, to dominate other lines of business too.

This is likely to have anti-competitive effects. Rather than competing on the merits, the company can enjoy the unfair advantage of massive network effects even though it may be starting from scratch in a new line of business. This stifles competition and hurts innovation and consumer choice.

Antitrust authorities in other jurisdictions have addressed this. In 2015, the Belgian National Lottery was fined for re-using personal information acquired through its monopoly for a different, and incompatible, line of business.

As lawmakers and regulators in Europe and in the United States start to think of “purpose specification” as a tool for anti-trust enforcement, tech giants should beware.


John Miller

John Miller is the VP for Global Policy and Law at the Information Technology Industry Council (ITI), a D.C. based advocate group for the high tech sector.  Miller leads ITI’s work on cybersecurity, privacy, surveillance, and other technology and digital policy issues.

Data has long been the lifeblood of innovation. And protecting that data remains a priority for individuals, companies and governments alike. However, as times change and innovation progresses at a rapid rate, it’s clear the laws protecting consumers’ data and privacy must evolve as well.

“Data has long been the lifeblood of innovation. And protecting that data remains a priority for individuals, companies and governments alike.”

As the global regulatory landscape shifts, there is now widespread agreement among business, government, and consumers that we must modernize our privacy laws, and create an approach to protecting consumer privacy that works in today’s data-driven reality, while still delivering the innovations consumers and businesses demand.

More and more, lawmakers and stakeholders acknowledge that an effective privacy regime provides meaningful privacy protections for consumers regardless of where they live. Approaches, like the framework ITI released last fall, must offer an interoperable solution that can serve as a model for governments worldwide, providing an alternative to a patchwork of laws that could create confusion and uncertainty over what protections individuals have.

Companies are also increasingly aware of the critical role they play in protecting privacy. Looking ahead, the tech industry will continue to develop mechanisms to hold us accountable, including recommendations that any privacy law mandate companies identify, monitor, and document uses of known personal data, while ensuring the existence of meaningful enforcement mechanisms.


Nuala O’Connor

Nuala O’Connor is president and CEO of the Center for Democracy & Technology, a global nonprofit committed to the advancement of digital human rights and civil liberties, including privacy, freedom of expression, and human agency. O’Connor has served in a number of presidentially appointed positions, including as the first statutorily mandated chief privacy officer in U.S. federal government when she served at the U.S. Department of Homeland Security. O’Connor has held senior corporate leadership positions on privacy, data, and customer trust at Amazon, General Electric, and DoubleClick. She has practiced at several global law firms including Sidley Austin and Venable. She is an advocate for the use of data and internet-enabled technologies to improve equity and amplify marginalized voices.

For too long, Americans’ digital privacy has varied widely, depending on the technologies and services we use, the companies that provide those services, and our capacity to navigate confusing notices and settings.

“Americans deserve comprehensive protections for personal information – protections that can’t be signed, or check-boxed, away.”

We are burdened with trying to make informed choices that align with our personal privacy preferences on hundreds of devices and thousands of apps, and reading and parsing as many different policies and settings. No individual has the time nor capacity to manage their privacy in this way, nor is it a good use of time in our increasingly busy lives. These notices and choices and checkboxes have become privacy theater, but not privacy reality.

In 2019, the legal landscape for data privacy is changing, and so is the public perception of how companies handle data. As more information comes to light about the effects of companies’ data practices and myriad stewardship missteps, Americans are surprised and shocked about what they’re learning. They’re increasingly paying attention, and questioning why they are still overburdened and unprotected. And with intensifying scrutiny by the media, as well as state and local lawmakers, companies are recognizing the need for a clear and nationally consistent set of rules.

Personal privacy is the cornerstone of the digital future people want. Americans deserve comprehensive protections for personal information – protections that can’t be signed, or check-boxed, away. The Center for Democracy & Technology wants to help craft those legal principles to solidify Americans’ digital privacy rights for the first time.


Chris Baker

Chris Baker is Senior Vice President and General Manager of EMEA at Box.

Last year saw data privacy hit the headlines as businesses and consumers alike were forced to navigate the implementation of GDPR. But it’s far from over.

“…customers will have trust in a business when they are given more control over how their data is used and processed”

2019 will be the year that the rest of the world catches up to the legislative example set by Europe, as similar data regulations come to the forefront. Organizations must ensure they are compliant with regional data privacy regulations, and more GDPR-like policies will start to have an impact. This can present a headache when it comes to data management, especially if you’re operating internationally. However, customers will have trust in a business when they are given more control over how their data is used and processed, and customers can rest assured knowing that no matter where they are in the world, businesses must meet the highest bar possible when it comes to data security.

Starting with the U.S., 2019 will see larger corporations opt-in to GDPR to support global business practices. At the same time, local data regulators will lift large sections of the EU legislative framework and implement these rules in their own countries. 2018 was the year of GDPR in Europe, and 2019 be the year of GDPR globally.


Christopher Wolf

Christopher Wolf is the Founder and Chair of the Future of Privacy Forum think tank, and is senior counsel at Hogan Lovells focusing on internet law, privacy and data protection policy.

With the EU GDPR in effect since last May (setting a standard other nations are emulating),

“Regardless of the outcome of the debate over a new federal privacy law, the issue of the privacy and protection of personal data is unlikely to recede.”

with the adoption of a highly-regulatory and broadly-applicable state privacy law in California last Summer (and similar laws adopted or proposed in other states), and with intense focus on the data collection and sharing practices of large tech companies, the time may have come where Congress will adopt a comprehensive federal privacy law. Complicating the adoption of a federal law will be the issue of preemption of state laws and what to do with the highly-developed sectoral laws like HIPPA and Gramm-Leach-Bliley. Also to be determined is the expansion of FTC regulatory powers. Regardless of the outcome of the debate over a new federal privacy law, the issue of the privacy and protection of personal data is unlikely to recede.

Dec
15
2018
--

The limits of coworking

It feels like there’s a WeWork on every street nowadays. Take a walk through midtown Manhattan (please don’t actually) and it might even seem like there are more WeWorks than office buildings.

Consider this an ongoing discussion about Urban Tech, its intersection with regulation, issues of public service, and other complexities that people have full PHDs on. I’m just a bitter, born-and-bred New Yorker trying to figure out why I’ve been stuck in between subway stops for the last 15 minutes, so please reach out with your take on any of these thoughts: @Arman.Tabatabai@techcrunch.com.

Co-working has permeated cities around the world at an astronomical rate. The rise has been so remarkable that even the headline-dominating SoftBank seems willing to bet the success of its colossal Vision Fund on the shift continuing, having poured billions into WeWork – including a recent $4.4 billion top-up that saw the co-working king’s valuation spike to $45 billion.

And there are no signs of the trend slowing down. With growing frequency, new startups are popping up across cities looking to turn under-utilized brick-and-mortar or commercial space into low-cost co-working options.

It’s a strategy spreading through every type of business from retail – where companies like Workbar have helped retailers offer up portions of their stores – to more niche verticals like parking lots – where companies like Campsyte are transforming empty lots into spaces for outdoor co-working and corporate off-sites. Restaurants and bars might even prove most popular for co-working, with startups like Spacious and KettleSpace turning restaurants that are closed during the day into private co-working space during their off-hours.

Before you know it, a startup will be strapping an Aeron chair to the top of a telephone pole and calling it “WirelessWorking”.

But is there a limit to how far co-working can go? Are all of the storefronts, restaurants and open spaces that line city streets going to be filled with MacBooks, cappuccinos and Moleskine notebooks? That might be too tall a task, even for the movement taking over skyscrapers.

The co-working of everything

Photo: Vasyl Dolmatov / iStock via Getty Images

So why is everyone trying to turn your favorite neighborhood dinner spot into a part-time WeWork in the first place? Co-working offers a particularly compelling use case for under-utilized space.

First, co-working falls under the same general commercial zoning categories as most independent businesses and very little additional infrastructure – outside of a few extra power outlets and some decent WiFi – is required to turn a space into an effective replacement for the often crowded and distracting coffee shops used by price-sensitive, lean, remote, or nomadic workers that make up a growing portion of the workforce.

Thus, businesses can list their space at little-to-no cost, without having to deal with structural layout changes that are more likely to arise when dealing with pop-up solutions or event rentals.

On the supply side, these co-working networks don’t have to purchase leases or make capital improvements to convert each space, and so they’re able to offer more square footage per member at a much lower rate than traditional co-working spaces. Spacious, for example, charges a monthly membership fee of $99-$129 dollars for access to its network of vetted restaurants, which is cheap compared to a WeWork desk, which can cost anywhere from $300-$800 per month in New York City.

Customers realize more affordable co-working alternatives, while tight-margin businesses facing increasing rents for under-utilized property are able to pool resources into a network and access a completely new revenue stream at very little cost. The value proposition is proving to be seriously convincing in initial cities – Spacious told the New York Times, that so many restaurants were applying to join the network on their own volition that only five percent of total applicants were ultimately getting accepted.

Basically, the business model here checks a lot of the boxes for successful marketplaces: Acquisition and transaction friction is low for both customers and suppliers, with both seeing real value that didn’t exist previously. Unit economics seem strong, and vetting on both sides of the market creates trust and community. Finally, there’s an observable network effect whereby suppliers benefit from higher occupancy as more customers join the network, while customers benefit from added flexibility as more locations join the network.

… Or just the co-working of some things

Photo: Caiaimage / Robert Daly via Getty Images

So is this the way of the future? The strategy is really compelling, with a creative solution that offers tremendous value to businesses and workers in major cities. But concerns around the scalability of demand make it difficult to picture this phenomenon becoming ubiquitous across cities or something that reaches the scale of a WeWork or large conventional co-working player.

All these companies seem to be competing for a similar demographic, not only with one another, but also with coffee shops, free workspaces, and other flexible co-working options like Croissant, which provides members with access to unused desks and offices in traditional co-working spaces. Like Spacious and KettleSpace, the spaces on Croissant own the property leases and are already built for co-working, so Croissant can still offer comparatively attractive rates.

The offer seems most compelling for someone that is able to work without a stable location and without the amenities offered in traditional co-working or office spaces, and is also price sensitive enough where they would trade those benefits for a lower price. Yet at the same time, they can’t be too price sensitive, where they would prefer working out of free – or close to free – coffee shops instead of paying a monthly membership fee to avoid the frictions that can come with them.

And it seems unclear whether the problem or solution is as poignant outside of high-density cities – let alone outside of high-density areas of high-density cities.

Without density, is the competition for space or traffic in coffee shops and free workspaces still high enough where it’s worth paying a membership fee for? Would the desire for a private working environment, or for a working community, be enough to incentivize membership alone? And in less-dense and more-sprawl oriented cities, members could also face the risk of having to travel significant distances if space isn’t available in nearby locations.

While the emerging workforce is trending towards more remote, agile and nomadic workers that can do more with less, it’s less certain how many will actually fit the profile that opts out of both more costly but stable traditional workspaces, as well as potentially frustrating but free alternatives. And if the lack of density does prove to be an issue, how many of those workers will live in hyper-dense areas, especially if they are price-sensitive and can work and live anywhere?

To be clear, I’m not saying the companies won’t see significant growth – in fact, I think they will. But will the trend of monetizing unused space through co-working come to permeate cities everywhere and do so with meaningful occupancy? Maybe not. That said, there is still a sizable and growing demographic that need these solutions and the value proposition is significant in many major urban areas.

The companies are creating real value, creating more efficient use of wasted space, and fixing a supply-demand issue. And the cultural value of even modestly helping independent businesses keep the lights on seems to outweigh the cultural “damage” some may fear in turning them into part-time co-working spaces.

And lastly, some reading while in transit:

Nov
20
2018
--

Our 3 favorite startups from Morgan Stanley’s 2nd Multicultural Innovation Lab Demo Day 

The Morgan Stanley Multicultural Innovation LabMorgan Stanley’s in-house accelerator focused on companies founded by multicultural and female entrepreneurs, hosted its second Annual Showcase and Demo Day. The event also featured companies from accelerators HearstLab, Newark Venture Partner Labs and PS27 Ventures. (Note: I was formerly employed by Morgan Stanley and have no financial ties.)

The showcase represented the culmination of the program’s second year, which followed an initial five-company class that has already seen two acquisitions. Through the six-month program, Morgan Stanley provides early-stage companies with a wide range of benefits, including an equity investment from Morgan Stanley, office space at Morgan Stanley headquarters, access to Morgan Stanley’s extensive network and others. Applications are now open for its third cohort of companies, with the application window closing on January 4th, 2019.

The 16 presenting startups, all led by a female or multicultural founder, offered solutions to structural inefficiencies across a wide array of categories, including fintech, developer tools and health. Though all of the companies offered impressive presentations and strong value propositions, here are three of the companies that stood out to us.

Hatch Apps

In hopes of democratizing software and app development, Hatch Apps provides a platform that allows users and companies to build iOS, Android and web applications without any code through pre-built templates and custom plug-and-play functions. In essence, Hatch Apps provides a solution for application building similar to what Squarespace or Wix provide for websites.

In the modern economy, every company is in one way or another a tech or tech-enabled company. Now the demand for strong engineers has made the fight for talent increasingly competitive and has made engineering quite costly, even when only needed for simple tasks. 

For an implementation and subscription fee, Hatch Apps allows companies with less sophisticated engineering DNA to reduce entering costs by launching native apps on their own, across platforms and often on faster timelines than those seen through third-party developers. Once an app is launched, Hatch Apps provides customers with detailed analytics and allows them to send targeted push notifications, export data and make in-app changes that can automatically go live in app stores.

The company initially took a bootstrapping approach to financing and raised funds by selling a 2016 election-themed “Cards Against Humanity”-style game created on the platform. Since then, Hatch Apps has already received funding from the Y Combinator Fellowship, Morgan Stanley and a number of other investors.

FreeWill

While estate planning is a topic many don’t like to think about, it’s a critical issue for managing cross-generational wealth. But will drafting can often be very complex, time-consuming and costly, requiring hours of legal consultation and coordination between various parties.

Founded by two former classmates at Stanford Business School, FreeWill looks to simplify the estate-planning process by providing a free online platform that automates will drafting, in a similar function to what TurboTax does for taxes. Using FreeWill, users can quickly set allocations for their estate and select personal recipients, charitable donations, executor specifications and other ancillary requests. The platform then creates a finalized legal document that is legally valid in all 50 states, to which users can also quickly make changes and replace without incurring expensive legal costs.

FreeWill is able to provide the platform to consumers for free due to the proceeds it receives from its nonprofit customers, who pay to be featured on the platform as a partner organization. FreeWill offers a compelling value proposition for partnering companies. By acting as a channel to funnel user donations to listed organizations, FreeWill has been able to drive a 600 percent increase in charitable giving to partner organizations on average. FreeWill also provides partner organizations with backing analytics that allow nonprofits to track bequests and donors through monthly reports. 

FreeWill currently boasts an impressive roster of 75 paying nonprofit partners that include American Red Cross, Amnesty International and many others. In the long-run it hopes to be the go-to solution for financial and legal end-of-life planning for investment advisors, life insurance and employee benefits providers.

Shoobs

Shoobs is looking to be the go-to platform for local “urban” events, which the company defined as events centered on local nightlife, comedy and concerts in the hip-hop, R&B and reggae genres to name a few. But unlike the genre-agnostic, transaction-focused event management platforms that can make the space seem pretty crowded, Shoobs focused on providing genre-specific even discovery. Shoobs matches urban event goers with artists of their choice and related smaller-scale events that can be harder to discover, acting as a form of curation, quality control and discovery.

For event organizers, Shoobs helps provide digital ticketing and promotion services, with event recommendation capabilities that target the most promising potential customers. Through its offering to event organizers, Shoobs is able to monetize its services through ticket sale commission, advertising and brand partnerships.

Since its initial launch in London, Shoobs notes it has become one of the top urban events platforms in the city, with an extensive base of recurring registered users and event organizers. After previously working with AEG for its London launch, Shoobs is looking to expand stateside with the help of organizers like Live Nation. Shoobs joins a long list of promising Y Combinator alumni companies with YC also acting as one of Shoobs’ initial investors.

Other presenting companies included:

Morgan Stanley Multicultural Innovation Lab

  • BeautyLynk “is an on-demand hair and makeup service provider, specializing in customizable services for women.”
  • Broadway Roulette “is an events marketplace that pairs consumers with surprise cultural events, beginning with Broadway theater.”
  • CariClub “is an enterprise software platform to connect young professionals with nonprofit opportunities.”
  • COI Energy Services “is an integrated platform for electric utilities and business users to optimize and manage energy usage.”
  • CoSign “is an API and application that allows anyone to create, distribute and monetize visual content.”
  • Goalsetter “is a goals-based gifting, savings and investing platform designed for children.”
  • myLAB Box “offers customizable at-home health-test kits and relevant telemedicine consultations / prescription services.”

HearstLab

  • Priori “is a global legal marketplace changing the way in-house teams find, hire and manage outside counsel.”
  • TRENCH “is an online fashion marketplace that makes use of the unworn items in every woman’s closet.”

Newark Venture Partners Labs

  • Floss Bar “is a new type of preventive brand for oral health care. The company offers high-quality, routine dental care across flexible locations at thoughtful prices.”
  • Upsider “is a software solution allowing recruiters to leverage AI technology to identify a comprehensive set of candidates who align with their business and role requirements, resulting in a more strategic understanding of the best possible talent for the job.”

PS27 Ventures

  • BlueWave Technologies “is a cleantech company and the creators of the BlueWave™ Cleaning System — a water-free, detergent-free and chemical-free plasma device that cleans items that are extremely hard or impossible to clean with a washer and dryer.”
  • OnPay Solutions “focuses exclusively on business-to-business payments. They create payment software and offer payment web services to enhance efficiency and productivity for Accounts Payable and Accounts Receivable.”

Oct
08
2018
--

WeWork taps Lemonade to offer insurance to WeLive members

WeWork has partnered with Lemonade to provide renters insurance to WeLive members.

WeLive is the residential offering from WeWork, offering members a fully-furnished apartment, complete with amenities like housekeeping, mailroom, and on-site laundry, on a flexible rental schedule. In other words, bicoastal workers or generally nomadic individuals can rent a short-term living space without worrying about all the extras.

As part of that package, WeLive is now referring new and existing WeLive members to Lemonade for renters insurance.

WeLive currently has two locations — one in New York and one in D.C. — collectively representing more than 400 units. WeWork says that both units are nearly at capacity. The company has plans to open a third location in Seattle Washington by Spring 2020.

Lemonade, meanwhile, is an up-and-coming insurance startup that is rethinking the centuries-old industry. The company’s first big innovation was the digitization of getting insurance. The company uses a chatbot to lead prospective customers through the process in under a minute.

The second piece of Lemoande’s strategy is rooted in the business model. Unlike incumbent insurance providers, Lemonade takes its profit up-front, raking away a percentage of customers’ monthly payments. The rest, however, is set aside to fulfill claims. Whatever goes unclaimed at the end of the year is donated to the charity of each customer’s choice.

To date, Lemonade has raised a total of $180 million. WeWork, on the other hand, has raised just over $9 billion, with a reported valuation as high as $35 billion.

Of course, part of the reason for that lofty valuation is the fact that WeWork is a real estate behemoth, with Re/Code reporting that the company is Manhattan’s second biggest private office tenant. But beyond sheer square footage, WeWork has spent the past few years filling its arsenal with various service providers for its services store.

With 175,000 members (as of end of 2017, so that number is likely much higher now), WeWork has a considerable userbase with which it can negotiate deals with service providers, from enterprise software makers to… well, insurance providers.

Lemonade is likely just the beginning of WeWork’s stretch into developing a suite of services and partnerships for its residential members.

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com