Dec
31
2018
--

Great things that happened with PostgreSQL in the Year 2018

PostgreSQL in 2018

In this blog post, we’ll look back at what’s been going on in the world of PostgreSQL in 2018.

Before we start talking about the good things that have happened in the PostgreSQL in  2018, we hope you had a wonderful year and we wish you a happy and prosperous 2019.

PostgreSQL has been a choice for those who are looking for a completely community-driven open source database that is feature-rich and extensible. We have seen tremendously great things happening in PostgreSQL for many years, with 2018 being a prime example. As you could see the following snippet from DB engine rankings, PostgreSQL has topped the chart for growth in popularity in the year 2018 compared to other databases.

PostgreSQL adoption growth has been increasing year over year, and 2018 has again been one such year as we can see.

Let’s start with a recap of some of the great PostgreSQL events, and look at what we should take away from 2018 in the PostgreSQL space.

PostgreSQL 11 Released

PostgreSQL 11 was a release that incorporated a lot of features offered in commercial database software governed by an enterprise license. For example, there are times when you are required to enforce the handling of embedded transactions inside a stored procedure in your application code. There are also times when you wish to partition a table with foreign keys or use hash partitioning. This used to require workarounds. The release of PostgreSQL 11 covers these scenarios.

There were many other add-ons as well, such as Just-In-Time compilation, improved query parallelism, partition elimination, etc. You can find out more in our blog post here, or the PostgreSQL 11 release notes (if you have not seen already). Special thanks to everyone involved in such a vibrant PostgreSQL release.

End of Life for PostgreSQL 9.3

9.3.25 was the last minor release that has happened for PostgreSQL 9.3 (on November 8, 2018). There will be no more minor releases supported by the community for 9.3. If you are still using PostgreSQL 9.3 (or a major earlier release than 9.3), it is the time to start planning to upgrade your database to take advantage of additional features and performance improvements.

Watch out for future Percona webinars (dates will be out soon) on PostgreSQL migrations and upgrades that will help handle situations such as downtime and other complexities involved in migrating your partitions built using table inheritance when you migrate from legacy PostgreSQL versions to the latest versions.

PostgreSQL Minor Releases

For minor PostgreSQL release, there was nothing new in what we saw this year compared to previous years. The PostgreSQL community aims for a minor version release for all the supported versions every quarter. However, we may see more minor releases due to critical bug fixes or security fixes. One of such release was done on March 3rd, 2018 for the CVE-2018-1058 security fix. This proves that you do not necessarily need to wait for specific release dates when a security vulnerability has been identified. You may see the fix released as a minor version as soon as the development, review and testing are completed for the fix.  

There have been five minor releases this year on the following dates.

Security Fixes in All the Supported PostgreSQL Releases This Year

The PostgreSQL Global Development Team and contributors handle security fixes very seriously. There have been several instances where we have received immediate responses after reporting a problem or a bug. Likewise, we have seen many security bug fixes as soon as they have been reported.

Following are a list of security fixes we have seen in the year 2018:

We thank all the Core team, Contributors, Hackers and Users involved in making it another great year for PostgreSQL and a huge WIN for the open source world.

If you would like to participate in sharing your PostgreSQL knowledge to a wider audience, or if you have got a great topic that you would like to talk about, please submit your proposal to one of the world’s biggest open source conferences: Percona Live Open Source Database Conference 2019 in Austin, Texas from May 28-30, 2019. The Call for Papers is open until Jan 20, 2019.

You can also submit blog articles to our Open Source Database Community Blog for publishing to a wider open source database audience.

Please subscribe to our blog posts to hear many more great things that are happening in the PostgreSQL space.

Dec
24
2018
--

Salesforce keeps rolling with another banner year in 2018

The good times kept on rolling this year for Salesforce with all of the requisite ingredients of a highly successful cloud company — the steady revenue growth, the expanding product set and the splashy acquisitions. The company also opened the doors of its shiny new headquarters, Salesforce Tower in San Francisco, a testament to its sheer economic power in the city.

Salesforce, which set a revenue goal of $10 billion a few years ago is already on its way to $20 billion. Yet Salesforce is also proof you can be ruthlessly good at what you do, while trying to do the right thing as an organization.

Make no mistake, Marc Benioff and Keith Block, the company’s co-CEOs, want to make obscene amounts of money, going so far as to tell a group of analysts earlier this year that their goal by 2034 is to be a $60 billion company. Salesforce just wants to do it with a hint of compassion as it rakes in those big bucks and keeps well-heeled competitors like Microsoft, Oracle and SAP at bay.

A look at the numbers

In the end, a publicly traded company like Salesforce is going to be judged by how much money it makes, and Salesforce it turns out is pretty good at this, as it showed once again this year. The company grew every quarter by over 24 percent YoY and ended up the year with $12.53 billion in revenue. Based on its last quarter of $3.39 billion, the company finished the year on a $13.56 billion run rate.

This compares with $9.92 billion in total revenue for 2017 with a closing run rate of $10.72 billion.

Even with this steady growth trajectory, it might be some time before it hits the $5 billion-a-quarter mark and checks off the $20 billion goal. Keep in mind that it took the company three years to get from $1.51 billion in Q12016 to $3.1 billion in Q12019.

As for the stock market, it has been highly volatile this year, but Salesforce is still up. Starting the year at $102.41, it was sitting at $124.06 as of publication, after peaking on October 1 at $159.86. The market has been on a wild ride since then and cloud stocks have taken a big hit, warranted or not. On one particularly bad day last month, Salesforce had its worst day since 2016 losing 8.7 percent in value,

Spending big

When you make a lot of money you can afford to spend generously, and the company invested some of those big bucks when it bought Mulesoft for $6.5 billion in March, making it the most expensive acquisition it has ever made. With Mulesoft, the company had a missing link between data sitting on-prem in private data centers and Salesforce data in the cloud.

Mulesoft helps customers build access to data wherever it lives via APIs. That includes legacy data sitting in ancient data repositories. As Salesforce turns its eyes toward artificial intelligence and machine learning, it requires oodles of data and Mulesoft was worth opening up the wallet to provide the company with that kind of access to a variety of enterprise data.

Salesforce 2018 acquisitions. Chart: Crunchbase.

But Mulesoft wasn’t the only thing Salesforce bought this year. It made five acquisitions in all. The other significant one came in July when it scooped up Dataorama for a cool $800 million, giving it a market intelligence platform.

What could be on board for 2019? If Salesforce sticks to its recent pattern of spending big one year, then regrouping the next, 2019 could be a slower one for acquisitions. Consider that it bought just one company last year after buying a dozen in 2016.

One other way to keep revenue rolling in comes from high-profile partnerships. In the past, Salesforce has partnered with Microsoft and Google, and this year it announced that it was teaming up with Apple. Salesforce also announced another high-profile arrangement with AWS to share data between the two platforms more easily. The hope with these types of cross pollination is that the companies can both increase their business. For Salesforce, that means using these partnerships as a platform to move the revenue needle faster.

Compassionate capitalism

Even while his company has made big bucks, Benioff has been preaching compassionate capitalism using Twitter and the media as his soap box.

He went on record throughout this year supporting Prop C, a referendum question designed to help battle San Francisco’s massive homeless problem by taxing companies with greater than $50 million in revenue — companies like Salesforce. Benioff was a vocal proponent of the idea, and it won. He did not find kindred spirits among some of his fellow San Francisco tech CEOs, openly debating Twitter CEO Jack Dorsey on Twitter.

Speaking about Prop C in an interview with Kara Swisher of Recode in November, Benioff talked in lofty terms about why he believed in the measure even though it would cost his company money.

“You’ve got to really be mindful and think about what it is that you want your company to be for and what you’re doing with your business and here at Salesforce, that’s very important to us,” he told Swisher in the interview.

He also talked about how employees at other tech companies were driving their CEOs to change their tune around social issues, including supporting Prop C, but Benioff had to deal with his own internal insurrection this year when 650 employees signed a petition asking him to rethink Salesforce’s contract with the U.S. Customs and Border Protection (CBP) in light of the current administration’s border policies. Benioff defended the contract, stating that that Salesforce tools were being used internally at CBP for staff recruiting and communication and not to enforce border policy.

Regardless, Salesforce has never lost its focus on meeting lofty revenue goals, and as we approach the new year, there is no reason to think that will change. The company will continue to look for new ways to expand markets and keep their revenue moving ever closer to that $20 billion goal, even as it continues to meld its unique form of compassion and capitalism.

Dec
21
2018
--

Announcing General Availability of Percona Server for MySQL 8.0

Percona Server for MySQL 8.0

Percona Server for MySQL 8.0

Percona has released Percona Server for MySQL 8.0 as Generally Available (GA). Our Percona Server for MySQL 8.0 software is the company’s free, enhanced, drop-in replacement for MySQL Community Edition. Percona Server for MySQL 8.0 includes all of the great features in MySQL Community Edition 8.0. It also includes enterprise-class features from Percona made available free and open source. Percona Server for MySQL is trusted by thousands of enterprises to meet their need for a mature, proven, cost-effective MySQL solution that delivers excellent performance and reliability.

Downloads are available on the Percona Website and in the Percona Software Repositories.

Features in Percona Server for MySQL 8.0

Percona Server for MySQL 8.0 includes all of the features available in MySQL 8.0 Community Edition in addition to enterprise-grade features developed by Percona for the community.

MySQL Community Edition 8.0 Features

Some of the highlights from MySQL 8.0 contained in Percona Server for MySQL 8.0 include:

  • MySQL Document Store—Combining NoSQL functionality within the X API along with JSON enhancements such as new operators and functions enables developers to use MySQL 8.0 for non-relational data storage without the need for a separate NoSQL database.
  • Stronger SQL—With the addition of Window Functions, Common Table Expressions, Unicode safe Regular Expressions, and other improvements MySQL 8.0 provide broader support for the range of SQL standard functionality.
  • Transactional Data Dictionary—Enables atomic and crash-safe DDL operations, enhancing reliability, and eliminating the need for metadata files.
  • Security—SQL Roles, SHA2 default authentication, fine-grained privileges, and other enhancements make MySQL 8.0 more secure and adaptable to your organization’s compliance needs.
  • Geospatial—New SRS aware spatial data types, spatial indexes, and spatial functions, enabling the use of MySQL 8.0 for complex GIS use-cases.

Percona Server for MySQL 8.0 Features

Building on the upstream MySQL 8.0 Community Edition, Percona Server for MySQL 8.0 brings many great features in this release, including the following:

  • Security and Compliance:
    • Audit Logging Plugin: Provides monitoring and logging of database activity to assist organizations in meeting their compliance objectives. This feature is comparable to MySQL Enterprise Auditing.
    • PAM-based Authentication Plugin: Assists enterprises in integrating Percona Server for MySQL with their single sign-on (SSO) and two-factor authentication (2FA) systems by integrating with standard PAM modules. This feature is comparable to MySQL Enterprise Authentication.
    • Enhanced Encryption: Improves upon Transparent Data Encryption (TDE) present in MySQL Community Edition. Enhanced encryption adds support for binary log encryption, temporary file encryption, encryption support for all InnoDB tablespace types and logs, encryption of the parallel doublewrite buffer, key rotation, and support for centralized key management using Hashicorp Vault. Please Note: Some of the encryption features are still considered experimental and are not yet suitable for production use. These features together are comparable to MySQL Enterprise TDE.
  • Performance and Scalability:
    • Threadpool: Supporting 10000+ connections, this feature provides significant performance benefits under heavy load. This feature is comparable to MySQL Enterprise Scalability.
    • InnoDB Engine Enhancements: Enables highly concurrent IO-bound workloads to see significant performance improvements through parallel doublewrite, multithreaded LRU flushers, and single page eviction. In a simple benchmark, we saw a 60% performance improvement in some workloads when comparing Percona Server for MySQL to MySQL Community Edition
    • MyRocks Storage Engine: Based on the RocksDB storage library, MyRocks brings MySQL into the 21st century by being optimized for modern hardware such as nVME SSDs. Utilizing strong compression, MyRocks reduces write-amplification and storage requirements on SSDs compared to InnoDB to lower TCO and increase ROI when working with large datasets. Improved throughput consistency compared to InnoDB enables scaling cloud resources for your databases more strategically.
  • Observability and Usability:
    • Improved Instrumentation: Percona Server for MySQL 8.0 offers more than double the available performance and stats counters compared to MySQL Community Edition, as well as support for gathering per-user and per-thread statistics, and extended slow query logging capabilities. Together with free tools like Percona Monitoring and Management these enhancements enable your DBAs to troubleshoot issues faster and effectively improve your application performance.
    • Reduced Backup Impact: Lighter weight Backup Locking reduces the impact to performance and availability of performing backups.  This feature makes your backups run faster and your applications perform better during long-running backups when used together with Percona XtraBackup 8.0.

Features Removed in Percona Server for MySQL 8.0

Some features were not ported forward from Percona Server for MySQL 5.7 to Percona Server for MySQL 8.0.  Features which are unused, have something comparable included upstream, or are no longer relevant in this major release have been removed. For more information see our documentation.

  • Slow Query Log Rotation and Expiration: Not widely used, can be accomplished using logrotate
  • CSV engine mode for standard-compliant quote and comma parsing
  • Expanded program option modifiers
  • The ALL_O_DIRECT InnoDB flush method: it is not compatible with the new redo logging implementation
  • XTRADB_RSEG table removed from INFORMATION_SCHEMA
  • InnoDB memory size information from SHOW ENGINE INNODB STATUS; the same information is available from Performance Schema memory summary tables
  • Query cache enhancements: The query cache is no longer present in MySQL 8.0

Features Being Deprecated in Percona Server for MySQL 8.0

  • TokuDB Storage Engine: TokuDB will be supported throughout the Percona Server for MySQL 8.0 release series, but will not be available in the next major release.  Percona encourages TokuDB users to explore the MyRocks Storage Engine which provides similar benefits for the majority of workloads and has better optimized support for modern hardware.

Additional Resources

Dec
21
2018
--

Release Notes for Percona Server for MySQL 8.0.13-3 GA

Percona Server for MySQL 8.0

Percona Server for MySQL 8.0

Percona announces the GA release of Percona Server for MySQL 8.0.13-3 on December 21, 2018 (downloads are available here and from the Percona Software Repositories). This release merges changes of MySQL 8.0.13, including all the bug fixes in it. Percona Server for MySQL 8.0.13-3 is now the current GA release in the 8.0 series. All of Percona’s software is open-source and free.

Percona Server for MySQL 8.0 includes all the features available in MySQL 8.0 Community Edition in addition to enterprise-grade features developed by Percona. For a list of highlighted features from both MySQL 8.0 and Percona Server for MySQL 8.0, please see the GA release announcement.

Note: If you are upgrading from 5.7 to 8.0, please ensure that you read the upgrade guide and the document Changed in Percona Server for MySQL 8.0.

Features Removed in Percona Server for MySQL 8.0

  • Slow Query Log Rotation and Expiration: Not widely used, can be accomplished using logrotate
  • CSV engine mode for standard-compliant quote and comma parsing
  • Expanded program option modifiers
  • The ALL_O_DIRECT InnoDB flush method: it is not compatible with the new redo logging implementation
  • XTRADB_RSEG table from INFORMATION_SCHEMA
  • InnoDB memory size information from SHOW ENGINE INNODB STATUS; the same information is available from Performance Schema memory summary tables
  • Query cache enhancements: The query cache is no longer present in MySQL 8.0

Features Deprecated in Percona Server for MySQL 8.0

  • TokuDB Storage Engine: the Percona Server for MySQL 8.0 release series supports TokuDB. We are deprecating TokuDB support in the next major release. Percona encourages TokuDB users to explore the MyRocks Storage Engine which provides similar benefits for the majority of workloads and has better-optimized support for modern hardware.

Issues Resolved in Percona Server for MySQL 8.0.13-3

Improvements

  • #5014: Update Percona Backup Locks feature to use the new BACKUP_ADMIN privilege in MySQL 8.0
  • #4805: Re-Implemented Compressed Columns with Dictionaries feature in PS 8.0
  • #4790: Improved accuracy of User Statistics feature

Bugs Fixed Since 8.0.12-2rc1

  • Fixed a crash in mysqldump in the --innodb-optimize-keys functionality #4972
  • Fixed a crash that can occur when system tables are locked by the user due to a lock_wait_timeout #5134
  • Fixed a crash that can occur when system tables are locked by the user from a SELECT FOR UPDATE statement #5027
  • Fixed a bug that caused innodb_buffer_pool_size to be uninitialized after a restart if it was set using SET PERSIST#5069
  • Fixed a crash in TokuDB that can occur when a temporary table experiences an autoincrement rollover #5056
  • Fixed a bug where marking an index as invisible would cause a table rebuild in TokuDB and also in MyRocks #5031
  • Fixed a bug where audit logs could get corrupted if the audit_log_rotations was changed during runtime. #4950
  • Fixed a bug where LOCK INSTANCE FOR BACKUP and STOP SLAVE SQL_THREAD would cause replication to be blocked and unable to be restarted. #4758 (Upstream #93649)

Other Bugs Fixed:

#5155#5139#5057#5049#4999#4971#4943#4918#4917#4898, and #4744.

Known Issues in Percona Server for MySQL 8.0.13-3

We have a few features and issues outstanding that should be resolved in the next release.

Pending Feature Re-Implementations and Improvements

  • #4892: Re-Implement Expanded Fast Index Creation feature.
  • #5216: Re-Implement Utility User feature.
  • #5143: Identify Percona features which can make use of dynamic privileges instead of SUPER

Notable Issues in Features

  • #5148: Regression in Compressed Columns Feature when using innodb-force-recovery
  • #4996: Regression in User Statistics feature where TOTAL_CONNECTIONS field report incorrect data
  • #4933: Regression in Slow Query Logging Extensions feature where incorrect transaction id accounting can cause an assert during certain DDLs.
  • #5206: TokuDB: A crash can occur in TokuDB when using Native Partitioning and the optimizer has index_merge_union enabled. Workaround by using SET SESSION optimizer_switch="index_merge_union=off";
  • #5174: MyRocks: Attempting to use unsupported features against MyRocks can lead to a crash rather than an error.
  • #5024: MyRocks: Queries can return the wrong results on tables with no primary key, non-unique CHAR/VARCHAR rows, and UTF8MB4 charset.
  • #5045: MyRocks: Altering a column or table comment cause the table to be rebuilt

Find the release notes for Percona Server for MySQL 8.0.13-3 in our online documentation. Report bugs in the Jira bug tracker.

Dec
21
2018
--

Backup and Restore a PostgreSQL Cluster With Multiple Tablespaces Using pg_basebackup

PostgreSQL backup cluster multiple tablespaces

PostgreSQL logopg_basebackup is a widely used PostgreSQL backup tool that allows us to take an ONLINE and CONSISTENT file system level backup. These backups can be used for point-in-time-recovery or to set up a slave/standby. You may want to refer to our previous blog posts, PostgreSQL Backup StrategyStreaming Replication in PostgreSQL and Faster PITR in PostgreSQL where we describe how we used pg_basebackup for different purposes. In this post, I’ll demonstrate the steps to restore a backup taken using pg_basebackup when we have many tablespaces that store databases or their underlying objects.

A simple backup can be taken using the following syntax.

Tar and Compressed Format
$ pg_basebackup -h localhost -p 5432 -U postgres -D /backupdir/latest_backup -Ft -z -Xs -P
Plain Format
$ pg_basebackup -h localhost -p 5432 -U postgres -D /backupdir/latest_backup -Fp -Xs -P

Using a tar and compressed format is advantageous when you wish to use less disk space to backup and store all tablespaces, data directory and WAL segments, with everything in just one directory (target directory for backup).

Whereas a plain format stores a copy of the data directory as is, in the target directory. When you have one or more non-default tablespaces, tablespaces may be stored in a separate directory. This is usually the same as the original location, unless you use

--tablespace-mapping

  to modify the destination for storing the tablespaces backup.

PostgreSQL supports the concept of tablespaces. In simple words, a tablespace helps us maintain multiple locations to scatter databases or their objects. In this way, we can distribute the IO and balance the load across multiple disks.

To understand what happens when we backup a PostgreSQL cluster that contains multiple tablespaces, let’s consider the following example. We’ll take these steps:

  • Create two tablespaces in an existing master-slave replication setup.
  • Take a backup and see what is inside the backup directory.
  • Restore the backup.
  • Conclude our findings

Create 2 tablespaces and take a backup (tar format) using pg_basebackup

Step 1 :

I set up a replication cluster using PostgreSQL 11.2. You can refer to our blog post Streaming Replication in PostgreSQL to reproduce the same scenario. Here are the steps used to create two tablespaces:

$ sudo mkdir /data_pgbench
$ sudo mkdir /data_pgtest
$ psql -c "CREATE TABLESPACE data_pgbench LOCATION '/data_pgbench'"
$ psql -c "CREATE TABLESPACE data_pgtest LOCATION '/data_pgtest'"
$ psql -c "select oid, spcname, pg_tablespace_location(oid) from pg_tablespace"
oid | spcname | pg_tablespace_location
-------+--------------+------------------------
1663 | pg_default |
1664 | pg_global |
16419 | data_pgbench | /data_pgbench
16420 | data_pgtest | /data_pgtest
(4 rows)

Step 2 :

Now, I create two databases in two different tablespaces, using pgbench to create a few tables and load some data in them.

$ psql -c "CREATE DATABASE pgbench TABLESPACE data_pgbench"
$ psql -c "CREATE DATABASE pgtest TABLESPACE data_pgtest"
$ pgbench -i pgbench
$ pgbench -i pgtest

In a master-slave setup built using streaming replication, you must ensure that the directories exist in the slave, before running a

"CREATE TABLESPACE ..."

  on the master. This is because, the same statements used to create a tablespace are shipped/applied to the slave through WALs – this is unavoidable. The slave crashes with the following message, when these directories do not exist:

2018-12-15 12:00:56.319 UTC [13121] LOG: consistent recovery state reached at 0/80000F8
2018-12-15 12:00:56.319 UTC [13119] LOG: database system is ready to accept read only connections
2018-12-15 12:00:56.327 UTC [13125] LOG: started streaming WAL from primary at 0/9000000 on timeline 1
2018-12-15 12:26:36.310 UTC [13121] FATAL: directory "/data_pgbench" does not exist
2018-12-15 12:26:36.310 UTC [13121] HINT: Create this directory for the tablespace before restarting the server.
2018-12-15 12:26:36.310 UTC [13121] CONTEXT: WAL redo at 0/9000448 for Tablespace/CREATE: 16417 "/data_pgbench"
2018-12-15 12:26:36.311 UTC [13119] LOG: startup process (PID 13121) exited with exit code 1
2018-12-15 12:26:36.311 UTC [13119] LOG: terminating any other active server processes
2018-12-15 12:26:36.314 UTC [13119] LOG: database system is shut down
2018-12-15 12:27:01.906 UTC [13147] LOG: database system was interrupted while in recovery at log time 2018-12-15 12:06:13 UTC
2018-12-15 12:27:01.906 UTC [13147] HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.

Step 3 :

Let’s now use pg_basebackup to take a backup. In this example, I use a tar format backup.

$ pg_basebackup -h localhost -p 5432 -U postgres -D /backup/latest_backup -Ft -z -Xs -P
94390/94390 kB (100%), 3/3 tablespaces

In the above log, you could see that there are three tablespaces that have been backed up: one default, and two newly created tablespaces. If we go back and check how the data in the two tablespaces are distributed to appropriate directories, we see that there are symbolic links created inside the pg_tblspc directory (within the data directory) for the oid’s of both tablespaces. These links are directed to the actual location of the tablespaces, we specified in Step 1.

$ ls -l $PGDATA/pg_tblspc
total 0
lrwxrwxrwx. 1 postgres postgres 5 Dec 15 12:31 16419 -> /data_pgbench
lrwxrwxrwx. 1 postgres postgres 6 Dec 15 12:31 16420 -> /data_pgtest

Step 4 :

Here are the contents inside the backup directory, that was generated through the backup taken in Step 3.

$ ls -l /backup/latest_backup
total 8520
-rw-------. 1 postgres postgres 1791930 Dec 15 12:54 16419.tar.gz
-rw-------. 1 postgres postgres 1791953 Dec 15 12:54 16420.tar.gz
-rw-------. 1 postgres postgres 5113532 Dec 15 12:54 base.tar.gz
-rw-------. 1 postgres postgres 17097 Dec 15 12:54 pg_wal.tar.gz

Tar Files :

16419.tar.gz

 and

16420.tar.gz

 are created as a backup for the two tablespaces. These are created with the same names as the OIDs of their respective tablespaces.

Let’s now take a look how we can restore this backup to completely different locations for data and tablespaces.

Restore a backup with multiple tablespaces

Step 1 :

In order to proceed further with the restore, let’s first extract the base.tar.gz file. This file contains some important files that help us to proceed further.

$ tar xzf /backup/latest_backup/base.tar.gz -C /pgdata
$ ls -larth /pgdata
total 76K
drwx------. 2 postgres postgres 18 Dec 14 14:15 pg_xact
-rw-------. 1 postgres postgres 3 Dec 14 14:15 PG_VERSION
drwx------. 2 postgres postgres 6 Dec 14 14:15 pg_twophase
drwx------. 2 postgres postgres 6 Dec 14 14:15 pg_subtrans
drwx------. 2 postgres postgres 6 Dec 14 14:15 pg_snapshots
drwx------. 2 postgres postgres 6 Dec 14 14:15 pg_serial
drwx------. 4 postgres postgres 36 Dec 14 14:15 pg_multixact
-rw-------. 1 postgres postgres 1.6K Dec 14 14:15 pg_ident.conf
drwx------. 2 postgres postgres 6 Dec 14 14:15 pg_dynshmem
drwx------. 2 postgres postgres 6 Dec 14 14:15 pg_commit_ts
drwx------. 6 postgres postgres 54 Dec 14 14:18 base
-rw-------. 1 postgres postgres 4.5K Dec 14 16:16 pg_hba.conf
-rw-------. 1 postgres postgres 208 Dec 14 16:18 postgresql.auto.conf
drwx------. 2 postgres postgres 6 Dec 14 16:18 pg_stat
drwx------. 2 postgres postgres 58 Dec 15 00:00 log
drwx------. 2 postgres postgres 6 Dec 15 12:54 pg_stat_tmp
drwx------. 2 postgres postgres 6 Dec 15 12:54 pg_replslot
drwx------. 4 postgres postgres 68 Dec 15 12:54 pg_logical
-rw-------. 1 postgres postgres 224 Dec 15 12:54 backup_label
drwx------. 3 postgres postgres 28 Dec 15 12:57 pg_wal
drwx------. 2 postgres postgres 4.0K Dec 15 12:57 global
drwx------. 2 postgres postgres 32 Dec 15 13:01 pg_tblspc
-rw-------. 1 postgres postgres 55 Dec 15 13:01 tablespace_map
-rw-------. 1 postgres postgres 24K Dec 15 13:04 postgresql.conf
-rw-r--r--. 1 postgres postgres 64 Dec 15 13:07 recovery.conf
-rw-------. 1 postgres postgres 44 Dec 15 13:07 postmaster.opts
drwx------. 2 postgres postgres 18 Dec 15 13:07 pg_notify
-rw-------. 1 postgres postgres 30 Dec 15 13:07 current_logfiles

Step 2 :

The files that we need to consider for our recovery are :

  • backup_label
  • tablespace_map

When you open the backup_label file, we see the start WAL location, backup start time, etc. These are some details that help us perform a point-in-time-recovery.

$ cat backup_label
START WAL LOCATION: 0/B000028 (file 00000001000000000000000B)
CHECKPOINT LOCATION: 0/B000060
BACKUP METHOD: streamed
BACKUP FROM: master
START TIME: 2018-12-15 12:54:10 UTC
LABEL: pg_basebackup base backup
START TIMELINE: 1

Now, let us see what is inside the

tablespace_map

 file.

$ cat tablespace_map
16419 /data_pgbench
16420 /data_pgtest

In the above log, you could see that there are two entries – one for each tablespace. This is a file that maps a tablespace (oid) to its location. When you start PostgreSQL after extracting the tablespace and WAL tar files, symbolic links are created automatically by postgres – inside the pg_tblspc directory for each tablespace – to the appropriate tablespace location, using the mapping done in this files.

Step 3 :

Now, in order to restore this backup in the same postgres server from where the backup was taken, you must remove the existing data in the original tablespace directories. This allows you to extract the tar files of each tablespaces to the appropriate tablespace locations.

The actual commands for extracting tablespaces from the backup in this case were the following:

$ tar xzf 16419.tar.gz -C /data_pgbench (Original tablespace location)
$ tar xzf 16420.tar.gz -C /data_pgtest  (Original tablespace location)

In a scenario where you want to restore the backup to the same machine from where the backup was originally taken, we must use different locations while extracting the data directory and tablespaces from the backup. In order to achieve that, tar files for individual tablespaces may be extracted to different directories than the original directories specified in

tablespace_map

 file, upon which we can modify the

tablespace_map

 file with the new tablespace locations. The next two steps should help you to see how this works.

Step 3a :

Create two different directories and extract the tablespaces to them.

$ tar xzf 16419.tar.gz -C /pgdata_pgbench (Different location for tablespace than original)
$ tar xzf 16420.tar.gz -C /pgdata_pgtest  (Different location for tablespace than original)

Step 3b :

Edit the

tablespace_map

 file with the new tablespace locations. Replace the original location of each tablespace with the new location, where we have extracted the tablespaces in the previous step. Here is how it appears after the edit.

$ cat tablespace_map
16419 /pgdata_pgbench
16420 /pgdata_pgtest

Step 4 :

Extract pg_wal.tar.gz from backup to pg_wal directory of the new data directory.

$ tar xzf pg_wal.tar.gz -C /pgdata/pg_wal

Step 5 :

Create

recovery.conf

 to specify the time until when you wish to perform a point-in-time-recovery. Please refer to our previous blog post – Step 3, to understand recovery.conf for PITR in detail.

Step 6 :

Once all of the steps above are complete you can start PostgreSQL.
You should see the following files renamed after recovery.

backup_label   --> backup_label.old
tablespace_map --> tablespace_map.old
recovery.conf  --> recovery.done

To avoid the exercise of manually modifying the tablespace_map file, you can use

--tablespace-mapping

 . This is an option that works when you use a plain format backup, but not with tar. Let’s see why you may prefer a tar format when compared to plain.

Backup of PostgreSQL cluster with tablespaces using plain format

Consider the same scenario where you have a PostgreSQL cluster with two tablespaces. You might see the following error when you do not use

--tablespace-mapping

 .

$ pg_basebackup -h localhost -p 5432 -U postgres -D /backup/latest_backup -Fp -Xs -P -v
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/22000028 on timeline 1
pg_basebackup: directory "/data_pgbench" exists but is not empty
pg_basebackup: removing contents of data directory "/backup/latest_backup"

What the above error means is that the pg_basebackup is trying to store the tablespaces in the same location as the original tablespace directory. Here

/data_pgbench

 is the location of tablespace :

data_pgbench.

 And, now, pg_basebackup is trying to store the tablespace backup in the same location. In order to overcome this error, you can apply tablespace mapping using the following syntax.

$ pg_basebackup -h localhost -p 5432 -U postgres -D /backup/latest_backup -T "/data_pgbench=/pgdata_pgbench" -T "/data_pgtest=/pgdata_pgtest" -Fp -Xs -P

-T

 is used to specify the tablespace mapping.

-T

 can be replaced by

--tablespace-mapping

.

The advantage of using -T (

--tablespace-mapping

 ) is that the tablespaces are stored separately in the mapping directories. In this example with plain format backup, you must extract all the following three directories in order to restore/recover the database using backup.

  • /backup/latest_backup
  • /pgdata_pgtest
  • /pgdata_pgbench

However, you do not need a

tablespace_map

  file in this scenario, as it is automatically taken care of by PostgreSQL.
If you take a backup in tar format, you see all the tar files for base, tablespaces and WAL segments stored in the same backup directory, and just this directory can be extracted for performing restore/recovery. However, you must manually extract the tablespaces and WAL segments to appropriate locations and edit the tablespace_map file, as discussed above.


Image based on Photos by Alan James Hendry on Unsplash   and  Tanner Boriack on Unsplash

Dec
21
2018
--

Percona Server for MongoDB Authentication Using Active Directory

authentication

mongodb authentication with active directoryThis article will walk you through using the SASL library to allow your Percona Server for MongoDB instance to authenticate with your company’s Active Directory server. Percona Server for MongoDB includes enterprise level features, such as LDAP authentication, audit logging and with the 3.6.8 release a beta version of data encryption at rest, all in its open source offering.

Pre set-up assumptions

In this article we will make a couple of assumptions:

  1. You have an Active Directory server up and running and that it is accessible to the server that you have Percona Server for MongoDB installed on.
  2. These machines are installed behind a firewall as the communications between the two servers will be in plain text. This is due the fact that we can only use the SASL mechanism of PLAIN when authenticating and credentials will be sent in plain text.
  3. You have sudo privilege on the server you are going to install Percona Server for MongoDB on.

Installing Percona Server for MongoDB

The first thing you are going to need to do is to install the Percona Server for MongoDB package. You can get this in a couple of different ways. You can either install from the Percona repositories, or you can download the packages and install them manually.

Once you have Percona Server for MongoDB installed, we want to start the mongod service and make sure it is set to run on restart.

sudo systemctl start mongod
sudo systemctl enable mongod

Now that the service is up and running, we want to open the mongo shell and add a database administrator user. This user will be authenticated inside of the MongoDB server itself and will not have any interactions with the Active Directory server.

To start the mongo shell up, type mongo from a terminal window. Once you do this you will see something similar to the following:

Percona Server for MongoDB shell version v3.6.8-2.0
connecting to: mongodb://127.0.0.1:27017
Percona Server for MongoDB server version: v3.6.8-2.0
Server has startup warnings:
2018-12-11T17:48:47.471+0000 I STORAGE [initandlisten]
2018-12-11T17:48:47.471+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-12-11T17:48:47.471+0000 I STORAGE [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-12-11T17:48:48.197+0000 I CONTROL [initandlisten]
2018-12-11T17:48:48.197+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-12-11T17:48:48.197+0000 I CONTROL [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-12-11T17:48:48.197+0000 I CONTROL [initandlisten] **          You can use percona-server-mongodb-enable-auth.sh to fix it.
2018-12-11T17:48:48.197+0000 I CONTROL [initandlisten]

Notice the second warning that access control is not enabled for the database. Percona Server for MongoDB comes with a script that you can run that will enable authentication for you, but we can also do this manually.

We will go ahead and manually add a user in MongoDB that has the root role assigned to it. This user will have permission to do anything on the server, so you will want to make sure to keep the password safe. You will also not want to use this user for doing your day to day work inside of MongoDB.

This user needs to be created in the admin database as it needs to have access to the entire system. To do this run the following commands inside of the mongo shell:

> use admin
switched to db admin
> db.createUser({"user": "admin", "pwd": "$3cr3tP4ssw0rd", "roles": ["root"]})
Successfully added user: { "user" : "admin", "roles" : [ "root" ] }

Now that we have a user created in MongoDB we can go ahead and enable authorization. To do this we need to modify the /etc/mongod.conf file and add the following lines:

security:
  authorization: enabled
setParameter:
  authenticationMechanisms: PLAIN,SCRAM-SHA-1

Notice that we have two mechanisms set up for authentication. The first one, PLAIN, is used for authenticating with Active Directory. The second one, SCRAM-SHA-1 is used for internal authentication inside of MongoDB.

Once you’ve made the changes, you can restart the mongod service by running the following command:

sudo systemctl restart mongod

Now if you were to run the mongo shell again, you wouldn’t see the access control warning any more, and you would need to log in as your new user to be able to run any commands.

If you were to try to get a list of databases before logging in you would get an error:

> show dbs;
2018-12-11T21:50:39.551+0000 E QUERY [thread1] Error: listDatabases failed:{
"ok" : 0,
"errmsg" : "not authorized on admin to execute command { listDatabases: 1.0, $db: \"admin\" }",
"code" : 13,
"codeName" : "Unauthorized"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:65:1
shellHelper.show@src/mongo/shell/utils.js:849:19
shellHelper@src/mongo/shell/utils.js:739:15
@(shellhelp2):1:1

Let’s go ahead and run the mongo shell and then log in with our admin user:

> use admin
switched to db admin
> db.auth("admin", "$3cr3tP4ssw0rd")
1

If you are successful you will get a return value of 1. If authentication fails, you will get a return value of 0. Failure is generally due to a mistyped username or password, but you could also be trying to authenticate in the wrong database. In MongoDB you must be in the database that the user was created in before trying to authenticate.

Now that we’ve logged in as the admin user, we will add a document that will be used to verify that our Active Directory based user can successfully access the data at the end of this post.

> use percona
switched to db percona
> db.test.insert({"message": "Active Directory user success!"})
WriteResult({ "nInserted" : 1 })

Install the Cyrus SASL packages

Now that we have a Percona Server for MongoDB instance set up and it is secured, we need to add some packages that will allow us to communicate properly with the Active Directory server.

For RedHat use the following command

sudo yum install -y cyrus-sasl cyrus-sasl-plain

For Ubuntu use this command

sudo app install -y sasl2-bin

Next we need to update the SASL configuration to use LDAP instead of PAM, which is the default. To do this we need to edit the file /etc/sysconfig/saslauthd, remembering to backup up your original file first.

For RedHat we use the following commands

sudo cp /etc/sysconfig/saslauthd /etc/sysconfig/saslauthd.bak
sudo sed -i -e s/^MECH=pam/MECH=ldap/g /etc/sysconfig/saslauthd

For Ubuntu we use these commands instead

sudo cp /etc/default/saslauthd /etc/default/saslauthd.bak
sudo sed -i -e s/^MECHANISMS="pam"/MECHANISMS="ldap"/g /etc/default/saslauthd 
sudo sed -i -e s/^START=no/START=yes/g /etc/default/saslauthd

We also need to create the file /etc/saslauthd.conf with contents similar to the following (replace values as necessary for your Active Directory installation):

ldap_servers: ldap://LDAP.EXAMPLE.COM
ldap_mech: PLAIN
ldap_filter: cn=%u,CN=Users,DC=EXAMPLE,DC=COM
ldap_search_base:CN=Users,DC=EXAMPLE,DC=COM
ldap_filter:(cn=%u)
ldap_bind_dn:CN=ADADMIN,CN=Users,DC=EXAMPLE,DC=COM
ldap_password:ADADMINPASSWORD

Now that we’ve got SASL set up, we can start the saslauthd process and set it to run on restart.

sudo systemctl start saslauthd
sudo systemctl enable saslauthd

Next we need to allow the mongod process to write to the saslauthd mux socket and change the permissions on the owning directory to 755 so MongoDB can write to it. This is the default on RedHat, but not for Ubuntu.

On Ubuntu you can either change the permissions on the folder

sudo chmod 755 /run/saslauthd

Or you could add the mongod user to the sasl group

sudo usermod -a -G sasl mongod

Test the users

The SASL installation provides us with a tool to test that our Active Directory users can be logged in from this machine. Let’s go ahead and test to see if we can authenticate with our Active Directory user.

sudo testsaslauthd -u aduser -p ADP@assword1

You should see 0: OK "Success." if authentication worked.

Create a SASL config file for MongoDB

To allow MongoDB to use SASL to communicate with Active Direcory, we need to create a configuration file.

Create the requisite directory if it doesn’t exist:

mkdir -p /etc/sasl2

And then we need to create the file /etc/sasl2/mongodb.conf and place the following contents into it:

pwcheck_method: saslauthd
saslauthd_path: /var/run/saslauthd/mux
log_level: 5
mech_list: plain

Add Active Directory user to MongoDB

Now we can finally add our Active Directory user to our MongoDB instance:

$ mongo
Percona Server for MongoDB shell version v3.6.8-2.0
connecting to: mongodb://127.0.0.1:27017
Percona Server for MongoDB server version: v3.6.8-2.0
> use admin
switched to db admin
> db.auth("admin", "$3cr3tP4ssw0rd")
1
> use $external
switched to db $external
> db.createUser({"user": "aduser", "roles": [{"role": "read", "db": "percona"}]})
Successfully added user: {
        "user" : "aduser",
        "roles" : [
                {
                        "role" : "read",
                        "db" : "percona"
                }
        ]
}

As you can see from the above, when we create the user that will be authenticated with Active Directory, we need to be in the special $external database and we don’t supply a password as we would when we create a MongoDB authenticated user.

Now let’s try to log in with our Active Directory based user. First we need exit our current mongo shell and restart it, and then we can log in with our Active Directory user:

> exit
bye
$ mongo
Percona Server for MongoDB shell version v3.6.8-2.0
connecting to: mongodb://127.0.0.1:27017
Percona Server for MongoDB server version: v3.6.8-2.0
> use $external
switched to db $external
> db.auth({"mechanism": "PLAIN", "user": "aduser", "pwd": "adpassword", "digestPassword ": false})
1
> use percona
switched to db percona
> db.test.find()
{ "_id" : ObjectId("5c12a47904a287e45fcb580e"), "message" : "Active Directory user success!" }

As you can see above our Active Directory based user was able to authenticate and then change over to the percona database and see the document we stored earlier.

You will notice that our auth() call above is different than the one we used to log in with MongoDB based users. In this case we need to pass in a document with not only the user and password, but also the mechanism to use. We also want to set digestPassword to false.

You can also log in directly from the command line with the following:

mongo percona --host localhost --port 27017 --authenticationMechanism PLAIN --authenticationDatabase \$external --username dduncan --p

There are a couple of things to note here if you’re not used to using the command line to log in:

  1. We place the --password option at the end of the command line and do not provide a password here. This will cause the application to prompt us for a password.
  2. You will also automatically be placed into the percona database, or whatever database name you provide after mongo.
  3. You need to escape the $external database name with a backslash (\) or the terminal will treat $external as an environment variable and you will most likely get an error.

Conclusion

In conclusion, it is easy to connection your Percona Server for MongoDB instance to your corporate Active Directory server. This allows your MongoDB users to use the same credentials to log into MongoDB as they do their corporate email and workstation.


Photo by Steve Halama on Unsplash

Dec
20
2018
--

Benchmark PostgreSQL With Linux HugePages

Benchmarking HugePages and PostgreSQL

Linux kernel provides a wide range of configuration options that can affect performance. It’s all about getting the right configuration for your application and workload. Just like any other database, PostgreSQL relies on the Linux kernel to be optimally configured. Poorly configured parameters can result in poor performance. Therefore, it is important that you benchmark database performance after each tuning session to avoid performance degradation. In one of my previous posts, Tune Linux Kernel Parameters For PostgreSQL Optimization, I described some of the most useful Linux kernel parameters and how those may help you improve database performance. Now I am going to share my benchmark results with you after configuring Linux Huge Page with different PostgreSQL workload. I have performed a comprehensive set of benchmarks for many different PostgreSQL load sizes and different number concurrent clients.

Benchmark Machine

  • Supermicro server:
    • Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
    • 2 sockets / 28 cores / 56 threads
    • Memory: 256GB of RAM
    • Storage: SAMSUNG  SM863 1.9TB Enterprise SSD
    • Filesystem: ext4/xfs
  • OS: Ubuntu 16.04.4, kernel 4.13.0-36-generic
  • PostgreSQL: version 11

Linux Kernel Settings

I have used default kernel settings without any optimization/tuning except for disabling Transparent HugePages. Transparent HugePages are by default enabled, and allocate a page size that may not be recommended for database usage. For databases generally, fixed sized HugePages are needed, which Transparent HugePages do not provide. Hence, disabling this feature and defaulting to classic HugePages is always recommended.

PostgreSQL Settings

I have used consistent PostgreSQL settings for all the benchmarks in order to record different PostgreSQL workloads with different settings of Linux HugePages. Here is the PostgreSQL setting used for all benchmarks:

shared_buffers = '64GB'
work_mem = '1GB'
random_page_cost = '1'
maintenance_work_mem = '2GB'
synchronous_commit = 'on'
seq_page_cost = '1'
max_wal_size = '100GB'
checkpoint_timeout = '10min'
synchronous_commit = 'on'
checkpoint_completion_target = '0.9'
autovacuum_vacuum_scale_factor = '0.4'
effective_cache_size = '200GB'
min_wal_size = '1GB'
wal_compression = 'ON'

Benchmark scheme

In the benchmark, the benchmark scheme plays an important role. All the benchmarks are run three times with thirty minutes duration for each run. I took the median value from these three benchmarks. The benchmarks were carried out using the PostgreSQL benchmarking tool pgbench.  pgbench works on scale factor, with one scale factor being approximately 16MB of workload. 

HugePages

Linux, by default, uses 4K memory pages along with HugePages. BSD has Super Pages, whereas Windows has Large Pages. PostgreSQL has support for HugePages (Linux) only. In cases where there is a high memory usage, smaller page sizes decrease performance. By setting up HugePages, you increase the dedicated memory for the application and therefore reduce the operational overhead that is incurred during allocation/swapping; i.e. you gain performance by using HugePages.

Here is the Hugepage setting when using Hugepage size of 1GB. You can always get this information from /proc.

AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:     100
HugePages_Free:       97
HugePages_Rsvd:       63
HugePages_Surp:        0
Hugepagesize:    1048576 kB

For more detail about HugePages please read my previous blog post.

https://www.percona.com/blog/2018/08/29/tune-linux-kernel-parameters-for-postgresql-optimization/

Generally, HugePages comes in sizes 2MB and 1GB, so it makes sense to use 1GB size instead of the much smaller 2MB size.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-transhuge
https://kerneltalks.com/services/what-is-huge-pages-in-linux/

Benchmark Results

This benchmark shows the overall impact of different sizes of HugePages. The first set of benchmarks was created with the default Linux 4K page size without enabling HugePages. Note that Transparent Hugepages were also disabled, and remained disabled throughout these benchmarks.

Then the second set of benchmarks was performed with 2MB HugePages. Finally, the third set of benchmarks is performed with HugePages set to 1GB in size.

All these benchmarks were executed with PostgreSQL version 11. The sets include a combination of different database sizes and clients. The graph below shows comparative performance results for these benchmarks with TPS (transactions per seconds) on the y-axis, and database size and the number of clients per database size on the x-axis.

 

Clearly, from the graph above, you can see that the performance gain with HugePages increases as the number of clients and the database size increases, as long as the size remains within the pre-allocated shared buffer.

This benchmark shows TPS versus clients. In this case, the database size is set to 48GB. On the y-axis, we have TPS and on the x-axis, we have the number of connected clients. The database size is small enough to fit in the shared buffer, which is set to 64GB.

With HugePages set to 1GB, the higher the number of clients, the higher the comparative performance gain.

The next graph is the same as the one above except for a database size of 96GB. This exceeds the shared buffer size, which is set to 64GB.

 

The key observation here is that the performance with 1GB HugePages improves as the number of clients increases and it eventually gives better performance than 2MB HugePages or the standard 4KB page size.

This benchmark shows the TPS versus database size. In this case, the number of connected clients it set to 32. On the y-axis, we have TPS and on the x-axis, we have database sizes.

As expected, when the database spills over the pre-allocated HugePages, the performance degrades significantly.

Summary

One of my key recommendations is that we must keep Transparent HugePages off. You will see the biggest performance gains when the database fits into the shared buffer with HugePages enabled. Deciding on the size of huge page to use requires a bit of trial and error, but this can potentially lead to a significant TPS gain where the database size is large but remains small enough to fit in the shared buffer.

Dec
20
2018
--

Percona Database Performance Blog 2018 Year in Review: Top Blog Posts

Percona Database Performance Blog

Percona Database Performance BlogLet’s look at some of the most popular Percona Database Performance Blog posts in 2018.

The closing of a year lends itself to looking back. And making lists. With the Percona Database Performance Blog, Percona staff and leadership work hard to provide the open source community with insights, technical support, predictions and metrics around multiple open source database software technologies. We’ve had nearly 4 million visits to the blog in 2018: thank you! We look forward to providing you with even better articles, news and information in 2019.

As 2018 moves into 2019, let’s take a quick look back at some of the most popular posts on the blog this year.

Top 10 Most Read

These posts had the most number of views (working down from the highest):

When Should I Use Amazon Aurora and When Should I use RDS MySQL?

Now that Database-as-a-service (DBaaS) is in high demand, there is one question regarding AWS services that cannot always be answered easily : When should I use Aurora and when RDS MySQL?

About ZFS Performance

ZFS has many very interesting features, but I am a bit tired of hearing negative statements on ZFS performance. It feels a bit like people are telling me “Why do you use InnoDB? I have read that MyISAM is faster.” I found the comparison of InnoDB vs. MyISAM quite interesting, and I’ll use it in this post.

Linux OS Tuning for MySQL Database Performance

In this post we will review the most important Linux settings to adjust for performance tuning and optimization of a MySQL database server. We’ll note how some of the Linux parameter settings used OS tuning may vary according to different system types: physical, virtual or cloud.

A Look at MyRocks Performance

As the MyRocks storage engine (based on the RocksDB key-value store http://rocksdb.org ) is now available as part of Percona Server for MySQL 5.7, I wanted to take a look at how it performs on a relatively high-end server and SSD storage.

How to Restore MySQL Logical Backup at Maximum Speed

The ability to restore MySQL logical backups is a significant part of disaster recovery procedures. It’s a last line of defense.

Why MySQL Stored Procedures, Functions and Triggers Are Bad For Performance

MySQL stored procedures, functions and triggers are tempting constructs for application developers. However, as I discovered, there can be an impact on database performance when using MySQL stored routines. Not being entirely sure of what I was seeing during a customer visit, I set out to create some simple tests to measure the impact of triggers on database performance. The outcome might surprise you.

AMD EPYC Performance Testing… or Don’t get on the wrong side of SystemD

Ever since AMD released their EPYC CPU for servers I wanted to test it, but I did not have the opportunity until recently, when Packet.net started offering bare metal servers for a reasonable price. So I started a couple of instances to test Percona Server for MySQL under this CPU. In this benchmark, I discovered some interesting discrepancies in performance between  AMD and Intel CPUs when running under systemd.

Tuning PostgreSQL Database Parameters to Optimize Performance

Out of the box, the default PostgreSQL configuration is not tuned for any particular workload. Default values are set to ensure that PostgreSQL runs everywhere, with the least resources it can consume and so that it doesn’t cause any vulnerabilities. It is primarily the responsibility of the database administrator or developer to tune PostgreSQL according to their system’s workload. In this blog, we will establish basic guidelines for setting PostgreSQL database parameters to improve database performance according to workload.

Using AWS EC2 instance store vs EBS for MySQL: how to increase performance and decrease cost

If you are using large EBS GP2 volumes for MySQL (i.e. 10TB+) on AWS EC2, you can increase performance and save a significant amount of money by moving to local SSD (NVMe) instance storage. Interested? Then read on for a more detailed examination of how to achieve cost-benefits and increase performance from this implementation.

Why You Should Avoid Using “CREATE TABLE AS SELECT” Statement

In this blog post, I’ll provide an explanation why you should avoid using the CREATE TABLE AS SELECT statement. The SQL statement “create table <table_name> as select …” is used to create a normal or temporary table and materialize the result of the select. Some applications use this construct to create a copy of the table. This is one statement that will do all the work, so you do not need to create a table structure or use another statement to copy the structure.

Honorable Mention:

Is Serverless Just a New Word for Cloud-Based?

Top 10 Most Commented

These posts generated some healthy discussions (not surprisingly, this list overlaps with the first):

Posts Worth Revisiting

Don’t miss these great posts that have excellent information on important topics:

Have a great end of the year celebration, and we look forward to providing more great blog posts in 2019.

Dec
20
2018
--

Cinven acquires One.com, one of Europe’s biggest hosting providers with 1.5M customers

One of the biggest providers of domain names and web hosting in Europe is changing hands today. One.com, which has around 1.5 million customers mainly across the north of the region, has been sold by private equity firm Accel-KKR to Cinven, another PE player that focuses on investments in Europe.

Terms of the deal are not being disclosed, but as a rough guide, Cinven once owned and sold another European hosting provider of comparable size: it acquired Host Europe Group in 2013 for $668 million and then sold it in 2016 for $1.8 billion to GoDaddy two years ago almost to the day. At the time of the sale, Host Europe Group also had about 1.5 million customers.

One.com and its business segment represent a significant, if not wildly evolving, part of the tech landscape: for as long as businesses and consumers continue to use the web, there will be a need for companies who sell and host domain names and provide services around that.

With a catchy domain name of its own, One.com has been riding the wave of that solidity of purpose for several years already. KKR-Accel says that organic growth at the company has been accelerating at a rate of 20 percent and that revenues under its four-year ownership doubled to €60 million ($69 million) with profitability growing 50x on a marketing pitch in which it positions itself as the ‘budget’ option to businesses.

“The vision of One.com since its founding has been to deliver value-added and easy-to-use solutions to small- and medium-sized businesses and prosumers,” said Jacob Jensen, Founder and CEO of One.com, in a statement. He is staying on to continue leading the company.

Cinven says it is interested in growth the business by way of acquisition, specifically: “There are opportunities to accelerate the growth of the business organically and through acquisition.”

In other words, expect some consolidation moves in the future where some of the smaller providers in Europe potentially get gobbled up to create a bigger entity with better economies of scale. That’s needed not just because GoDaddy has ramped up its presence here, but because the likes of Amazon has only grown in stature and provides a number of other services to users to make its offerings more sticky.

“We are very excited to invest in One.com alongside Jacob. It is a high quality business with an attractive brand and scalable technology platform, operating in a market with structural growth drivers,” said Thomas Railhac, Partner at Cinven, in a statement. “This is a subsector we know well through Cinven’s successful investment in HEG in Fund 5, continuing to invest in both the organic growth story and targeted acquisitions.”

Dec
19
2018
--

Crew, a Workplace and Slack messaging rival for shift workers, raises $35M, adds enterprise version

When it comes to shift workers communicating with each other in the workplace when they are not face-to-face, gone are the days of cork announcement boards. Now, the messaging app is the medium, and today one of the startups tackling that opportunity in a unique way has raised a round of funding to get to the next stage of growth.

Crew, a chat app that specifically targets businesses that employ shift workers who do not typically sit at computers all day, has now raised $35 million in Series C funding from DAG Ventures, Tenaya Capital, and previous backers Greylock Partners, Sequoia Capital, Harrison Metal Capital and Aspect Ventures. With the funding news, it’s also announcing the launch of a new feature called Crew Enterprise, which helps businesses better manage messaging across large groups of these workers.

The funding and new product come on the heels of the company hitting 25,000 organizations using its service — many of them multi-store retailers with an emphasis in the food industry, household names like Domino’s Pizza and Burger King — with some strong engagement. Its users are together sending some 25 million messages or responses to other messages each week, on average six times per day per user, with more than 55 percent of its whole user base logging in on an average day.

There are quite a lot of messaging apps out in the market today, but the majority of them are aimed at so-called knowledge workers, people who might be using a number of apps throughout their day, who often sit at desks and use computers alongside their phones and tablets. Crew takes a different approach in that it targets the vast swathe of other workers in the job market and their priorities.

As it turns out, co-founder and CEO Danny Leffel tells me that those priorities are focused around a few specific things that are not the same as those for the other employment sector. One is to get the latest shift schedules for work, especially when they are not at work; another is to be able to swap those shifts when they need to; and a third, largely coming from the management end, is to make sure that everything gets communicated to the staff even when they are not in for work to attend a staff meeting.

“Some of the older practices feel like versions of a Rube Goldberg machine,” he said. “The stories we hear are quite insane.” Shift schedules, he said, are an example. “Lots of workplaces have rules, where you can’t call in to check the schedule because it causes employees to come off the floor. One hotel manager told us he couldn’t hold staff meetings with everyone there because he runs a 24/7 workplace so some people would have to come in especially. One store GM from a supermarket chain told us that the whole store has only one email address, so when an announcement goes out, the GM prints that and hands it to everyone. And the problems just compound when you talk to them.”

Crew is by no means the only business internal messaging service that is aiming to provide a product specifically for shift workers. Workplace, Facebook’s own take on enterprise communications, has also positioned itself as a platform for “every worker,” and has snagged a clutch of huge clients such as Walmart (2.2 million employees globally) and Starbucks (254,000) to fill out that vision.

Leffel, however, paints a sightly different picture of how this is playing out, since in many cases even when a company has been “won” as a global customer that hasn’t translated to a global roll out.

“Starbucks is theoretically using Workplace, but it’s been deployed only to managers,” he said. “We have almost 1,000 Starbucks locations using Crew. We knew we had a huge presence there, and we were worried when Facebook won them, but we haven’t seen even a dent in our business so far.”

Leffel has had previous some experience of getting into the ring with Facebook — although it hasn’t ended with him the winner. His previous startup, Yardsellr, positioned itself as the “eBay of Facebook,” working as a layer on top of the big social network for people to sell items. It died a death in 2013, when Facebook took a less friendly turn to Yardsellr using Facebook’s social graph to grow its own business (it was a time when it was cutting off apps from Zynga for similar reasons). Today, Facebook itself owns the experience of selling on its platform via Marketplace.

Crew seems to have found a strong foothold among enterprises in terms of its usefulness, not just use, which is one sign of how it might have more staying power.

survey it conducted among 50,000 of its users found that 63 percent of leaders who use Crew report fewer missed shifts and 70 percent see increased motivation on their team. Crew worked out that among respondents, it is generating time savings of four or more hours per week for 93 percent of surveyed managers. And because of better communication, people are working faster when handing off things to each other on the front line, with a Domino’s Pizza franchisee sped up delivery punctuality by 23 percent as one example. (The company offers services on three tiers, ranging from free for small teams, Pro at $10 per month per location, to Enterprise priced on negotiation.)

Crew’s new enterprise tier is aiming to take the company to the next step. Today, Leffel says that a lot of its customers are buying on a location-by-location basis. The idea with Crew Enterprise is that larger organizations will be able to provide a more unified experience across all of those locations (not to mention pay more for the functionality). Managers can use the service to message out details about promotions, and they have a better ability to manage conversations across the platform and also get more feedback from people who are directly interacting with customers. Meanwhile, admins also gain better ability to manage compliance.

If some of this sounds familiar, it’s not just because Workplace is the only one who is also targeting the same users. Dynamic Signal and Zinc (formerly Cotap) are two other startups that are also trying to provide better messaging-based communications to more than just white-collar knowledge workers. Crew will have its work cut out for it, but there is a lot of room for now for multiple players.

“We are seeing a shift in the marketplace, going from absolutely don’t use your phone at work to don’t use it when customers are present,” Leffel said of the opportunity. “Some have started to change the rules to allow workers to use their own phones to perform price checks. We are solving for this evolving workflow.”

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com