Nov
14
2019
--

Adobe announces GA of customer data platform

The customer data platform (CDP) is the newest tool in the customer experience arsenal as big companies try to help customers deal with data coming from multiple channels. Today, Adobe announced the general availability of its CDP.

The CDP is like a central data warehouse for all the information you have on a single customer. This crosses channels like web, email, text, chat and brick and mortar in-person visits, as well as systems like CRM, e-commerce and point of sale. The idea is to pull all of this data together into a single record to help companies have a deep understanding of the customer at an extremely detailed level. They then hope to leverage that information to deliver highly customized cross-channel experiences.

The idea is to take all of this information and give marketers the tools they need to take advantage of it. “We want to make sure we create an offering that marketers can leverage and makes use of all of that goodness that’s living within Adobe Experience platform,” Nina Caruso, product marketing manager for Adobe Audience Manager, explained.

She said that would involve packaging and presenting the data in such a way to make it easier for marketers to consume, such as dashboards to deliver the data they want to see, while taking advantage of artificial intelligence and machine learning under the hood to help them find the data to populate the dashboards without having to do the heavy lifting.

Beyond that, having access to real-time streaming data in one place under the umbrella of the Adobe Experience Platform should enable marketers to create much more precise market segments. “Part of real-time CDP will be building productized primo maintained integrations for marketers to be able to leverage, so that they can take segmentations and audiences that they’ve built into campaigns and use those across different channels to provide a consistent customer experience across that journey life cycle,” Caruso said.

As you can imagine, bringing all of this information together, while providing a platform for customization for the customer, raises all kinds of security and privacy red flags at the same time. This is especially true in light of GDPR and the upcoming California privacy law. Companies need to be able to enforce data usage rules across the platform.

To that end, the company also announced the availability of Adobe Experience Platform Data Governance, which helps companies define a set of rules around the data usage. This involves “frameworks that help [customers] enforce data usage policies and facilitate the proper use of their data to comply with regulations, obligations and restrictions associated with various data sets,” according to the company.

“We want to make sure that we offer our customers the controls in place to make sure that they have the ability to appropriately govern their data, especially within the evolving landscape that we’re all living in when it comes to privacy and different policies,” Caruso said.

These tools are now available to Adobe customers.

Nov
14
2019
--

Salesforce announces it’s moving Marketing Cloud to Microsoft Azure

In the world of enterprise software, there are often strange bedfellows. Just yesterday, Salesforce announced a significant partnership with AWS around the Cloud Information Model. This morning, it announced it was moving its Marketing Cloud to Microsoft Azure. That’s the way that enterprise partnerships shimmy and shake sometimes.

The companies also announced they were partnering around Microsoft Teams, integrating Teams with Salesforce Sales Cloud and Service Cloud.

Salesforce plans to move Marketing Cloud, which has been running in its own data centers, to Microsoft Azure in the coming months, although the exact migration plan timeline is not clear yet. This is a big deal for Microsoft, which competes fiercely with AWS for customers. AWS is the clear market leader in the space, but Microsoft has been a strong second for some time now, and bringing Salesforce on board as a customer is certainly a quality reference for the company.

Brent Leary, founder at CRM Essentials, who has been watching the market for many years, says the partnership says a lot about Microsoft’s approach to business today, and that it’s willing to partner broadly to achieve its goals. “I think the bigger news is that Salesforce chose to go deeper with Microsoft over Amazon, and that Microsoft doesn’t fear strengthening Salesforce at the potential expense of Dynamics 365 (its CRM tool), mainly because their biggest growth driver is Azure,” Leary told TechCrunch.

Microsoft and Salesforce have always had a complex relationship. In the Steve Ballmer era, they traded dueling lawsuits over their CRM products. Later, Satya Nadella kindled a friendship of sorts by appearing at Dreamforce in 2015. The relationship has ebbed and flowed since, but with this announcement, it appears the frenemies are closer to friends than enemies again.

Let’s not forget though, that it was just yesterday that Salesforce announced a partnership with AWS around the Cloud Information Model, one that competes directly with a different partnership between Adobe, Microsoft and SAP; or that just last year Salesforce announced a significant partnership with AWS around data integration.

These kinds of conflicting deals are confusing, but they show that in today’s connected cloud world, companies that will compete hard with one another in one part of the market may still be willing to partner in other parts when it makes sense for both parties and for customers. That appears to be the case with today’s announcement from these companies.

Nov
14
2019
--

Moveworks snags $75M Series B to resolve help desk tickets with AI

Moveworks, a startup using AI to help resolve help desk tickets in an automated fashion, announced a $75 million Series B investment today.

The round was led by Iconiq Capital, Kleiner Perkins and Sapphire Ventures. Existing investors Lightspeed Venture Partners, Bain Capital Ventures and Comerica Bank also participated. The round also included a personal investment from John W. Thompson, a partner at LightSpeed Venture Partners and chairman at Microsoft. Today’s investment brings the total raised to $105 million, according to the company.

That’s a lot of money for an early-stage company, but CEO and co-founder Bhavin Shah says his company is solving a common problem using AI. “Moveworks is a machine learning platform that uses natural language understanding to take tickets that are submitted by employees every day to their IT teams for stuff they need, and we understand [the content of the tickets], interpret them, and then we take the actions to resolve them [automatically],” Shah explained.

He said the company decided to focus on help desk tickets because they saw data when they were forming the company that suggested a common set of questions, and that would make it easier to interpret and resolve these issues. In fact, they are currently able to resolve 25-40% of all tickets autonomously.

He says this should lead to greater user satisfaction because some of their problems can be resolved immediately, even when IT personnel aren’t around to help. Instead of filing a ticket and waiting for an answer, Moveworks can provide the answer, at least part of the time, without human intervention.

Aditya Agrawal, a partner at Iconiq, says that the company really captured his attention. “Moveworks is not just transforming IT operations, they are building a more modern and enlightened way to work. They’ve built a platform that simplifies and streamlines every interaction between employees and IT, enabling both to focus on what matters,” he said in a statement.

The company was founded in 2016, and in the early days was only resolving 2% of the tickets autonomously, so it has seen major improvement. It already has 115 employees and dozens of customers (although Shah didn’t want to provide an exact number).

Nov
14
2019
--

Eigen nabs $37M to help banks and others parse huge documents using natural language and ‘small data’

One of the bigger trends in enterprise software has been the emergence of startups building tools to make the benefits of artificial intelligence technology more accessible to non-tech companies. Today, one that has built a platform to apply the power of machine learning and natural language processing to massive documents of unstructured data has closed a round of funding as it finds strong demand for its approach.

Eigen Technologies, a London-based startup whose machine learning engine helps banks and other businesses that need to extract information and insights from large and complex documents like contracts, is today announcing that it has raised $37 million in funding, a Series B that values the company at around $150 million – $180 million.

The round was led by Lakestar and Dawn Capital, with Temasek and Goldman Sachs Growth Equity (which co-led its Series A) also participating. Eigen has now raised $55 million in total.

Eigen today is working primarily in the financial sector — its offices are smack in the middle of The City, London’s financial center — but the plan is to use the funding to continue expanding the scope of the platform to cover other verticals such as insurance and healthcare, two other big areas that deal in large, wordy documentation that is often inconsistent in how its presented, full of essential fine print, and typically a strain on an organisation’s resources to be handled correctly — and is often a disaster if it is not.

The focus up to now on banks and other financial businesses has had a lot of traction. It says its customer base now includes 25% of the world’s G-SIB institutions (that is, the world’s biggest banks), along with others that work closely with them, like Allen & Overy and Deloitte. Since June 2018 (when it closed its Series A round), Eigen has seen recurring revenues grow sixfold with headcount — mostly data scientists and engineers — double. While Eigen doesn’t disclose specific financials, you can see the growth direction that contributed to the company’s valuation.

The basic idea behind Eigen is that it focuses what co-founder and CEO Lewis Liu describes as “small data.” The company has devised a way to “teach” an AI to read a specific kind of document — say, a loan contract — by looking at a couple of examples and training on these. The whole process is relatively easy to do for a non-technical person: you figure out what you want to look for and analyse, find the examples using basic search in two or three documents and create the template, which can then be used across hundreds or thousands of the same kind of documents (in this case, a loan contract).

Eigen’s work is notable for two reasons. First, typically machine learning and training and AI requires hundreds, thousands, tens of thousands of examples to “teach” a system before it can make decisions that you hope will mimic those of a human. Eigen requires a couple of examples (hence the “small data” approach).

Second, an industry like finance has many pieces of sensitive data (either because it’s personal data, or because it’s proprietary to a company and its business), and so there is an ongoing issue of working with AI companies that want to “anonymise” and ingest that data. Companies simply don’t want to do that. Eigen’s system essentially only works on what a company provides, and that stays with the company.

Eigen was founded in 2014 by Dr. Lewis Z. Liu (CEO) and Jonathan Feuer (a managing partner at CVC Capital Partners, who is the company’s chairman), but its earliest origins go back 15 years earlier, when Liu — a first-generation immigrant who grew up in the U.S. — was working as a “data-entry monkey” (his words) at a tire manufacturing plant in New Jersey, where he lived, ahead of starting university at Harvard.

A natural computing whiz who found himself building his own games when his parents refused to buy him a games console, he figured out that the many pages of printouts he was reading and re-entering into a different computing system could be sped up with a computer program linking up the two. “I put myself out of a job,” he joked.

His educational life epitomises the kind of lateral thinking that often produces the most interesting ideas. Liu went on to Harvard to study not computer science, but physics and art. Doing a double major required working on a thesis that merged the two disciplines together, and Liu built “electrodynamic equations that composed graphical structures on the fly” — basically generating art using algorithms — which he then turned into a “Turing test” to see if people could detect pixelated actual work with that of his program. Distill this, and Liu was still thinking about patterns in analog material that could be re-created using math.

Then came years at McKinsey in London (how he arrived on these shores) during the financial crisis where the results of people either intentionally or mistakenly overlooking crucial text-based data produced stark and catastrophic results. “I would say the problem that we eventually started to solve for at Eigen became tangible,” Liu said.

Then came a physics PhD at Oxford where Liu worked on X-ray lasers that could be used to decrease the complexity and cost of making microchips, cancer treatments and other applications.

While Eigen doesn’t actually use lasers, some of the mathematical equations that Liu came up with for these have also become a part of Eigen’s approach.

“The whole idea [for my PhD] was, ‘how do we make this cheaper and more scalable?,’ ” he said. “We built a new class of X-ray laser apparatus, and we realised the same equations could be used in pattern matching algorithms, specifically around sequential patterns. And out of that, and my existing corporate relationships, that’s how Eigen started.”

Five years on, Eigen has added a lot more into the platform beyond what came from Liu’s original ideas. There are more data scientists and engineers building the engine around the basic idea, and customising it to work with more sectors beyond finance. 

There are a number of AI companies building tools for non-technical business end-users, and one of the areas that comes close to what Eigen is doing is robotic process automation, or RPA. Liu notes that while this is an important area, it’s more about reading forms more readily and providing insights to those. The focus of Eigen is more on unstructured data, and the ability to parse it quickly and securely using just a few samples.

Liu points to companies like IBM (with Watson) as general competitors, while startups like Luminance is another taking a similar approach to Eigen by addressing the issue of parsing unstructured data in a specific sector (in its case, currently, the legal profession).

Stephen Nundy, a partner and the CTO of Lakestar, said that he first came into contact with Eigen when he was at Goldman Sachs, where he was a managing director overseeing technology, and the bank engaged it for work.

“To see what these guys can deliver, it’s to be applauded,” he said. “They’re not just picking out names and addresses. We’re talking deep, semantic understanding. Other vendors are trying to be everything to everybody, but Eigen has found market fit in financial services use cases, and it stands up against the competition. You can see when a winner is breaking away from the pack and it’s a great signal for the future.”

Nov
13
2019
--

Messaging app Wire confirms $8.2M raise, responds to privacy concerns after moving holding company to the US

Big changes are afoot for Wire, an enterprise-focused end-to-end encrypted messaging app and service that advertises itself as “the most secure collaboration platform”. In February, Wire quietly raised $8.2 million from Morpheus Ventures and others, we’ve confirmed — the first funding amount it has ever disclosed — and alongside that external financing, it moved its holding company in the same month to the US from Luxembourg, a switch that Wire’s CEO Morten Brogger described in an interview as “simple and pragmatic.”

He also said that Wire is planning to introduce a freemium tier to its existing consumer service — which itself has half a million users — while working on a larger round of funding to fuel more growth of its enterprise business — a key reason for moving to the US, he added: There is more money to be raised there.

“We knew we needed this funding and additional to support continued growth. We made the decision that at some point in time it will be easier to get funding in North America, where there’s six times the amount of venture capital,” he said.

While Wire has moved its holding company to the US, it is keeping the rest of its operations as is. Customers are licensed and serviced from Wire Switzerland; the software development team is in Berlin, Germany; and hosting remains in Europe.

The news of Wire’s US move and the basics of its February funding — sans value, date or backers — came out this week via a blog post that raises questions about whether a company that trades on the idea of data privacy should itself be more transparent about its activities.

Specifically, the changes to Wire’s financing and legal structure were only communicated to users when news started to leak out, which brings up questions not just about transparency, but about the state of Wire’s privacy policy, given the company’s holding company now being on US soil.

It was an issue picked up and amplified by NSA whistleblower Edward Snowden . Via Twitter, he described the move to the US as “not appropriate for a company claiming to provide a secure messenger — claims a large number of human rights defenders relied on.”

“There was no change in control and [the move was] very tactical [because of fundraising],” Brogger said about the company’s decision not to communicate the move, adding that the company had never talked about funding in the past, either. “Our evaluation was that this was not necessary. Was it right or wrong? I don’t know.”

The other key question is whether Wire’s shift to the US puts users’ data at risk — a question that Brogger claims is straightforward to answer: “We are in Switzerland, which has the best privacy laws in the world” — it’s subject to Europe’s General Data Protection Regulation framework (GDPR) on top of its own local laws — “and Wire now belongs to a new group holding, but there no change in control.”

In its blog post published in the wake of blowback from privacy advocates, Wire also claims it “stands by its mission to best protect communication data with state-of-the-art technology and practice” — listing several items in its defence:

  • All source code has been and will be available for inspection on GitHub (github.com/wireapp).
  • All communication through Wire is secured with end-to-end encryption — messages, conference calls, files. The decryption keys are only stored on user devices, not on our servers. It also gives companies the option to deploy their own instances of Wire in their own data centers.
  • Wire has started working on a federated protocol to connect on-premise installations and make messaging and collaboration more ubiquitous.
  • Wire believes that data protection is best achieved through state-of-the-art encryption and continues to innovate in that space with Messaging Layer Security (MLS).

But where data privacy and US law are concerned, it’s complicated. Snowden famously leaked scores of classified documents disclosing the extent of US government mass surveillance programs in 2013, including how data-harvesting was embedded in US-based messaging and technology platforms.

Six years on, the political and legal ramifications of that disclosure are still playing out — with a key judgement pending from Europe’s top court which could yet unseat the current data transfer arrangement between the EU and the US.

Privacy versus security

Wire launched at a time when interest in messaging apps was at a high watermark. The company made its debut in the middle of February 2014, and it was only one week later that Facebook acquired WhatsApp for the princely sum of $19 billion.

We described Wire’s primary selling point at the time as a “reimagining of how a communications tool like Skype should operate had it been built today” rather than in in 2003. That meant encryption and privacy protection, but also better audio tools and file compression and more.

It was a pitch that seemed especially compelling considering the background of the company. Skype co-founder Janus Friis and funds connected to him were the startup’s first backers (and they remain the largest shareholders);Wire was co-founded in by Skype alums Jonathan Christensen and Alan Duric (former no longer with the company, latter is its CTO); and even new investor Morpheus has Skype roots.

Yet even with that Skype pedigree, the strategy faced a big challenge.

“The consumer messaging market is lost to the Facebooks of the world, which dominate it,” Brogger said today. “However, we made a clear insight, which is the core strength of Wire: security and privacy.”

That, combined with trend around the consumerization of IT that’s brought new tools to business users, is what led Wire to the enterprise market in 2017 — a shift that’s seen it pick up a number of big names among its 700 enterprise customers, including Fortum, Aon, EY and SoftBank Robotics.

But fast forward to today, and it seems that even as security and privacy are two sides of the same coin, it may not be so simple when deciding what to optimise in terms of features and future development, which is part of the question now and what critics are concerned with.

“Wire was always for profit and planned to follow the typical venture backed route of raising rounds to accelerate growth,” one source familiar with the company told us. “However, it took time to find its niche (B2B, enterprise secure comms).

“It needed money to keep the operations going and growing. [But] the new CEO, who joined late 2017, didn’t really care about the free users, and the way I read it now, the transformation is complete: ‘If Wire works for you, fine, but we don’t really care about what you think about our ownership or funding structure as our corporate clients care about security, not about privacy.’”

And that is the message you get from Brogger, too, who describes individual consumers as “not part of our strategy”, but also not entirely removed from it, either, as the focus shifts to enterprises and their security needs.

Brogger said there are still half a million individuals on the platform, and they will come up with ways to continue to serve them under the same privacy policies and with the same kind of service as the enterprise users. “We want to give them all the same features with no limits,” he added. “We are looking to switch it into a freemium model.”

On the other side, “We are having a lot of inbound requests on how Wire can replace Skype for Business,” he said. “We are the only one who can do that with our level of security. It’s become a very interesting journey and we are super excited.”

Part of the company’s push into enterprise has also seen it make a number of hires. This has included bringing in two former Huddle C-suite execs, Brogger as CEO and Rasmus Holst as chief revenue officer — a bench that Wire expanded this week with three new hires from three other B2B businesses: a VP of EMEA sales from New Relic, a VP of finance from Contentful; and a VP of Americas sales from Xeebi.

Such growth comes with a price-tag attached to it, clearly. Which is why Wire is opening itself to more funding and more exposure in the US, but also more scrutiny and questions from those who counted on its services before the change.

Brogger said inbound interest has been strong and he expects the startup’s next round to close in the next two to three months.

Nov
13
2019
--

AWS, Salesforce join forces with Linux Foundation on Cloud Information Model

Last year, Adobe, SAP and Microsoft came together and formed the Open Data Initiative. Not to be outdone, this week, AWS, Salesforce and Genesys, in partnership with The Linux Foundation, announced the Cloud Information Model.

The two competing data models have a lot in common. They are both about bringing together data and applying a common open model to it. The idea is to allow for data interoperability across products in the partnership without a lot of heavy lifting, a common problem for users of these big companies’ software.

Jim Zemlin, executive director at The Linux Foundation, says this project provides a neutral home for the Cloud Information model, where a community can work on the problem. “This allows for anyone across the community to collaborate and provide contributions under a central governance model. It paves the way for full community-wide engagement in data interoperability efforts and standards development, while rapidly increasing adoption rate of the community,” Zemlin explained in a statement.

Each of the companies in the initial partnership is using the model in different ways. AWS will use it in conjunction with its AWS Lake Formation tool to help customers move, catalog, store and clean data from a variety of data sources, while Genesys customers can use its cloud and AI products to communicate across a variety of channels.

Patrick Stokes from Salesforce says his company is using the Cloud Information Model as the underlying data model for his company’s Customer 360 platform of products. “We’re super excited to announce that we’ve joined together with a few partners — AWS, Genesys and The Linux Foundation — to actually open-source that data model,” Stokes told TechCrunch.

Of course, now we have two competing “open” data models, and it’s going to create some friction until the two competing projects find a way to come together. The fact is that many companies use tools from each of these companies, and if there continues to be these competing approaches, it’s going to defeat the purpose of creating these initiatives in the first place.

As Satya Nadella said in 2015, “It is incumbent upon us, especially those of us who are platform vendors to partner broadly to solve real pain points our customers have.” If that’s the case, having competing models is not really achieving that.

Nov
13
2019
--

Freshworks raises $150M Series H on $3.5B valuation

Freshworks, a company that makes a variety of business software tools, from CRM to help-desk software, announced a $150 million Series H investment today from Sequoia Capital, CapitalG (formerly Google Capital) and Accel on a hefty $3.5 billion valuation. The late-stage startup has raised almost $400 million, according to Crunchbase data.

The company has been building an enterprise SaaS platform to give customers a set of integrated business tools, but CEO and co-founder Girish Mathrubootham says they will be investing part of this money in R&D to keep building out the platform.

To that end, the company also announced today a new unified data platform called the “Customer-for-Life Cloud” that runs across all of its tools. “We are actually investing in really bringing all of this together to create the “Customer-for-Life Cloud,” which is how you take marketing, sales, support and customer success — all of the aspects of a customer across the entire life cycle journey and bring them to a common data model where a business that is using Freshworks can see the entire life cycle of the customer,” Mathrubootham explained.

While Mathrubootham was not ready to commit to an IPO, he said they are in the process of hiring a CFO and are looking ahead to one day becoming a public company. “We don’t have a definite timeline. We want to go public at the right time. We are making sure that as a company that we are ready with the right processes and teams and predictability in the business,” he said.

In addition, he says he will continue to look for good acquisition targets, and having this money in the bank will help the company fill in gaps in the product set should the right opportunity arise. “We don’t generally acquire revenue, but we are looking for good technology teams both in terms of talent, as well as technology that would help give us a jumpstart in terms of go-to-market.” It hasn’t been afraid to target small companies in the past, having acquired 12 already.

Freshworks, which launched in 2010, has almost 2,500 employees, a number that’s sure to go up with this new investment. It has 250,00 customers worldwide, including almost 40,000 paying customers. These including Bridgestone Tires, Honda, Hugo Boss, Toshiba and Cisco.

Nov
13
2019
--

Atlassian expands Jira Service Desk beyond IT teams

Atlassian today announced a set of new templates and workflows for Jira Service Desk that were purpose-built for HR, legal and facilities teams. Service Desk started six years ago as a version of Jira that was mostly meant for IT departments. Atlassian, however, found that other teams inside the companies that adopted it started to use it as well, including various teams at Twitter and Airbnb, for example. With today’s update, it’s now making it easier for these teams, at least in legal, HR and facilities, to get started with Jira Service Desk without having to customize the product themselves.

“Over the last six years, one of the observations that we’ve made was that we need to provide really good services — the idea that we can provide great services to employees is really something that is really on the rise,” said Edwin Wong, the head of the company’s IT products. “I think in the past, maybe we were a bit more forgiving in terms of what employees expected from services departments. But today you’re just so used to great experiences in your consumer life and when you come to work, you expect the same.”

But lots of service teams, he argues, didn’t have the tools to provide this experience, yet they were looking for tools to streamline their workflows (think onboarding for HR teams, for example) and to move from manual processes to something more automated and modern. Jira was already flexible enough to allow them to do this, but the new set of templates now codifies these processes for them.

Wong stressed this isn’t just about tracking but also managing work across teams and providing them a more centralized hub for information. “One of the big challenges that we’ve seen from many of the customers that we’ve spoken to is the challenge of just figuring out where to go when you want something,” he said. “When I have a new employee, where do I go to ask for a new laptop? Is that the same process as telling my facilities teams that perhaps there is an issue with a bathroom?”

Atlassian is starting with these three templates because that’s where it saw the most immediate need. Over time, I’m sure we’ll see the company get into other verticals as well.

Nov
13
2019
--

Mirantis acquires Docker Enterprise

Mirantis today announced that it has acquired Docker’s Enterprise business and team. Docker Enterprise was very much the heart of Docker’s product lineup, so this sale leaves Docker as a shell of its former, high-flying unicorn self. Docker itself, which installed a new CEO earlier this year, says it will continue to focus on tools that will advance developers’ workflows. Mirantis will keep the Docker Enterprise brand alive, though, which will surely not create any confusion.

With this deal, Mirantis is acquiring Docker Enterprise Technology Platform and all associated IP: Docker Enterprise Engine, Docker Trusted Registry, Docker Unified Control Plane and Docker CLI. It will also inherit all Docker Enterprise customers and contracts, as well as its strategic technology alliances and partner programs. Docker and Mirantis say they will both continue to work on the Docker platform’s open-source pieces.

The companies did not disclose the price of the acquisition, but it’s surely nowhere near Docker’s valuation during any of its last funding rounds. Indeed, it’s no secret that Docker’s fortunes changed quite a bit over the years, from leading the container revolution to becoming somewhat of an afterthought after Google open-sourced Kubernetes and the rest of the industry coalesced around it. It still had a healthy enterprise business, though, with plenty of large customers among the large enterprises. The company says about a third of Fortune 100 and a fifth of Global 500 companies use Docker Enterprise, which is a statistic most companies would love to be able to highlight — and which makes this sale a bit puzzling from Docker’s side, unless the company assumed that few of these customers were going to continue to bet on its technology.

Update: for reasons only known to Docker’s communications team, we weren’t told about this beforehand, but the company also today announced that it has raised a $35 million funding round from Benchmark. This doesn’t change the overall gist of the story below, but it does highlight the company’s new direction.

Here is what Docker itself had to say. “Docker is ushering in a new era with a return to our roots by focusing on advancing developers’ workflows when building, sharing and running modern applications. As part of this refocus, Mirantis announced it has acquired the Docker Enterprise platform business,” Docker said in a statement when asked about this change. “Moving forward, we will expand Docker Desktop and Docker Hub’s roles in the developer workflow for modern apps. Specifically, we are investing in expanding our cloud services to enable developers to quickly discover technologies for use when building applications, to easily share these apps with teammates and the community, and to run apps frictionlessly on any Kubernetes endpoint, whether locally or in the cloud.”

Mirantis itself, too, went through its ups and downs. While it started as a well-funded OpenStack distribution, today’s Mirantis focuses on offering a Kubernetes-centric on-premises cloud platform and application delivery. As the company’s CEO Adrian Ionel told me ahead of today’s announcement, today is possibly the most important day for the company.

So what will Mirantis do with Docker Enterprise? “Docker Enterprise is absolutely aligned and an accelerator of the direction that we were already on,” Ionel told me. “We were very much moving towards Kubernetes and containers aimed at multi-cloud and hybrid and edge use cases, with these goals to deliver a consistent experience to developers on any infrastructure anywhere — public clouds, hybrid clouds, multi-cloud and edge use cases — and make it very easy, on-demand, and remove any operational concerns or burdens for developers or infrastructure owners.”

Mirantis previously had about 450 employees. With this acquisition, it gains another 300 former Docker employees that it needs to integrate into its organization. Docker’s field marketing and sales teams will remain separate for some time, though, Ionel said, before they will be integrated. “Our most important goal is to create no disruptions for customers,” he noted. “So we’ll maintain an excellent customer experience, while at the same time bringing the teams together.”

This also means that for current Docker Enterprise customers, nothing will change in the near future. Mirantis says that it will accelerate the development of the product and merge its Kubernetes and lifecycle management technology into it. Over time, it will also offer a managed services solutions for Docker Enterprise.

While there is already some overlap between Mirantis’ and Docker Enterprise’s customer base, Mirantis will pick up about 700 new enterprise customers with this acquisition.

With this, Ionel argues, Mirantis is positioned to go up against large players like VMware and IBM/Red Hat. “We are the one real cloud-native player with meaningful scale to provide an alternative to them without lock-in into a legacy or existing technology stack.”

While this is clearly a day the Mirantis team is celebrating, it’s hard not to look at this as the end of an era for Docker, too. The company says it will share more about its future plans today, but didn’t make any spokespeople available ahead of this announcement.

Nov
12
2019
--

Watch Out for Disk I/O Performance Issues when Running EXT4

Performance Issues When Running EXT4

Recently, at Percona Live Europe 2019, Dimitri Kravchuk from Oracle mentioned that he observed some unclear drop in performance for MySQL on an ext4 filesystem with the latest Linux kernels. I decided to check this case out on my side and found out that indeed, starting from linux kernel 4.9, there are some cases with notable (up to 2x) performance drops for ext4 filesystem in direct i/o mode.

So what’s wrong with ext4? It started in 2016 from the patch that was pushed to kernel 4.9: “ext4: Allow parallel DIO reads”. The purpose of that patch was to help to improve read scalability in direct i/o mode. However, along with improvements in pure read workloads, it also introduced regression in intense mixed random read/write scenarios. And it’s quite weird, but this issue had not been noticed for 3 years. Only this summer was performance regression reported and discussed in LKML. As a result of this discussion, there is an attempt to fix it, but from my current understanding that fix will be pushed only to upcoming 5.4/5.5 kernels. Below I will describe what this regression looks like, how it affects MySQL workloads, and what workarounds we can apply to mitigate this issue.

ext4 Performance Regression

Let’s start by defining the scope of this ext4 performance regression. It will only have an impact if the setup/workload meets following conditions:
– fast ssd/nvme
– linux kernel>=4.9
– files resides on ext4 file system
– files opened with O_DIRECT flag
– at least some I/O should be synchronous

In the original report to LKML, the issue was observed/reproduced with a mixed random read/write scenario with sync I/O and O_DIRECT. But how do these factors relate to MySQL? The only files opened by InnoDB in O_DIRECT mode are tablespaces (*.ibd files), and I/O pattern for tablespaces consists of following operations:

– reads ibd data in synchronous mode
– writes ibd data in asynchronous mode
– posix_allocate to extend tablespace file followed by a synchronous write
– fsync

There are also extra I/O from WAL log files:

– writes data to log files in synchronous mode
– fsync

So in the case of InnoDB tablespaces that are opened with O_DIRECT, we have a mix of sync reads and async writes and it turned out that such a combination along with sync writes to innodb log file is enough to cause notable performance regression as well. I have sketched the workload for fio tool (see below) that simulates the I/O access pattern for InnoDB and have run it for SSD and NVMe drives for linux kernels 4.4.0, 5.3.0, and 5.3.0 with ext4 scalability fix.

[global]
filename=tablespace1.ibd:tablespace2.ibd:tablespace3.ibd:tablespace4.ibd:tablespace5.ibd
direct=1
bs=16k
iodepth=1

#read data from *.ibd tablespaces
[ibd_sync_read]
rw=randread
ioengine=psync

#write data to *.ibd tavlespaces
[ibd_async_write]
rw=randwrite
ioengine=libaio

#write data to ib* log file
[ib_log_sync_write]
rw=write
bs=8k
direct=0
ioengine=psync
fsync=1
filename=log.ib
numjobs=1

fio results on the chart:

Observations:

– for SATA/SSD drive there is almost no difference in throughtput, and only at 16 threads do we see a drop in reads for ext4/kernel-5.3.0. For ext4/kernel-5.3.0 mounted with dioread_nolock (that enables scalability fixes), we see that IOPS back and even look better.
– for NVMe drive the situation looks quite different – until 8 i/o threads IOPS for both reads and writes are more/less similar but after increasing pressure on i/o we see a notable spike for writes and similar drop for reads. And again mounting ext4 with dioread_nolock helps to get the same throughput as and for kernels < 4.9.

The similar performance data for the original issue reported to LKML (with more details and analysis) can be found in the patch itself.

How it Affects MySQL

O_DIRECT

Now let’s check the impact of this issue on an IO-bound sysbench/OLTP_RW workload in O_DIRECT mode. I ran a test for the following setup:

– filesystem: xfs, ext4/default, ext4/dioread_nolock
– drives: SATA/SSD and NVMe
– kernels: 4.4.0, 5.3.0, 5.3.0+ilock_fix

Observations

– in the case of SATA/SSD, the ext4 scalability issue has an impact on tps rate after 256 threads and drop is 10-15%
– in the case of NVMe and regular ext4 with kernel 5.3.0 causes performance drop in ~30-80%. If we apply a fix by mounting ext4 with dioread_nolock or use xfs,  throughput looks good.

O_DSYNC

As ext4 regression affects O_DIRECT, let’s replace O_DIRECT with O_DSYNC and look at results of the same sysbench/OLTP_RW workload on kernel 5.3.0:

Note: In order to make results between O_DIRECT and O_DSYNC comparable, I have limited available memory for MySQL instance by cgroup.

Observations:

In the case of O_DSYNC and regular ext4, the performance is just 10% less than for O_DIRECT/ext4/dioread_nolock and O_DIRECT/xfs and ~35% better than for O_DIRECT/ext4. That means that O_DSYNC can be used as a workaround for cases when you have fast storage and ext4 as filesystem but can’t switch to xfs or upgrade kernel.

Conclusions/workarounds

If your workload/setup is affected, there are the following options that you may consider as a workaround:

– downgrade linux kernel to 4.8
– install kernel 5.3.0 with fix and mount ext4 with dioread_nolock option
– if O_DIRECT is important, switch to xfs filesystem
– if changing filesystem is not an option,  replace O_DIRECT with O_DSYNC+cgroup

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com