Jul
11
2018
--

Enterprise software investments may be tepid now, but they’re poised to engage

Have we reached “peak software”?

Just like the idea of “peak oil” — the hypothetical point at which global oil production could max out — you could say we’re approaching a saturation point for venture-capital investments in software companies.

Recent data from PitchBook shows that venture investing in software companies has plateaued: The amount of VC money invested in these companies — $32 billion last year — remained roughly constant over the last four years. The actual number of venture-backed software investments, mostly for business-focused companies, has actually declined, from 4,068 in 2014 to 2,980 last year.

But software is not, in fact, a declining industry. As I explore with my colleague Neeraj Agrawal in a recent report called Software 2018, released last month, a closer look at the PitchBook data shows that the fall-off in software deal volumes is primarily in the Bay Area, where an overheated market has boosted valuations and caused some investors to temporarily pull back. Investment in other U.S. regions, and globally, is actually going up. Investment in software companies based in Europe, Canada and Australia/New Zealand, for example, was $5.4 billion in 2017, up nearly 69 percent from the previous year.

Perhaps more important, a number of broader, global mega trends continue to fuel software innovation today, promising more new companies and more new jobs. These trends include everything from the rise of artificial intelligence, which is pushing software into new fields like autonomous driving, to the recent corporate tax cuts in the U.S., which could free up hundreds of billions of dollars for big corporations to buy up software startups.

Mary Meeker just released her annual, consumer-focused Internet Trends report in May. But here are some of the key trends we see shaping the global, mostly business-focused software market this year:

(Photo by Tomohiro Ohsumi/Getty Images)

SoftBank: Not just for consumer companies anymore

SoftBank’s new, $100 billion Vision Fund has had a huge impact on the technology industry already, given the Japanese firm’s ability to essentially play kingmaker in a given technology market by making a huge investment of hundreds of millions of dollars in one company. This, obviously, makes it extremely difficult for competitors to keep up in terms of building market share. And if a company declines SoftBank’s money, there’s the potentially lethal possibility that SoftBank could fund a competitor, essentially snuffing out the first company.

What’s less noticed, however, is that SoftBank is investing in many business-focused software companies, not just big consumer names like Uber, FlipKart and SoFi. Softbank recently put $2.25 billion into GM’s Cruise business unit for autonomous driving and $250 million into secondary storage vendor Cohesity, for example, and has backed other B2B players such as construction/building-software outfit Katerra; real-estate software company Compass; and workplace chat app Slack.

With these investments and others, SoftBank is accelerating the pace of growth in many key software markets and likely also dampening these companies’ IPO prospects, since companies receiving several hundred million dollars from the Japanese company face less of a financial need to go public. SoftBank is essentially taking the place of an IPO.

Image: Bryce Durbin/TechCrunch

More software means less hardware, more robots

The continuing march of software innovation isn’t great for everyone — losers in this picture could include hardware vendors and people with jobs that can be automated by smart, software-powered robots. (Yes, even lawyers and doctors could be affected — it’s not just truck drivers.)

The implications of artificial intelligence on the job market, and the auto industry, have been widely discussed. Less noticed, though, are the shifting growth rates in cloud-based IT gear versus traditional IT hardware, the technology that powers large corporations and other organizations. IDC predicts that by 2020, corporate spending on cloud-infrastructure software will finally exceed spending on non-cloud IT infrastructure — meaning all those boxes inside corporate data centers from vendors like Dell, IBM, Cisco, H-P etc. Many of those companies are trying to figure out their cloud services approach to stay relevant. 

Lower taxes = more software M&A

Not everyone loves the Trump administration’s policies, but if you’re a software CEO, you might be a fan of the administration’s new tax bill. That’s because the 2017 bill could be a boon for software-industry M&A. Two key components of the new law — the reduced rate charged to companies to repatriate cash from overseas and the lowering of the corporate tax rate to 21 percent from 35 percent — could leave many big tech acquirers with new war chests, analysts believe.

According to investment bank Qatalyst Partners, both changes could leave a group of the largest traditional tech-company acquirers with an additional $400 billion to spend, if they repatriate money from overseas. This would be enough to buy 50 leading software companies today, according to Qatalyst. We have already seen some of this with the recent acquisitions of GitHub by Microsoft ($7.5 billion) and Adaptive Insight by Workday ($1.55 billion) and Q1 deals like MuleSoft by Salesforce ($6.5 billion) and CallidusCloud by SAP ($2.4 billion).

The traditional tech acquirers could be more receptive to acquisitions than ever these days, given that the easy, low-cost cloud business model has allowed a range of young tech upstarts to attack many parts of their businesses from all angles. Often, the easiest solution is for the big tech companies to buy the upstarts.

Niche is nice for software

As software transforms big, well-known corporate markets — like data center software, and technology for functions like human resources, sales and marketing — it is also making inroads into much more narrow industries and corporate functions. The low cost of the cloud makes it easy for every industry, from physical therapy to prison management to mortgage lending, to grow its own, customized software, usually deployed for tasks like operations and customer management. Often there are multiple firms vying for customers (and investor dollars) today in these specialized fields.

Similarly, software is fueling extremely specialized companies to serve business needs inside companies today. These include companies as varied as DocuSign, which has built a multi-billion dollar public company focusing exclusively on document signing, and Carta, which sells technology to help companies manage their financial cap tables.

Mary Meeker is right that consumer internet trends like the rise of online wallets, subscription services for certain goods and increasing oversight of social media by regulators will have big economic implications in the years to come. But we humbly offer that business software is a pretty big economic driver too — you just have to work a little harder to figure out the implications for businesses and the markets.

Apr
17
2018
--

Enterprise AI will make the leap — who will reap the benefits?

This year, artificial intelligence will further elevate the enterprise by transforming the way we work, securing digital assets, increasing collaboration and ushering in a new era of AI-powered innovation. Enterprise AI is rapidly moving beyond hype and into reality, and is primed to become one of the most consequential technological segments. Although startups have already realized AI’s power in redefining industries, enterprise executives are still in the process of understanding how it will transform their business and reshape their teams across all departments.

Throughout the past year, early adopting businesses of all sizes and industries began to reap benefits. AI applications with AI-powered capabilities introduced opportunities to change the way the enterprise engaged customers, segmented markets, assessed sales leads and engaged influencers. Enterprises are on the edge of taking this a step further because of the amount of knowledge and tools leveraging the potential of AI within their entire organization.

“New breakthroughs in AI, enabled by new hardware architectures, will create new intelligent business models for enterprises,” says Nigel Toon, co-founder and CEO at U.K.-based Graphcore. “Companies that can build an initial knowledge model and launch an initial intelligent service or product, then use this first product to capture new data and improve the knowledge model on a continuing basis, will quickly create clear class-leading products and services that competitors will struggle to keep up with.”

The category is evolving, and large companies are finding distinct ways to innovate. They can uniquely tap into decades of industry experience to develop horizontal AI, built for specific industries like healthcare, financial services, automotive, retail and more. These implementations, though, require deep industry expertise and industry-specific design, training, monitoring, security and implementation to meet the high-stakes IT requirements of global organizations.

“In 2018, AI is entering the enterprise. I believe we will see many enterprises adopt AI technology, but the (few) leaders will be those that can align AI with their strategic business goals,” says Ronny Fehling, associate director of Gamma Artificial Intelligence at BCG.

2018: AI will start separating the winners from the losers

Early industry successes (and failures) proved AI’s inevitability, but also the reality that wide-scale adoption would come through incremental progress only. This year, we’ll see AI move from influencing product or business functions to an organization-wide AI strategy. Expect the winners to move fast and remain nimble to keep implementing off-the-shelf and proprietary AI.

The companies that win the AI talent war will gain exponential advantages, given the category’s rapid growth.

Hans-Christian Boos, CEO and founder of Germany-based Arago, adds: “2018 will be a make or break year for enterprise and the established economy in general. I believe AI is the only viable path for innovation, new business models and digital disruption in companies from the industrial era. General AI can enable these enterprises to finally make use of the only advantage they have in the battle against new business models and giants from the Silicon Valley, or rather giants from the new age of knowledge based business models.”

The AI talent challenge

A boon in enterprise AI will also mean a further shortage of talent. Industries like telecommunications, financial services and manufacturing will feel the talent squeeze the most. The companies that win the AI talent war will gain exponential advantages, given the category’s rapid growth.

Hence, enterprises will try to attract talent by offering a powerful vision, a track record of product success, a bench of early client implementations and the potential to impact the masses. It’s about developing high-functioning and reliable solutions that become a new foundation for clients.

Developers and data scientists, however, are only the beginning. Winning enterprises must adopt their organizational structures that attract a new generation of product managers, sales, marketing, communications and other delivery teams that understand AI. This requires an informed, passionate and forward-thinking group of professionals that will help customers understand the future of work and customer engagement powered by AI.

AI adoption and employee training

Digital transformation, powered in large part by new AI capabilities, requires enterprises to understand how to extract data and utilize data-driven intelligence. Data is one of the greatest assets and essentials in maximizing the value in an AI application, yet data is often underutilized and misunderstood. Executives must establish teams and hold individuals across departments accountable for the successful and ongoing implementation of digital tools that extract full value from available internal and external data.

This transformation into an AI-native organization requires it to hire, train and re-skill all levels of employees, and provide the resources for individuals to adopt AI-powered disciplines that enhance their performance. Most workforce, from top to bottom, should be encouraged to rethink and evolve their role by incorporating new digital tools, often enabled by AI itself.

Expect AI and other digital technologies to become more prevalent in all business disciplines, not only at the application layer, as Vishal Chatrath, co-founder and CEO of U.K.-based Prowler.io emphasises. “Decision-making in enterprise is dominated by expert-systems that are born obsolete. The AI tools available till now that rely on deep-neural nets which are great for classification problems (identifying cats, dogs, words etc.) are not really fit for purpose for decision-making in large, complex and dynamic environments, because they are very data inefficient (needs millions of data points) and effectively act like black-boxes. 2018 will see Enterprise AI move beyond classification to decision-making.”

What’s next

However, the spotlight will shine on data governance as businesses adjust entire departments and workflows around data. In turn, data management and integrity will be an essential component of success as consumers and enterprises gain greater awareness about how companies use customers’ data. This opens a large field of opportunities, but also will require transparency in how companies are using, sharing and building applications on top of customer data to ensure trust.

“Every single industry will be enhanced with AI in the coming years. In the last years there was a lot of foundation work on gathering standardized data and now we can start to use some of the advanced AI techniques to bring huge efficiency and quality gains to enterprise companies,” says Rasmus Rothe, co-founder and CTO of Germany-based research lab and venture builder Merantix. “Enterprises should therefore thoroughly analyze their business units to understand how AI can help them to improve. Partnering with external AI experts instead of trying to build everything yourself is often more capital efficient and also leads to better results.”

The shift toward AI-native enterprises is in a defining phase. The pie of the AI-enabled market will continue to grow and everyone has an opportunity to take a slice. Enterprises need to quickly leverage their assets and extract the value of their data as AI algorithms themselves will become the most valuable part when data has become a commodity. The question is, who will move first, and who will have the biggest appetite.

Apr
08
2018
--

In a Slack world, Microsoft bets on Teams and Yammer

The growth of Windows has slowed as Microsoft’s mobile platform goals have faded and the PC market matured. As a result, Microsoft has had to seek new revenue outside of its operating system.

In 2017, as part of that effort to grow, Microsoft announced a new subscription product called Microsoft 365, bringing together Windows, the company’s cloud-centered productivity suite Office 365 and enterprise tooling into a single package.

The introduction of Microsoft 365 presaged the company’s re-organization which, to quote CNBC, “rebuilt the company around the cloud instead of Windows.” This seems reasonable; if Windows isn’t going to return to growth, other services have to keep adding top line revenue. Microsoft’s evolution to a cloud-powered, services-focused company is therefore set to continue.

In the pursuit of new, non-Windows top line, Microsoft wagered that it could expand its “commercial cloud” revenue to a $20 billion run rate by the end of its fiscal 2018. It beat the goal, reaching the $20 billion mark far ahead of the calendar-equivalent date of mid-Summer of this year.

One of those products, Teams, is a component to Office 365 and part of what Microsoft CEO Satya Nadella called a “growth opportunity” that is “a lot bigger than anything [his company has] achieved.”

Today we’re going to explore Microsoft’s current actions in one part of the cloud productivity space through the lens of Teams.

Microsoft Teams

Microsoft’s Teams product is a communications tool often compared to Slack . TechCrunch, for example, recently called the software service “Microsoft’s Slack competitor.” ComputerWorld, in a news item earlier this year, wrote that “Microsoft turn[ed] up [the] heat on Slack” when it announced new Teams features.

It goes on and on, allowing us to comfortably hold up Microsoft Teams as Redmond’s answer to Slack, a company famous for its quick growth, impressive mind share and its independent status from any major tech company. That last fact remains true despite rumored acquisition interest from Microsoft itself, along with pretty much every big company in the sector you can name.

To see Microsoft invest in its own tool that competes with Slack isn’t surprising. There is a large market for the product, and Redmond is loath to let any rival service cut in on its productivity revenue.

Therefore, if there is a hot productivity tool in the market and Microsoft isn’t going to buy it, it might as well build one of its own. Unsurprisingly, the company has been hard at work doing just that.

Joining a big company when you are a comparatively small company can be arduous.

News that Teams could release a free version made headlines. Teams also picked up guest access in February, its introduction of Cortana integration made it into mainstream tech publications and this week Microsoft announced new “retention policies” for Teams.

All that and Microsoft bought Teams a friend this year in the form of Chalkup, a collaboration company focused on the education world.

In short, Teams is adding new features while building its org chart and expanding access. All good things, certainly. However, it was not too long ago Microsoft spent quite a lot of money to buy a different, distinct collaboration tool. What happened to it?

Yammer

Microsoft bought Yammer in 2012 for $1.2 billion, building out what TechCrunch called, at the time, its “Social Enterprise Strategy.” And while the Yammer-Microsoft deal was “great news” for the company and its investors, it also marked the beginning of the “tough part” for the newly acquired startup.

Joining a big company when you are a comparatively small company can be arduous. And if you do so when the larger company is undergoing a massive change in leadership (Microsoft hired a new CEO two years after the Yammer deal) and a business model change-up (Microsoft bought Nokia in 2014, also two years after the Yammer deal, before closing that strategic idea out years later), it’s probably even harder to integrate.

Externally, that difficulty showed. Following the Microsoft deal, Yammer search volume grew before stagnating and later slipping. The product was eventually switched on for free for Office 365 customers in early 2016, four years after it was purchased. Office 365 itself launched a half-decade before, making the moment a bit long in the works.

But all that is the past, and, notably, Microsoft is putting more emphasis on Yammer today than it has in recent years. That may feel odd, given what we just went over concerning Teams.

To dig into that, Crunchbase News got Microsoft’s Seth Patton on the phone, who explained the company’s thinking. According to the 15-year company veteran who now works on Office 365, Microsoft has two separate views for Teams and Yammer. Teams is built for what Patton calls inner-loop communication: stuff for teams, smaller companies and the like; Yammer, in contrast, is better for outer-loop communication: less tactical decisions and more company-wide communications.

The split between Slack and Teams products and the Yammers and Convos of the world isn’t hokum or mere corporate-speak. I’ve worked in newsrooms that used the mix of tools to allow for simple direct messaging between individuals (Slack) and team-wide threaded communications (Yammer). It takes a little getting used to, but it can flow well if you need that level of inter-party discussion.

Even more interesting than the fact that Yammer is not dead is that Microsoft is actively investing in it. According to Patton, Microsoft’s chiefs “doubled down” on Yammer while Teams was being brought into the market in late 2016. This gave Yammer about a year of redoubled investment and attention.

Taking all that together, Microsoft is investing in two communications products at the same time, both of which are baked into its productivity suite. So why the huge push now?

Slack: Software’s favorite rocket ship

You are no doubt familiar with Slack’s growth arc. It’s been a nearly chronic narrative in tech for the past few years. And I don’t mean that in a pejorative sense. (I’m as guilty as anyone else.)

But, in case you have a life, here are some highlights: Slack reached ARR of $50 million in December of 2015. In October of 2016, Slack hit the $100 million ARR mark. Then the company bested $200 million last September. That’s darn quick, and investors took notice, showering the company with cash and ever-rising valuations.

One way to get acquired, after all, is to stick out by worrying the biggest companies in the market through growth.

Fueling Slack’s continued growth is a push into the realm of bigger companies. The firm launched Slack Enterprise Grid last January, bringing enterprise-grade management tools to Slack’s product. With Enterprise Grid, Slack can keep going after bigger accounts. (To that point, IBM has more than 200,000 active users on Slack that use Enterprise Grid.)

That quick growth has made Slack an acquisition target. One way to get acquired, after all, is to stick out by worrying the biggest companies in the market through growth. It’s just hard as heck to do, as incumbent revenue numbers are so large that, well, you have to grow fast to become interesting.

An even bigger scrap

As we know, Slack has rebuffed acquisition offers. As a result, we’re seeing Microsoft, the dominant player in the world of productivity, attempt to slow down Slack in an effort to not lose future users and future dollars. Hell, even Google is in on the race. Its Slack competitor launched for early users in February. Facebook is also tinkering around the edges. It’s fun to watch.

But productivity is Microsoft’s cash cow. For Google, it’s a big side project, but nothing compared to its advertising revenue. That puts Microsoft and Slack more up against one another in the enterprise chat fight.

(In mid-March, Microsoft announced that 200,000 organizations now use Teams, up from 125,000 in September of 2017. That’s 60 percent growth in a half-year or so — a quick growth pace, too.)

What we’ll learn over the next few years is if Microsoft’s enormous enterprise channel can be leveraged enough to slow Slack’s growth, or if Slack’s momentum can actually capture a piece of the productivity market and hold onto it.

It’s a startup against a platform company, a classic enough battle. But with big tech bigger, richer and more powerful than ever, it’s a more relevant business case than we might think at first blush. More when one draws blood or Slack goes public.

Mar
23
2018
--

Storytelling for B2B startups: Avoiding ‘buzzword bingo’ to make your wonky enterprise company worth talking about

If there’s one thing I learned from my time as both a journalist at The Wall Street Journal and Forbes and, now, advising a global venture capital firm on communications, it’s that storytelling can make or break a company.

This is especially true the more complicated and arcane a company’s technology is. Stories about online-dating and burrito-delivery apps are easily understood by most people. But if a company specializes in making technology for hybrid-cloud data centers, or parsing specialized IT alerts and cybersecurity warnings, the storytelling task becomes much harder — but, I would argue, even more important.

Sure, a wonky company will still be able to talk easily to its customers and chat up nerdy CIOs at trade shows. But what happens when they raise a Series C or D round of financing and actually need to reach a broader audience — like really big, potential business partners, potential acquirers, public investors or high-level business reporters? Often, they’re stuck.

It can be painful to watch. When I was a reporter, I was amazed at the buzzwords thrown at me by some technology companies trying to get me to write about them. For fun, my colleagues and I would put some of these terms into online “buzzword bingo” websites just to see what indecipherable company descriptions they would spit out. (Example: “An online, cloud-based, open-source hyperconverged Kubernetes solution.”) Often, when pressed, PR representatives couldn’t explain to me what these companies actually did.

These companies obviously never made it into my stories. And I would argue that many of them suffered more broadly from their overall lack of high-profile press coverage; large business publications like the ones for which I worked target the very big-company executives and investors these later-stage startups were trying to reach.

Now, of course, I’m on the other side of that reporter/company equation — and I often feel like a big chunk of my job is working as a technology translator.

A natural-born storyteller

So why is this B2B storytelling problem so common, and arguably getting worse? Lots of reasons. Many of these hard-to-understand companies are founded by highly technical engineers for whom storytelling is (not surprisingly) not a natural skill. In many cases, their marketing departments are purely data-driven, focused on demand generation, ROI and driving prospects to an online sales funnel — not branding and high-level communications. As marketing technology has gotten more and more advanced and specialized, so have marketing departments.

As a result, many B2B and enterprise-IT companies are often laser-focused on talking about their products’ specific bells and whistles, staying in “sell mode” for a technical audience and cranking out wonky whitepapers and often-boring product press releases. They’re less adept at taking a step back to address the actual business benefits their product enables. Increasingly, this tech-talk also plays well with the legions of hyper-specialized, tech-news websites that have proliferated to serve every corner of the technology market, making some executives think there’s no need to target higher-level press.

Everyone has a story to tell. It’s up you to figure out what your company’s is, and how to tell that story in a compelling, understandable fashion.

One prominent marketing and PR consultant I know, who has worked with hundreds of Silicon Valley startups since the 1980s, says she is “shocked” by how poorly many senior tech industry CEOs today communicate their companies’ stories. Many tend to “shun” communications, considering it too “soft” in this new era of data-obsessed marketing, the consultant Jennifer Jones, recently told me. But in the end, poor communications and storytelling can create or exacerbate business problems, and often affect a company’s valuation.

So how do you get to a point where you can talk about your company in plain terms, and reach the high-level audiences you’re targeting?

One tactic, obviously, is to ditch the jargon when you need to. The pitch you use on potential customers — who likely already have an intimate understanding of your market and the specific problems you’re trying to solve — is not as relevant for other audiences.

A big fund manager at Fidelity or T. Rowe Price, or a national business journalist, probably knows, for example, that cloud computing is a big trend now, or that companies are buying more technology to battle complex cybersecurity attacks. But do they really understand the intricacies of “hybrid-cloud” data center setups? Or what a “behavioral attack detection solution” does? Probably not.

The David versus Goliath angle

Another tip is to put your company story in a larger, thematic context. People can better understand what you do if you can explain how you fit into larger technology and societal trends. These might include the rise of free, open-source software, or the growing importance of mobile computing.

It’s also helpful to talk about what you do in relation to larger, more established players. Are you nipping away at the slow-growing, legacy business of Oracle/EMC/Dell/Cisco? As a journalist, I once wrote a story about a small public networking company called F5 Networks that specialized in making “application delivery controllers.” But the story mostly focused on F5’s battle with a much larger competitor; in fact, the editors titled the story “One-Upping Cisco.” That’s the angle most readers were likely to care about. Journalists, particularly, love these David versus Goliath type stories, and national business publications are full of them.

Start focusing on high-level storytelling earlier, not when you’ve already raised $100 million in venture funding and have several hundred employees.

Another key storytelling strategy is leveraging your customers. If your business is boring to the average person, try to get one of your household-name customers to talk publicly about how they use your technology. Does your supply-chain software help L’Oréal sell more lipstick, or UPS make faster package deliveries?

One of our portfolio companies had a nice business-press hit a few years ago by talking about how their software helped HBO stream “Game of Thrones” episodes. (The service had previously crashed because too many people were trying to watch the show.) You can leverage these highly visible customers for case studies on your website. These can be great fodder for your sales team as well as later press interviews, as long as they’re well-written and understandable. Try to get more customers to agree to this type of content when you sign the contract with them.

From “Mad Men” to math men

Finally, there’s the issue of marketing leadership inside tech companies. In my experience, most smaller, B2B or enterprise IT-focused startups have CMOs or VPs of marketing who are more focused on data and analytics than brand communications — more “math men” than “Mad Men.” This isn’t surprising, as these companies often sell data-rich products and have business models where PR and general advertising don’t directly drive sales (unlike, say, a company making a food-delivery app). The CEOs of these companies value data and analytics, too.

I encourage B2B tech CEOs to focus on hiring CMOs with some brand/communications experience, or at least a willingness to outsource it to competent partners who are experts in that area. After a couple of early rounds of funding, you should be outgrowing your highly specialized PR firm (if you even have one) that focuses on a narrow brand of trade publications, for example. These firms usually don’t have contacts at the bigger, national business and technology outlets that are read by big mutual fund managers, and the business development folks at Cisco or Oracle. Hiring ex-journalists — not technical experts — to write content and develop messaging can be a good idea, too.

In other words, start focusing on high-level storytelling earlier, not when you’ve already raised $100 million in venture funding and have several hundred employees. By that point, it can simply be too late: Your company has already been typecast by the trade press and written off by higher-level reporters, and sometimes even potential business partners, as too niche-y and hard to understand.

As a journalist, I learned that everyone has a story to tell. It’s up you to figure out what your company’s is, and how to tell that story in a compelling, understandable fashion. If you do, I’m pretty sure the business benefits will follow.

Jun
28
2017
--

The next generational shift in enterprise infrastructure has arrived

 Cloud computing is driving growth at 3 of the 5 most valuable companies in the world. AI will impact jobs only as quickly as AI-powered business software evolves. These are just two of the ramifications of disruptions in enterprise technology permeating mainstream media. Yet the inner workings of the tightly knit enterprise software industry are rarely publicized. Read More

Jun
15
2017
--

VR’s killer app: business services

 Enterprise adoption is trumping entertainment uses and will spring VR and AR into the mainstream. Read More

Apr
07
2017
--

Tracking the explosive growth of open-source software

 Many hot new enterprise technologies are centered around free, “open-source” technology. But how can corporate customers, and investors, evaluate all these new open-source offerings? These questions are especially tough to answer because most open-source companies are still private. That’s why we created a detailed index to track popular open-source software projects. Read More

Dec
29
2016
--

How viral open-source startups can build themselves into enterprise-IT powerhouses

Striped Halftone Pattern Hordes of new enterprise-IT upstarts have popped up in Silicon Valley, with some drawing lofty valuations from investors. They’re driven by new, more-advanced technologies in areas such as databases, software development, networking and cloud computing. And many are taking aim at incumbents like Dell, EMC, Oracle and IBM. But will these new companies ever be as valuable as those big names? Read More

Dec
14
2016
--

Row Store and Column Store Databases

Row Store and Column Store

Row Store and Column StoreIn this blog post, we’ll discuss the differences between row store and column store databases.

Clients often ask us if they should or could be using columnar databases. For some applications, a columnar database is a great choice; for others, you should stick with the tried and true row-based option.

At a basic level, row stores are great for transaction processing. Column stores are great for highly analytical query models. Row stores have the ability to write data very quickly, whereas a column store is awesome at aggregating large volumes of data for a subset of columns.

One of the benefits of a columnar database is its crazy fast query speeds. In some cases, queries that took minutes or hours are completed in seconds. This makes columnar databases a good choice in a query-heavy environment. But you must make sure that the queries you run are really suited to a columnar database.

Data Storage

Let’s think about a basic database, like a stockbroker’s transaction records. In a row store, each client would have a record with their basic information – name, address, phone number, etc. – in a single table. It’s likely that each record would have a unique identifier. In our case, it would probably be an

account_number

.

There is another table that stored stock transactions. Again, each transaction is uniquely identified by something like a

transaction_id

. Each transaction is associated to one

account_number

, but each

account_number

 is associated with multiple transactions. This provides us with a one-to-many relationship, and is a classic example of a transactional database.

We store all these tables on a disk and, when we run a query, the system might access lots of data before it determines what information is relevant to the specific query. If we want to know the

account_number

,

first_name

,

last_name

,

stock

, and

purchase_price

 for a given time period, the system needs to access all of the information for the two tables, including fields that may not be relevant to the query. It then performs a join to relate the two tables’ data, and then it can return the information. This can be inefficient at scale, and this is just one example of a query that would probably run faster on a columnar database.

With a columnar database, each field from each table is stored in its own file or set of files. In our example database, all

account_number

 data is stored in one file, all

transaction_id

 data is stored in another file, and so on. This provides some efficiencies when running queries against wide tables, since it is unlikely that a query needs to return all of the fields in a single table. In the query example above, we’d only need to access the files that contained data from the requested fields. You can ignore all other fields that exist in the table. This ability to minimize i/o is one of the key reasons columnar databases can perform much faster.

Normalization Versus Denormalization

Additionally, many columnar databases prefer a denormalized data structure. In the example above, we have two separate tables: one for account information and one for transaction information. In many columnar databases, a single table could represent this information. With this denormalized design, when a query like the one presented is run, no joins would need to be processed in the columnar database, so the query will likely run much faster.

The reason for normalizing data is that it allows data to be written to the database in a highly efficient manner. In our row store example, we need to record just the relevant transaction details whenever an existing customer makes a transaction. The account information does not need to be written along with the transaction data. Instead, we reference the

account_number

 to gain access to all of the fields in the accounts table.

The place where a columnar database really shines is when we want to run a query that would, for example, determine the average price for a specific stock over a range of time. In the case of the columnar database, we only need a few fields – 

symbol

,

price

, and

transaction_date

– in order to complete the query. With a row store, we would gather additional data that was not needed for the query but was still part of the table structure.

Normalization of data also makes updates to some information much more efficient in a row store. If you change an account holder’s address, you simply update the one record in the accounts table. The updated information is available to all transactions completed by that account owner. In the columnar database, since we might store the account information with the transactions of that user, many records might need updating in order update the available address information.

Conclusion

So, which one is right for you? As with so many things, it depends. You can still perform data analysis with a row-based database, but the queries may run slower than they would on a column store. You can record transactions in a column-based model, but the writes may takes longer to complete. In an ideal world, you would have both options available to you, and this is what many companies are doing.

In most cases, the initial write is to a row-based system. We know them, we love them, we’ve worked with them forever. They’re kind of like that odd relative who has some real quirks. We’ve learned the best ways to deal with them.

Then, we write the data (or the relevant parts of the data) to a column based database to allow for fast analytic queries.

Both databases incurred write transactions, and both also likely incur read transactions. Due to the fact that a column-based database has each column’s data in a separate file, it is less than ideal for a “SELECT * FROM…” query, since the request must access numerous files to process the request. Similarly, any query that selects a single or small subset of files will probably perform better in a row store. The column store is awesome for performing aggregation over large volumes of data. Or when you have queries that only need a few fields from a wide table.

It can be tough to decide between the two if you only have one database. But it is more the norm that companies support multiple database platforms for multiple uses. Also, your needs might change over time. The sports car you had when you were single is less than optimal for your current family of five. But, if you could, wouldn’t you want both the sports car and the minivan? This is why we often see both database models in use within a single company.

Dec
10
2016
--

Software is due for a bundling event

Overhead shot of a small group of people, wearing monochromatic colors, pulling at ropes from opposing directions We are approaching a new phase of enterprise software, where every niche of Software-as-a-Service has been filled and cloud companies are being consolidated into larger companies. Markets have a tendency to cycle from bundling to unbundling, and software is due for a bundling event. Read More

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com