Personal blog of Yzmir Ramirez

Artificial intelligence technology holds a huge amount of promise for enterprises — as a tool to process and understand their data more efficiently; as a way to leapfrog into new kinds of services and products; and as a critical stepping stone into whatever the future might hold for their businesses. But the problem for many enterprises is that they are not tech businesses at their core, so bringing on and using AI will typically involve a lot of heavy lifting. Today, one of the startups building AI services is announcing a big round of funding to help bridge that gap.

SambaNova — a startup building AI hardware and integrated systems that run on it that only officially came out of three years in stealth last December — is announcing a huge round of funding today to take its business out into the world. The company has closed on $676 million in financing, a Series D that co-founder and CEO Rodrigo Liang has confirmed values the company at $5.1 billion.

The round is being led by SoftBank, which is making the investment via Vision Fund 2. Temasek and the government of Singapore Investment Corp. (GIC), both new investors, are also participating, along with previous backers BlackRock, Intel Capital, GV (formerly Google Ventures), Walden International and WRVI, among other unnamed investors. (Sidenote: BlackRock and Temasek separately kicked off an investment partnership yesterday, although it’s not clear if this falls into that remit.)

Co-founded by two Stanford professors, Kunle Olukotun and Chris Ré, and Liang, who had been an engineering executive at Oracle, SambaNova has been around since 2017 and has raised more than $1 billion to date — both to build out its AI-focused hardware, which it calls DataScale, and to build out the system that runs on it. (The “Samba” in the name is a reference to Liang’s Brazilian heritage, he said, but also the Latino music and dance that speaks of constant movement and shifting, not unlike the journey AI data regularly needs to take that makes it too complicated and too intensive to run on more traditional systems.)

SambaNova on one level competes for enterprise business against companies like Nvidia, Cerebras Systems and Graphcore — another startup in the space which earlier this year also raised a significant round. However, SambaNova has also taken a slightly different approach to the AI challenge.

In December, the startup launched Dataflow-as-a-Service as an on-demand, subscription-based way for enterprises to tap into SambaNova’s AI system, with the focus just on the applications that run on it, without needing to focus on maintaining those systems themselves. It’s the latter that SambaNova will be focusing on selling and delivering with this latest tranche of funding, Liang said.

SambaNova’s opportunity, Liang believes, lies in selling software-based AI systems to enterprises that are keen to adopt more AI into their business, but might lack the talent and other resources to do so if it requires running and maintaining large systems.

“The market right now has a lot of interest in AI. They are finding they have to transition to this way of competing, and it’s no longer acceptable not to be considering it,” said Liang in an interview.

The problem, he said, is that most AI companies “want to talk chips,” yet many would-be customers will lack the teams and appetite to essentially become technology companies to run those services. “Rather than you coming in and thinking about how to hire scientists and hire and then deploy an AI service, you can now subscribe, and bring in that technology overnight. We’re very proud that our technology is pushing the envelope on cases in the industry.”

To be clear, a company will still need data scientists, just not the same number, and specifically not the same number dedicating their time to maintaining systems, updating code and other more incremental work that comes managing an end-to-end process.

SambaNova has not disclosed many customers so far in the work that it has done — the two reference names it provided to me are both research labs, the Argonne National Laboratory and the Lawrence Livermore National Laboratory — but Liang noted some typical use cases.

One was in imaging, such as in the healthcare industry, where the company’s technology is being used to help train systems based on high-resolution imagery, along with other healthcare-related work. The coincidentally-named Corona supercomputer at the Livermore Lab (it was named after the 2014 lunar eclipse, not the dark cloud of a pandemic that we’re currently living through) is using SambaNova’s technology to help run calculations related to some COVID-19 therapeutic and antiviral compound research, Marshall Choy, the company’s VP of product, told me.

Another set of applications involves building systems around custom language models, for example in specific industries like finance, to process data quicker. And a third is in recommendation algorithms, something that appears in most digital services and frankly could always do to work a little better than it does today. I’m guessing that in the coming months it will release more information about where and who is using its technology.

Liang also would not comment on whether Google and Intel were specifically tapping SambaNova as a partner in their own AI services, but he didn’t rule out the prospect of partnering to go to market. Indeed, both have strong enterprise businesses that span well beyond technology companies, and so working with a third party that is helping to make even their own AI cores more accessible could be an interesting prospect, and SambaNova’s DataScale (and the Dataflow-as-a-Service system) both work using input from frameworks like PyTorch and TensorFlow, so there is a level of integration already there.

“We’re quite comfortable in collaborating with others in this space,” Liang said. “We think the market will be large and will start segmenting. The opportunity for us is in being able to take hold of some of the hardest problems in a much simpler way on their behalf. That is a very valuable proposition.”

The promise of creating a more accessible AI for businesses is one that has eluded quite a few companies to date, so the prospect of finally cracking that nut is one that appeals to investors.

“SambaNova has created a leading systems architecture that is flexible, efficient and scalable. This provides a holistic software and hardware solution for customers and alleviates the additional complexity driven by single technology component solutions,” said Deep Nishar, senior managing partner at SoftBank Investment Advisers, in a statement. “We are excited to partner with Rodrigo and the SambaNova team to support their mission of bringing advanced AI solutions to organizations globally.”

Another huge financing round is coming in for an AI company today, this time for a startup called Mythic getting a fresh $40 million as it appears massive deals are closing left and right in the sector.

Mythic particularly focuses on the inference side of AI operations — basically making the calculation on the spot for something based off an extensively trained model. The chips are designed to be low power, small, and achieve the same kind of performance you’d expect from a GPU in terms of the lightning-fast operations that algorithms need to perform to figure out whether or not that thing your car is about to run into is a cat or just some text on the road. SoftBank Ventures led this most-recent round of funding, with a strategic investment also coming from Lockheed Martin Ventures. ARM executive Rene Haas will also be joining the company’s board of directors.

“The key to getting really high performance and really good energy efficiency is to keep everything on the chip,” Henry said. “The minute you have to go outside the chip to memory, you lose all performance and energy. It just goes out the window. Knowing that, we found that you can actually leverage flash memory in a very special way. The limit there is, it’s for inference only, but we’re only going after the inference market — it’s gonna be huge. On top of that, the challenge is getting the processors and memory as close together as possible so you don’t have to move around the data on the chip.”

Mythic, like other startups, is looking to ease the back-and-forth trips to memory on the processors in order to speed things up and lower the power consumption, and CEO Michael Henry says the company has figured out how to essentially do the operations — based in a field of mathematics called linear algebra — on flash memory itself.

Mythic’s approach is designed to be what Henry calls more analog. To visualize how it might work, imagine a set-up in Minecraft, with a number of different strings of blocks leading to an end gate. If you flipped a switch to turn 50 of those strings on with some unit value, leaving the rest off, and joined them at the end and saw the combined final result of the power, you would have completed something similar to an addition operation leading to a sum of 50 units. Mythic’s chips are designed to do something not so dissimilar, finding ways to complete those kinds of analog operations for addition and multiplication in order to handle the computational requirements for an inference operation. The end result, Henry says, consumes less power and dissipates less heat while still getting just enough accuracy to get the right solution (more technically: the calculations are 8-bit results).

After that, the challenge is sticking a layer on top of that to make it look and behave like a normal chip to a developer. The goal is to, like other players in the AI hardware space, just plug into frameworks like TensorFlow. Those frameworks abstract out all the complicated tooling and tuning required for such a specific piece of hardware and make it very approachable and easy for developers to start building machine learning projects. Andrew Feldman, CEO of another AI hardware startup called Cerebras Systems, said at the Goldman Sachs Technology and Internet conference last month that frameworks like TensorFlow had  most of the value Nvidia had building up an ecosystem for developers on its own system.

Henry, too, is a big TensorFlow fan. And for good reason: it’s because of frameworks like TensorFlow that allow next-generation chip ideas to even get off the ground in the first place. These kinds of frameworks, which have become increasingly popular with developers, have abstracted out the complexity of working with specific low-level hardware like a field programmable gate array (FPGA) or a GPU. That’s made building machine learning-based operations much easier for developers and led to an explosion of activity when it comes to machine learning, whether it’s speech or image recognition among a number of other use cases.

“Things like TensorFlow make our lives so much easier,” Henry said. “Once you have a neural network described on TensorFlow, it’s on us to take that and translate that onto our chip. We can abstract that difficulty by having an automatic compiler.”

While many of these companies are talking about getting massive performance gains over a GPU — and, to be sure, Henry hopes that’ll be the case — the near term goal for Mythic is to match the performance of a $1,000 GPU while showing it can take up less space and consume less power. There’s a market for the card that customers can hot swap in right away. Henry says the company is focused on using a PCI-E interface, a very common plug-and-play system, and that’s it.

The challenge for Mythic, however, is going to get into the actual design of some of the hardware that comes out. It’s one thing to sell a bunch of cards that companies can stick into their existing hardware, but it’s another to get embedded into the actual pieces of hardware themselves — which is what’s going to need to happen if it wants to be a true workhorse for devices on the edge, like security cameras or things handling speech recognition. That makes the buying cycle a little more difficult, but at the same time, there will be billions of devices out there that need advanced hardware to power their inference operations.

“If we can sell a PCI card, you buy it and drop it in right away, but those are usually for low-volume, high-selling price products,” Henry said. “The other customers we serve design you into the hardware products. That’s a longer cycle, that can take upwards of a year. For that, typically the volumes are much higher. The nice thing is that you’re really really sticky. If they design you into a product you’re really sticky. We can go after both, we can go after board sales, and then go after design.”

There are probably going to be two big walls to Mythic, much less any of the other players out there. The first is that none of these companies have shipped a product. While Mythic, or other companies, might have a proof-of-concept chip that can drop on the table, getting a production-ready piece of next-generation silicon is a dramatic undertaking. Then there’s the process of not only getting people to buy the hardware, but actually convincing them that they’ll have the systems in place to ensure that developers will build on that hardware. Mythic says it plans to have a sample for customers by the end of the year, with a production product by 2019.

That also explains why Mythic, along with those other startups, are able to raise enormous rounds of money — which means there’s going to be a lot of competition amongst all of them. Here’s a quick list of what fundraising has happened so far: SambaNova Systems raised $56 million last week; Graphcore raised $50 million in November last year; Cerebras Systems’s first round was $25 million in December 2016; and this isn’t even counting an increasing amount of activity happening among companies in China. There’s still definitely a segment of investors that consider the space way too hot (and there is, indeed, a ton of funding) or potentially unnecessary if you don’t need the bleeding edge efficiency or power of these products.

And there are, of course, the elephants in the room in the form of Nvidia and to a lesser extent Intel. The latter is betting big on FPGA and other products, while Nvidia has snapped up most of the market thanks to GPUs being much more efficient at the kind of math needed for AI. The play for all these startups is they can be faster, more efficient, or in the case of Mythic, cheaper than all those other options. It remains to be seen whether they’ll unseat Nvidia, but nonetheless there’s an enormous amount of funding flowing in.

“The question is, is someone going to be able to beat Nvidia when they have the valuation and cash reserves,” Henry said. “But the thing, is we’re in a different market. We’re going after the edge, we’re going after things embedded inside phones and cars and drones and robotics, for applications like AR and VR, and it’s just really a different market. When investors analyze us they have to think of us differently. They don’t think, is this the one that wins Nvidia, they think, are one or more of these powder keg markets explode. It’s a different conversation for us because we’re an edge company.”

Another massive financing round for an AI chip company is coming in today, this time for SambaNova Systems — a startup founded by a pair of Stanford professors and a longtime chip company executive — to build out the next generation of hardware to supercharge AI-centric operations.

SambaNova joins an already quite large class of startups looking to attack the problem of making AI operations much more efficient and faster by rethinking the actual substrate where the computations happen. The GPU has become increasingly popular among developers for its ability to handle the kinds of lightweight mathematics in very speedy fashion necessary for AI operations. Startups like SambaNova look to create a new platform from scratch, all the way down to the hardware, that is optimized exactly for those operations. The hope is that by doing that, it will be able to outclass a GPU in terms of speed, power usage, and even potentially the actual size of the chip. SambaNova today said it has raised a massive $56 million series A financing round was co-led by GV and Walden International, with participation from Redline Capital and Atlantic Bridge Ventures.

SambaNova is the product of technology from Kunle Olukotun and Chris Ré, two professors at Stanford, and led by former Oracle SVP of development Rodrigo Liang, who was also a VP at Sun for almost 8 years. When looking at the landscape, the team at SambaNova looked to work their way backwards, first identifying what operations need to happen more efficiently and then figuring out what kind of hardware needs to be in place in order to make that happen. That boils down to a lot of calculations stemming from a field of mathematics called linear algebra done very, very quickly, but it’s something that existing CPUs aren’t exactly tuned to do. And a common criticism from most of the founders in this space is that Nvidia GPUs, while much more powerful than CPUs when it comes to these operations, are still ripe for disruption.

“You’ve got these huge [computational] demands, but you have the slowing down of Moore’s law,” Olukotun said. “The question is, how do you meet these demands while Moore’s law slows. Fundamentally you have to develop computing that’s more efficient. If you look at the current approaches to improve these applications based on multiple big cores or many small, or even FPGA or GPU, we fundamentally don’t think you can get to the efficiencies you need. You need an approach that’s different in the algorithms you use and the underlying hardware that’s also required. You need a combination of the two in order to achieve the performance and flexibility levels you need in order to move forward.”

While a $56 million funding round for a series A might sound massive, it’s becoming a pretty standard number for startups looking to attack this space, which has an opportunity to beat massive chipmakers and create a new generation of hardware that will be omnipresent among any device that is built around artificial intelligence — whether that’s a chip sitting on an autonomous vehicle doing rapid image processing to potentially even a server within a healthcare organization training models for complex medical problems. Graphcore, another chip startup, got $50 million in funding from Sequoia Capital, while Cerebras Systems also received significant funding from Benchmark Capital.

Olukotun and Liang wouldn’t go into the specifics of the architecture, but they are looking to redo the operational hardware to optimize for the AI-centric frameworks that have become increasingly popular in fields like image and speech recognition. At its core, that involves a lot of rethinking of how interaction with memory occurs and what happens with heat dissipation for the hardware, among other complex problems. Apple, Google with its TPU, and reportedly Amazon have taken an intense interest in this space to design their own hardware that’s optimized for products like Siri or Alexa, which makes sense because dropping that latency to as close to zero as possible with as much accuracy as possible in the end improves the user experience. A great user experience leads to more lock-in for those platforms, and while the larger players may end up making their own hardware, GV’s Dave Munichiello — who is joining the company’s board — says this is basically a validation that everyone else is going to need the technology soon enough.

“Large companies see a need for specialized hardware and infrastructure,” he said. “AI and large-scale data analytics are so essential to providing services the largest companies provide that they’re willing to invest in their own infrastructure, and that tells us more investment is coming. What Amazon and Google and Microsoft and Apple are doing today will be what the rest of the Fortune 100 are investing in in 5 years. I think it just creates a really interesting market and an opportunity to sell a unique product. It just means the market is really large, if you believe in your company’s technical differentiation, you welcome competition.”

There is certainly going to be a lot of competition in this area, and not just from those startups. While SambaNova wants to create a true platform, there are a lot of different interpretations of where it should go — such as whether it should be two separate pieces of hardware that handle either inference or machine training. Intel, too, is betting on an array of products, as well as a technology called Field Programmable Gate Arrays (or FPGA), which would allow for a more modular approach in building hardware specified for AI and are designed to be flexible and change over time. Both Munichiello’s and Olukotun’s arguments are that these require developers who have a special expertise of FPGA, which is a sort of niche-within-a-niche that most organizations will probably not have readily available.

Nvidia has been a massive benefactor in the explosion of AI systems, but it clearly exposed a ton of interest in investing in a new breed of silicon. There’s certainly an argument for developer lock-in on Nvidia’s platforms like Cuda. But there are a lot of new frameworks, like TensorFlow, that are creating a layer of abstraction that are increasingly popular with developers. That, too represents an opportunity for both SambaNova and other startups, who can just work to plug into those popular frameworks, Olukotun said. Cerebras Systems CEO Andrew Feldman actually also addressed some of this on stage at the Goldman Sachs Technology and Internet Conference last month.

“Nvidia has spent a long time building an ecosystem around their GPUs, and for the most part, with the combination of TensorFlow, Google has killed most of its value,” Feldman said at the conference. “What TensorFlow does is, it says to researchers and AI professionals, you don’t have to get into the guts of the hardware. You can write at the upper layers and you can write in Python, you can use scripts, you don’t have to worry about what’s happening underneath. Then you can compile it very simply and directly to a CPU, TPU, GPU, to many different hardwares, including ours. If in order to do that work, you have to be the type of engineer that can do hand-tuned assembly or can live deep in the guts of hardware, there will be no adoption… We’ll just take in their TensorFlow, we don’t have to worry about anything else.”

(As an aside, I was once told that Cuda and those other lower-level platforms are really used by AI wonks like Yann LeCun building weird AI stuff in the corners of the Internet.)

There are, also, two big question marks for SambaNova: first, it’s very new, having started in just November while many of these efforts for both startups and larger companies have been years in the making. Munichiello’s answer to this is that the development for those technologies did, indeed, begin a while ago — and that’s not a terrible thing as SambaNova just gets started in the current generation of AI needs. And the second, among some in the valley, is that most of the industry just might not need hardware that’s does these operations in a blazing fast manner. The latter, you might argue, could just be alleviated by the fact that so many of these companies are getting so much funding, with some already reaching close to billion-dollar valuations.

But, in the end, you can now add SambaNova to the list of AI startups that have raised enormous rounds of funding — one that stretches out to include a myriad of companies around the world like Graphcore and Cerebras Systems, as well as a lot of reported activity out of China with companies like Cambricon Technology and Horizon Robotics. This effort does, indeed, require significant investment not only because it’s hardware at its base, but it has to actually convince customers to deploy that hardware and start tapping the platforms it creates, which supporting existing frameworks hopefully alleviates.

“The challenge you see is that the industry, over the last ten years, has underinvested in semiconductor design,” Liang said. “If you look at the innovations at the startup level all the way through big companies, we really haven’t pushed the envelope on semiconductor design. It was very expensive and the returns were not quite as good. Here we are, suddenly you have a need for semiconductor design, and to do low-power design requires a different skillset. If you look at this transition to intelligent software, it’s one of the biggest transitions we’ve seen in this industry in a long time. You’re not accelerating old software, you want to create that platform that’s flexible enough [to optimize these operations] — and you want to think about all the pieces. It’s not just about machine learning.”

 While some of the largest chip manufacturers are looking to shift their focus onto the GPU for their biggest machine learnings, there’s a blooming ecosystem of new chip startups looking to rethink the way processing for AI works That includes a European-based startup called Graphcore, which said today that it has raised $50 million in new financing led by Sequoia Capital — following… Read More