How Cerebras is breaking the GPU bottleneck on AI inference

Nvidia has lengthy dominated the market in compute {hardware} for AI with its graphics processing items (GPUs). Nonetheless, the Spring 2024 launch of Cerebras Techniques’ mature third-generation chip, based mostly on their flagship wafer-scale engine know-how, is shaking up the panorama by providing enterprises an progressive and aggressive various.

This text explores why Cerebras’ new product issues, the way it stacks up towards each Nvidia’s choices and people of Groq, one other new startup offering superior AI-specialized compute {hardware} and highlights what enterprise decision-makers ought to take into account when navigating this evolving panorama.

First, a be aware on why the timing of Cerebras’ and Groq’s problem is so vital. Till now, a lot of the processing for AI has been within the coaching of enormous language fashions (LLMs), not in really making use of these fashions for actual functions. Nvidia’s GPUs have been extraordinarily dominant throughout that interval. Nonetheless, within the subsequent 18 months, {industry} specialists count on the market to achieve an inflection level because the AI tasks that many firms have been coaching and creating will lastly be deployed. At that time, AI workloads shift from coaching to what the {industry} calls inference, the place pace and effectivity turn out to be way more vital. Will Nvidia’s line of GPUs be capable to preserve prime place?

Let’s take a deeper look. Inference is the method by which a skilled AI mannequin evaluates new information and produces outcomes– for instance, throughout a chat with an LLM, or as a self-driving automobile maneuvers via visitors–as an alternative of coaching, when the mannequin is being formed behind the scenes earlier than being launched. Inference is vital to all AI purposes, from split-second real-time interactions to the information analytics that drive long-term decision-making. The AI inference market is on the cusp of explosive progress, with estimates predicting it should attain $90.6 billion by 2030.

Traditionally, AI inference has been carried out on GPU chips. This was on account of GPUs basic superiority over CPU on the parallel computing wanted for environment friendly coaching over large datasets. Nonetheless, as demand for heavy inference workloads will increase, GPUs devour vital energy, generate excessive ranges of warmth and are costly to take care of.

Cerebras, based in 2016 by a crew of AI and chip design specialists, is a pioneer within the area of AI inference {hardware}. The corporate’s flagship product, the Wafer-Scale Engine (WSE), is a revolutionary AI processor that units a brand new bar for inference efficiency and effectivity. The lately launched third-generation CS-3 chip boasts 4 trillion transistors, making it the bodily largest neural community chip ever produced–at 56x bigger than the most important GPUs it’s nearer in dimension to a dinner plate than a postage stamp. It incorporates 3000x extra on-chip reminiscence. Because of this particular person chips can deal with enormous workloads with out having to community, an architectural innovation that permits sooner processing speeds, better scalability, and diminished energy consumption.

The CS-3 excels with LLMs; stories point out that Cerebras’ chip can course of an eye-watering 1,800 tokens per second for the Llama 3.1 8B mannequin, far outpacing present GPU-based options. Furthermore, with pricing beginning at simply 10 cents per million tokens, Cerebras is positioning itself as a aggressive resolution.

The necessity for pace

Given the demand for AI inference, it’s no shock that Cerebras’ spectacular stats are drawing {industry} consideration. Certainly, the corporate has had sufficient early traction that its press package cites a number of {industry} leaders lauding its know-how.

“Speed and scale change everything,” in keeping with Kim Branson, SVP of AI/ML at GlaxoSmithKline, the place the enhance supplied by Cerebras’ CS-3 has reportedly improved the corporate’s means to deal with large datasets for drug discovery and evaluation.

Denis Yarats, CTO of Perplexity, sees ultra-fast inference as the important thing to reshaping serps and consumer experiences. “Lower latencies drive higher user engagement,” stated Yarats. “With Cerebras’ 20x speed advantage over traditional GPUs, we believe user interaction with search and intelligent answer engines will be fundamentally transformed.”

Russell d’Sa, CEO of LiveKit, highlighted how Cerebras’ ultra-fast inference has enabled his firm to develop next-gen multimodal AI purposes with voice and video-based interactions. “Combining Cerebras’ best-in-class compute with LiveKit’s global edge network has allowed us to create AI experiences that feel more human, thanks to the system’s ultra-low latency.”

The aggressive panorama: Nvidia vs. Groq vs. Cerebras

Regardless of the facility of its know-how, Cerebras faces a aggressive market. Nvidia’s dominance within the AI {hardware} market is properly established, with its Hopper GPUs being a staple in coaching and operating AI fashions. Compute on Nvidia’s GPUs is obtainable via cloud suppliers equivalent to Amazon Net Companies, Google Cloud Platform, or Microsoft Azure and Nvidia’s established market presence offers it a major edge when it comes to ecosystem assist and buyer belief.

Nonetheless, the AI {hardware} market is evolving, and competitors is intensifying. Groq, one other AI chip startup, has additionally been making waves with its personal inference-focused language processing unit (LPU). Primarily based on proprietary Tensor Streaming Processor (TSP) know-how, Groq additionally boasts spectacular efficiency benchmarks, vitality effectivity and aggressive pricing.

Regardless of the spectacular efficiency of Cerebras and Groq, many enterprise decision-makers could not have heard a lot about them but, primarily as a result of they’re new entrants to the sector and are nonetheless increasing their distribution channels, whereas Nvidia GPUs can be found from all main cloud suppliers. Nonetheless, each Cerebras and Groq now provide sturdy cloud computing options and promote their {hardware}. Cerebras Cloud supplies versatile pricing fashions, together with per-model and per-token choices, permitting customers to scale their workloads with out heavy upfront investments. Equally, Groq Cloud provides customers entry to its cutting-edge inference {hardware} through the cloud, boasting that customers can “switch from other providers like OpenAI by switching three lines of code”. Each firms’ cloud choices permit decision-makers to experiment with superior AI inference applied sciences at a decrease value and with better flexibility, making it comparatively simple to get began regardless of their smaller market presence in comparison with Nvidia.

How do the choices stack up?

Nvidia

• Efficiency: GPUs just like the H100 excel in parallel processing duties, however can not match the pace of the specialised CS-3 and LPU for AI inference.
• Vitality Effectivity: Whereas Nvidia has made strides in enhancing the vitality effectivity of its GPUs, they continue to be power-hungry in comparison with Cerebras and Groq’s choices.
• Scalability: GPUs are extremely scalable, with well-established strategies for connecting a number of GPUs to work on giant AI fashions.
• Flexibility: Nvidia provides in depth customization via its CUDA programming mannequin and broad software program ecosystem. This flexibility permits builders to tailor their GPU setups to a variety of computational duties past AI inference and coaching.
  Cloud Compute Entry: Nvidia GPU compute as a service is obtainable at hyperscale via many cloud suppliers, equivalent to GCP, AWS and Azure.

Cerebras

• Energy: CS-3 is a record-breaking powerhouse with 900,000 AI-optimized cores and 4 trillion transistors, able to dealing with AI fashions with as much as 24 trillion parameters. It provides peak AI efficiency of 125 petaflops, making it exceptionally environment friendly for large-scale AI fashions.
• Vitality Effectivity: The CS-3’s large single-chip design reduces the necessity for visitors between parts, which considerably lowers vitality utilization in comparison with massively networked GPU alternate options.
• Scalability: Cerebras’ WSE-3 is extremely scalable, able to supporting clusters of as much as 2048 programs, which ship as much as 256 exaflops of AI compute.
• Strategic Partnerships: Cerebras is integrating with main AI instruments like LangChain, Docker and Weights and Biases, offering a sturdy ecosystem that helps fast AI software growth.
• Cloud Compute Entry: At the moment solely out there via Cerebras Cloud, which provides versatile per-model or by per-token pricing.

Groq

What enterprise decision-makers ought to do subsequent

Given the quickly evolving panorama of AI {hardware}, enterprise decision-makers ought to take a proactive strategy to evaluating their choices. Whereas Nvidia stays the market chief, the emergence of Cerebras and Groq provides compelling alternate options to look at. Lengthy the gold customary of AI compute, Nvidia GPU now seems as a basic instrument made to do a job, reasonably than a specialised instrument optimized for its function. Function-designed AI chips such because the Cerebras CS-3 and Groq LPU could symbolize the longer term.

Listed below are some steps that enterprise leaders can take to navigate this altering panorama:

1. Assess Your AI Workloads: Decide whether or not your present and deliberate AI workloads may gain advantage from the efficiency benefits supplied by Cerebras or Groq. In case your group depends closely on LLMs or real-time AI inference, these new applied sciences might present vital advantages.
2. Assess Cloud and {Hardware} Choices: As soon as your workloads are clearly outlined, consider the cloud and {hardware} options supplied by every vendor. Take into account whether or not utilizing cloud-based compute providers, investing in on-premises {hardware}, or taking a hybrid strategy will most fit your wants.
3. Consider Vendor Ecosystems: Nvidia GPU compute is broadly out there from cloud suppliers, and its {hardware} and software program developer ecosystems are sturdy, whereas Cerebras and Groq are new gamers within the area.
4. Keep Agile and Knowledgeable: Keep agility in your decision-making course of, and guarantee your crew stays knowledgeable concerning the newest developments in AI {hardware} and cloud providers.

The entry of startup chip-makers Cerebras and Groq into the sector of AI inference modifications the sport considerably. Their specialised chips just like the CS-3 and LPU outperform the Nvidia GPU processors which have been the {industry} customary. Because the AI inference know-how market continues to evolve, enterprise decision-makers ought to frequently consider their wants and techniques.