AMD is more complex than Nvidia as I can simply say “near-monopoly on GPUs” or “deep moat from CUDA” and that summarizes quite nicely why Nvidia is an AI leader. Of course, there is much more to the products, and that complexity has worked in our favor.
However, AMD is far less straight forward, and coupled with the complexity of the chip market, I am not surprised there was a muted reaction to the AI conference this month. Wall Street was quite late to Nvidia’s H100 release, in fact, Nvidia’s stock was at a deep discount the very month the H100 shipped. One can only hope that AMD will be at a similar deep discount when the MI300 ships in volume in Q4 2023.
I’ve said “I’m Still Feeling Zen” with AMD during the PC-slump – a play on AMD’s Zen Architecture — and I’ve also called the company “The Dark Horse,” which refers to being an underestimated competitor. The Dark Horse reference is becoming less noteworthy given its penetration in the data center has grown 5-6X since we first started referring to the company as underestimated.
In October, I had said Nvidia was Ready to Rumble. At the exact time that Nvidia has been named AI King, I’m going to say that AMD is Ready to Rival.
Below, I go through the product lines you can expect to be Phase 1 of AMD’s AI Acceleration strategy. I believe what is described below will take us through the next two years of gains as AMD will primarily rely on the MI300A and MI300X to nibble at Nvidia’s GPU monopoly. There is also exciting things happening in CPUs with the Zen 4/4c release in the second half of the year, including a cloud optimized processor.
Later down the line, in Phase 2, AMD will benefit from recurring software revenue, hybrid AI, edge computing, FPGAs/Xilinx and Automotive. However, maintaining CPU growth coupled with a competitive GPU strategy is most important right now and I think it’s prudent to focus entirely here for the time being as our current position depends on this.
Notably, the revenue potential from Phase 2 will be quite substantial after the GPU strategy materializes. The main point to know is that major design companies will do quite well outside of the data center, so what you’re seeing now and next year will only multiply.
The goal of this particular analysis is to describe what AMD is setting out to accomplish with GPUs in as lucid a manner as possible. The opportunity in front of AMD is exciting. But, let’s first start with the risks before we go into the larger analysis as I want to make sure the risks are fully understood.
Here are the hurdles that AMD must clear to become a major AI contender – I expand on these points below.
- Lacks a popular software platform and CUDA competitor. AMD’s recently released software platform ROCM is promising but is no CUDA.
- AMD is later to market on AI acceleration in terms of GPUs. Although AMD has accomplished what is nearly impossible by being a second-place contender that crushed first-place Intel, the reality is that being in second place is a major obstacle.
- On that note, the company has its hands full competing against Intel on CPUs. It will now go up against Nvidia on GPUs. Lisa Su is one of the best CEOs in the history of the tech industry, but can she and her team take on both at the same time?
There are also a few major positives that are in AMD’s favor – I expand on these points below.
- The MI300s should be able to compete on performance once the GPUs are benchmarked as AMD’s GPUs power the world’s largest supercomputers.
- AMD is exceptional at undercutting on price. This is primarily how AMD overtook Intel coupled with a better design (the Zen 2 architecture)
- AMD’s designs are excellent at improving power efficiency. Power efficiency is important for total cost of ownership. Not only will AMD’s GPUs likely be cheaper (no confirmation on pricing just yet) but they will also cost less to own over a four-year life span.
- Hyperscalers will support competition to Nvidia. You can think of Nvidia as more of a frenemy to Big Tech. This is due to pricing power, CUDA being closed source, and also now Nvidia will be competing with Big Tech in some areas. For example, Omniverse will compete with Meta’s metaverse ambitions. I don’t think it’s a coincidence that one of AMD’s largest customers is Meta. For the MI300 release, AMD is primarily focused on hyperscalers with the CDNA GPUs and not consumer-level RDNA GPUs.
- Victor Peng, former CEO of Xilinx and now President of AMD, is an ace of spades in AMD’s pocket, as is Forrest Norrod and Jean Hu. As any epic CEO should do, Lisa Su has loaded up her team with a strong C-suite.
Brief Background on AMD-Powered Supercomputers
To understand AMD’s beginnings on AI acceleration, we have to start the discussion with supercomputers.
Supercomputers are the world’s most powerful computers and are government owned by the Department of Energy at national laboratories. Currently, Frontier is the world’s fastest supercomputer, and this is powered by AMD’s EPYC Milan CPUs and AMD’s MI250X GPUs. Infinity Fabric is essential to AMD’s architecture as it links functions of the CPUs and GPUs by providing interconnects for purposes of data exchange and memory.
Supercomputers are important for national defense purposes. Per the announcement from the Lawrence laboratory: “Besides supporting the nuclear stockpile, El Capitan will perform secondary national security missions, including nuclear nonproliferation and counterterrorism. NNSA laboratories are building machine learning and AI into computational techniques and analysis that will benefit NNSA’s primary missions and unclassified projects such as climate modeling and cancer research for DOE.”
Last year, the AMD-powered Frontier supercomputer broke Japan’s record at a speed of 1.1 exaflops, which is two times faster than the record held by Japan for two years. By breaking the 1.0 ExaFLOP/s barrier in the HPL benchmark test, AMD-powered Frontier became the world’s first exascale computer. This speed is greater than a quintillion calculations per second.
This year, AMD will be powering the launch of a new and highly anticipated supercomputer called El Capitan located in Livermore, California. The ambitious goal for El Capitan is to exceed 2 exaFLOPS of “double-precision” processing power. This supercomputer is powered by AMD EPYC Genoa CPUs and AMD’s MI300A GPUs. El Capitan will also feature AMD’s ROCm open compute software platform.
AMD’s first real-go at competing with Nvidia on commercialized AI acceleration will be this year, however, the company has been powering the world’s top performing computer for five years. I think answering this question — why did the Frontier and El Capitan projects choose AMD — is critical for understanding why AMD can rival Nvidia on GPUs in the near future. The analysis below is aimed at discussing a few advantages AMD has in terms of design.
Also, investors should note that AMD’s new GPUs will be shipping around the same time that El Capitan will launch (ETA: Oct/Nov 2023).
AMD’s CPU Zen Architecture
We’ve covered AMD’s Zen Architecture in depth a few times, including about two years ago in a 1-hour webinar on AMD and three years ago in a premium report here. The 2020 report is important because it was real-time on discussing the bullish thesis that AMD could take substantial market share from Intel in the data center. At the time, the company had 4% CPU server market share and now has over 20% market share.
We need to revisit how AMD was able to take on an 800 lb. gorilla in order to piece together how AMD plans to do it again.
Here is what was said a couple of years back – I’ve bolded what is important for the purposes of this analysis:
“In August of 2019, AMD released a competitive 7nm chip while Intel was still producing 14nm chips with a 10nm chip on the way. Essentially, AMD leapfrogged the incumbent with a product that is more power efficient and allows for more cores per chip. Because 7nm are twice as dense as 14nm, AMD was able to release a 64-core server chip and 128 threads rather than AMD’s previous 32-core server chip. Up until early 2019, Intel’s offering has been a 28-core server chip and 64 threads. […] AMD has blatantly stated the second-generation EPYC server processors had 1.8 to 2 times the performance advantage of Intel’s Xeon processor line and is half the cost in some instances.”
Here is a recent statement from Microsoft on the dramatic results from AMD’s Zen Architecture:
Source: Twitter
So, how was AMD able to outpace Intel on computing power, memory and energy use — at half the cost?
The Zen-2 architecture introduced a multi-chip module that used the most advanced technology where it’s needed most by combining 7nm chiplets with a 14nm die. This was quite a competitive leap as Intel was still using a monolithic design.
In this case, the 14nm was leveraged for memory controllers because the central hub runs input/output (I/O) and memory better. This helped AMD beat Intel on memory bandwidth. The design also greatly improved performance by putting the L2 cache on the core and the L3 cache across the core. Overall, these design improvements lower the power required while increasing the performance as it requires fewer NUMA hops, which in turn, increases instructions per clock, and this ultimately reduces latency.
From there, AMD undercut Intel on price, which becomes a virtuous cyclebecomes a virtuous cycle as driving down costs means more chips will be bought from AMD. We also mentioned in the 2021 webinar that at the time, a third-party analyst named Michael Larabel benchmarked AMD as being 14% faster than Intel while costing about 30% less.
In what can a be an industry full of jargon, this is most important point in my previous AMD analysis as to why AMD went from 2-4% CPU server share to 20%+ when AMD’s Rome went up against Intel Xeon Cascade Lake:
“It’s estimated that for every $1.00 in Rome chip sales, Intel loses $2.25 on average in Intel Xeon SP sales. The savings are then deployed to buy more Rome chips, which can further depress Intel’s revenue.”$1.00 in Rome chip sales, Intel loses $2.25 on average in Intel Xeon SP sales. The savings are then deployed to buy more Rome chips, which can further depress Intel’s revenue.”
The older Rome Series Zen 2 architecture is what was discussed in our webinar. Meanwhile, the Milan Series is the current series driving forward AMD’s data center growth. The Milan Series Zen 3 architecture has made improvements in performance largely due to 3D stacking. By incorporating 3D stacking in Zen 3, AMD was able to triple the L3 cache size while only adding four clock cycles of latency. When 3D stacking is incorporated with GPUs, the result is computers that train neural nets up to 40 percent faster with 16 percent less energy.
Next up is Bergamo, the CPU line specifically designed for cloud native workloads. In this case, Bergamo will have less cache and more performance per watt. During the conference, Meta was on stage with AMD to attest to Bergamo having 2.5X better throughput performance. Part of the upcoming release includes Siena, which will drive more dollar performance per watt at the edge for telco customers. Notably, Genoa and Genoa-X will continue to provide more cache for general purpose workloads.
Total Cost of Ownership:
Total cost of ownership (TCO) refers to the total cost to own and operate equipment over its useful life span. TCO is a motivating factor for hyperscalers when evaluating equipment as it factors in not only the acquisition cost but also the costs associated with owning and operating the equipment over the hardware life cycle.
For example, Spiceworks reported that “in 2010, the research firm Gartner estimated that for a desktop PC priced at $1,000, the total cost of ownership works out to at least $2,680 per year when you factor in things like capital expenditures, labor expenses of supporting the computer, and indirect costs such as lost end-user productivity due to downtime.” Over a four-year lifespan, the costs are estimated to be $9,800 to $17,600.
We will have to get the benchmarks after the MI300s are released, but power consumption is a major contributor to higher TCO. In addition to this, if a company has to buy more GPUs to train and run similar size LLMs, then this would (theoretically) also contribute to a higher TCO for Nvidia equipment.
In the past, AMD advertised up to 20% Capex savings compared to Intel based on Epyc processors delivering more performance from a single chip compared to Intel’s dual-processor powered by two CPUs. Big Tech has capex budgets into the tens of billions. Although it’s not specifically disclosed exactly how much goes toward AI acceleration, we know that Big Tech is driving forward Nvidia’s GPU sales at $8 billion per quarter or $35 billion to $40 billion per year.
Here is the thesis in a nutshell: If a competitor can deliver 20% savings on this kind of budget with similar performance, then it will turn heads. We can geek out all day long on the computing performance of Nvidia’s H100 GPU, however, if the MI300s drive down total cost of ownership through low unit pricing, better power efficiency and reducing the number of GPUs required, then hyperscalers will line up to support this.
What Google, Amazon, Microsoft, Meta and large enterprises want most of all is to build incredible AI systems but at a manageable cost. This goes back to the virtuous cycle. The more they save, the more they can build.
If Big Tech capex goes further with AMD, then that will be something Nvidia will be forced to address. Nvidia is unrivaled right now on GPUs, which means their pricing power is unrivaled. The H100 is in short supply, and the A100 may be being stockpiled by China before United States sanctions take effect. The lead times on the H100 and A100 are into 2024 at this point, which means there is no better time for AMD to enter the market and undercut on price/TCO than right now.
In the next section, we discuss how the MI300A and MI300X GPUs may be following a similar path as the Rome, Milan and Genoa CPUs.
MI300A and MI300X GPUs
The El Capitan Supercomputer is expected to launch this Fall. When it launches, El Capitan is expected to follow a similar system as Frontier which is (1) AMD Epyc CPU with (4) MI300 GPUs with Infinity Fabric. The “A” in the MI300 stands for APU, which refers to a CPU being combined with a GPU. Nvidia has only recently attempted this at the HPC level with the Grace CPU and H100 GPU, but this is technically two discrete devices with separate memories.
By having a fully shared, coherent memory, the MI300A architecture reduces latency while enabling high bandwidth. The high-speed, low-latency unified memory helps improve speed while allowing the CPU and GPU to do what they do best. By allowing both processor types to access shared memory, HPC programming is more efficient.
Notably, AMD has been successful in releasing APUs for PCs and gaming. Technically, APUs underperform GPUs when it comes to gaming but outperform on PCs as they don’t use as much power as dedicated GPUs.
Here are the specs:
- 24 CPU Cores comprised of three Genoa eight-core chiplets
- 128 GB HBM3 memory with approx. 5TB/second of memory bandwidth
- (9) 5nm compute logic chiplets and (4) 6nm base dies
- Shared memory
MI300X
According to AMD, the MI300X will have 2.4X the memory density of the H100 and 1.6X the memory bandwidth. The reason that the MI300X was able to run the popular Falcon-40B large language model (LLM) with 40 parameters is because the neural network was ran entirely in memory without the need to move data back-and-forth with the external memory. AMD also stated the MI300X will be able to run up to 80B parameters on a single chip.
- (8) 5nm GPUs and (4) 6nm base dies
- 153 billion transistors
- 192GB of HMB3 memory. 5TB/second of memory bandwidth
- Ran a 40B parameter large language model (LLM) on a single GPU (unprecedented)
- Can scale up to 8 accelerators in a single package for cutting-edge generative AI LLMs
The MI300X requires more power than its predecessor MI250X at 750 watts, and this is higher than Nvidia’s H100 at 700 watts. However, it’s not an apples-to-apples because what the MI300X promises to deliver is running compute-intensive large language models with fewer GPUs than is required with the H100s due to offering roughly double the memory.
The need for fewer GPUs is accomplished by running LLMs in the memory. The image below shows why having 2.4X memory at 192GB compared to Nvidia’s 80GB will reduce the number of GPUs required for running popular large language models.
Here is what Dr. Lisa Su said at the recent AI conference:
“For the largest models, that actually reduces the number of GPUs you need, significantly speeding up the performance, especially for inference, as well as reducing the total cost of ownership."
In terms of the MI250X versus the MI300X, the newer model is powering nearly three times more transistors than its predecessor. There is also a lower power variant of the MI300X expected in early 2024. In addition to this, AMD offers chiplet and packaging technologies that reduce power requirements.
To compete with Nvidia’s DGX systems, AMD is also releasing the AMD Instinct Platform which will combine eight MI300X systems with 1.5B terabytes of HBM3 memory. The server utilizes the Open Compute Platform specifications so that it’s compatible with existing hyperscaler infrastructure (more on this below).
Quick note on AMD Radeon RX Series Gaming GPUs
We are invested in AMD for the company’s AI potential. However, it makes sense to touch base on gaming as graphics processing units (GPUs) are first and foremost gaming chips. At the time that Nvidia was founded in the early 1990s, and up until recently, gaming was one of the most computationally challenging use cases for hardware.
Nvidia is the inventor of GPUs, and for the first two decades or so, NVDA was primarily a gaming stock. Part of our original thesis was that the AI era would be upon us when AI revenue overtakes gaming revenue for Nvidia, as this would help pinpoint when Nvidia’s industry had officially changed. It’s a big moment when something like this happens (Mac/PC overtaken by iPhone/Mobile revenue, etc.).
Since gaming is what led to Nvidia’s GPU positioning for AI/ML, it makes sense to note that AMD is a decent contender on gaming GPUs. Here’s a snapshot of how gaming GPUs rank in 2023 from industry-expert Tom’s Hardware:
PC Gamer stated that AMD has 9% of market share compared to Nvidia’s roughly 82%. This is hard to rely on for concrete numbers as GPUs are down nearly 50% year-over-year and both Nvidia and AMD’s bigger gaming releases occurred in Q4. The upcoming 2023 numbers will represent the market share a bit better. In the past, we’ve reported that Peddie’s numbers were 25% AMD and 75% Nvidia. According to PC Gamer, Intel may be eating into AMD’s market share, but we won’t know this definitively until there’s reports on Radeon RX 7900 numbers.
Open Accelerator Module (OAM):
In 2019, Meta and Microsoft led a coalition to create open standards that allow for choice in processor and accelerator. The OAM came together to design packaging and motherboard socketing technology that allows accelerators with different sockets and thermals to be consistently deployed.
The OAM form factor replaces the PCIe form factor accelerator cards. The OAM form factor is also ideal for interoperability across custom silicon, such as ASICs, which experience excessive signal insertion loss to PCIe connectors and the baseboard.
AMD’s Instinct GPU accelerators feature OAM baseboards. These universal hardware design standards allow IT departments to choose a new GPU architecture with a more simplified installation and provides the ability to upgrade at any time. The IT departments may or may not choose AMD over Nvidia but the process is easier with OAMs and open standards.
ROCm Open Software (AMD) versus CUDA (Nvidia) & Other AMD Weaknesses
Are you feeling bulled up? If only it were so simple! The predominant weakness AMD must address is ROCm open software versus CUDA. Our original thesis on Nvidia in 2018 pointed toward developers/CUDA being the primary moatthe primary moat. The CUDA software moat will be tougher to disrupt than Nvidia’s A100 and H100 hardware lead. This is because developers have to install new drivers, compilers, and will have to learn new libraries and tools. Meanwhile, Nvidia’s closed source CUDA has everything a developer needs to support code development. So, why would a developer switch? Answering this will not be so easy for AMD to answer compared to GPU performance, power efficiency, and total cost of ownership.
The long history of support CUDA that CUDA offers will be very hard for AMD to shake as it requires time to build up proper support, including libraries and frameworks. AMD’s ROCm has only a fraction of the libraries that CUDA offers. Meanwhile, software engineers in AI/ML lack only one thing: time. It’s a very competitive and fast-moving space, and AI engineers are in high demand. AI startups are getting more funding than any other type of startup right now. Money is tight for startups everywhere – except in AI.
Speaking of startups, AMD is particularly lacking in terms of bottom-up adoption and revenue, which refers to employees at a lower level helping to drive adoption rather than top-down from the C-suite. This is because AMD is weaker in terms of its consumer-level RDNA cards as they lack matrix cores for machine learning purposes.
In December, the company released RDNA3 which has more matrix operations, but does not compare to dedicated matrix cores. The CDNA GPUs from AMD are aimed at hyperscalers and fully satisfy AI/ML operations in this regard, however, the price is prohibitive for individual developers and smaller startups.
An example of ROCm lacking support is that the open source development platform underperforms with the popular 3D modeling program called Blender and does not offer support for bugs. Currently there is a statement on the wiki linux page: “ROCm HIP is known to currently have issuesissues with cycles in the while in the 3D Viewport (refer to the "issue" cited before to find workarounds), however rendering with Render > Render Image or F12 should work fine.”
Overall, developers not only must take the time to learn a new development platform but are likely to encounter roadblocks in terms of support for popular programs.
Pytorch is a popular, deep learning framework that has natively supported ROCm since 2021 and also supports CUDA directly in the interface. In June of 2022, ROCm 5.3 moved from Beta to Stable on the Pytorch 1.12 framework. This a good date to work with (June of 2022) in terms of when ROCm officially launched for AI/ML development.
PyTorch was founded by Meta’s AI research team, and it quickly overtook Google’s framework TensorFlow. The current version PyTorch 2.0 utilizes OpenAI’s Triton software stack. OpenAI open-sourced Triton in an effort to circumvent Nvidia’s closed source CUDA libraries.
Pictured Above: Beta tested throughput performance for AMD’s CDNA architecture with ROCm software platform and PyTorch Framework and libraries. You can expect this to increase dramatically with the MI300.throughput performance for AMD’s CDNA architecture with ROCm software platform and PyTorch Framework and libraries. You can expect this to increase dramatically with the MI300.
What we discussed earlier in the analysis is that the MI300A GPUs are unique due to the unified, coherent memory. It’s expected that AMD will offer memory models on the PyTorch framework to help developers optimize memory usage in AI acceleration.
ROCm is currently only supported on Linux but is expected to support Windows soon. This further prevents adoption as a serious contender to CUDA should be supported on both major operating systems.
Conclusion:
The MI300s are on the way and there is pent-up demand for GPUs. Prices are high, lead times are long, and the race for AI is fierce. AMD is an equation where strong management + strong track record on CPUs + lower prices, lower power usage, and fewer GPUs for LLMs = the one and only contender that can take on Nvidia. The MI300s have been in development for longer than one might assume and they arrive in volume in Q4.
Let’s get ready to rumble! –Michael Buffer