The focus of our recent AMD deep dive in July helped emphasize how memory has become a point of fierce competition among AI accelerators.
“The MI300X requires more power than its predecessor MI250X at 750 watts, and this is higher than Nvidia’s H100 at 700 watts. However, it’s not an apples-to-apples because what the MI300X promises to deliver is running compute-intensive large language models with fewer GPUs than is required with the H100s due to offering roughly double the memory.”due to offering roughly double the memory.”
We discussed this further in the Lam Research: Wafer Fab Equipment Leader & HBM/DRAM Memory deep dive when it was stated:
“The impetus is AI acceleration, which is in a stage where it’s fiercely competing on memory […] In fact, according to Lam, AI servers use 8X DRAM and 3X NAND compared to an enterprise class server […]according to Lam, AI servers use 8X DRAM and 3X NAND compared to an enterprise class server […]
Point being, memory is becoming an important component in the AI arms race, and Lam is a major equipment supplier and beneficiary of capital intensity that follows each new generation of HBM.memory is becoming an important component in the AI arms race, and Lam is a major equipment supplier and beneficiary of capital intensity that follows each new generation of HBM.
In the analysis, we touched on the GH200 super chip from Nvidia, which “combines the H100 GPU with Grace 72-core Arm CPUs for an increase of memory capacity by 3.5X and memory bandwidth by 3X. Per the press release: HBM3e memory, which is 50% faster than current HBM3, delivers a total of 10TB/sec of combined bandwidth, allowing the new platform to run models 3.5x larger than the previous version, while improving performance with 3x faster memory bandwidth.”
Since then, more information has been revealed about Nvidia’s H200 GPUs, which will have 141GB total of HBM3e memory with 4.8 TB/s of bandwidth across six HBM3e stacks. This compares to the H100 with 80GB of HBM3 and 3.35 TB/s of bandwidth. To see how this compares to the MI300X from AMD, you can read more here.
If 2023 was the year AI accelerators made their importance known, then 2024 will be the year that memory and HBM3/HBM3E makes its importance known as the competition is going head-to-head at memory capacity and bandwidth per GPU rather than compute performance. This further translates to mean the AI race is more focused on inference for the next generation of GPUs as the neural network can be run entirely in memory without the need to move data back-and-forth with the external memory.
In fact, to drive the point further as to how important memory will be in the next generation of GPUs, the compute performance from the H100 to the H200 is not changing much. According to what the industry has seen so far from Nvidia’s GPU HGX 200 systems, there will be “32 PLOPS FP8” performance, which would be achieved through eight H100s with 3,958 teraflops of FP8 each. The translation is that Nvidia’s H200 upgrade is strategically focused on memory, which also translates to AMD having a strong sense of direction on design as it forced Nvidia to answer to the MI300X’s memory capacity and bandwidth. It’s also quite strategic for AMD to go directly toward competing on LLM inferencing performance as these are the workloads that will be most in demand.
Memory is Tough
As we review the memory market, and also segue to the PC market in this analysis, it bears mentioning that the memory is a tough market. It can often be a race to the bottom on pricing, there are rollercoaster-like cyclical patterns, it’s heavily exposed to consumer devices, and it has many global leaders not trading on the Nasdaq. Memory leaders are concentrated in APAC and so any United States stock needs to be measured against overseas competitors.
With that said, over the past year, the memory, smartphone, and PC markets have been going through inventory corrections, but these markets are close to bottoming based on recent earnings commentary from several semiconductor companies such as Samsung, Microsoft, AMD, Intel and others. Sometime in the medium term, PCs will go through a super cycle driven by AI. We want to look more closely at the timing of this and who the major players might be.
In the Lam Research analysis, it was stated that the pummeled consumer market is deceiving as eventually hybrid AI will bring AI processing capabilities to the edge, including consumer devices. Memory is a key component in the competitive AI race both inside and outside the data center. Although the introduction to this analysis emphasized data center AI accelerators, we want to start to turn our focus toward the edge.
It's also important to point out that what matters most for determining a good stock is dollar content per chip. For example, HBM3e is priced five to six times higher than typical DRAM. This means that if the shipment volume is 2% of total DRAM, then its sales ratio reaches 12%. Our goal is to find the semiconductors that can charge more for their chips in the AI super cycle for PCs rather than simply PC players.
Memory Rebound
HBM3 sales are expected to explode. According to Trend Force, HBM3 could grow to $8.9 billion in 2024 for a 127% YoY increase.
SK Hynix is projecting 100% growth in HBM demand this year and next, revised up from 50% growth. According to SK Hynix, the AI chip boom will drive an 82% CAGR for HBM3 by 2027. SK Hynix has reclaimed its spot as the number two memory company globally due to HBM sales, and was the preferred supplier of HBM3 for Nvidia’s H100 GPUs.
Samsung, the world’s largest manufacturer of DRAM and NAND by revenue, has shown a trend of QoQ improvements in revenue and operating profits since 4Q22. As long as this trend continues, then it’s likely the memory market has bottomed. According to the Korea Economic Daily, Samsung executives stated at a conference: “Our customers’ current (HBM) orders have more than doubled from last year.”
Micron shows something similar to where the company was reporting deeply negative growth and is rebounding significantly on revenue. This is easily seen on a QoQ basis pictured below.
As a percentage of revenue, Micron’s QoQ Profit Improvement is also quite clear with a recovery by the second half of 2024.
Although HBM3 will participate, the major memory suppliers are primarily rebounding from lapping a trough in a consumer cycle that had peaked following Covid. Typically, memory stocks would be on a deep discount given the steep cycle, yet SMH returns are > QQQ returns. This has lifted the tide of all boats, and memory stocks such as Micron and Lam are not trading where they’d typically trade, which makes a near-term buy less likely for the I/O Fund until we see a pullback. This is where the I/O Fund is unique, not only do we strive to be early in our research, such as to the importance of memory in the next release of AI accelerators, but we are also careful with our timing.
PC Rebound
Similar to the memory and smartphone markets, the PC market is in the process of normalizing inventory. Silicon Motion, one of the largest manufacturers of NAND flash controllers, which go into PCs and smartphones, commented in its recent earnings call that inventory for the PC and smartphone markets is normalizing and these markets are poised to return to growth in CY24.
“We saw inventory level begin to normalize across the majority of end markets and OEM order activity pick up in the third quarter leading to a strong revenue growth in the quarter. We expect this trend to continue and are confident they will lead to strong sequential growth in the fourth quarter. While the first half of 2023 was challenging due to the global macro economy weakness and excess inventory in the channels the inventory level across our end market is normalizing and OEM demand continue to improve.” -Silicon Motion, Q3 Earnings Call Nov 2023
“By end market standpoint, excess inventory in the PC and smartphone markets have plagued the industry since late 2022, when the global economy weakened and demand slowed. It has taken nearly a year where we believe the inventory level in both the PC and smartphone markets are normalizing. We are seeing more consistent order pattern from our customers and better visibility that are more closely aligned with end market demand. We are optimistic that this trend will continue and that the industry is well positioned to return to growth in 2024.” -Silicon Motion, Q3 Earnings Call Nov 2023
This commentary by Silicon Motion was also supported by AMD in its recent earnings call, in which management commented that it expects growth for the PC market in CY24, and the PC market will return to normal seasonality levels in demand.
“Year-over-year, we expect revenue for the Data Center and the Client segments to be up by strong double-digit percentage, the Gaming segment to decline, given where we are in the console cycle, and the Embedded segment to decline due to additional softening of demand in the embedded market.
Sequentially, we expect Data Center segment to grow by strong double-digit percentage, Client segment revenue to increase and the Gaming and Embedded segment to decline by double-digit percentage […]
Sales of our Ryzen 7000 processors featuring our industry-leading Ryzen AI on-chip accelerator, grew significantly in the quarter as inventory levels in the PC market normalized and demand began returning to seasonal patterns.” –AMD Q3 2023 Earnings Call
For the PC market, HP believes AI can help double the PC market’s growth over the next 3 years.
“The biggest opportunity we see is with in Personal Systems. The ability to run generative AI applications on a PC will enable personalized experiences, improved latency, provide better security and privacy protections, and reduce costs. And as we begin commercializing AI-enabled devices, we believe the overall PC category growth rate can double over the next three years.” -HPQ Analyst Day
Intel has ambitious plans to take advantage of the AI opportunity in PCs. Through its AI PC Acceleration Program, Intel is focused on having AI on more than 100M PCs by 2025. Intel is working with over 100 independent software vendors on more than 300 accelerated AI features, including well-known companies such as Adobe, Webex, and Zoom.
On December 14th, Intel will launch the Core Ultra Processors. The new processors are a 7nm chiplet design which makes updates easier, and allows for the most efficient use of the chip tiles. Especially for AI purposes, chiplets are replacing monolithic circuits, where the overall system is divided into smaller parts so that 3nm or 5nm nodes can be replaced when needed. This avoids having to replace all of the components as nodes shrink and design companies otherwise battle Moore’s Law. With chiplets, AI platforms can scale by adding computing power and reduce total cost of ownership.
The Meteor Lake architecture will come with a neural processing unit (NPU) to execute AI workloads. The goal of AI edge devices (PCs, mobile devices, and edge servers) is to diversify AI execution for both speed and power efficiency. By running the workloads across NPUs, GPUs and CPUs, the Core Ultra will drive better efficiency.
Per Intel’s management in the most recent earnings call:
“Built on Intel 4, the Intel Core Ultra has been shipping to customers for several weeks and will officially launch on December 14 alongside our 5th Gen Xeon. The Ultra represents the first client chiplet design enabled by Foveros Advanced 3D packaging technology, delivering improved power efficiency and graphics performance.
It is also the first Intel client processor to feature our integrated neural processing unit, or NPU, that enables dedicated low-power compute for AI workloads. Next year, we will deliver Arrow Lake as well as Lunar Lake, which offers our next-gen NPU, ultra-low power mobility and breakthrough performance per watt.”
Intel and also Qualcomm are set to release WiFi 7 in Q4. WiFi 7 will be two to four times faster than WiFi 6 and will have twice as many data streams. Although AI applications are not mainstream yet, WiFi 7 makes it possible to develop and distribute AI applications due to the high capacity, low latency, and extended range. The high bandwidth consumption that AI applications require, along with moving AI workloads to AI devices means WiFi 7 will be instrumental for AI-powered PCs.
Note: We will discuss Qualcomm again when we look more closely at the mobile rebound.
2024 Refresh: Windows 12
“We actually think 2024 is going to be a pretty good year for client, in particular because of the Windows refresh. We still think that the install base is pretty old, and does require a refresh. We think next year may be the start of that given the Windows catalyst.” -Intel, Citi Analyst Conference, Sept 6th
In September, a CoPilot update was rolled-out for Windows 11 with over 150 features such as CoPilot in Windows, which is a toolbar that allows generative AI to be used across any task across the Windows operating system. The new update also added AI-powered Bing search to the taskbar.
Windows 11 was an exciting update, yet the next Windows release is expected to make a much bigger impact. The Intel statement was the first to hint at the Windows refresh, which is generally understood to be Windows 12, due to come out in 2024.
We had noted that AMD also referenced a Windows refresh in the last earnings call. Per management’s opening remarks noted in our Post-ER writeup: “Looking forward, we are executing on a multiyear Ryzen AI road map to deliver leadership compute capabilities built on top of Microsoft's Windows software ecosystem to enable the new generation of AI PCs that will fundamentally redefine the computing experience over the coming years.”
It's likely that Intel and AMD are leaking the next Windows refresh because Microsoft is working closely with hardware partners to optimize chips to handle the AI workloads. The Ryzen 7000 mobile processors are the first x86 chips to contain a dedicated AI engine to support Microsoft’s Windows Studio Effects. Typically, this requires Arm-based hardware with a dedicated neural processing unit (NPU).
AI-powered PCs will ultimately change the trajectory for AI, to where more people can access AI-powered applications, which in turn, will help AI developers be able to build a bigger ecosystem. There is a major bottleneck right now for AI applications to where client devices are not powerful enough or energy efficient enough to leverage AI capabilities.
Look for Windows 12 to be a major release for 2024, and to be the official moment AI-powered PCs kickoff.
What Industry Analysts are Saying:
Gartner:
Per Gartner, “The global PC market is expected to rebound in the fourth quarter of 2023 as the beleaguered industry begins a return to growth. Worldwide PC shipments totaled 64.3 million in the third quarter of 2023, according to analysis from Gartner, marking a 9% decrease compared to the same period last year and the eighth consecutive quarterly decline.”
Additional Management Commentary:
AMD:
“Sales of our Ryzen 7000 processors, featuring our industry-leading Ryzen AI on-chip accelerator, grew significantly in the quarter as inventory levels in the PC market normalized and demand began returning to seasonal patterns.” -AMD, Q3 Earnings
“Question – Blayne Curtis: Thanks. And then I just wanted to ask on the PC market, I think you and Intel have seen you were under-shipping in the first half. Maybe you're kind of over-shipping a little bit now, restocking. I'm just kind of curious your perspective of what that normalized run rate is in terms of the size of the PC market and kind of any perspective, if inventory levels are starting to move back up.
Answer – Lisa T. Su: Yeah, I would say again, Blayne, when we look at sort of the third quarter and sort of the environment that we're on now, I think inventory levels are relatively normalized, and so the sell-in and consumption are fairly close. We were building up for holiday season that is a strong season for us overall. When I think about the size of the market, I think from a consumption standpoint, this year is probably somewhere like 250 to 255 million units or so.”
“Jean Hu
Yeah. Hi, Stacy. I'll say the first thing is, if you look at 2023, it's a very unusual year for the industry, right, especially the PC market. It's one of the worst down cycles during the last 3 decades. So during that kind of a down cycle, definitely, we had headwinds on gross margin side, on our Client business, which we have made significant progress in Q3 and Q4 in second half.”
Microsoft:
“In our consumer business, PC market unit volumes are returning to pre-pandemic levels.”
“Windows OEM revenue increased 4% year-over-year, significantly ahead of expectations driven by stronger-than-expected consumer channel inventory builds and the stabilizing PC market demand noted earlier, particularly in commercial.”
Previous Cyclical Behavior
According to Deutsche Bank, past semiconductor cycles have lasted on average of ~28 months (2 years, 4 months).
If we look at the beginning of 2018 to the end of Feb 2020, the last down cycle, the SOXX was down 14% off of its high due to the US and China trade war, which included bans on Huawei and ZTE. Semiconductor companies were negatively impacted by these bans. By the end of February 2020, semiconductor stocks were down the following from their highs:
- QRVO: down 15%
- HP: down 21%
- SWKS: down 22%
- QCOM: down 18%
- DELL: down 42%
Conclusion:
As we look toward 2024, memory and PCs are shaping up to be a predominant theme. This broad analysis looks at management commentary across a range of companies that all seem to point toward the bottom being in for PCs and memory. In the coming weeks, we will look closer at specific companies that will participate as we narrow our focus for 2024.
Recommended Reading: