Super Micro, also known as Supermicro, is sandwiched in the AI trend between hyperscalers and major chip design companies. The company is a server maker that started off by making motherboards and other components before it began making complete systems. The company is unique in that it sits between being an equipment manufacturer (Dell, HP) and being a design manufacturer (Foxconn).
The company competes with Dell, IBM, Hewlett Packard, and China’s Inspur. The chart below gives you a general idea of the landscape although Supermicro has doubled its revenue since 2021. The server maker is now a $6.8 billion company and ended 2021 as a $3.5B company (Counterpoint estimates were about $1B off) but the chart below is useful in visualizing the competitors.
The word “competitor” is being used loosely here as many of these companies will not necessarily be able to compete on liquid cooling for AI development platforms, or with Supermicro’s Building Block design. The companies pictured above have stagnated and this has worked out for Supermicro, a company that could have stagnated but continued to innovate instead.
According to IDC, the worldwide server market forecast is expected to deceleratedecelerate from 20% to 0%. Supermicro declined in revenue going into 2023, however, according to management, this is due to supply chain issues. This is important to distinguish as the 2023 bull case for Supermicro rests on the high demand the company is seeing, that due to supply chain issues, the company is unable to fill. The prevailing bull thesis is that Supermicro’s supply chain issues will ease in the near term.
According to the most recent Investor’s Presentation, Supermicro grew 5X faster compared to the industry average for subsystems and server systems. While there is pressure for tech management teams to cut costs, Supermicro may be more insulated by working closely on making AI systems with the most cutting-edge chips. This includes AMD 4th Generation Zen Epyc processors, Intel Xeon and soon Sapphire Rapids, Nvidia Grace CPUs and Ampere Arm-based third-gen CPUs, Nvidia’s A100 and H100s GPUs, AMD’s MI250 and MI300 GPUs and Intel’s Ponte Vecchio GPUs.
Another piece to the bull case for Supermicro is the near-term goal to reach $10 billion, which will put the company behind Dell and Hewlett Packard. Should the company reach its $20 billion long-term goal, then it could very well be the leader or at least a strong rival to Dell and HP. If/When this happens, it’ll be due to AI systems. It was stated on the call that AI/GPU and rack-scale solutions represented 29% of our total revenue and the company expects “significant future growth.”
Supermicro’s revenue quickly accelerated last year due to one large customer in Q3 2022 to Q1 2023. This one large customer, which was later identified as Meta, accounted for upward of 20% of revenue in June and September of 2022. By December, Meta had accounted for 10% of revenue. This was the subject of a short report. However, if you invest in a small cap or low mid-cap semiconductor company, there is going to be high customer concentration.
There was also a hint on the call that another customer may be ramping: “an existing Cloud Service Provider customer represented more than 10% of revenues for the first time.”
Liquid Cooling
As the performance of CPUs and GPUs increase, the heat these systems generate increases. Liquid cooling is becoming a popular alternative to air cooling to sustain maximum performance with the added benefit of driving down costs for supercomputers. According to a press release in 2021, liquid cooling can improve data center power usage effectiveness (PUE) and total cost of ownership (TCO) “by over 40% on power costs.”
Here's a quote from the CEO on the importance of this competitive advantage:
“The power consumption and thermal challenges of these new technologies have risen dramatically and 40KW or even 80KW rack solution demands are getting stronger and popular for computing hungry DC and industries. Having high power efficiency and air/liquid thermal expertise has become one of our key differentiators of success.”Having high power efficiency and air/liquid thermal expertise has become one of our key differentiators of success.”
In 2022, Supermicro stated that liquid cooling is being used in 10% of supercomputers but will grow to be used in the “vast majority” in order to offset the heat generated by power-consuming components. The company offers Direct to Chip cooling, Immersion cooling and Rear-door Heat Exchanger cooling. This design works better than air cooling, which needs air conditioning and server fans to run constantly.
Direct to Chip Cooling: Running a cold liquid over the top of a running chip by using a pump to circulate liquid. This is a closed loop system, or also known as a self-contained cooling system.
Immersion Cooling: The system is immersed in liquid for cooling.
Rear Door Heat Exchanger: Uses a specialized rear door to the rack where coolant absorbs the heat.
Water removes heat better than air. Liquid molecules are closer together than air molecules, which results in higher heat transfer. Artificial intelligence/Machine Learning and Big Data require massive amounts of data processing, and as future generations of CPUs and GPUs are released, these systems will exceed air cooling capacity.
Liquid cooling also solves CPU throttling, which occurs when CPUs and GPUs overheat and are throttled back to avoid damage to the chip.
AI Development Platforms
AI development platforms remove the need for disparate hardware and software by offering an end-to-end platform. Supermicro has partnered with Nvidia to offer Certified systems with the new H100 GPUs for the Nvidia AI Enterprise Platform.
These systems come with 3-year AI enterprise software subscriptions, and include workflows, frameworks, pretrained models and infrastructure optimization, in the cloud, in the data center and at the edge.
Supermicro is closely partnered with Nvidia on the H100 GPU rollout with air flow designs that reduce fan speeds, lower power consumption, lower noise levels and lower the total cost of ownership.
The most recent system announced in March enables AI workloads to be run in offices and the system can be rack-mounted, as well, for data center environments. The self-contained cooling system reduces operating expenses and helps the machine to run quietly for AI, deep learning, machine learning and high-performance computing (HPC) applications.
Supermicro is able to deliver workload optimized products quickly because of its building block design. The AI market is moving quickly, and SMCI can build and validate systems partly due to a modular design that allows systems to be updated from new products, such as when Nvidia, AMD or Intel have new design releases.
Financials:
Supermicro saw strong price action due to strong guidance for next quarter and due to strong guidance of 20% revenue growth for fiscal year 2024, which begins in July.
The current quarter revenue and EPS missed management guidance and analyst estimates. SMCI reported $1.28 billion for fiscal Q3 growth of (5%), which missed guidance of $1.48 billion, at the midpoint. GAAP EPS was $1.53 and Adjusted EPS was $1.71. This compares to management guidance of: “GAAP diluted net income per share of $1.75 to $2.02 and non-GAAP diluted net income per share of $1.88 to $2.14.”
This was Supermicro’s first miss dating back to 2019. Management said the following about the miss: “The shortfall was primarily due to key new component shortages for Supermicro’s new generation server platforms which have been mostly resolved to-date.”
The fiscal Q4 revenue guide was for $1.7B to $1.9B, which is above the $1.64B that analysts were expecting. The EPS forecast form management is for $2.21 to $2.71, compared to analyst expectations of $1.76.
Per the earnings call: “If supply conditions improve sooner, we expect to be above that range, despite some economic headwinds ahead. In other words, I continue to expect our fiscal year 2024 revenue to be at least 20% year-over-year growth and we are accelerating to reach our mid- to-long-term growth objectives of $20 billion per year.”
According to a previous earnings call, the CFO stated: “GPU prices and CPU prices are going up, especially with the new refreshes that are coming out. So we anticipate that [average selling prices] will continue to go up.”
At 20% growth, the company will surpass $8 billion in revenue next fiscal year, ending in June. The FY2025 analyst estimates are for growth of 11%.
Margins:
Server solutions and systems come with thin gross margins. Despite thin margins, Supermicro is a company with strong earnings with EPS of $10.73 for FY2023. Please see below for questions from analysts on the call regarding gross margin.
Gross Margin of 17.6% compares to 15.5% in the year ago quarter. The company stated GM was lower due to product mix and new platform ramps.
GAAP operating margin of 7.7% compares to 6.6% in the year ago quarter. This is lower compared to previous quarters this year in the 9% to 10% range. The company stated it was lower due to lower revenue.
GAAP net margin of 6.7% is up from 5.7% a year ago. This is lower compared to previous quarters this year in the 9% to 10% range.
Cash Flow:
The company reported strong cash flow in the current quarter with a margin of 15.5% for operating cash flow and 14.8% margin on free cash flow. This equals $198 million and $190 million, respectively. There is $362 million on the balance sheet and $187 million in debt.
Notably, the company has lumpy free cash flow with FY2022 and FY2019 ending negative. Here’s a snapshot of the most recent quarters:
According to management, Supermicro’s decline in revenue growth is due to supply chain issues. Per the CFO’s opening remarks:
“Fiscal Q3, 2023 revenues were $1.28 billion, down 5% year-over-year and down 29% quarter-over-quarter, which was below our initial guidance range of $1.42 billion to $1.52 billion. The shortfall was primarily due to key new component shortages for Supermicro’s new generation server platforms which have been mostly resolved to-date.
We note that our shipments against a record backlog may be constrained by supply chain bottlenecks due to high demand for our advanced AI server platforms.”
Supermicro builds complete systems, and the supply chain issues can extend beyond CPUs, GPUs and memory to also include difficulty obtaining metal-oxide-semiconductor field-effect transistors (MOSFETs), diodes and capacitors. If there is low availability with these components, the systems won’t be complete in order to ship. According to The Register, lead times were at 26 weeks back in October compared to the 10-14 weeks that is the target for a healthy supply chain. This is an improvement from 40 weeks a year ago.
Here was another comment on the earnings call:
“These new AI product demands from top-tier companies have led us to challenges in terms of new key components availability.
Compounded with the economic headwind, our Q3 results were reflective of these difficult yet opportune conditions. The good news is that we have already started to address these component shortage pressures over the past few months and we are in a much-improved situation going forward. We have started to produce and ship some back orders since April.”
Risks:
Investors risk entering a frothy AI market. Most tech investors have mastered the extreme exuberance followed by extreme fear that tech seems to oscillate between. We are nearing extreme exuberance on AI with social media exploding over Chat-GPT and Bard. It’s rarely a good sign, and I’m saying that as someone who is exposed to AI-related stocks and benefits from this exuberance. Entering AI stocks right now should be done with a stop in mind.
This company had a short report out earlier this year that caused the stock to selloff. You can read the concerns here.
Regarding fundamentals, the gross margin is the primary concern raised in the earnings calls.
Yeah. So, Nehal, we — yeah. Back two years ago, we gave a 17% to 21% — 23% topline growth. Obviously, we’re in there at a minimum of 20%. And for the gross margins, we continue to, like I said, to wrestle with taking market share and also balancing that against gross margins.
But we’re confident with our new manufacturing facilities coming online that we will be able to improve our gross margins. And we also, as we come out of this quarter and we begin to ramp our new product offerings that we will be able to improve margins as well.
Here was another question about the gross margin:
Ananda Baruah:
But I would love to get a better understanding of how you guys are thinking about sort of the gross margin manifestation if we think about the continued layering in of larger footprint, which may come at a slightly lower margin. Is it really that over time, we just expect a greater presence of that lower margin business with some efficiency gains or is it just in the beginning here, the margin will be lower for the new business, but then collectively, the P&L gross margin expands over time?
David Weigand:
Yeah. So we’re looking at it and on — in the — your latter alternative, Ananda, and here’s why. So right now, there’s three things that we’ve been facing. We’re having to face more air transportation costs in order to make our deliveries. So that impacts our margin. And also, we’re having to pay other expedite fees. That impacts our margin.
Number two, we ran a lot less through our factories than in Q3 than we did in Q2. So your margin efficiency, your ability to spread your fixed costs, it’s tremendously impacted on a smaller scale. So as we scale up, we improve our margins.
Thirdly, the — as we ramped our new product offerings, there is an efficiency on these new — on the production of these new products. So we are going to improve the efficiency of these products, which will improve the margin. And so those three things alone speak to margin improvements.
Charles Liang
I can add some color. I mean, as I shared, I mean, we are building a $20 billion of revenue, hopefully in midterm and that’s why a grow our capacity and support a large customer is very important to us. Once our volume becomes higher, our costs will be improved and then business operation efficiency will be higher.
So we are doing better great way to grow our revenue. And so, I mean, once we start to reach that number under $10 billion to $20 billion, I guess, our gross margin will start to grow, because we won’t always invest for big growth after that.
Valuation:
Supermicro has an old school semiconductor top line valuation that reflects its roots as a server maker. Interesting enough, it’s trading at its previous 2015 high. The 5-year median is 0.46 but there’s been too big of a product pivot to rely on this for the future valuation.
On the bottom line, SMCI trades in line with Intel. The bottom line is probably a better gauge for this stock than top line. I took a screenshot with July 2022 marked so you could see where AMD typically trades when it’s not going through a major cyclical event with PCs. It also shows where Intel and SMCI were back in July versus now – about 50% higher valuations.
The 5-year median for SMCI is 15 and has been up to a 20 5-year median in the past.
Conclusion:
The AI market is frothy but we may take a shot at entering. If we stop out, then no big deal. We’d like exposure to Supermicro for its ambitions to overtake the incumbents, plus the clear path the company will take to do so.
In the past, we have written about the importance of Big Tech’s capex programs and its impact on demand for semiconductors. Particularly in 2021 and 2022, where there was a significant increase in data center and cloud computing related capex. It has been our position that Big Tech capex – which includes Google, Meta, Amazon and Microsoft – is a leading indicator for AI semiconductor companies and has been a secular tailwind for our holdings such as Nvidia and AMD. Now that Big Tech have reported their fiscal 2022 earnings, we thought it’d be a good time to review the 2023 capital expenditure outlook for the IT market and Big Tech.
2023 IT Market Spending Forecasts
In January 2023, Gartner released their 2023 forecasts for overall IT spending. Gartner forecasts growth of $4.5 trillion, an increase of 2.2% from 2022. Looking at the breakdown, Software and IT services continue to see meaningful y/y growth. Meanwhile, after exhibiting healthy 12% growth in 2022, Data Centers is forecasted to be almost flat at 0.7% in 2023. Devices continues to be negatively impacted by inflationary pressures impacting consumer demand.
In contrast to Gartner’s 2023 forecast of flat growth in overall Data Center spending. The growth in Hyperscale Data Centers is forecasted to grow at levels that vastly outpaces Data Centers. Hyperscale Data Centers are large data centers operated by Amazon, Microsoft and Google.
According to Precedence Research, The global hyperscale data center market size was estimated at USD 62 billion in 2021 and is expected to hit around USD 593 billion by 2030, a forecasted growth rate (CAGR) of 28.52% during the forecast period 2022 to 2030.
This growth is also reflected in forecasts for the Artificial Intelligence Chip market. In December 2022, Allied Market Research forecasts that the global artificial intelligence chip market will grow from $11.2 billion in 2021 to reach $263.6 billion by 2031, growing at a CAGR of 37.1% from 2022 to 2031. AI chips – supplied by Nvidia and AMD – will provide the computing power necessary to drive these hyperscale data centers.
Big Tech FY2023 Earnings Commentary
How did the recent Big Tech commentary on 2023 capex align with these market forecasts? Overall, Big Tech has forecasted capex to be flat to slightly down y/y. However, an important theme was a shift toward higher ROI capex such as technical infrastructure and reduction in lower ROI capex, such as office facilities. After embarking on an aggressive capex program in 2021 and 2022, Big Tech has taken a pause to reassess their cost base and to reprioritize capex in light of the current macro environment.
Put another way, the size of the capex pie isn’t expected to grow in 2023 compared to 2022, but the slice spent on technical infrastructure (i.e. Cloud and AI), will grow at the expense of labor, office facilities etc. A change in capex mix that we believe is supportive in the medium-term of NVDA and AMD.
In 2016, Big Tech in total spent $30b in capex, in 2022 they spent $150b, a five-fold increase. Big Tech commentary indicates 2023 capex will be flat to slightly lower than 2022.
What did FAAMG say about 2023?
Alphabet:
Google spent $31.5b on capex in 2022 compared to $24.6b in 2021 and forecasted 2023 to be at a similar level to 2022. Although the forecasted growth rate in capex is lower than historical levels. Management commentary around capex was very telling on where the priorities lay. On the Q422 call, management referenced AI a total of 56 times in relation to its importance to the future growth of the company. Here are a few snippets that stood out with an emphasis on AI being Google’s #1 priority.
Sundar Pichai, CEO
I'll focus on two major things today in a bit more detail, and then I'll give a shorter-than-usual quarterly snapshot from across our business. First, how we unlock the incredible opportunities AI enables for consumers, our partners and for our business; and second, how we focus our investments and make necessary decisions as a company to get there … the AI opportunity ahead. AI is the most profound technology we are working on today. Our talented researchers, infrastructure and technology make us extremely well positioned, as AI reaches an inflection point.
Our AI is a powerful enabler for businesses and organizations of all sizes and we have much more to come here. There's a few flavors of this. Google Cloud is making our technological leadership in AI available to customers via our Cloud AI platform, including infrastructure and tools for developers and data scientists like Vertex AI.
AI also continues to improve Google's other products dramatically
On the AI side, it is a really exciting time. I think we've been investing for a while, and it's clear that the market is ready. Consumers are interested in trying out new experiences. I think I feel comfortable with all the investments we have made in making sure we can develop AI responsibly.
Philip Schindler, CMO
Going forward, we are focused on growing revenues on top of this higher base through AI-driven innovation. Sundar highlighted the incredible opportunities underway with AI and the transformative impact it will have on businesses. Already, breakthroughs in everything from natural language understanding to generative AI are fueling our ability to deliver results that drive meaningful performance for advertisers and are useful to users.
Ruth Porat, CFO
And as I indicated in opening comments, when we look at capex for 2023, we do expect it's going to be generally in line with 2022 with an important mix shift. We're increasing our investments in technical infrastructure. And that's not just for AI. That's to support investments across Alphabet, in particular in Cloud as well. And at the same time, we're meaningfully decreasing our capex for office facilities.
With AI, this is obviously an Alphabet strategic priority, and we see huge opportunity ahead
Meta:
For Meta, capital expenditures, including principal payments on finance leases, was $32b billion for 2022 compared to $19.3b in 2021. 2022 capex was driven by investments in servers, data centers and network infrastructure. Meta forecasted 2023 capex to be between $30-33b down from their prior guidance of $34-37b. Similar to Google, management commentary around AI and capex was very telling on where the priorities lay.
Mark Zuckerberg, CEO
Now before getting into our product priorities, I want to discuss my management theme for 2023, which is the Year of Efficiency. We closed last year with some difficult layoffs and restructuring some teams. And when we did this, I said clearly that this was the beginning of our focus on efficiency and not the end. And since then, we have taken some additional steps, like working with our infrastructure team on how to deliver our roadmap while spending less on capex
And next, I want to give some updates on our priority areas. Our priorities haven’t changed since last year. The two major technological waves driving our roadmap are AI today and over the longer term, the metaverse.
AI, it’s the foundation of our discovery engine and our ads business. And we also think that it’s going to enable many new products and additional transformations in our apps. Generative AI is an extremely exciting new area with so many different applications. And one of my goals for Meta is to build on our research to become a leader in generative AI in addition to our leading work in recommendation AI.
Yes, I can start with generative AI. Yes, I think this is a really exciting area. And I mean, I’d say the two biggest themes that focused on for this year and one is efficiency and then the kind of the new product area is going to be the generative AI work.
A lot of the trends that we are seeing here is, we are using larger models, which require more computation. We have shifted the models from being more CPU-based to being GPU-based
There is a positive readthrough on Zuckerberg’s comment on the shift from CPU to GPU models. This could potentially benefit Nvidia and their H100 GPU.
Susan Li, CFO
Turning now to the capex outlook for 2023, we expect capital expenditures to be in the range of $30 billion to $33 billion, lowered from our prior estimate of $34 billion to $37 billion. The reduced outlook reflects our updated plans for lower data center construction spend in 2023 as we shift to a new data center architecture that is more cost efficient and can support both AI and non-AI workloads
So we’re shifting our data centers to a new architecture that can more efficiently support both AI and non-AI workloads. And that’s going to give us more optionality as we better understand our demand for AI over time. Additionally, we’re expecting that the new design will be cheaper and faster to build than previous data center architecture. Along with the new data center architecture, we’re going to optimize our approach to building data centers. So we have a new phased approach that allows us to build base plans with less initial capacity and less initial capital outlay, but then flex up future capacity quickly if needed. We’re still planning to grow AI capacity significantly, and that connects
The current surge in capex is really due to the building out of AI infrastructure, which we really began last year and are continuing into this year. We will be measuring the ROI of these AI investments, and their returns will continue to inform our future spend. Our intention is still to bring capex as a percent of revenue down, but capital intensity in the nearest term is really going to depend, in part, on the revenue outlook and our needs to further build AI capacity for future demand
Javier Olivan, COO
I think if you look at the strategy on ads, we really have two parts, which is continue investing in AI and that’s where we are seeing a lot of the improvement in ads relevance.
Microsoft:
For Microsoft FY 2022 capex, including assets acquired under financial leases, was $29.2 and compared $24.2 to FY 2021. For FY 2023, Microsoft has stated “… we expect a sequential decrease on a dollar basis with normal quarterly spend variability in the timing of our cloud infrastructure buildout.”
Satya Nadella – Chairman and Chief Executive Officer
The age of AI is upon us and Microsoft is powering it. We are witnessing non-linear improvements in capability of foundation models, which we are making available as platforms. And as customers select their cloud providers and invest in new workloads, we are well positioned to capture that opportunity as a leader in AI. We have the most powerful AI supercomputing infrastructure in the cloud. It’s being used by customers and partners like OpenAI to train state-of-the-art models and services, including ChatGPT.
Amazon:
For Amazon, capex including equipment financial leases, was $58.3b in 2022 compared to $55b in 2021. These expenditures primarily reflect investments in technology infrastructure. In the past, management has indicated that about 50% of total capex has gone toward infrastructure. Management gave no guidance for 2023 other that these investments will continue.
Conclusions
Big Tech is not immune to the weaker macroeconomy nor consumer. This has been evident in their earnings releases. For Big Tech’s next capex act, their commentary focused on shifting capex to higher ROI investments with a focus on cost efficiency. These comments have increased our conviction that investments in AI are a key strategic priority and will continue.
From an investing perspective, it supports our investment thesis in Nvidia and AMD. Nvidia’s new H100 GPU chip has positioned it to benefit from the buildout in AI related and hyperscale data center infrastructure. Critically, given their dominant market position in AI chips, this will enable Nvidia to then monetize and gain a greater share in the software stack. In addition, AMD plans to commercially release its MI300 GPU this year.
Per the most recent AMD earnings call:
“MI300 will be the industry's first data center chip that combines a CPU, GPU and memory into a single integrated design, delivering 8x more performance and 5x better efficiency for HPC and AI workloads, compared to our MI250 accelerator currently powering the world's fastest supercomputer. MI300 is on track to begin sampling to lead customers later this quarter and launch in the second half of 2023.”
In the most recent earnings report, Nvidia management commented that the H100 adoption rate and software monetization at the enterprise level is happening faster than expected. We will further outline how Nvidia is well positioned to benefit from this spending in AI and what to look for in Nvidia’s upcoming earnings report. We’ve recently covered AMD here.
There were a few key things discussed on the call:
Why the H100 will ramp faster than the A100 with Q1 being the estimated time when we should see the H100 driving forward data center growth (we should get an acceleration this quarter in the DC segment).
How the H100 helps drive enterprise software revenue as it’s been optimized with Nvidia’s software stack.
The strength of Nvidia’s networking business following the acquisition of Mellanox.
The CEO believes even if hyperscalers slowdown spending in 2023, that Nvidia is more insulated because their systems are optimized for AI acceleration, which is a top priority within capex spending, and because their systems reduce costs and improve efficiency.
My specific investment thesis is this: “the A100 GPU is what led the company’s gains since Q2 2020 (detailed here) and the Hopper H100 GPU is what will lead the company’s gains for the next two years.”I detail what I think is the most important Q&A from the call below plus other moments that provide a glimpse into what Nvidia investors can expect from here.
Note: I did not cover gaming in this analysis because I am covering Nvidia’s gaming bottom for my free newsletter. I’ll make sure to post this separate analysis on the forum early next week. The H100 will be absent since we are entering an actionable phase with the CEOs discussions about Q1. Actionable analysis, for the most part, is reserved for premium.
Financials:
Note: calendar months are provided to avoid confusion due to Nvidia’s off calendar fiscal year. This upcoming report will be Q3 FY 2023
Nvidia reported as expected for Q3 ending in October with revenue of $5.93 billion for growth of (17%) which matched management guidance of $5.90 billion. Analyst consensus for revenue was $5.85 billion, or (17.7%) growth.
Fiscal Q4 ending in January was a slight miss with guidance of $6 billion compared to analyst consensus of $6.17 billion. This represents growth of (21%). The guidance has led to slightly lower estimates for Q1 ending in April with revisions from (19.8%) to (22.5%) for revenue of $6.41 billion.
As stated on the forum in the pre-earnings write-up, analysts have Nvidia returning to positive growth by July 2023 and to strong growth of 35%+ by October of 2023. This is helped by the low comps that we are currently experiencing.
Nvidia reported adjusted EPS of $0.58 which missed adjusted EPS estimates of $0.71. This compares to the July quarter of $0.51 adjusted EPS. Management indicated that profitability will increase from here: [GAAP and non-GAAP operating expenses were] primarily due to higher compensation expenses related to headcount growth and salary increases and higher data center infrastructure expenses. Sequentially, both GAAP and non-GAAP operating expense growth was in the single-digit percent, and we plan to keep it relatively flat at these levels over the coming quarters.”
An analyst did bring up that stock based compensation has been increasing each quarter at $700 million in the current quarter, up from $648 million and $578 million in the two previous quarters.
In Q3, the GAAP gross margin was 53.6% and the adjusted gross margin 56.1%. This was a miss from management Q3 guidance of 62.4%. The reason for the miss related to China: “Gross margins reflect $702 million in inventory charges largely related to lower data center demand in China, partially offset by a warranty benefit of approximately $70 million.” Nvidia is signaling the gross margin will return to normal next quarter with a guide for GM of 63.2%.
The company reported operating profit of $601 million for an operating margin of 10.1%. This compares to management’s guidance for an operating margin of 18.5% with Nvidia’s typical OM at 37% to 38%.
The adjusted operating margin of 25.9% is down from the typical range of 47%.
GAAP net margin of 11.5% for net profit of $680 million was up from a GAAP net margin of 9.8% in the previous quarter. The adjusted net margin of 24.5% for an adjusted profit of $1.46 billion compared to 19.3% in the previous quarter.
For the most part, Nvidia’s bottom line showed signs that last quarter was a bottom for the company with marginal, yet crucial improvement sequentially.
Nvidia had lower cash flow margins than it did last quarter at a 6.61% operating margin for operating cash flow of $392 million compared to a margin of 18.9% last quarter for operating cash flow of $1.27 billion. The free cash flow margin was (2.6%) for free cash flow of ($156) million compared to a 12% margin last quarter for free cash flow of $824 million.
The company had stock-based compensation of $745 million in the quarter, up from $648 million last quarter. There is $13.14 billion in cash and $10.95 billion in debt. The company returned $3.75 billion to shareholders with share repurchases and cash dividends. There is $8.3 billion remaining under the share repurchase authorization through December 2023.
Nvidia Discusses Why the H100 Will Ramp Faster than the A100
Since our thesis is that the H100 will drive sales and the stock price over the next couple of years, similar to the A100, we want to make sure we are getting confirmation of how the H100 is performing now that it has been on the market for about a month.
Why the H100 is Special
1. Enterprise Software
The first question from C.J. Muse discussed how the H100 is bundled with Enterprise Software, and the timing of when software monetization will begin to occur. The answer from the CEO was effectively “now.”
Here is what Huang said:
“Every company we’re talking to would like to have the agility and the scale, flexibility of clouds. And so, over the last year or so, we’ve been working on moving all of our software stacks to the cloud – all of our platform and software stacks to the cloud. And so today, we announced that Microsoft and ourselves are going to standardize on the NVIDIA stack, for a very large part of the work that we’re doing together so that we could take a full stack out to the world’s enterprise. That’s all software included.
If they would like to use it in the cloud, it’s per GPU instance hour; if they would like to utilize our software on-prem, they could do it through software license and so — license and subscription. And so, in both cases, we now have software available practically everywhere you would like to engage it.
2. The CEO stated H100 is going to Ramp faster than the A100
This was the discussion I felt was most important to Nvidia investors on the call. Second place would be the discussion around the Gaming bottom. Enterprise software is certainly important to as the software stack will eclipse hardware at some point. However, today, Nvidia is a hardware company and visibility into the pace of H100 adoption is key for our 2023 position and allocation.
Notably, I believe there will be positive surprises in the data center segment as we go along into 2023. It’s prudent for analysts to be cautious as we don’t have big tech capex numbers yet and the H100 has only been out for a month. Eventually, enthusiasm for Nvidia will return and it’ll the H100 that drives the positive sentiment.
Here was the question, which is being quoted in full due to its importance to our thesis:
William Stein:
I’m hoping you can discuss the pace of H100 growth as we progress over the next year. We’ve gotten a lot of questions as to whether the ramp in this product should look like a sort of traditional product cycle where there’s quite a bit of pent-up demand for this significant improved performance product and that there’s supply available as well. So, does this rollout sort of look relatively typical from that perspective, or should we expect a more perhaps delayed start of the growth trajectory where we see maybe substantially more growth in, let’s say, second half of ‘23?”
Jensen Huang
H100 ramp is different than the A100 ramp in several ways. The first is that the TCO, the cost benefits, the operational cost benefits because of the energy savings because every data center is now power limited, and because of this incredible transformer engine that’s designed for the latest AI models.
The performance over Ampere is so significant that I — and because of the pent-up demand for Hopper because of these new models that are — that I spoke about earlier, deep recommender systems and large language models and generative AI models. Customers are clamoring to ramp Hopper as quickly as possible, and we are trying to do the same. We are all hands on deck to help the cloud service providers stand up the supercomputers.
Remember, NVIDIA is the only company in the world that produces and ships semi-custom supercomputers in high volume. It’s a miracle to ship one supercomputer every three years. It’s unheard of to ship supercomputers to every cloud service provider in a quarter. And so, we’re working hand in glove with every one of them, and every one of them are racing to stand up Hoppers. We expect them to have Hopper cloud services stood up in Q1. And so, we are expecting to ship some volume — we’re expecting to ship production in Q4, and then we’re expecting to ship large volumes in Q1. That’s a faster transition than Ampere. And so, it’s because of the dynamics that I described.
My translation: Per the CEO, Q1 should be good to us Nvidia investors!
3. Grace Hopper and the CPU, GPU, DPU Trifecta
Grace Hopper is Nvidia’s new CPU that is meant to further accelerate and be integrated with Nvidia’s GPUs and DPUs. Notably, AMD is doing the same – where their CPUs are optimized and integrated to further accelerate AMD’s GPUs and DPUs.
Mark Lipacis
Jensen, I think for you, you’ve articulated a vision for the data center where a solution with an integrated solution set of a CPU, GPU and DPU is deployed for all workloads or most workloads, I think. Could you just give us a sense of — or talk about where is this vision in the penetration cycle? And maybe talk about Grace — Grace’s importance for realizing that vision, what will Grace deliver versus an off-the-shelf x86 [CPU], do you have a sense of where Grace will get embraced first or the fastest within that vision? Thank you.
Jensen Huang
Thanks Mark. Grace’s data moving capability is off the charts. Grace also is memory coherent to our GPU, which allows our GPU to expand its effective GPU memory, fast GPU memory by a factor of 10. That’s not possible without special capabilities that are designed between Hopper and Grace and the architecture of Grace […] It all needs to be fast, so that you can make a recommendation within milliseconds to hundreds of millions of people using your service.”
Networking is Showing Surprising Strength
According to an analyst on the call, their calculations show networking driving most of the sequential growth. He is referencing Mellanox acquisition which we covered a few years ago in this analysis.
Ambrish Srivastava
I actually had a couple of clarifications. Colette, on the data center side, is it a fair assumption that compute was down Q-over-Q in the reported quarter because the quarter before, Mellanox or the networking business was up as it was called out. And again, you said it grew quarter-over-quarter. So, is that a fair assumption?
Collette Kress
So, looking at our compute for the quarter is about flattish. Yes, we’re seeing also growth, growth in terms of our networking, but you should look at our Q3, compute is about flattish with last quarter.
Additional comments on Networking:
“Your data center networking business, I believe, is driving about $800 million per quarter in sales, very, very strong growth over the past few years” – Harlan Sur
“Jensen, can you help us understand like where your InfiniBand networking sits relative to like traditional data center switching?” – Aaron Rakers
“Yes. Thanks, Aaron. The math is like this. If you’re going to spend $20 billion on an infrastructure and the efficiency of that overall data center is improved by 10%, the numbers are huge. And when we do these large language models and recommender systems, the processing is done across the entire data center. And so, we distribute the workload across multiple GPUs, multiple nodes and it runs for a very long time. And so, the importance of the network can’t be overemphasized.”
We’ve been using Big Tech capex as a proxy for our semiconductor positions. According to one analyst on the call, the presumption is capex from the Big 3 will be flat in 2023. These are still sizable budgets, but the concern is if capex flatlines in 2023, what level of growth will the data center segment be capable of?
Here was the question on the call from Vivek Arya:
“And then, Jensen, the question for you. A lot of concerns about large hyperscalers cutting their spending and pointing to a slowdown. So if, let’s say, U.S. cloud CapEx is flat or slightly down next year, do you think your business can still grow in the data center and why?”
The answer from the CEO focused on Nvidia driving growth from AI acceleration, rather than general purpose compute, which implies that Capex can be flat while Nvidia will be serving the most valuable piece in the stack. AI acceleration, according to the CEO, will not be flat or down.
“Vivek, our data center business is indexed to two fundamental dynamics. The first has to do with general purpose computing no longer scaling. And so, acceleration is necessary to achieve the necessary level of cost efficiency scale and energy efficiency scale, so that we can continue to increase workloads while saving money and saving power. Accelerated computing is recognized generally as the path forward as general purpose computing slows. The second dynamic is AI. And we’re seeing surging demand in some very important sectors of AIs and important breakthroughs in AI.”
The CEO discussed deep recommender systems, large language models driven by Transformers, and generative AI for generating images and videos. He ended the answer with this: “And so, you could see that our company is indexed to two things, both of which are more important than ever, which is power efficiency, cost efficiency and then, of course, productivity. And these things are more important than ever. And my expectation is that we’re seeing all the strong demand and surging demand for AI and for these reasons.”
My translation: Capex can be flat and the CEO foresees Nvidia will take a higher percentage of this capex because they’re serving demand where few companies can across the three major AI breakthroughs he pointed out. My other comment would be that we won’t have a full picture of capex for next year until we get Q1 reports and 2023 full year guides around end of January. This is when we did a deep dive analysis on capex spending last year, and we will revisit this. So, keep an eye out for that.
Note: Nvidia Expected to Change Reporting on Data Center
There was a discussion on the call that Nvidia plans to start breaking out the data center segment to account for internet service companies in addition to hyperscalers. My understanding is internet service providers would mean 5G providers or other internet services related to edge computing. This was not directly stated but it makes the most sense given where edge computing is headed, which could rival the hyperscalers.
Matt Ramsay
I guess, Colette, I heard in your script that you had you talked about maybe a new way of commenting on or reporting hyperscaler revenue in your data center business. And I wondered if you could maybe give us a little bit more detail about what you’re thinking there and what sort of drove the decision?
Jensen Huang:
[…] And these are internet service companies that offer services, but they’re not public cloud computing companies. The second factor has to do with cloud computing […] [hyperscalers] are two things to us, therefore, a hyperscaler can be a sell to customer; they are also a sell with partner.”
Conclusion:
The market has been discouraging this year. The gaming selloff for Nvidia and PC selloff for AMD were brutal. But if you listen to these calls, it is crystal clear something monumental is going on. We want to capture this as fully as possible. Perhaps we will have 40% allocation in two positions (NVDA or AMD) or perhaps we will have to trim to 15% across two positions and layer back up to 30% allocation. We will do this as skillfully as possible.
If 2021 to 2022 taught us anything, it’s that only the strong survive. That goes for stocks/companies and investors. There is no doubt that NVDA and AMD will weather what’s ahead and we want to stay close to our AI bellwethers. Whatever the tide brings us, you can expect us to obsessively cover these companies and to actually increase our coverageincrease our coverage as we go along. There is no limit to the research needed if we are building positions with conviction.
If you were to guess, when do you think we wrote the following paragraph?
“When a thesis is not reflected in the revenue segments yet, there are typically lower entry points and ongoing volatility. You’ll see in the technical analysis that although I could not be more bullish on this stock long-term, there is weakness in the semiconductor sector and we hope this translates to a lower entry point for our readers.
The market is also in a fierce debate between AMD, Intel, and Nvidia and is also distracted by other chips, such as Micron and NXP. In my analysis, I look for growth. How big is the market relative to how big the company is now?
You can ignore Nvidia’s gaming revenue and other segments for the main trajectory that we are focused on. Gaming is great for stability and earnings reports, but the growth will not be from gaming (a market where Nvidia is already a mature, market leader). I’m also not focused on PC sales or the CPU-powered cloud, as the first is not a growth market and the second is not the piece in the cloud stack that will accelerate future technologies.”
That was written in 2019 yet the far majority of those concerns could be stated verbatim right now. Do we care about PC sales or gaming consoles? No, although our stance is that we have to expect these concerns will affect our semiconductor positions at times. The good news for Nvidia and AMD investors is that as time goes on, the less consumer-related hardware will have an impact. The 2022 Nvidia Investors Presentation provided numbers which show in detail how consumer exposure will become less of a concern in the future for these AI heavyweights.
When do you think we wrote this analysis?
“Over the past few weeks, I have read many lagging explanations on the chip shortage – too many fabless semiconductor companies, too few foundries, automobile manufacturers paused ordering in March and didn’t prepare for the sharp rebound, tensions with China, and even a fire at the Asahi Kasei plant that specifically manufactures sensing devices for the automobile industry.
While all of these are true, the overarching issue is that the role of semiconductors has changed from a commodity to the primary accelerant of future technologies. This is because connectivity, automation, and ultimately AI, will disrupt every corner of every industry.
We saw this happen with data and cloud but now we must accelerate this to the next level for AI/ML and the common denominator is semiconductors. Automotive is only the beginning. We can add renewables to the list and even e-commerce as AR/VR and AI/ML attempt to prop up the leaders who are competitive enough to add these features first.
As a tech stock analyst, I don’t have the luxury of lagging analysis of any kind. My subscribers require (and deserve) forward-looking, and with my intense focus on semiconductor chips, I don’t think my readers are surprised that semis are under pressure due to an increasingly important role.
I have repeated (perhaps too many times) that there is no way forward without the semis. We are seeing this manifest in automotive right now, but as investors, we should get used to hearing about semiconductor shortages.
You and I can debate Palantir, Snowflake or C3.AI, for example, and the valuations or the right angle for AI/ML-driven software, but the common denominator to these companies is the need for semiconductors to drive forward AI and 5G.
Now, we add the enormous push for auto manufacturers to compete with Tesla, Apple, Lucid Motors and what we have is a bottle neck where the automotive industry filters into semiconductors.
My guess is the demand won’t be letting up for many years as we are no longer in the cyclical pattern that semis are notorious for. Instead, demand will outpace supply for years to come.
Is this a bad thing or a good thing for our stocks? As investors, we can either listen to the news or listen to management. In this case, they are not aligned. Machines trade off news and natural language processing (NLP) but as human investors, we have the advantage of looking deeper into the issues.
I have written volumes of analysis leading up to the triple-digit growth we are seeing now in the data center from AI accelerator chips. Most of this was written when data center growth was negative. For instance, my Nvidia thesis was set end of 2018 — and in 2019 Nvidia reported negative data center revenue year-over-year for four quarters in a row.reported negative data center revenue year-over-year for four quarters in a row.
I mention this because following a trend’s trajectory is more important than immediate gratification from the market. The trend will always win out over time.
I have maintained that chips will eventually lead the AI market and are the best angle for investing in edge computing. I have also defended our stocks against custom silicon. Now we have the first of what I predict will be many semiconductor shortages and bullish to me.
The shortage is that there are hundreds (thousands really) of companies that rely on semiconductors. This will come to a head with AI and 5G as those who go-to-market soon with these features will have an enormous competitive advantage.”
That was written at the height of the bull market in February of 2021. My goal is to illustrate there has always been headlines to worry about for the semiconductors. We’ve firmly held these stocks and bought during dips. In the past, from 2018-2019, I focused on the GPU-powered cloud and the CUDA moat here and here. Our 2020 coverage centered on the A100 GPU which we discussed at time of launch for premium here and continued coverage on the A100 about a year later on the free side.
Here is background on the A100:
“Nvidia released the Ampere architecture and A100 GPU as an upgrade from the Volta architecture. The A100 GPUs are able to unify training and inference on a single chip, whereas in the past Nvidia’s GPUs were mainly used for training. This allows Nvidia a competitive advantage by offering both training and inferencing. The result is a 20x performance boost from a multi-instance GPU that allows many GPUs to look like one GPU. The A100 offers the largest leap in performance to date over the past 8 generations.”
Nvidia's AI Dominance Will be Propelled Forward by Software:
I wanted to go back through a bit of Nvidia’s history – what was the thesis and how did the thesis evolve? – before I go into how Nvidia will continue to dominate. In my opinion, I believe this is the most important analysis I have ever written on Nvidia because the company is changing rapidly into a software company.
The shift that Nvidia is going through has gone unnoticed and that’s to our benefit. Because we have been hell bent on finding what companies will dominate AI hardware, I’ve been asked frequently who do I think will dominate AI software (Palantir? Snowflake? Google?)
I’m prepared to give you that answer today: I believe Nvidia will be one of the biggest or perhaps the biggest AI software stack company in the world.the biggest AI software stack company in the world. The analysis below kickstarts our in-depth coverage on this new thesis — and I fully believe I will be quoting this analysis in five years from now when we check back on how the AI software thesis played out.
Before I go into semiconductor jargon where I risk losing your attention, I want to make sure our Members are fully aware that the segment where Nvidia will dominate with AI software is automotive. I am not talking about a few OEMs that trickle into a little bump in revenue. I am saying that Automotive is scheduled to become Nvidia’s number one segment – even over data centers – and to the tune of it being 3X larger than its gaming segment.
Don’t take my word for it because the CFO said exactly that (more on this below) and there is ample evidence that this is happening, which I also detail for you. Wall Street won’t be giving this the credit it deserves until 2023 at the earliest but you will hear non-stop “Nvidia Automotive” coverage by 2024-2026 as this segment ramps. I go over why those are the target dates below.
But first, let’s talk about the H100 and how this new GPU architecture will also help Nvidia lead on AI software at the enterprise level. There is plenty going on outside of Automotive that we need to cover so I kept automotive for last.
GTC Highlights: The Hopper H100 GPU
In March at GTC 2022, Nvidia announced the Hopper H100 GPU with 80 billion transistors and will be released in Q3 of this year. For reference, the A100 has 54 billion transistors. This is Nvidia’s solid attempt to keep their stake in the ground in leading high-performance computing over AMD’s Instinct MI250/250X and the newly announced MI210.
It’s easy to focus on hardware with Nvidia (and AMD) yet these companies are becoming more software-driven each year. By owning the majority of the market for AI accelerators, these two companies are afforded an opportunity to also own the software layer as a means to lower the barrier to entry for training models, deploying inference across various frameworks, and other workloads related to deep learning, conversational AI, video conferencing algorithms, and more. By supplying the hardware, these companies have natural inroads to machine learning operations (MLOps).
The H100 is the New Artificial Intelligence Infrastructure
DGX, DGX Pods and DGX SuperPods:
The H100 will power all AI and high-performance computing systems including the PCI express accelerator for mainstream servers and many H100 GPUs can be combined to power advanced AI through the following systems: DGX, DGX Pod and DGX SuperPod.
The difference between the A100 and H100 is the performance will be two to three times faster. The H100 GPUs and the DGX H100 server pods and super pods offer Nvidia the next leg-up as the company has solved an important bandwidth issue.
Hopper tackles some of the bigger issues around previous generations like speeding up algorithms by offering dynamic programming on GPUs to break down problems to simpler subproblems, boosting bandwidth by 3X with SHARP in-networking computing and Infiniband Switches, and the H100 can leverage NVLink to connect eight H100s into one giant GPU for 640 billion transistors, 32 petaflops, 640GB of HBM3, and 24 terabytes per second of memory bandwidth.
The chip is custom built by Taiwan Semiconductors with a 4nm design making it the world’s fastest 4nm GPU. The H100 has about 50% more memory and interface bandwidth than the A100. That’s 1.5X more bandwidth with the NVLink connection and PCIe 5.0 doubling the bandwidth of PCIe 4.0. The H100 will ship with support for 80GB of HBM3 memory at 3 TB/s speed.
The NVLink is now able to link together server nodes to build a data center-sized GPU. NVLink was originally designed to bypass the PCIe slot and has become an important tool for chip-to-chip connectivity, especially for high-speed operations. There is a dedicated chip called the NVSwitch which has increased the NVLink’s bandwidth. The ultimate goal is to run 32 servers with their own operating systems to run a single job. NVLink will complement the InfiniBand networking for high-performance computing and NVLink will be default for all of Nvidia’s chips, including GPUs, CPUs, DPUs and SoCs.
Where the H100 really stands apart is the leap in performance with about 3X more performance than the A100 and the H100 is up to 6X faster. The leap in performance is measured by H100’s ability to deliver up to 4,000 TFLOPS of FP8 compute, 2,000 TFLOPS of FP16 compute and 1,000 TFLOPS of TF32 compute and 60 TLOPS of general purpose FP64 compute. The A100 lacked support for FP8 compute at default whereas the H100 will leverage a transformer engine to switch between FP8 and FP16, depending on the workload.
According to Nvidia, the H100 delivers 9X more throughput in AI training, and 16X to 30X more inference performance. The company also states in HPC application-specific workloads, the H100 is 7X faster. The goal of the H100 was not only to add more transistors and make the H100 faster, but to also offer function-specific optimizations. This is achieved through the transformer engine.
The architecture aims to answer one of the bigger challenges facing superfast compute, which is that moving data into traditional servers overloads the CPU and system memory and becomes bottlenecked by PCI-Express.
By improving the bandwidth issue, Nvidia’s goal is to create more demand for their DGX Pod and SuperPod Systems, which in turn, will create more demand for their software.
The H100 DGX Pod is a 32-node, 256-GPU system. The H100 DGX Pod connects 32 DGX systems using the NVLink Switch System to scale into a super-GPU capable of 768 terabytes per second. To compare, the entire internet requires 100 terabytes per second. This results in 1 exaflop of AI computing.
From there, multiple H100 DGX Pods can connect through the Infiniband Switch to scale DGX Superpods with thousands of H100 GPUs. DGX SuperPods are turnkey systems that power enterprise AI. DGX SuperPods were also available with the A100 yet the H100 will have 6X better performance with 1 exaflop of FP8 AI performance to run trillions of parameters (more on this below).
Spectrum-4 Ethernet Platform
Perhaps one of Nvidia’s most important advancements for the H100 is the ability to attach the network directly to the GPU to avoid bottlenecks at the CPU. This is accomplished by sending data with direct memory access at 50 gigabytes per second. Hopper HGX and DGX are networking and interconnects that facilitate moving data with an advanced networking processor called the CX7. The result is the H100 CNX that avoids bandwidth bottlenecks and frees the CPU and system memory to process other parts of the application.
The Spectrum Ethernet platform, which consists of a Spectrum-4 Switch, CX7 SmartNIC and Bluefield-3 DPU will be used for several of Nvidia’s AI platforms, such as Riva, Merlin and Omniverse. These workloads include natural language processing, recommenders, and digital twins and will be supported by a networking system that helps exchange large databases between nodes. Whereas traditional workloads required many connections exchanging small amounts of data, the workloads of the future will require data to be shared quickly between GPUs and storage. This is accomplished by bypassing the CPU and sending data directly to the GPU while using the network hardware to move the data.
This is ideal for enterprise use cases where people are more likely to use Ethernet while AI and HPC workloads continue to use the Quantum-2 based off Mellanox’s InfiniBand. Quantum-2 allows for in-network computing to do data reductions in the network. It’s also more likely that Ethernet is used for 5G and sensors.
Eos: The First Hopper AI Factory
Nvidia is building AI factories to compete with AI supercomputers, which are blueprints for AI infrastructure that can be adopted by cloud partners and enterprises.
Eos is built with 18 H100 SuperPods, with 576 DGX H100 systems and 360 NVLink Switches. Nvidia states EOS is 1.4X faster than the fastest supercomputer and offers 4X the AI processing of the world’s fastest supercomputer. This will deliver 18 EFLOPS of FP8 AI compute or 9 EFLOPS of FP16 compute.
Previously, FP16 was the standard for AI whereas FP8 is gaining more support to become the industry standard. Depending on what AI compute you use, benchmarks will not be apples-to-apples if FP8 is compared to FP64 performance. Right now, AMD’s Frontier supercomputer is #1 with 1.1 exaflops of FP64 performance compared to the upcoming Venado supercomputer’s 10 exaflops of FP8 performance.
The difference is that the smaller bit size allows for an economical way to achieve more speed when giving up a small amount of accuracy doesn’t make a critical difference. This also helps in the face of a slowing Moore’s Law. FP8 is most commonly used for inference yet may be used for training in the future due to boosting throughput. Following the release of the Hopper H100, Intel released Gaudi2 which supports FP8. Chip makers Graphcore, AMD and Qualcomm have recently pushed for an industry-standard for the low precision floating point format FP8 rather than integer formats.
Here is what Nvidia said in the GTC keynote:
But the trend in AI computing has been toward developing neural nets that lean on the lowest precision that will still yield an accurate result. The smaller formats compute faster and more efficiently, and they require less memory and memory bandwidth. The addition of 8-bit floating-point units in the H100 leads to a significant speedup—double the throughput compared to its 16-bit units”
DPX Instructions (ISA):
The H100 improves dynamic programming with DPX Instructions that will help specific AI Algorithms to perform up to 7X faster than previous GPUs and 40X faster than CPU-based algorithms. As algorithms require more complexity, the new set of DPX instructions will help break the complex problems down into simpler subproblems using GPUs instead of CPUs or FPGAs.
The DPX ISA are expected to be broadly available with the CUDA 12.0 release. Examples of where this will be useful include disease research and drug discovery where the process can be sped up 35X for real-time processing to match the rate of DNA sequencing. Route optimization and finding the shortest distance between destinations for use in factories and autonomous driving systems, or Floyd-Warshall acceleration, is boosted up to 40X compared to CPU-only servers. These instructions will also be used for quantum computing and SQL queries as dynamic programming can help find the optimal order for joining a set of tables.
GPU Confidential Computing:
Data is encrypted at-rest and in-transit yet is often unprotected during use. Meanwhile, the data used to train AI models is worth millions in investments and is trained from domain knowledge and company-proprietary data. The new H100 offers confidential computing whereas previously only CPUs offered the protection of both data and applications during use.
Nvidia is Becoming a Leading AI Software Company
It would be easy to read the information above and to assume Nvidia is improving its hardware. However, the company’s future resides in software which will remove some of the cyclicality of hardware revenue. I believe once Nvidia’s software revenue begins to reveal itself in earnings reports, the market will finally piece together the true potential of this AI powerhouse.
It’s both the hardware and the software stack that led me to say Nvidia will surpass Apple in 5 years. You know this story well: the relationship between a hardware company leveraging their position to capture the lion’s share of the software — because that’s exactly what Apple did.
There are four layers to Nvidia’s full-stack accelerated computing: hardware, system software, platform software and applications. Below, I discuss a few ways that Nvidia is capturing more of the software stack due to vendor lock-in effects from their dominance in hardware.
As stated, in the past, our focus was the GPU-powered data center. This was a four-year thesis from 2018 and we doubled up on the thesis in June of 2020 for the A100 release. I want to make sure and emphasize that Nvidia’s lesser-known catalyst is actually the software.
The H100 is helpful in maintaining a lead in GPUs, which is critical turf to protect with GPUs being the most popular AI accelerator, however — the AI/ML catalyst will be further fueled by the Nvidia’s lead in software. This is why the majority of who will remain the AI leader will be up to developers and not the C-suite partnerships on hardware that characterized Intel’s lead over the past few decades. The developers choose the frameworks, the SDKs, libraries and the other parts of the software stack, and because of this, they also choose the GPUs they build on rather than IT departments.
Transformers
The transformer engine is one of the key aspects of the H100. Transformers are becoming one of the most popular neural-network models by applying self-attention to detect how data elements in a series influence and depend on one another.
Sequential text, images and video data are used for self-supervised learning and pattern recognition, which results in more data being used to create better models. Prior to transformer models, labeled datasets had to be used to train neural networks. Transformer models eliminate this need by finding patterns between elements mathematically, which substantially opens up what datasets can be used and how quickly. Transformers are partial to the parallel processing that GPUs offer.
Google first introduced transformer models in 2017 and transformers are used in Google and Bing Search. Transformers also led to BERT models, which stands for Bidirectional Encoder Representations from Transformers, and is commonly used for text sequences. Transformers are also used in GPT-3 (it’s the T in GPT) which improved from 1.5 billion parameters to 175 billion parameters. GPT-3 has the ability to report on queries it has not been specifically trained on.
Nvidia and Microsoft recently worked on a Mega transformer model with 530 billion parameters and the future for AI engineers is trillion-parameter transformers and applications. The H100 is already prepping for this. According to Nvidia, the training needs for transformer models will increase 275-fold every two years compared to 8-fold for other models. The H100 GPU with its Transformer Engine supports the FP8 format to speed up training to support trillion-parameter models. This leads to transformer models that go from taking 5 days to train to becoming 6X faster to only taking 19 hours to train.
The transformer engine is software combined with the new hardware in the H100’s tensor cores. As discussed, the A100 was designed for floating-point numbers to 16 bits while the H100 is designed for 8 bits. This is helpful because AI models are moving toward neural nets that lean on the lowest precision and yet still yields an accurate result. In this case, 8 bits double the throughput of 16-bit units, compute faster and more efficiently, and they require less memory and memory bandwidth.
The main feature from the Transformer Engine is the ability to choose what precision is needed for each layer in the neural network at each step, transitioning between 8-bits, 16-bits, 32-bits, plus the H100 is able to do matrix math with two forms of 8-bit numbers with either 5-bits as the exponent or 4-bits as the exponent: E5M2 and E4M3. This is important because the E4M3 may be favored for back propagation while E5M2 may be favored for inferencing.
Pictured above: Nvidia is prepped to support model sizes growing up to 275X every two years
Triton Inference Server:
Nvidia offers AI frameworks to reduce time for developers throughout the AI workflow from data processing and ETL to deep learning model training and large-scale inferencing. These libraries include Dali, Rapids, Triton and Magnum I/O. The library supports all popular frameworks and offers pre-trained models and data pipelines.
Triton is open-source inference software that helps developers deploy models across GPUs and CPUs, it supports Tensor Flow and PyTorch, any query type and any model – such as Transformers or CNNs (used for image recognition) and RNNs (used in speech recognition). The inference engine helps developers take AI development from experimentation to production by removing the need for multiple inference servers and simplifying machine learning infrastructure on the backend.
MLOps (machine learning operations) helps developers with less ML expertise to train and deploy models yet there were limited use cases with little help in deploying custom models. Triton offers high performance inference and scalability with Dockers and Kubernetes while serving up to hundreds of models with the model control API. By supporting all popular frameworks, Triton helps developers avoid framework lock-in due to the consistent interface regardless of training framework or hardware.
Nvidia will Power the Lion’s Share of Automotive – and that means software licensing
Nvidia’s lead in automotive across dozens of OEMs requires its own deep dive. The reason I haven’t prioritized this is because Hyperion 8 is shipping in 2024 and Hyperion 9 will ship in 2026. However, as long-term investors, we should touch base now on the long-term vision for yet another large and sweeping revenue segment. In fact, automotive promises to be Nvidia’s largest segment by 2030 – so on that alone, imagine what Nvidia investors have in front of us.
Nvidia’s Orin SoC (system-on-a-chip) is designed for the neural networks that run robots and AVs at the edge. This is the central computer for the car. The Orin SoC is capable of 254 trillion operations per second by combining Nvidia GPUs with Arm CPU cores and TensorRT APIs. The goal is to help OEMs move from Level 2 autonomous systems to the elusive Level 5 and it stiffens the competition with Tesla’s FSD. Notably, at the release two years ago, Tesla pointed towards Orin’s power consumption as a potential issue for EV batteries but this has not stopped many competing EVs from adopting Nvidia’s in-vehicle hardware and DRIVE software stack.
The EV manufacturers that have already moved forward with Nvidia DRIVE Orin include: Nio, Xpeng, BYD, Lucid Group, Mercedes and Land Rover, GM Cruise — you name it, it’s probably in production with Nvidia at this moment. The company’s current automotive pipeline exceeds $11 billion over the next six years – expect this small blip of pipeline to grow exponentially.
Nvidia recently announced an upgrade to Orin called Atlan with 1,000 TOPS on one chip, or more than then Level 5 compute in AVs today. This chip will catapult forward the computing performance of AVs and is expected to be released in 2023.
Nvidia DRIVE is the operating system and software stack for vehicles that offers an execution environment and includes both security and over-the-air updates. DriveWorks is an SDK that enables self-driving applications. Drive AV offers key ingredients to an autonomous system, such as perception, mapping and planning modules. Regarding mapping, Nvidia DRIVE Map is a multi-modal drive engine that can map independently and has two map engines. Drive IX is open-source software that offers vision, voice and graphics for the user experience. (I will do a separate deep dive on Nvidia Automotive in 2023).
The entire autonomous platform is called Hyperion, which includes the compute and sensor toolkit. This includes the hardware plus a 360-degree camera, radar, lidar and ultrasonic sensor suite. As stated, Hyperion 8 ships in 2024 with Hyperion 9 shipping in 2026, which will double the processing speed and offer an increase in sensors. Nvidia offers open-source developer kits to help increase its compatibility across various projects.
Rather than train the vehicles on the road, Nvidia trains in simulation and can create virtual world obstacles for the vehicles to learn from. This is a different approach from companies like Tesla who have millions of cars on the road collecting data which they then augment for unusual events with a photorealistic simulator.
Tesla has the most data of any car manufacturer which helps the company competitively as more data equals better performing models especially in terms of object detection. More data from millions of cars on the roads also helps with prediction as Tesla collects data from incorrect predictions that can be added to the training set. By leveraging a prediction neural network, Tesla does not need to use human labeling or annotation and can instead use what’s called a temporal sequence of events — in other words, Tesla rewinds events and labels objects automatically with the use of a supercomputer.
The advantage here is that training neural networks correlates with the miles (which again, are substantial due to size of fleet on the road compared to competitors) rather than correlating with the need for human labeling. The result of automatic labeling is that Tesla is able to predict rare situations with more accuracy.
Where Nvidia delivers a strong advantage is the company has decades of history with graphics and simulation due to its gaming roots. As stated, Tesla also uses imitation learning and has a photorealistic simulator which uses vector space for labels and functions like a game engine. However, Nvidia has been quietly working on their simulation platform for many years internally despite only recently marketing Omniverse to the public. In this case, Nvidia has such a high-level of confidence in their simulation skills that they forego the real-life fleet to primarily train virtual 3D models. The company is also packaging the simulation platform for many other uses cases, such as AI factories, 5G networks, power plants and climate research. Developers can work with 3D tools through Python-based development.
To keep it simple, Tesla’s primary advantage is the data they have collected as no other EV/AV has collected this level of data from real drivers. To contrast, Nvidia has arguably the best simulation platform due to decades of graphics work. These digital twins are only now being widely marketed despite being in development for over 5 years. The license costs $9,000 and Nvidia has estimated its current addressable market is 20 million engineers. Notably, Nvidia’s Hyperion will also be deployed in millions of vehicles over time, offering similar levels of data as Tesla’s fleet.
The Tesla VS Nvidia debates have not formally begun but they are certainly in our future … so brace yourself. Ultimately, the way Nvidia stands apart is the company does not directly compete on manufacturing vehicles. This is something anyone can agree on. That means many OEMs will use Nvidia’s DRIVE system whereas Tesla is less likely to commercialize their software as they’re viewed as a main competitor.
As long as Nvidia continues to innovate and maintain a lead, the popularity of its DRIVE system is likely to remain due to the company’s strategic advantages in AI and supercomputing. The company did an excellent job of tackling the edge computing use case of autonomous vehicles first.
Hardware is only part of the equation. The long-term plan is for Nvidia to license software for autonomous vehicles, which will create a recurring revenue stream. The licensing fees will go well beyond Omniverse to include the actual owner of the vehicle paying a subscription fee to Nvidia for its software. Tesla does this with their AutoPilot software which has grown from $5,000 to $12,000 per vehicle.
$150 billion from Omniverse software – fees are charged to both users and robots/digital twins
$300 billion from Automotive – primarily software
What Nvidia is communicating is that software revenue will surpass hardware revenue long-term.
Here is what Kress stated: "Our software content per vehicle can be in the thousands of dollars over the lifetime of the vehicle compared to the hundreds of dollars for the hardware. And second, software scales with the installed base of vehicles, not annual production.”
Note on CUDA:
The software discussion on Nvidia is not complete without a mention of CUDA. We called this Nvidia’s moat back in 2018 and we continue to believe it provides an important moat. The CUDA-related libraries include frameworks that span quantum computing, robotics, 5G networks, cybersecurity and drug discovery. The universal skills taught around CUDA and Nvidia’s SDKs help to drive more business for Nvidia’s GPUs.
The primary risk right now is valuation as Nvidia trades 2X higher than its peers on both the top line sales valuations and on the bottom line with earnings and cash-based valuations. There’s probably equal risk in waiting for Nvidia to drop another 50% as there is in buying Nvidia at the 2X valuation. One reason Nvidia may be valued here is because it’s slowly becoming a software company. Regardless, Knox’s technicals help immensely in determining if the market will continue to award Nvidia it’s gold medal valuation or if the market will discount Nvidia based on sentiment-driven headlines. This is a position we plan to keep on building so you can keep an eye out for those trade alerts over the next few years.
Conclusion:
Finding great companies is only half the battle, fighting negative sentiment is the other half – and semis have no shortage of this in any market – hence our beginning quotes from 2019 and also 2021.
Nvidia is the strongest company in terms of product on the market today. That doesn’t mean semis won’t be a roller coaster – we should fully expect that semis will undulate in sentiment and price while we hold our stocks over many years. We can’t change the way Wall Street works — which is a pendulum that swings between value stocks and growth stocks — but we can describe in great detail why concerns around gaming and consumer electronics slowing down is not going to change our position. We do not care to perfectly time entries or to find a perfect bottom – you’ll be hard pressed to find any legendary investor recommend that this be an investor’s goal. What we care about is finding quality companies and building those positions over time. Nvidia fits this description.
If you were to guess, when do you think we wrote the following paragraph?
“When a thesis is not reflected in the revenue segments yet, there are typically lower entry points and ongoing volatility. You’ll see in the technical analysis that although I could not be more bullish on this stock long-term, there is weakness in the semiconductor sector and we hope this translates to a lower entry point for our readers.
The market is also in a fierce debate between AMD, Intel, and Nvidia and is also distracted by other chips, such as Micron and NXP. In my analysis, I look for growth. How big is the market relative to how big the company is now?
You can ignore Nvidia’s gaming revenue and other segments for the main trajectory that we are focused on. Gaming is great for stability and earnings reports, but the growth will not be from gaming (a market where Nvidia is already a mature, market leader). I’m also not focused on PC sales or the CPU-powered cloud, as the first is not a growth market and the second is not the piece in the cloud stack that will accelerate future technologies.”
That was written in 2019 yet the far majority of those concerns could be stated verbatim right now. Do we care about PC sales or gaming consoles? No, although our stance is that we have to expect these concerns will affect our semiconductor positions at times. The good news for Nvidia and AMD investors is that as time goes on, the less consumer-related hardware will have an impact. The 2022 Nvidia Investors Presentation provided numbers which show in detail how consumer exposure will become less of a concern in the future for these AI heavyweights.
When do you think we wrote this analysis?
“Over the past few weeks, I have read many lagging explanations on the chip shortage – too many fabless semiconductor companies, too few foundries, automobile manufacturers paused ordering in March and didn’t prepare for the sharp rebound, tensions with China, and even a fire at the Asahi Kasei plant that specifically manufactures sensing devices for the automobile industry.
While all of these are true, the overarching issue is that the role of semiconductors has changed from a commodity to the primary accelerant of future technologies. This is because connectivity, automation, and ultimately AI, will disrupt every corner of every industry.
We saw this happen with data and cloud but now we must accelerate this to the next level for AI/ML and the common denominator is semiconductors. Automotive is only the beginning. We can add renewables to the list and even e-commerce as AR/VR and AI/ML attempt to prop up the leaders who are competitive enough to add these features first.
As a tech stock analyst, I don’t have the luxury of lagging analysis of any kind. My subscribers require (and deserve) forward-looking, and with my intense focus on semiconductor chips, I don’t think my readers are surprised that semis are under pressure due to an increasingly important role.
I have repeated (perhaps too many times) that there is no way forward without the semis. We are seeing this manifest in automotive right now, but as investors, we should get used to hearing about semiconductor shortages.
You and I can debate Palantir, Snowflake or C3.AI, for example, and the valuations or the right angle for AI/ML-driven software, but the common denominator to these companies is the need for semiconductors to drive forward AI and 5G.
Now, we add the enormous push for auto manufacturers to compete with Tesla, Apple, Lucid Motors and what we have is a bottle neck where the automotive industry filters into semiconductors.
My guess is the demand won’t be letting up for many years as we are no longer in the cyclical pattern that semis are notorious for. Instead, demand will outpace supply for years to come.
Is this a bad thing or a good thing for our stocks? As investors, we can either listen to the news or listen to management. In this case, they are not aligned. Machines trade off news and natural language processing (NLP) but as human investors, we have the advantage of looking deeper into the issues.
I have written volumes of analysis leading up to the triple-digit growth we are seeing now in the data center from AI accelerator chips. Most of this was written when data center growth was negative. For instance, my Nvidia thesis was set end of 2018 — and in 2019 Nvidia reported negative data center revenue year-over-year for four quarters in a row.reported negative data center revenue year-over-year for four quarters in a row.
I mention this because following a trend’s trajectory is more important than immediate gratification from the market. The trend will always win out over time.
I have maintained that chips will eventually lead the AI market and are the best angle for investing in edge computing. I have also defended our stocks against custom silicon. Now we have the first of what I predict will be many semiconductor shortages and bullish to me.
The shortage is that there are hundreds (thousands really) of companies that rely on semiconductors. This will come to a head with AI and 5G as those who go-to-market soon with these features will have an enormous competitive advantage.”
That was written at the height of the bull market in February of 2021. My goal is to illustrate there has always been headlines to worry about for the semiconductors. We’ve firmly held these stocks and bought during dips. In the past, from 2018-2019, I focused on the GPU-powered cloud and the CUDA moat here and here. Our 2020 coverage centered on the A100 GPU which we discussed at time of launch for premium here and continued coverage on the A100 about a year later on the free side.
Here is background on the A100:
“Nvidia released the Ampere architecture and A100 GPU as an upgrade from the Volta architecture. The A100 GPUs are able to unify training and inference on a single chip, whereas in the past Nvidia’s GPUs were mainly used for training. This allows Nvidia a competitive advantage by offering both training and inferencing. The result is a 20x performance boost from a multi-instance GPU that allows many GPUs to look like one GPU. The A100 offers the largest leap in performance to date over the past 8 generations.”
Nvidia's AI Dominance Will be Propelled Forward by Software:
I wanted to go back through a bit of Nvidia’s history – what was the thesis and how did the thesis evolve? – before I go into how Nvidia will continue to dominate. In my opinion, I believe this is the most important analysis I have ever written on Nvidia because the company is changing rapidly into a software company.
The shift that Nvidia is going through has gone unnoticed and that’s to our benefit. Because we have been hell bent on finding what companies will dominate AI hardware, I’ve been asked frequently who do I think will dominate AI software (Palantir? Snowflake? Google?)
I’m prepared to give you that answer today: I believe Nvidia will be one of the biggest or perhaps the biggest AI software stack company in the world.the biggest AI software stack company in the world. The analysis below kickstarts our in-depth coverage on this new thesis — and I fully believe I will be quoting this analysis in five years from now when we check back on how the AI software thesis played out.
Before I go into semiconductor jargon where I risk losing your attention, I want to make sure our Members are fully aware that the segment where Nvidia will dominate with AI software is automotive. I am not talking about a few OEMs that trickle into a little bump in revenue. I am saying that Automotive is scheduled to become Nvidia’s number one segment – even over data centers – and to the tune of it being 3X larger than its gaming segment.
Don’t take my word for it because the CFO said exactly that (more on this below) and there is ample evidence that this is happening, which I also detail for you. Wall Street won’t be giving this the credit it deserves until 2023 at the earliest but you will hear non-stop “Nvidia Automotive” coverage by 2024-2026 as this segment ramps. I go over why those are the target dates below.
But first, let’s talk about the H100 and how this new GPU architecture will also help Nvidia lead on AI software at the enterprise level. There is plenty going on outside of Automotive that we need to cover so I kept automotive for last.
GTC Highlights: The Hopper H100 GPU
In March at GTC 2022, Nvidia announced the Hopper H100 GPU with 80 billion transistors and will be released in Q3 of this year. For reference, the A100 has 54 billion transistors. This is Nvidia’s solid attempt to keep their stake in the ground in leading high-performance computing over AMD’s Instinct MI250/250X and the newly announced MI210.
It’s easy to focus on hardware with Nvidia (and AMD) yet these companies are becoming more software-driven each year. By owning the majority of the market for AI accelerators, these two companies are afforded an opportunity to also own the software layer as a means to lower the barrier to entry for training models, deploying inference across various frameworks, and other workloads related to deep learning, conversational AI, video conferencing algorithms, and more. By supplying the hardware, these companies have natural inroads to machine learning operations (MLOps).
The H100 is the New Artificial Intelligence Infrastructure
DGX, DGX Pods and DGX SuperPods:
The H100 will power all AI and high-performance computing systems including the PCI express accelerator for mainstream servers and many H100 GPUs can be combined to power advanced AI through the following systems: DGX, DGX Pod and DGX SuperPod.
The difference between the A100 and H100 is the performance will be two to three times faster. The H100 GPUs and the DGX H100 server pods and super pods offer Nvidia the next leg-up as the company has solved an important bandwidth issue.
Hopper tackles some of the bigger issues around previous generations like speeding up algorithms by offering dynamic programming on GPUs to break down problems to simpler subproblems, boosting bandwidth by 3X with SHARP in-networking computing and Infiniband Switches, and the H100 can leverage NVLink to connect eight H100s into one giant GPU for 640 billion transistors, 32 petaflops, 640GB of HBM3, and 24 terabytes per second of memory bandwidth.
The chip is custom built by Taiwan Semiconductors with a 4nm design making it the world’s fastest 4nm GPU. The H100 has about 50% more memory and interface bandwidth than the A100. That’s 1.5X more bandwidth with the NVLink connection and PCIe 5.0 doubling the bandwidth of PCIe 4.0. The H100 will ship with support for 80GB of HBM3 memory at 3 TB/s speed.
The NVLink is now able to link together server nodes to build a data center-sized GPU. NVLink was originally designed to bypass the PCIe slot and has become an important tool for chip-to-chip connectivity, especially for high-speed operations. There is a dedicated chip called the NVSwitch which has increased the NVLink’s bandwidth. The ultimate goal is to run 32 servers with their own operating systems to run a single job. NVLink will complement the InfiniBand networking for high-performance computing and NVLink will be default for all of Nvidia’s chips, including GPUs, CPUs, DPUs and SoCs.
Where the H100 really stands apart is the leap in performance with about 3X more performance than the A100 and the H100 is up to 6X faster. The leap in performance is measured by H100’s ability to deliver up to 4,000 TFLOPS of FP8 compute, 2,000 TFLOPS of FP16 compute and 1,000 TFLOPS of TF32 compute and 60 TLOPS of general purpose FP64 compute. The A100 lacked support for FP8 compute at default whereas the H100 will leverage a transformer engine to switch between FP8 and FP16, depending on the workload.
According to Nvidia, the H100 delivers 9X more throughput in AI training, and 16X to 30X more inference performance. The company also states in HPC application-specific workloads, the H100 is 7X faster. The goal of the H100 was not only to add more transistors and make the H100 faster, but to also offer function-specific optimizations. This is achieved through the transformer engine.
The architecture aims to answer one of the bigger challenges facing superfast compute, which is that moving data into traditional servers overloads the CPU and system memory and becomes bottlenecked by PCI-Express.
By improving the bandwidth issue, Nvidia’s goal is to create more demand for their DGX Pod and SuperPod Systems, which in turn, will create more demand for their software.
The H100 DGX Pod is a 32-node, 256-GPU system. The H100 DGX Pod connects 32 DGX systems using the NVLink Switch System to scale into a super-GPU capable of 768 terabytes per second. To compare, the entire internet requires 100 terabytes per second. This results in 1 exaflop of AI computing.
From there, multiple H100 DGX Pods can connect through the Infiniband Switch to scale DGX Superpods with thousands of H100 GPUs. DGX SuperPods are turnkey systems that power enterprise AI. DGX SuperPods were also available with the A100 yet the H100 will have 6X better performance with 1 exaflop of FP8 AI performance to run trillions of parameters (more on this below).
Spectrum-4 Ethernet Platform
Perhaps one of Nvidia’s most important advancements for the H100 is the ability to attach the network directly to the GPU to avoid bottlenecks at the CPU. This is accomplished by sending data with direct memory access at 50 gigabytes per second. Hopper HGX and DGX are networking and interconnects that facilitate moving data with an advanced networking processor called the CX7. The result is the H100 CNX that avoids bandwidth bottlenecks and frees the CPU and system memory to process other parts of the application.
The Spectrum Ethernet platform, which consists of a Spectrum-4 Switch, CX7 SmartNIC and Bluefield-3 DPU will be used for several of Nvidia’s AI platforms, such as Riva, Merlin and Omniverse. These workloads include natural language processing, recommenders, and digital twins and will be supported by a networking system that helps exchange large databases between nodes. Whereas traditional workloads required many connections exchanging small amounts of data, the workloads of the future will require data to be shared quickly between GPUs and storage. This is accomplished by bypassing the CPU and sending data directly to the GPU while using the network hardware to move the data.
This is ideal for enterprise use cases where people are more likely to use Ethernet while AI and HPC workloads continue to use the Quantum-2 based off Mellanox’s InfiniBand. Quantum-2 allows for in-network computing to do data reductions in the network. It’s also more likely that Ethernet is used for 5G and sensors.
Eos: The First Hopper AI Factory
Nvidia is building AI factories to compete with AI supercomputers, which are blueprints for AI infrastructure that can be adopted by cloud partners and enterprises.
Eos is built with 18 H100 SuperPods, with 576 DGX H100 systems and 360 NVLink Switches. Nvidia states EOS is 1.4X faster than the fastest supercomputer and offers 4X the AI processing of the world’s fastest supercomputer. This will deliver 18 EFLOPS of FP8 AI compute or 9 EFLOPS of FP16 compute.
Previously, FP16 was the standard for AI whereas FP8 is gaining more support to become the industry standard. Depending on what AI compute you use, benchmarks will not be apples-to-apples if FP8 is compared to FP64 performance. Right now, AMD’s Frontier supercomputer is #1 with 1.1 exaflops of FP64 performance compared to the upcoming Venado supercomputer’s 10 exaflops of FP8 performance.
The difference is that the smaller bit size allows for an economical way to achieve more speed when giving up a small amount of accuracy doesn’t make a critical difference. This also helps in the face of a slowing Moore’s Law. FP8 is most commonly used for inference yet may be used for training in the future due to boosting throughput. Following the release of the Hopper H100, Intel released Gaudi2 which supports FP8. Chip makers Graphcore, AMD and Qualcomm have recently pushed for an industry-standard for the low precision floating point format FP8 rather than integer formats.
Here is what Nvidia said in the GTC keynote:
But the trend in AI computing has been toward developing neural nets that lean on the lowest precision that will still yield an accurate result. The smaller formats compute faster and more efficiently, and they require less memory and memory bandwidth. The addition of 8-bit floating-point units in the H100 leads to a significant speedup—double the throughput compared to its 16-bit units”
DPX Instructions (ISA):
The H100 improves dynamic programming with DPX Instructions that will help specific AI Algorithms to perform up to 7X faster than previous GPUs and 40X faster than CPU-based algorithms. As algorithms require more complexity, the new set of DPX instructions will help break the complex problems down into simpler subproblems using GPUs instead of CPUs or FPGAs.
The DPX ISA are expected to be broadly available with the CUDA 12.0 release. Examples of where this will be useful include disease research and drug discovery where the process can be sped up 35X for real-time processing to match the rate of DNA sequencing. Route optimization and finding the shortest distance between destinations for use in factories and autonomous driving systems, or Floyd-Warshall acceleration, is boosted up to 40X compared to CPU-only servers. These instructions will also be used for quantum computing and SQL queries as dynamic programming can help find the optimal order for joining a set of tables.
GPU Confidential Computing:
Data is encrypted at-rest and in-transit yet is often unprotected during use. Meanwhile, the data used to train AI models is worth millions in investments and is trained from domain knowledge and company-proprietary data. The new H100 offers confidential computing whereas previously only CPUs offered the protection of both data and applications during use.
Nvidia is Becoming a Leading AI Software Company
It would be easy to read the information above and to assume Nvidia is improving its hardware. However, the company’s future resides in software which will remove some of the cyclicality of hardware revenue. I believe once Nvidia’s software revenue begins to reveal itself in earnings reports, the market will finally piece together the true potential of this AI powerhouse.
It’s both the hardware and the software stack that led me to say Nvidia will surpass Apple in 5 years. You know this story well: the relationship between a hardware company leveraging their position to capture the lion’s share of the software — because that’s exactly what Apple did.
There are four layers to Nvidia’s full-stack accelerated computing: hardware, system software, platform software and applications. Below, I discuss a few ways that Nvidia is capturing more of the software stack due to vendor lock-in effects from their dominance in hardware.
As stated, in the past, our focus was the GPU-powered data center. This was a four-year thesis from 2018 and we doubled up on the thesis in June of 2020 for the A100 release. I want to make sure and emphasize that Nvidia’s lesser-known catalyst is actually the software.
The H100 is helpful in maintaining a lead in GPUs, which is critical turf to protect with GPUs being the most popular AI accelerator, however — the AI/ML catalyst will be further fueled by the Nvidia’s lead in software. This is why the majority of who will remain the AI leader will be up to developers and not the C-suite partnerships on hardware that characterized Intel’s lead over the past few decades. The developers choose the frameworks, the SDKs, libraries and the other parts of the software stack, and because of this, they also choose the GPUs they build on rather than IT departments.
Transformers
The transformer engine is one of the key aspects of the H100. Transformers are becoming one of the most popular neural-network models by applying self-attention to detect how data elements in a series influence and depend on one another.
Sequential text, images and video data are used for self-supervised learning and pattern recognition, which results in more data being used to create better models. Prior to transformer models, labeled datasets had to be used to train neural networks. Transformer models eliminate this need by finding patterns between elements mathematically, which substantially opens up what datasets can be used and how quickly. Transformers are partial to the parallel processing that GPUs offer.
Google first introduced transformer models in 2017 and transformers are used in Google and Bing Search. Transformers also led to BERT models, which stands for Bidirectional Encoder Representations from Transformers, and is commonly used for text sequences. Transformers are also used in GPT-3 (it’s the T in GPT) which improved from 1.5 billion parameters to 175 billion parameters. GPT-3 has the ability to report on queries it has not been specifically trained on.
Nvidia and Microsoft recently worked on a Mega transformer model with 530 billion parameters and the future for AI engineers is trillion-parameter transformers and applications. The H100 is already prepping for this. According to Nvidia, the training needs for transformer models will increase 275-fold every two years compared to 8-fold for other models. The H100 GPU with its Transformer Engine supports the FP8 format to speed up training to support trillion-parameter models. This leads to transformer models that go from taking 5 days to train to becoming 6X faster to only taking 19 hours to train.
The transformer engine is software combined with the new hardware in the H100’s tensor cores. As discussed, the A100 was designed for floating-point numbers to 16 bits while the H100 is designed for 8 bits. This is helpful because AI models are moving toward neural nets that lean on the lowest precision and yet still yields an accurate result. In this case, 8 bits double the throughput of 16-bit units, compute faster and more efficiently, and they require less memory and memory bandwidth.
The main feature from the Transformer Engine is the ability to choose what precision is needed for each layer in the neural network at each step, transitioning between 8-bits, 16-bits, 32-bits, plus the H100 is able to do matrix math with two forms of 8-bit numbers with either 5-bits as the exponent or 4-bits as the exponent: E5M2 and E4M3. This is important because the E4M3 may be favored for back propagation while E5M2 may be favored for inferencing.
Pictured above: Nvidia is prepped to support model sizes growing up to 275X every two years
Triton Inference Server:
Nvidia offers AI frameworks to reduce time for developers throughout the AI workflow from data processing and ETL to deep learning model training and large-scale inferencing. These libraries include Dali, Rapids, Triton and Magnum I/O. The library supports all popular frameworks and offers pre-trained models and data pipelines.
Triton is open-source inference software that helps developers deploy models across GPUs and CPUs, it supports Tensor Flow and PyTorch, any query type and any model – such as Transformers or CNNs (used for image recognition) and RNNs (used in speech recognition). The inference engine helps developers take AI development from experimentation to production by removing the need for multiple inference servers and simplifying machine learning infrastructure on the backend.
MLOps (machine learning operations) helps developers with less ML expertise to train and deploy models yet there were limited use cases with little help in deploying custom models. Triton offers high performance inference and scalability with Dockers and Kubernetes while serving up to hundreds of models with the model control API. By supporting all popular frameworks, Triton helps developers avoid framework lock-in due to the consistent interface regardless of training framework or hardware.
Nvidia will Power the Lion’s Share of Automotive – and that means software licensing
Nvidia’s lead in automotive across dozens of OEMs requires its own deep dive. The reason I haven’t prioritized this is because Hyperion 8 is shipping in 2024 and Hyperion 9 will ship in 2026. However, as long-term investors, we should touch base now on the long-term vision for yet another large and sweeping revenue segment. In fact, automotive promises to be Nvidia’s largest segment by 2030 – so on that alone, imagine what Nvidia investors have in front of us.
Nvidia’s Orin SoC (system-on-a-chip) is designed for the neural networks that run robots and AVs at the edge. This is the central computer for the car. The Orin SoC is capable of 254 trillion operations per second by combining Nvidia GPUs with Arm CPU cores and TensorRT APIs. The goal is to help OEMs move from Level 2 autonomous systems to the elusive Level 5 and it stiffens the competition with Tesla’s FSD. Notably, at the release two years ago, Tesla pointed towards Orin’s power consumption as a potential issue for EV batteries but this has not stopped many competing EVs from adopting Nvidia’s in-vehicle hardware and DRIVE software stack.
The EV manufacturers that have already moved forward with Nvidia DRIVE Orin include: Nio, Xpeng, BYD, Lucid Group, Mercedes and Land Rover, GM Cruise — you name it, it’s probably in production with Nvidia at this moment. The company’s current automotive pipeline exceeds $11 billion over the next six years – expect this small blip of pipeline to grow exponentially.
Nvidia recently announced an upgrade to Orin called Atlan with 1,000 TOPS on one chip, or more than then Level 5 compute in AVs today. This chip will catapult forward the computing performance of AVs and is expected to be released in 2023.
Nvidia DRIVE is the operating system and software stack for vehicles that offers an execution environment and includes both security and over-the-air updates. DriveWorks is an SDK that enables self-driving applications. Drive AV offers key ingredients to an autonomous system, such as perception, mapping and planning modules. Regarding mapping, Nvidia DRIVE Map is a multi-modal drive engine that can map independently and has two map engines. Drive IX is open-source software that offers vision, voice and graphics for the user experience. (I will do a separate deep dive on Nvidia Automotive in 2023).
The entire autonomous platform is called Hyperion, which includes the compute and sensor toolkit. This includes the hardware plus a 360-degree camera, radar, lidar and ultrasonic sensor suite. As stated, Hyperion 8 ships in 2024 with Hyperion 9 shipping in 2026, which will double the processing speed and offer an increase in sensors. Nvidia offers open-source developer kits to help increase its compatibility across various projects.
Rather than train the vehicles on the road, Nvidia trains in simulation and can create virtual world obstacles for the vehicles to learn from. This is a different approach from companies like Tesla who have millions of cars on the road collecting data which they then augment for unusual events with a photorealistic simulator.
Tesla has the most data of any car manufacturer which helps the company competitively as more data equals better performing models especially in terms of object detection. More data from millions of cars on the roads also helps with prediction as Tesla collects data from incorrect predictions that can be added to the training set. By leveraging a prediction neural network, Tesla does not need to use human labeling or annotation and can instead use what’s called a temporal sequence of events — in other words, Tesla rewinds events and labels objects automatically with the use of a supercomputer.
The advantage here is that training neural networks correlates with the miles (which again, are substantial due to size of fleet on the road compared to competitors) rather than correlating with the need for human labeling. The result of automatic labeling is that Tesla is able to predict rare situations with more accuracy.
Where Nvidia delivers a strong advantage is the company has decades of history with graphics and simulation due to its gaming roots. As stated, Tesla also uses imitation learning and has a photorealistic simulator which uses vector space for labels and functions like a game engine. However, Nvidia has been quietly working on their simulation platform for many years internally despite only recently marketing Omniverse to the public. In this case, Nvidia has such a high-level of confidence in their simulation skills that they forego the real-life fleet to primarily train virtual 3D models. The company is also packaging the simulation platform for many other uses cases, such as AI factories, 5G networks, power plants and climate research. Developers can work with 3D tools through Python-based development.
To keep it simple, Tesla’s primary advantage is the data they have collected as no other EV/AV has collected this level of data from real drivers. To contrast, Nvidia has arguably the best simulation platform due to decades of graphics work. These digital twins are only now being widely marketed despite being in development for over 5 years. The license costs $9,000 and Nvidia has estimated its current addressable market is 20 million engineers. Notably, Nvidia’s Hyperion will also be deployed in millions of vehicles over time, offering similar levels of data as Tesla’s fleet.
The Tesla VS Nvidia debates have not formally begun but they are certainly in our future … so brace yourself. Ultimately, the way Nvidia stands apart is the company does not directly compete on manufacturing vehicles. This is something anyone can agree on. That means many OEMs will use Nvidia’s DRIVE system whereas Tesla is less likely to commercialize their software as they’re viewed as a main competitor.
As long as Nvidia continues to innovate and maintain a lead, the popularity of its DRIVE system is likely to remain due to the company’s strategic advantages in AI and supercomputing. The company did an excellent job of tackling the edge computing use case of autonomous vehicles first.
Hardware is only part of the equation. The long-term plan is for Nvidia to license software for autonomous vehicles, which will create a recurring revenue stream. The licensing fees will go well beyond Omniverse to include the actual owner of the vehicle paying a subscription fee to Nvidia for its software. Tesla does this with their AutoPilot software which has grown from $5,000 to $12,000 per vehicle.
$150 billion from Omniverse software – fees are charged to both users and robots/digital twins
$300 billion from Automotive – primarily software
What Nvidia is communicating is that software revenue will surpass hardware revenue long-term.
Here is what Kress stated: "Our software content per vehicle can be in the thousands of dollars over the lifetime of the vehicle compared to the hundreds of dollars for the hardware. And second, software scales with the installed base of vehicles, not annual production.”
Note on CUDA:
The software discussion on Nvidia is not complete without a mention of CUDA. We called this Nvidia’s moat back in 2018 and we continue to believe it provides an important moat. The CUDA-related libraries include frameworks that span quantum computing, robotics, 5G networks, cybersecurity and drug discovery. The universal skills taught around CUDA and Nvidia’s SDKs help to drive more business for Nvidia’s GPUs.
The primary risk right now is valuation as Nvidia trades 2X higher than its peers on both the top line sales valuations and on the bottom line with earnings and cash-based valuations. There’s probably equal risk in waiting for Nvidia to drop another 50% as there is in buying Nvidia at the 2X valuation. One reason Nvidia may be valued here is because it’s slowly becoming a software company. Regardless, Knox’s technicals help immensely in determining if the market will continue to award Nvidia it’s gold medal valuation or if the market will discount Nvidia based on sentiment-driven headlines. This is a position we plan to keep on building so you can keep an eye out for those trade alerts over the next few years.
Conclusion:
Finding great companies is only half the battle, fighting negative sentiment is the other half – and semis have no shortage of this in any market – hence our beginning quotes from 2019 and also 2021.
Nvidia is the strongest company in terms of product on the market today. That doesn’t mean semis won’t be a roller coaster – we should fully expect that semis will undulate in sentiment and price while we hold our stocks over many years. We can’t change the way Wall Street works — which is a pendulum that swings between value stocks and growth stocks — but we can describe in great detail why concerns around gaming and consumer electronics slowing down is not going to change our position. We do not care to perfectly time entries or to find a perfect bottom – you’ll be hard pressed to find any legendary investor recommend that this be an investor’s goal. What we care about is finding quality companies and building those positions over time. Nvidia fits this description.
If you were to guess, when do you think we wrote the following paragraph?
“When a thesis is not reflected in the revenue segments yet, there are typically lower entry points and ongoing volatility. You’ll see in the technical analysis that although I could not be more bullish on this stock long-term, there is weakness in the semiconductor sector and we hope this translates to a lower entry point for our readers.
The market is also in a fierce debate between AMD, Intel, and Nvidia and is also distracted by other chips, such as Micron and NXP. In my analysis, I look for growth. How big is the market relative to how big the company is now?
You can ignore Nvidia’s gaming revenue and other segments for the main trajectory that we are focused on. Gaming is great for stability and earnings reports, but the growth will not be from gaming (a market where Nvidia is already a mature, market leader). I’m also not focused on PC sales or the CPU-powered cloud, as the first is not a growth market and the second is not the piece in the cloud stack that will accelerate future technologies.”
That was written in 2019 yet the far majority of those concerns could be stated verbatim right now. Do we care about PC sales or gaming consoles? No, although our stance is that we have to expect these concerns will affect our semiconductor positions at times. The good news for Nvidia and AMD investors is that as time goes on, the less consumer-related hardware will have an impact. The 2022 Nvidia Investors Presentation provided numbers which show in detail how consumer exposure will become less of a concern in the future for these AI heavyweights.
When do you think we wrote this analysis?
“Over the past few weeks, I have read many lagging explanations on the chip shortage – too many fabless semiconductor companies, too few foundries, automobile manufacturers paused ordering in March and didn’t prepare for the sharp rebound, tensions with China, and even a fire at the Asahi Kasei plant that specifically manufactures sensing devices for the automobile industry.
While all of these are true, the overarching issue is that the role of semiconductors has changed from a commodity to the primary accelerant of future technologies. This is because connectivity, automation, and ultimately AI, will disrupt every corner of every industry.
We saw this happen with data and cloud but now we must accelerate this to the next level for AI/ML and the common denominator is semiconductors. Automotive is only the beginning. We can add renewables to the list and even e-commerce as AR/VR and AI/ML attempt to prop up the leaders who are competitive enough to add these features first.
As a tech stock analyst, I don’t have the luxury of lagging analysis of any kind. My subscribers require (and deserve) forward-looking, and with my intense focus on semiconductor chips, I don’t think my readers are surprised that semis are under pressure due to an increasingly important role.
I have repeated (perhaps too many times) that there is no way forward without the semis. We are seeing this manifest in automotive right now, but as investors, we should get used to hearing about semiconductor shortages.
You and I can debate Palantir, Snowflake or C3.AI, for example, and the valuations or the right angle for AI/ML-driven software, but the common denominator to these companies is the need for semiconductors to drive forward AI and 5G.
Now, we add the enormous push for auto manufacturers to compete with Tesla, Apple, Lucid Motors and what we have is a bottle neck where the automotive industry filters into semiconductors.
My guess is the demand won’t be letting up for many years as we are no longer in the cyclical pattern that semis are notorious for. Instead, demand will outpace supply for years to come.
Is this a bad thing or a good thing for our stocks? As investors, we can either listen to the news or listen to management. In this case, they are not aligned. Machines trade off news and natural language processing (NLP) but as human investors, we have the advantage of looking deeper into the issues.
I have written volumes of analysis leading up to the triple-digit growth we are seeing now in the data center from AI accelerator chips. Most of this was written when data center growth was negative. For instance, my Nvidia thesis was set end of 2018 — and in 2019 Nvidia reported negative data center revenue year-over-year for four quarters in a row.reported negative data center revenue year-over-year for four quarters in a row.
I mention this because following a trend’s trajectory is more important than immediate gratification from the market. The trend will always win out over time.
I have maintained that chips will eventually lead the AI market and are the best angle for investing in edge computing. I have also defended our stocks against custom silicon. Now we have the first of what I predict will be many semiconductor shortages and bullish to me.
The shortage is that there are hundreds (thousands really) of companies that rely on semiconductors. This will come to a head with AI and 5G as those who go-to-market soon with these features will have an enormous competitive advantage.”
That was written at the height of the bull market in February of 2021. My goal is to illustrate there has always been headlines to worry about for the semiconductors. We’ve firmly held these stocks and bought during dips. In the past, from 2018-2019, I focused on the GPU-powered cloud and the CUDA moat here and here. Our 2020 coverage centered on the A100 GPU which we discussed at time of launch for premium here and continued coverage on the A100 about a year later on the free side.
Here is background on the A100:
“Nvidia released the Ampere architecture and A100 GPU as an upgrade from the Volta architecture. The A100 GPUs are able to unify training and inference on a single chip, whereas in the past Nvidia’s GPUs were mainly used for training. This allows Nvidia a competitive advantage by offering both training and inferencing. The result is a 20x performance boost from a multi-instance GPU that allows many GPUs to look like one GPU. The A100 offers the largest leap in performance to date over the past 8 generations.”
Nvidia's AI Dominance Will be Propelled Forward by Software:
I wanted to go back through a bit of Nvidia’s history – what was the thesis and how did the thesis evolve? – before I go into how Nvidia will continue to dominate. In my opinion, I believe this is the most important analysis I have ever written on Nvidia because the company is changing rapidly into a software company.
The shift that Nvidia is going through has gone unnoticed and that’s to our benefit. Because we have been hell bent on finding what companies will dominate AI hardware, I’ve been asked frequently who do I think will dominate AI software (Palantir? Snowflake? Google?)
I’m prepared to give you that answer today: I believe Nvidia will be one of the biggest or perhaps the biggest AI software stack company in the world.the biggest AI software stack company in the world. The analysis below kickstarts our in-depth coverage on this new thesis — and I fully believe I will be quoting this analysis in five years from now when we check back on how the AI software thesis played out.
Before I go into semiconductor jargon where I risk losing your attention, I want to make sure our Members are fully aware that the segment where Nvidia will dominate with AI software is automotive. I am not talking about a few OEMs that trickle into a little bump in revenue. I am saying that Automotive is scheduled to become Nvidia’s number one segment – even over data centers – and to the tune of it being 3X larger than its gaming segment.
Don’t take my word for it because the CFO said exactly that (more on this below) and there is ample evidence that this is happening, which I also detail for you. Wall Street won’t be giving this the credit it deserves until 2023 at the earliest but you will hear non-stop “Nvidia Automotive” coverage by 2024-2026 as this segment ramps. I go over why those are the target dates below.
But first, let’s talk about the H100 and how this new GPU architecture will also help Nvidia lead on AI software at the enterprise level. There is plenty going on outside of Automotive that we need to cover so I kept automotive for last.
GTC Highlights: The Hopper H100 GPU
In March at GTC 2022, Nvidia announced the Hopper H100 GPU with 80 billion transistors and will be released in Q3 of this year. For reference, the A100 has 54 billion transistors. This is Nvidia’s solid attempt to keep their stake in the ground in leading high-performance computing over AMD’s Instinct MI250/250X and the newly announced MI210.
It’s easy to focus on hardware with Nvidia (and AMD) yet these companies are becoming more software-driven each year. By owning the majority of the market for AI accelerators, these two companies are afforded an opportunity to also own the software layer as a means to lower the barrier to entry for training models, deploying inference across various frameworks, and other workloads related to deep learning, conversational AI, video conferencing algorithms, and more. By supplying the hardware, these companies have natural inroads to machine learning operations (MLOps).
The H100 is the New Artificial Intelligence Infrastructure
DGX, DGX Pods and DGX SuperPods:
The H100 will power all AI and high-performance computing systems including the PCI express accelerator for mainstream servers and many H100 GPUs can be combined to power advanced AI through the following systems: DGX, DGX Pod and DGX SuperPod.
The difference between the A100 and H100 is the performance will be two to three times faster. The H100 GPUs and the DGX H100 server pods and super pods offer Nvidia the next leg-up as the company has solved an important bandwidth issue.
Hopper tackles some of the bigger issues around previous generations like speeding up algorithms by offering dynamic programming on GPUs to break down problems to simpler subproblems, boosting bandwidth by 3X with SHARP in-networking computing and Infiniband Switches, and the H100 can leverage NVLink to connect eight H100s into one giant GPU for 640 billion transistors, 32 petaflops, 640GB of HBM3, and 24 terabytes per second of memory bandwidth.
The chip is custom built by Taiwan Semiconductors with a 4nm design making it the world’s fastest 4nm GPU. The H100 has about 50% more memory and interface bandwidth than the A100. That’s 1.5X more bandwidth with the NVLink connection and PCIe 5.0 doubling the bandwidth of PCIe 4.0. The H100 will ship with support for 80GB of HBM3 memory at 3 TB/s speed.
The NVLink is now able to link together server nodes to build a data center-sized GPU. NVLink was originally designed to bypass the PCIe slot and has become an important tool for chip-to-chip connectivity, especially for high-speed operations. There is a dedicated chip called the NVSwitch which has increased the NVLink’s bandwidth. The ultimate goal is to run 32 servers with their own operating systems to run a single job. NVLink will complement the InfiniBand networking for high-performance computing and NVLink will be default for all of Nvidia’s chips, including GPUs, CPUs, DPUs and SoCs.
Where the H100 really stands apart is the leap in performance with about 3X more performance than the A100 and the H100 is up to 6X faster. The leap in performance is measured by H100’s ability to deliver up to 4,000 TFLOPS of FP8 compute, 2,000 TFLOPS of FP16 compute and 1,000 TFLOPS of TF32 compute and 60 TLOPS of general purpose FP64 compute. The A100 lacked support for FP8 compute at default whereas the H100 will leverage a transformer engine to switch between FP8 and FP16, depending on the workload.
According to Nvidia, the H100 delivers 9X more throughput in AI training, and 16X to 30X more inference performance. The company also states in HPC application-specific workloads, the H100 is 7X faster. The goal of the H100 was not only to add more transistors and make the H100 faster, but to also offer function-specific optimizations. This is achieved through the transformer engine.
The architecture aims to answer one of the bigger challenges facing superfast compute, which is that moving data into traditional servers overloads the CPU and system memory and becomes bottlenecked by PCI-Express.
By improving the bandwidth issue, Nvidia’s goal is to create more demand for their DGX Pod and SuperPod Systems, which in turn, will create more demand for their software.
The H100 DGX Pod is a 32-node, 256-GPU system. The H100 DGX Pod connects 32 DGX systems using the NVLink Switch System to scale into a super-GPU capable of 768 terabytes per second. To compare, the entire internet requires 100 terabytes per second. This results in 1 exaflop of AI computing.
From there, multiple H100 DGX Pods can connect through the Infiniband Switch to scale DGX Superpods with thousands of H100 GPUs. DGX SuperPods are turnkey systems that power enterprise AI. DGX SuperPods were also available with the A100 yet the H100 will have 6X better performance with 1 exaflop of FP8 AI performance to run trillions of parameters (more on this below).
Spectrum-4 Ethernet Platform
Perhaps one of Nvidia’s most important advancements for the H100 is the ability to attach the network directly to the GPU to avoid bottlenecks at the CPU. This is accomplished by sending data with direct memory access at 50 gigabytes per second. Hopper HGX and DGX are networking and interconnects that facilitate moving data with an advanced networking processor called the CX7. The result is the H100 CNX that avoids bandwidth bottlenecks and frees the CPU and system memory to process other parts of the application.
The Spectrum Ethernet platform, which consists of a Spectrum-4 Switch, CX7 SmartNIC and Bluefield-3 DPU will be used for several of Nvidia’s AI platforms, such as Riva, Merlin and Omniverse. These workloads include natural language processing, recommenders, and digital twins and will be supported by a networking system that helps exchange large databases between nodes. Whereas traditional workloads required many connections exchanging small amounts of data, the workloads of the future will require data to be shared quickly between GPUs and storage. This is accomplished by bypassing the CPU and sending data directly to the GPU while using the network hardware to move the data.
This is ideal for enterprise use cases where people are more likely to use Ethernet while AI and HPC workloads continue to use the Quantum-2 based off Mellanox’s InfiniBand. Quantum-2 allows for in-network computing to do data reductions in the network. It’s also more likely that Ethernet is used for 5G and sensors.
Eos: The First Hopper AI Factory
Nvidia is building AI factories to compete with AI supercomputers, which are blueprints for AI infrastructure that can be adopted by cloud partners and enterprises.
Eos is built with 18 H100 SuperPods, with 576 DGX H100 systems and 360 NVLink Switches. Nvidia states EOS is 1.4X faster than the fastest supercomputer and offers 4X the AI processing of the world’s fastest supercomputer. This will deliver 18 EFLOPS of FP8 AI compute or 9 EFLOPS of FP16 compute.
Previously, FP16 was the standard for AI whereas FP8 is gaining more support to become the industry standard. Depending on what AI compute you use, benchmarks will not be apples-to-apples if FP8 is compared to FP64 performance. Right now, AMD’s Frontier supercomputer is #1 with 1.1 exaflops of FP64 performance compared to the upcoming Venado supercomputer’s 10 exaflops of FP8 performance.
The difference is that the smaller bit size allows for an economical way to achieve more speed when giving up a small amount of accuracy doesn’t make a critical difference. This also helps in the face of a slowing Moore’s Law. FP8 is most commonly used for inference yet may be used for training in the future due to boosting throughput. Following the release of the Hopper H100, Intel released Gaudi2 which supports FP8. Chip makers Graphcore, AMD and Qualcomm have recently pushed for an industry-standard for the low precision floating point format FP8 rather than integer formats.
Here is what Nvidia said in the GTC keynote:
But the trend in AI computing has been toward developing neural nets that lean on the lowest precision that will still yield an accurate result. The smaller formats compute faster and more efficiently, and they require less memory and memory bandwidth. The addition of 8-bit floating-point units in the H100 leads to a significant speedup—double the throughput compared to its 16-bit units”
DPX Instructions (ISA):
The H100 improves dynamic programming with DPX Instructions that will help specific AI Algorithms to perform up to 7X faster than previous GPUs and 40X faster than CPU-based algorithms. As algorithms require more complexity, the new set of DPX instructions will help break the complex problems down into simpler subproblems using GPUs instead of CPUs or FPGAs.
The DPX ISA are expected to be broadly available with the CUDA 12.0 release. Examples of where this will be useful include disease research and drug discovery where the process can be sped up 35X for real-time processing to match the rate of DNA sequencing. Route optimization and finding the shortest distance between destinations for use in factories and autonomous driving systems, or Floyd-Warshall acceleration, is boosted up to 40X compared to CPU-only servers. These instructions will also be used for quantum computing and SQL queries as dynamic programming can help find the optimal order for joining a set of tables.
GPU Confidential Computing:
Data is encrypted at-rest and in-transit yet is often unprotected during use. Meanwhile, the data used to train AI models is worth millions in investments and is trained from domain knowledge and company-proprietary data. The new H100 offers confidential computing whereas previously only CPUs offered the protection of both data and applications during use.
Nvidia is Becoming a Leading AI Software Company
It would be easy to read the information above and to assume Nvidia is improving its hardware. However, the company’s future resides in software which will remove some of the cyclicality of hardware revenue. I believe once Nvidia’s software revenue begins to reveal itself in earnings reports, the market will finally piece together the true potential of this AI powerhouse.
It’s both the hardware and the software stack that led me to say Nvidia will surpass Apple in 5 years. You know this story well: the relationship between a hardware company leveraging their position to capture the lion’s share of the software — because that’s exactly what Apple did.
There are four layers to Nvidia’s full-stack accelerated computing: hardware, system software, platform software and applications. Below, I discuss a few ways that Nvidia is capturing more of the software stack due to vendor lock-in effects from their dominance in hardware.
As stated, in the past, our focus was the GPU-powered data center. This was a four-year thesis from 2018 and we doubled up on the thesis in June of 2020 for the A100 release. I want to make sure and emphasize that Nvidia’s lesser-known catalyst is actually the software.
The H100 is helpful in maintaining a lead in GPUs, which is critical turf to protect with GPUs being the most popular AI accelerator, however — the AI/ML catalyst will be further fueled by the Nvidia’s lead in software. This is why the majority of who will remain the AI leader will be up to developers and not the C-suite partnerships on hardware that characterized Intel’s lead over the past few decades. The developers choose the frameworks, the SDKs, libraries and the other parts of the software stack, and because of this, they also choose the GPUs they build on rather than IT departments.
Transformers
The transformer engine is one of the key aspects of the H100. Transformers are becoming one of the most popular neural-network models by applying self-attention to detect how data elements in a series influence and depend on one another.
Sequential text, images and video data are used for self-supervised learning and pattern recognition, which results in more data being used to create better models. Prior to transformer models, labeled datasets had to be used to train neural networks. Transformer models eliminate this need by finding patterns between elements mathematically, which substantially opens up what datasets can be used and how quickly. Transformers are partial to the parallel processing that GPUs offer.
Google first introduced transformer models in 2017 and transformers are used in Google and Bing Search. Transformers also led to BERT models, which stands for Bidirectional Encoder Representations from Transformers, and is commonly used for text sequences. Transformers are also used in GPT-3 (it’s the T in GPT) which improved from 1.5 billion parameters to 175 billion parameters. GPT-3 has the ability to report on queries it has not been specifically trained on.
Nvidia and Microsoft recently worked on a Mega transformer model with 530 billion parameters and the future for AI engineers is trillion-parameter transformers and applications. The H100 is already prepping for this. According to Nvidia, the training needs for transformer models will increase 275-fold every two years compared to 8-fold for other models. The H100 GPU with its Transformer Engine supports the FP8 format to speed up training to support trillion-parameter models. This leads to transformer models that go from taking 5 days to train to becoming 6X faster to only taking 19 hours to train.
The transformer engine is software combined with the new hardware in the H100’s tensor cores. As discussed, the A100 was designed for floating-point numbers to 16 bits while the H100 is designed for 8 bits. This is helpful because AI models are moving toward neural nets that lean on the lowest precision and yet still yields an accurate result. In this case, 8 bits double the throughput of 16-bit units, compute faster and more efficiently, and they require less memory and memory bandwidth.
The main feature from the Transformer Engine is the ability to choose what precision is needed for each layer in the neural network at each step, transitioning between 8-bits, 16-bits, 32-bits, plus the H100 is able to do matrix math with two forms of 8-bit numbers with either 5-bits as the exponent or 4-bits as the exponent: E5M2 and E4M3. This is important because the E4M3 may be favored for back propagation while E5M2 may be favored for inferencing.
Pictured above: Nvidia is prepped to support model sizes growing up to 275X every two years
Triton Inference Server:
Nvidia offers AI frameworks to reduce time for developers throughout the AI workflow from data processing and ETL to deep learning model training and large-scale inferencing. These libraries include Dali, Rapids, Triton and Magnum I/O. The library supports all popular frameworks and offers pre-trained models and data pipelines.
Triton is open-source inference software that helps developers deploy models across GPUs and CPUs, it supports Tensor Flow and PyTorch, any query type and any model – such as Transformers or CNNs (used for image recognition) and RNNs (used in speech recognition). The inference engine helps developers take AI development from experimentation to production by removing the need for multiple inference servers and simplifying machine learning infrastructure on the backend.
MLOps (machine learning operations) helps developers with less ML expertise to train and deploy models yet there were limited use cases with little help in deploying custom models. Triton offers high performance inference and scalability with Dockers and Kubernetes while serving up to hundreds of models with the model control API. By supporting all popular frameworks, Triton helps developers avoid framework lock-in due to the consistent interface regardless of training framework or hardware.
Nvidia will Power the Lion’s Share of Automotive – and that means software licensing
Nvidia’s lead in automotive across dozens of OEMs requires its own deep dive. The reason I haven’t prioritized this is because Hyperion 8 is shipping in 2024 and Hyperion 9 will ship in 2026. However, as long-term investors, we should touch base now on the long-term vision for yet another large and sweeping revenue segment. In fact, automotive promises to be Nvidia’s largest segment by 2030 – so on that alone, imagine what Nvidia investors have in front of us.
Nvidia’s Orin SoC (system-on-a-chip) is designed for the neural networks that run robots and AVs at the edge. This is the central computer for the car. The Orin SoC is capable of 254 trillion operations per second by combining Nvidia GPUs with Arm CPU cores and TensorRT APIs. The goal is to help OEMs move from Level 2 autonomous systems to the elusive Level 5 and it stiffens the competition with Tesla’s FSD. Notably, at the release two years ago, Tesla pointed towards Orin’s power consumption as a potential issue for EV batteries but this has not stopped many competing EVs from adopting Nvidia’s in-vehicle hardware and DRIVE software stack.
The EV manufacturers that have already moved forward with Nvidia DRIVE Orin include: Nio, Xpeng, BYD, Lucid Group, Mercedes and Land Rover, GM Cruise — you name it, it’s probably in production with Nvidia at this moment. The company’s current automotive pipeline exceeds $11 billion over the next six years – expect this small blip of pipeline to grow exponentially.
Nvidia recently announced an upgrade to Orin called Atlan with 1,000 TOPS on one chip, or more than then Level 5 compute in AVs today. This chip will catapult forward the computing performance of AVs and is expected to be released in 2023.
Nvidia DRIVE is the operating system and software stack for vehicles that offers an execution environment and includes both security and over-the-air updates. DriveWorks is an SDK that enables self-driving applications. Drive AV offers key ingredients to an autonomous system, such as perception, mapping and planning modules. Regarding mapping, Nvidia DRIVE Map is a multi-modal drive engine that can map independently and has two map engines. Drive IX is open-source software that offers vision, voice and graphics for the user experience. (I will do a separate deep dive on Nvidia Automotive in 2023).
The entire autonomous platform is called Hyperion, which includes the compute and sensor toolkit. This includes the hardware plus a 360-degree camera, radar, lidar and ultrasonic sensor suite. As stated, Hyperion 8 ships in 2024 with Hyperion 9 shipping in 2026, which will double the processing speed and offer an increase in sensors. Nvidia offers open-source developer kits to help increase its compatibility across various projects.
Rather than train the vehicles on the road, Nvidia trains in simulation and can create virtual world obstacles for the vehicles to learn from. This is a different approach from companies like Tesla who have millions of cars on the road collecting data which they then augment for unusual events with a photorealistic simulator.
Tesla has the most data of any car manufacturer which helps the company competitively as more data equals better performing models especially in terms of object detection. More data from millions of cars on the roads also helps with prediction as Tesla collects data from incorrect predictions that can be added to the training set. By leveraging a prediction neural network, Tesla does not need to use human labeling or annotation and can instead use what’s called a temporal sequence of events — in other words, Tesla rewinds events and labels objects automatically with the use of a supercomputer.
The advantage here is that training neural networks correlates with the miles (which again, are substantial due to size of fleet on the road compared to competitors) rather than correlating with the need for human labeling. The result of automatic labeling is that Tesla is able to predict rare situations with more accuracy.
Where Nvidia delivers a strong advantage is the company has decades of history with graphics and simulation due to its gaming roots. As stated, Tesla also uses imitation learning and has a photorealistic simulator which uses vector space for labels and functions like a game engine. However, Nvidia has been quietly working on their simulation platform for many years internally despite only recently marketing Omniverse to the public. In this case, Nvidia has such a high-level of confidence in their simulation skills that they forego the real-life fleet to primarily train virtual 3D models. The company is also packaging the simulation platform for many other uses cases, such as AI factories, 5G networks, power plants and climate research. Developers can work with 3D tools through Python-based development.
To keep it simple, Tesla’s primary advantage is the data they have collected as no other EV/AV has collected this level of data from real drivers. To contrast, Nvidia has arguably the best simulation platform due to decades of graphics work. These digital twins are only now being widely marketed despite being in development for over 5 years. The license costs $9,000 and Nvidia has estimated its current addressable market is 20 million engineers. Notably, Nvidia’s Hyperion will also be deployed in millions of vehicles over time, offering similar levels of data as Tesla’s fleet.
The Tesla VS Nvidia debates have not formally begun but they are certainly in our future … so brace yourself. Ultimately, the way Nvidia stands apart is the company does not directly compete on manufacturing vehicles. This is something anyone can agree on. That means many OEMs will use Nvidia’s DRIVE system whereas Tesla is less likely to commercialize their software as they’re viewed as a main competitor.
As long as Nvidia continues to innovate and maintain a lead, the popularity of its DRIVE system is likely to remain due to the company’s strategic advantages in AI and supercomputing. The company did an excellent job of tackling the edge computing use case of autonomous vehicles first.
Hardware is only part of the equation. The long-term plan is for Nvidia to license software for autonomous vehicles, which will create a recurring revenue stream. The licensing fees will go well beyond Omniverse to include the actual owner of the vehicle paying a subscription fee to Nvidia for its software. Tesla does this with their AutoPilot software which has grown from $5,000 to $12,000 per vehicle.
$150 billion from Omniverse software – fees are charged to both users and robots/digital twins
$300 billion from Automotive – primarily software
What Nvidia is communicating is that software revenue will surpass hardware revenue long-term.
Here is what Kress stated: "Our software content per vehicle can be in the thousands of dollars over the lifetime of the vehicle compared to the hundreds of dollars for the hardware. And second, software scales with the installed base of vehicles, not annual production.”
Note on CUDA:
The software discussion on Nvidia is not complete without a mention of CUDA. We called this Nvidia’s moat back in 2018 and we continue to believe it provides an important moat. The CUDA-related libraries include frameworks that span quantum computing, robotics, 5G networks, cybersecurity and drug discovery. The universal skills taught around CUDA and Nvidia’s SDKs help to drive more business for Nvidia’s GPUs.
The primary risk right now is valuation as Nvidia trades 2X higher than its peers on both the top line sales valuations and on the bottom line with earnings and cash-based valuations. There’s probably equal risk in waiting for Nvidia to drop another 50% as there is in buying Nvidia at the 2X valuation. One reason Nvidia may be valued here is because it’s slowly becoming a software company. Regardless, Knox’s technicals help immensely in determining if the market will continue to award Nvidia it’s gold medal valuation or if the market will discount Nvidia based on sentiment-driven headlines. This is a position we plan to keep on building so you can keep an eye out for those trade alerts over the next few years.
Conclusion:
Finding great companies is only half the battle, fighting negative sentiment is the other half – and semis have no shortage of this in any market – hence our beginning quotes from 2019 and also 2021.
Nvidia is the strongest company in terms of product on the market today. That doesn’t mean semis won’t be a roller coaster – we should fully expect that semis will undulate in sentiment and price while we hold our stocks over many years. We can’t change the way Wall Street works — which is a pendulum that swings between value stocks and growth stocks — but we can describe in great detail why concerns around gaming and consumer electronics slowing down is not going to change our position. We do not care to perfectly time entries or to find a perfect bottom – you’ll be hard pressed to find any legendary investor recommend that this be an investor’s goal. What we care about is finding quality companies and building those positions over time. Nvidia fits this description.
Throwback: Nvidia will Surpass Apple’s Valuation in 4.5 Years
Last August, I predicted that Nvidia could surpass Apple on market cap. Here is what I said in my Forbes article: “I believe Nvidia is capable of out-performing all five FAAMG stocks and will surpass even Apple’s valuation in the next five years” and I expanded on this by stating, “We believe [Nvidia] can surpass Apple by capitalizing on the artificial intelligence economy, which will add an estimated $15 trillion to GDP. This is compared to the mobile economy that brought us the majority of the gains in Apple, Google and Facebook, and contributes $4.4 trillion to GDP.”
As strong as Nvidia has been on price action, Apple will not allow my prediction to be an easy slam dunk as the heavyweight briefly claimed a $3 trillion market cap.
Currently, Nvidia has a market cap of $690 billion and Apple has a market cap of around $2.9 trillion. Nvidia’s market cap rose about 22% compared to Apple’s 17% since my publication of the article. I made this prediction in August of 2021, and during the month of November, we were beginning to make headway with a diversion between semiconductors and big tech.
One of the main reasons for me to make the bold statement that Nvidia will surpass Apple’s valuation is that the market opportunity for Nvidia is vast when compared to the mobile economy, which benefitted Apple.
“Artificial intelligence will touch every aspect of both industry and commerce, including consumer, enterprise, and small-to-medium sized businesses, and will do so by disrupting every vertical similar to cloud. To be more specific, AI will be similar to cloud by blazing a path that is defined by lowering costs and increasing productivity.”
When we began covering Nvidia, we were stating the company would become a leader on AI while most analysts were stuck on the gaming storyline as this was Nvidia’s core product for many decades. This caused many investors to miss out on the top supplier for AI accelerator chips in the data center. We had predicted this three years ago when we wrote: Nvidia has two impenetrable moats – which are developer adoption and the GPU-powered cloud. Notice, we did not mention gaming or crypto mining despite this being the only two narratives on this company at the time.
Sign up for I/O Fund's free newsletter with gains of up to 1100% – Click hereSign up for I/O Fund's free newsletter with gains of up to 1100% – Click hereClick here
We published this again in 2019 for premium members when we stated:
Nvidia’s acceleration may happen one or two years earlier as they are the core piece in the stack that is required for the computing power for the front-runners referenced in the graph above. There is a chance Nvidia reflects data center growth as soon as 2020-2021. -published August 2019, Premium I/O Fund.
Since the original 2018 publication on the two impenetrable moats, Nvidia has greatly outperformed FAAMG. We believe the same will be true over the next five years.
One reason for this is that last year, Nvidia released the Ampere architecture and A100 GPU as an upgrade from the Volta architecture. The A100 GPUs are able to unify training and inference on a single chip, whereas in the past Nvidia’s GPUs were mainly used for training. This allows Nvidia a competitive advantage by offering both training and inferencing. The result is a 20x performance boost from a multi-instance GPU that allows many GPUs to look like one GPU. The A100 offers the largest leap in performance to date over the past 8 generations.
One year later and the Ampere architecture is becoming one of the best-selling GPU architectures in the company’s history. This quarter, Microsoft Azure recently announced the availability of Azure ND A100 v4 Cloud GPU which is powered by NVIDIA A100 Tensor Core GPUs. The company claims it to be the fastest public cloud supercomputer. The news follows the launch by Amazon Web Services and Google Cloud general availability in prior quarters. The company has been extending its leadership in supercomputing. The latest top 500 list shows that Nvidia power 342 of the world’s top 500 supercomputers, including 70 percent of all new systems and eight of the top 10. This is a remarkable update from the company.
There are many other catalysts that will help Nvidia become the world’s most valuable company to prove my prediction true, including the metaverse, automotive, data analytics such as Spark with GPU acceleration, virtual machines for AI workloads and perhaps edge devices by licensing (or acquiring) Arm architecture.
We only have to wait until August of 2026 to see if Nvidia did indeed pass up Apple’s market cap. However, the wait should be an easy one if Nvidia continues to treat investors to the smooth gains (like butter) we’ve seen as of late.
Disclaimer: This is not financial advice. Please consult with your financial advisor in regards to any stocks you buy.
In November, Beth Kindig discussed winning tech stock picks on the Fox Business News show ‘”Making Money With Charles Payne.” Below are video previews of her discussion and an overview of what the two of them discussed.
Despite the current tech rout, we still believe that Roku has the top operating system, is priced reasonably and fits well with smart TVs. Another advantage for Roku is that it has the first-mover advantage in the advertising-based video on demand (AVOD). Roku not only benefits from the cord-cutters but also the brand advertisers. When Beth wrote her initial thesis in 2018, she used to hear questions, “What about Google, Amazon, Netflix, etc.?” Now, the stock has been a multi-bagger for her readers.
Asana is another excellent example of our stock-picking strategy. In this article, you can review how I/O Fund used to blend both fundamental and technical analysis to make gains on this cloud stock. You can also view the webinar from our portfolio manager Knox Ridley where he explains the patterns. Our premium subscribers receive regular trade notifications, trade setups, and market updates from Knox, which helps them to navigate the uncertainties in the markets. The I/O Fund went onto recently closed ASAN for a 285% gain in less than 10 months.
We have been building our positions with one or two strong picks in a year. For example, we have a position in Nvidia since 2018. We had rightly predicted that Nvidia would be a major player in Data Center and Artificial Intelligence. The market questioned our thesis back then. However, we have been firm and increased our position in the stock. Rather than rest on our laurels with long-ago entries from 2018, we continually buy and release our entries. For example, we were able to buy back into NVDA multiple times in this range, which led to 57% returns in less than 2 months. On 9/28 we stated in our service “If we see price move between $200 – $196 starting today and into Friday, that's a strong buy.”
Sign up for I/O Fund's free newsletter with gains of up to 1100% – Click hereSign up for I/O Fund's free newsletter with gains of up to 1100% – Click hereClick here
As you notice in the recent results, the company’s data center revenue accelerated by 55% YoY to $2.94B. Since we entered the stock very early, we have had big gains in Nvidia. Nvidia is also forefront in the Metaverse. We have covered earlier this year and we continue to be bullish on Nvidia.
We always like to start with small positions in a company and then build large positions. Our portfolio manager Knox Ridley guides us to entry and exit positions since he is an expert in technical analysis. Knox also tracks the broader market which helps us to develop a good risk management strategy.
Zoom is another cloud winner, which we started coverage well before the Covid-19. Beth has been bullish on the company due to the great product fit. In her own words, “Product-market fit is what led me to call Zoom Video the best IPO of the year in 2019, why I encouraged investors to know their winners during the cloud selloff, and why we reiterated a buy signal on my research site when Zoom Video was at $65.”
I/O Fund was also one of the early investors of Bitcoin. We started to build positions in Bitcoin in 2019. You can view our sample entries and exits here plus our press release on how Bitcoin contributed to our gains this year.
Short selling of ARKK
There are reports that investors are betting against ARKK Fund and the short interest in the ARKK has been increasing. Tuttle Capital Short Innovation ETF was launched to give an inverse return of ARKK. We believe that the investors are betting against the fund due to the macro environment shifting towards heightened inflation/slowing growth. High beta, low quality, and heightened risk assets tend to underperform in these environments, and ARKK has express no interest in pivoting their portfolio for this macro picture. However, we have a diversified risk management strategy in place. We recently have begun to book gains in high fliers and cut losers that would continue to struggle in a low growth environment. Since the secular bull market began in March of 2009, we have seen 3 slow down periods, which culminated in deep corrections. Each time the FED was forced to reflate the economy, and thus shifting risk-on assets back in the driver seat. We believe that we are entering one of these environments now. Regardless of the noise, and expected volatility, the bull market is not over yet. We always root for Ark and also for innovation so the I/O Fund (respectfully) hopes the shorts get burned!
I/O Fund is comprised of a team of analysts who share their research publicly as they build a portfolio of 30 stocks. Our team has record results for a retail Fund and we also have four-digit gains on some of our free newsletter coverage. You can learn more about our premium service by clicking here or sign up for our free newsletter here. clicking here or sign up for our free newsletter here.
Disclaimer: This is not financial advice. Please consult with your financial advisor in regards to any stocks you buy.
Last week, I wrote about the Nvidia-Arm acquisition, the politics involved as well as its chances of success. Nvidia announced last September that it reached a deal with Softbank to acquire U.K. based semiconductor and software design company Arm Ltd. for $40 billion, making it one of the biggest acquisitions in tech.
Behemoths like Google and Microsoft are opposed to the deal, which requires approval from authorities in at least four governments, including the U.S., U.K., E.U., and China. Qualcomm is fighting the acquisition, as the company relies heavily on Arm for microprocessor intellectual property (IP).
For reference, the name Arm stands for “Acorn RISC Machine” and comes from founders Sophie Wilson and Steve Furber discovering that a CPU can run faster on a small set of instructions. The discovery these two made in the early 1980s was to let the operating system break down tasks rather than add more instructions to the processor. While most CPU designs were adding more instructions to chips, Arm patented the technique of using fewer instructions that run more quickly and efficiently. Due to power constraints of the mobile device, which was introduced much later, Arm found a massive market where it dominates at 90%.
The company works quietly in the background with 22 billion chips shipped globally in 2019 and a cumulative total of 166 billion chips in 2019. The company hit 180 billion chips as of the press release on the acquisition.
Revenue is generated from licenses for Arm’s technology and royalties that come from the subsequent sale of the licensees’ chips that contain Arm’s technology. To compare, Nvidia announced it had shipped 1 billion processors in 2011 and there has not been an update for the 2 billion mark yet, with estimates of Nvidia’s shipments hitting around 100 million chips per year.
Arm is most dominant in mobile with 90% market share in mobile processors, and dominates in-vehicle infotainment and advanced-driver assistance system (ADAS) processor market at 75%. The overall share of Arm’s related markets is 34%.
Arm-based technology is found in electronic devices and PCs, including Microsoft’s Arm-based Surface and Apple’s custom CPUs for Macs. The majority of tablets and digital TVs also use Arm’s architecture.
In fact, Arm offers the most popular CPU architecture in the world. The company’s dominant market share is achieved through its developer ecosystem, which fits neatly into one of our primary theses for Nvidia, its GPU-powered cloud and developer ecosystem.
Arm will bring an estimated 15 million software programmers to Nvidia. In Nvidia’s recent earnings report, the company announced it had doubled from 1 million to 2 million developers – so Arm increases Nvidia’s reach by 7X.
Please note that small cap stocks can be extremely volatile and high risk. Atomera has a market cap of $160 million with a current price-to-sales of 254. The forward price-to-sales is 29. This illustrates that a small cap needs very little revenue growth to move from an outsized valuation to one that is more aligned with the market. This also represents a fair amount of speculation as the forward price-to-sales is determined from a consensus of two analysts who are counting on deals moving through the pipeline. There is no guarantee this will happen.
Atomera’s extreme volatility was on display last week. The stock climbed 24% before erasing those gains by market close. Last month, Atomera offered 1.76 million shares for $5.00 per share to raise $8.8 million. This led to a 13% drop. Additionally, Atomera is expecting no revenue in Q2 due to the effects of the coronavirus. This could add to volatility.
I am covering Atomera because I feel like there are some gains to be had in the breakup between Huawei and Western countries. There is a major restructuring going on. I also like how the market is attempting to price this company right now. We may wait until after the first Phase 4 deal (see below), as there will be plenty of runway left.
Technical analysis can often be less important when a trend is in play and the story is well known. However, for stocks like Atomera, technical analysis is crucial. Knox will be updating our readers this weekend and as we go along on this company.
Please note, as one reader pointed out on the forum, there are bearish comments online about design challenges around MST. These comments are likely correct to some extent and the question is if the company can overcome them. I’ve included more information under the subheading “Design Challenges” below.
I am still initiating coverage and asking Knox to track this stock for an entry because I am comfortable with the iteration process for technologies that solve big problems. The semiconductor market is old fashioned and moves very slowly at times around new processes. However, I am especially keen to find worthy stocks that help strengthen the domestic semiconductor market as China tensions heat up.
Financials
Atomera’s revenue in Q1 2020 was $62,000 compared to $71,000 in the year-ago quarter. For Q4, the revenue was $138,000 compared to $150,000 in the year-ago quarter.
The company’s net loss is $3.6 million per share, or negative ($0.22) EPS. Adjusted EBITDA was a loss of $2.9 million.
The company has cash and cash equivalents of $11.4 million as of March 31, 2020. As mentioned, ATOM recently issued shares at $5.00, for gross proceeds of $8.8 million.
For the full year 2019, revenue was $533,000 compared to $246,000 in fiscal 2018. Net loss was ($0.84) EPS in FY 2019 compared to ($1.02) EPS in FY 2018. TTM revenue is slightly down from FY 2019 at $520,000 and ($0.83) EPS.
The median forward revenue estimate from two analysts is $786,000 for 2020 with one analyst projecting $5.45 million in 2021. It’s the second estimate that makes Atomera exciting although the road may be bumpy between now and 2021.
Roth Capital is the bullish analyst: “We regard ATOM as a highly differentiated silicon enhancement IP vendor that is gaining traction with large semiconductor supply chain companies. We believe the company continues to make solid progress across its significant base of engagements. We expect ATOM to continue to convert additional engagements to licensing revenue over the next few quarters. We maintain our Buy rating.”
Forward EPS estimates for 2020 is ($0.71) and ($0.61).
Integrated License Agreements
Revenue is generated from integrated license agreements. Customers pay a licensing fee to use MST technology in the manufacturing of silicon wafers. Royalties are paid for each silicon wafer or device that incorporates the MST technology (see below for more on MST). The company also generates revenue through engineering services revenue.
According to the 2019 Investors Presentation, the company has an addressable market of $6-$7 billion in royalty fees primarily driven by FinFET and Advanced Nodes ($6 billion), RF SOI ($50 million) and 5V Analog ($660 million).
Please note: I’ve reached out to Investor Relations to confirm these numbers have not changed from any design challenges and will update as we go along.
Total addressable market as of 2018 was $7 billion at 2-3% licensing fees, or $140 million. The 5V analog market adds another $660 million to the addressable market figured on a market size of $33 billion.
Valuation
The gamble that Phase 4 deals will go through is best understood when looking at the spread between forward PS ratio of 200 with the PS ratio in 2021 at 29. Due to Atomera’s tiny revenue of roughly $520,000-$530,000 per year, Phase 4 deals are imperative to reach the 1-year forward price-to-sales (i.e. this is an all or nothing stock).
Customer Pipeline
As of now, there are 19 customers with 26 engagements with 16 in Phase 3 (integration). The more bullish moves around this stock are due to the Phase 3 deals the company has in the pipeline. It would be easy to presume the Phase 3 deals are with larger semiconductor companies as Atomera has a very specific use for its product. TMSC is used often as an example in their Investor Presentations.
According to the March 2020 Investors Presentation, the company is engaged with 50% of the world’s top semiconductor markets. The company states Asahi Kasei Microdevices and STMicroelectronics have licensed the technology plus “a large fabless semiconductor company” for mobile 5G markets.
The coronavirus is a challenge for Atomera as the customers in Phase 3 are cautious with budgest. There is also a slight delay in R&D engineering for new programs. As the company explained on the earnings call, the production personnel are in the fab, but development engineers continue to work from home, which can limit the ability to start new R&D lots.
Per the management, “where some customers would normally be starting wafers, they may be holding back until their engineers get back into the office to start pulling the levers on new lots.”
Here is what the company said about the coronavirus: “Due to the delays created by the coronavirus travel restrictions and the impact on our customer’s business, we are now expecting to have no revenue in Q2 2020. But as Scott indicated in his remarks, none of our customers have ceased work on MST and progress on the JDA contracts has been delayed but not canceled.”
Therefore, any investment in Atomera is a gamble on the company moving one or more customers to Phase 4 (installation). If/when the company moves to Phase 4 through Phase 6 (production), revenue levels become much more attractive.
Notably, it can be viewed as concerning that the company held a secondary offering at $5.00 a share before securing a Phase 4 deal. On the other hand, this may be to buy time and create a necessary financial cushion as covid-19 delays budgets and capex/R&D spending.
Product
Mears Silicon Technology (MST) is a performance enhancing technology that the company believes helps integrated circuits overcome a number of key engineering challenges.
Primarily, MST enhances transistor capabilities and reduces chip size. According to Atomera and a third-party study by PMIC published on their website, MST can result in a 16-21% reduced chip size.
MST allows DRAM designers to reduce chip size without moving to a new technology node. According to Atomera, the IDM process/development is $10 million and the foundry equipment upgrade is $30 to $50 million. Meanwhile, a foundry for a new node can cost billions.
MST is also beneficial in stopping dopant diffusion in high temperature manufacturing, which makes it helpful in chip designs. According to Tailwinds research, MST could become an essential element in FinFET production processes, as dopant diffusion is a major issue. The three major companies who have explored FinFET are Intel, Samsung and TSMC.
According to Atomera, Mixed-Signal/RF devices can also achieve a 10-11% die size reduction. This is achieved with a lattice design that increases horizontal current flow and reduces vertical leakage.
The CEO grew a $1 billion-plus division at Broadcom and also worked as an SVP and GM at Altera. The CTO has been inventing and working on patents for broadband networks for 30 years. I don’t see any flags with the management (a common issue with small caps).
Design Challenges
Atomera is as high-tech as a company and concept can be. Obviously, there are design challenges to overcome or the company would be generating more sales and would no longer be a tiny cap company. What I look for here is whether there will be enough demand to overcome the design challenges and to support the iteration process. I believe there is with the recent pressure on more domestic semiconductor manufacturing.
There is an ongoing debate sparked on a thread by an anonymous commenter on Seeking Alpha. The comment asserts: “High temperatures of older nodes won’t let their concept survive. Finer nodes with finfets/nanowires don't need it.” The comments state that Atomera’s advantage lies in “surface inversion devices whereas finfet/nanowires are volume inverted.”
I try to stay as neutral as possible and weigh both sides of a debate like this. The truth is this company could go either way – boom or bust — but probably not much in-between.
One thing I like about passionate bulls/investors and passionate bears/short sellers is they always bring to the surface the major catalysts or risks.
Here is what Atomera’s Investor Relations team said when I inquired about these issues: “MST1 and MST2 have different properties for handling thermal budgets, depending on the application. There is a lot of variability in customer processes and thermal budgets, and Atomera has worked with enough to have a good sense of how to navigate these types of engineering challenges.”
Below is what one of Atomera’s investors said (who also writes analysis). I’m pasting sections from his blog on the topic below. You can read the full blog here: “Are All the Atoms Aligned for Atomera?”
“The first potential issue to be addressed relates to diffusion of oxygen in high-temperature processes. Which translates roughly into the question of can MST be applied to chips that have high heat during manufacturing? The concern here is that many production lines have stages in which chips are subject to annealing at very high temperatures. If the thin layer of oxygen that Atomera applies were to be diffused in these processes, this would greatly diminish if not alleviate all the benefits of MST. Taking it one step further, if MST is not going to be used in high-heat process manufacturing, the applicable market for MST would be greatly diminished making this truly only a niche product.
From my digging, it appears that this concern is very overblown but not completely without merit.
Atomera has developed a work-around solution whereby they can apply MST at later stages in the process, thereby missing out on the high temperature steps that could diffuse the oxygen. This actually was discussed on ATOM’s Q2 2018 earnings call. Here’s what they said about this issue…
“I’m very pleased to tell you that during the last quarter, Atomera has started testing an optimized version of our film that shows remarkable potential by attacking the problem in a new way. Our approach has been to find a new material construction method that’s better at oxygen retention so it’s able to withstand a wider spectrum of processes surrounding MST…This should make it easier for customers to see better results earlier in their integration process and gain higher confidence in MST’s ability to withstand manufacturing variances during mass production which is a critical factor in their decision of whether to adopt our technology.”“I’m very pleased to tell you that during the last quarter, Atomera has started testing an optimized version of our film that shows remarkable potential by attacking the problem in a new way. Our approach has been to find a new material construction method that’s better at oxygen retention so it’s able to withstand a wider spectrum of processes surrounding MST…This should make it easier for customers to see better results earlier in their integration process and gain higher confidence in MST’s ability to withstand manufacturing variances during mass production which is a critical factor in their decision of whether to adopt our technology.”
I can also confirm that during conversations with management, they have specifically noted that oxygen diffusion is a potential issue, but they are very confident in their ability to deal with this. However, applying MST after annealing will certainly place constraints on the process. Some potential customers will need annealing too late, in the process, for MST to help them. Dopant implantation messes with the crystals, annealing heals them. Is it possible for doping to be accomplished without high energy, destructive implants? Maybe. But if not, how many customers are impacted by this?
At this time, I’m sure Atomera knows how big this potential issue is through their customer interactions, but it’s impossible for outsiders to have this knowledge. Having listened to the Company and learned what I can about semiconductor manufacturing processes, it seems likely that the truth lies somewhere between MST being a niche product and it working on any and all processes. I lean in favor of the market being much larger than some concerns have expressed.
Another issue mentioned by investors related to the stage of the product. Namely, all the white papers reporting the great gains of MST were based on simulations. Which implies that the process has yet to be replicated in the real world. Once again, here’s CEO Scott Bibaud, this time on last quarter’s conference call…
“These papers are based on simulation models and only limited experimental results since the advanced process nodes are not widely accessible and are extremely expensive, but they all show impressive performance improvements with MST. Over the last few months, we’ve had multiple test results from actual silicon runs, which have validated those fundamental mechanisms.”“These papers are based on simulation models and only limited experimental results since the advanced process nodes are not widely accessible and are extremely expensive, but they all show impressive performance improvements with MST. Over the last few months, we’ve had multiple test results from actual silicon runs, which have validated those fundamental mechanisms.”
So, yeah, it’s true that the papers were based on simulations, but Atomera has run their process on customer silicon many times. The theoretical gains were seen in these runs. However, “customer silicon” is, by definition, not Atomera’s. The data obtained from these trials is not Atomera’s to share with the world, and I don’t think a semiconductor company would be sharing their test results
from a new process with their competitors. So, we’ve not seen the actual numbers from silicon runs and will need to base confidence here on management’s statements that those numbers are consistent with simulations.”
Conclusion:
This is a true Hail Mary small cap idea. The stock’s potential hinges on Phase 4 deals coming through (already a gamble) meanwhile the coronavirus may have delayed orders.
However, one or two Phase 4 deals can really move the stock price for this company. The June 2019 investors report provided two scenarios showing $6.7 million in revenue up to $29 million in revenue as a result of signing one large customer.
From a high-level overview, I like this company right now (given the ample risks) because I am keen to invest in the increasing pressure on semiconductor companies to reduce dependency on China and to make up for Huawei’s dominance. As one Seeking Alpha comment had also pointed out, the semiconductor industry can be slow to adopt new technologies. This is very true, however, the geopolitical tensions will put pressure on the manufacturing process.
It’s important to emphasize that if Atomera signs a Phase 4 deal, there will still be time to invest. To reduce risk, we may explore waiting until the first Phase 4 deal is signed OR we will wait for a very clear technical breakout. We are not front-running this stock based on fundamentals.
Knox is essential to navigate this and he will write a blog update on the technicals (and what he’s looking for) this weekend.
