There are three important trends we weave together in this report to draw conclusions around potential winners in big data and analytics. We’ve recently covered MongoDB, Elastic, and we’ve discussed Confluent. What do they all have in common and why are these companies important right now? That’s what we aim to answer in this write-up.
Before we go into where we are with big data and analytics right now, I’ll quickly touch on cloud IaaS and especially why hybrid and multi-cloud are leading this space and why investors should not be concerned with tech giants that offer competing products in the data and analytics space.
When we talk about Big Data, the main driver is machine learning, which is performed through supervised learning with the use of historical data, or unsupervised learning, with clustering models and associations to identify rules. There is also reinforcement learning that is trained from feedback. Here are the three trends we are going to weave together to form a full picture of Big Data and Analytics.
- The migration to the cloud — but more specifically multi-cloud and hybrid
- Why multi-cloud drives demand for best-of-breed, i.e., generally speaking, we do not need to be overly concerned when tech giants that offer competing products
- How Apache Spark helped catalyze the AI/ML market with efficient data processing
- How we plan to invest right now given that #2 and #3 are prepping the market for us
Hybrid and Multi-Cloud are Driving the Cloud IaaS Market:
For cloud IaaS, we don’t want to only focus on CAGR but also the budget allocation that cloud IaaS is capturing. According to IDC, the IaaS market will reach $112.9 billion at a CAGR of 11.3% through 2025 and will account for 66.1% of total compute and storage infrastructure spend. Two-thirds of IaaS spend is on the public cloud.
To compare, the on-premise market (i.e., not hybrid) will grow at 0.3% CAGR for a total of $57.9 billion. According to the most recent Denodo survey, hybrid cloud drove 35% of the workloads worldwide. Private cloud expanded from 16.6% of workloads to 24% percent of workloads worldwide. Meanwhile, the public cloud had flat growth.
Hybrid cloud is a mix of public and private clouds or a mix of cloud and on-premise. Enterprise companies that choose hybrid deployments are motivated to not share intellectual property or data with a vendor, known as data residency, plus other security implications that come with storing data on another company’s servers. Other companies find moving to the cloud to be time and resource-intensive and prefer to keep some workloads on the servers they own.
Recently, a report came out that repatriation, or moving some workloads back to on-premise, has resulted in quite a bit of cost savings for companies like Dropbox, Crowdstrike and Zscaler, who use hybrid approaches. The report is quite surprising as the conclusion is that $100 billion to $500 billion in market value is lost on cloud deployments in terms of margins. One use case that is detailed is Dropbox, a company that reported savings of $75 million in two years after repatriation, which in turn, helped the company’s gross margins increase from 33% to 67%. Meanwhile, companies like Asana and Datadog spend about 60% of their revenue towards committed cloud spend. This report, among others, shows why hybrid is likely to be the chosen deployment for many enterprises into the near future.
We had previously formed a Microsoft thesis in 2018 based on the trend towards hybrid cloud and why a focus on a hybrid strategy for governments and enterprises was important to Azure’s growth rate. Microsoft is especially well suited to serve the hybrid market because of the company’s deep roots with on-premise enterprise software. When the I/O Fund first covered hybrid cloud as a major driver of cloud IaaS in 2018, Amazon’s AWS did not even have a publicly available hybrid product. The company later publicly released Outposts in 2019 to compete with Azure. If you want more information about how these two compete on hybrid on-prem deployments specifically, the in-depth analysis I published in the past is found on Seeking Alpha and also Forbes.
Multi-cloud refers to using more than one cloud provider, which is usually done to avoid vendor lock-in and to choose best-of-breed products. It also helps to avoid downtime should one cloud provider go down or become overwhelmed with demand.

Source: Statista
Multi-cloud is the dominant strategy today and is used by 80% to 90% of organizations. In 2019, Gartner stated 81% of respondents were using two or more cloud providers. The top reason was to avoid vendor lock-in by the “megavendors.” Therefore, this is why investors should not be concerned with tech giants offering competing products in the data and analytics space. The far majority of companies are taking strides to avoid vendor lock-in as multi-cloud technically requires more work yet increases agility and flexibility. The end result is these companies will use best-in-breed products.
According to IBM, 98% of companies plan to use multiple hybrid clouds and 85% operate in multi-cloud environment. There is substantial evidence that organizations are preferring a mix of cloud providers. Most importantly to our thesis and this particular analysis, only 40% use management tools and/or have implemented DevOps practices. The migration to the cloud was happening slowly over time and this migration is under-served in terms of management tools, data and analytics. This would be a sufficient tailwind on its own yet we also have the additional tailwind of data-intensive industries that are moving into machine learning.

The motivation behind cloud IaaS growth and especially hybrid and multi-cloud growth is partially driven by the need for analytics, and also newer trends, such as stream processing. Stream processing is a continuous stream of events that is processed in real-time as it’s received. This allows applications to respond to events as they occur. It combines real-time analytics, inferencing and monitoring to achieve things like optimizing transportation routes, understanding traffic patterns, anomaly detection in cyber security, making real-time predictions powered by machine learning, and even location-based advertising.
In terms of architecture, we’ve covered how microservices and containers are also driving the multi-cloud trend as microservices often span multiple clouds. You can find this write-up here on Forbes and also here on Medium, where we discussed a background on Google Cloud and how the company was the first to automate orchestration across containers. This write-up provides a great overview of where the major cloud IaaS providers are today and where they might go next strategically speaking.
Big Data and Analytics will Explode because of AI/ML Applications
There is an oft-quoted statistic that 90% of the world’s data was created in the last two years – and this stat is from 2018. The world produces 44 zetabytes of data across the digital universe as of 2020 and there is expected to be 200+ zetabytes of data in cloud storage by 2025. Each zettabyte has 21 zeroes or is 1,000 bytes to the 7th power. By these estimates, we can expect to see up to 5X growth specifically in data centers. Statista places the number at 181 zetabytes by 2025 up from 64.2 zettabytes in 2020.
In regards to data integration in the cloud, this spans from data lakes, to ETL pipelines, cloud data warehouses and object storage. Data fabrics and data virtualization is key to both hybrid and multi-cloud strategies.
Here's how Datadog’s CEO describes what is going on in terms of big data in the most recent earnings call: “it's almost a given that there will need to be a different way of charging for capturing some of the value provided to customers that can't just be attached to the straight volumes of data that are being exchanged because those volume of data are exploding exponentially while our customers' revenues are not going to explode exponentially.”
Generating the data is not the issue (clearly), and distributed data storage has been largely solved with Hadoop. I think it’s worth going through what Hadoop is and how it came about, and then we can look at how Apache Spark helped accelerate data processing, including for Machine Learning. Notably, most open-source projects are not “easy” and this is why companies do well that simplify how to work with Apache Spark and other frameworks, like Kafka.
Background on Hadoop and Data Storage:
Hadoop became instrumental in helping companies store large amounts of structured data, semi-structured and unstructured data through distributed storage and compute. The result was that data storage became cheap enough to retain any/all data that was generated rather than only the essential data due to its distributed file system. The distributed file system was designed to store and process billions of search engine pages across thousands of nodes. The project was created in 2006 by a team of engineers at Yahoo, who had worked previously on a search engine in the early 2000s with the goal of indexing 1 billion pages.
You can think of search engines as some of the first projects that needed to utilize Big Data. The original search engine project “Nutch was limited to 20-to-40 node clusters, and for this amount of data, more clusters were needed. At Yahoo, the team separated the distributed computing parts from Nutch and renamed the project Hadoop, which successfully worked on thousands of nodes. Parallelism was key for the data processing model as Yahoo’s algorithm would need to be run on multiple nodes at the same time and it had to scale linearly. It was then released in 2008 as an open-source project with up to 4000 nodes with distributed capacity with contributors such as Facebook and LinkedIN.
Distributed systems and parallel computing didn’t begin with Yahoo, of course, it began with Google. The paper “MapReduce: Simplified Data Processing on Large Clusters” is considered a defining moment in how programming models handled large data sets. MapReduce was a key moment because it was specifically designed to handle Big Data in terabytes and petabytes due to its framework for parallel computation using a key-value pair.
By 2012, Hadoop’s clusters were up to 42,000 nodes and the number of contributors had reached nearly 1500. Apache Hive is a ETL and data warehouse tool that uses SQL, but Hadoop can manage and process large volumes of data that are structured, unstructured or semi-structured data depending on the database that is chosen. Therefore, you can use many tools with Hadoop, such as Spark.
Background on Apache Spark and Data Processing for Machine Learning:
In 2014, Apache Spark was released which took over the MapReduce model primarily because of its speed. By working with data in-memory, the parallel processing framework can push queries 100X faster and on-disk queries run 10X faster. After the extract, transform and load the data (ETL) process, with Spark you can run a training algorithm on the same in-memory data. This helps Spark reach peak performance over competitors for ETL and relational queries, but also for machine learning. Spark’s goal was to become (and now remain) the general platform for distributed programmers where many specialized systems have one interface and one system to install and manage. Apache Spark also reduces code volume by using APIs for Scala, Java and Python. The framework offers a unified API for fault-tolerant stream processing, which reduces the number of APIs to learn. Spark ML and SparkML are the two APIs that are offered for machine learning pipelines.
Hadoop helped solve some of the data storage issues and reduced the cost for expensive storage and compute. Therefore, the next issue is who can work with these databases and can this be simplified. Apache Spark simplifies who can work with the framework by supporting libraries, which can be executed to interact with data shared across many libraries. The data processing engine is extremely fast because it processes and keeps the data in-memory without reading or writing to disk. This has resulted in Apache Spark becoming popular for machine learning and AI applications with the support of Apache’s very large community of contributors.
Overview of Public Companies in the Big Data and Analytics Space
Databricks and Snowflake:
I’m starting with Databricks and Snowflake simply because we discussed Apache Spark in this analysis. The founders of Apache Spark are from Berkeley and later went onto become the founders of Databricks. We covered this company in-depth on our Snowflake analysis because we feel this is Snowflake’s strongest competitor (i.e., not traditional SQL warehouses or Big Tech). Databricks is not public right now but plans to go public soon.
Here is a summary of the explanation we published in April as to how these two companies compare:
The major difference between Snowflake and Databricks from a customer standpoint is that Snowflake is laser-focused on the public cloud/cloud native while Databricks is differentiated in that it can build information pipelines across silos, including on-premise and hybrid architectures. As we know from this analysis, hybrid is key moving forward.
Snowflake's main value proposition is to reduce the time required to prep and monitor data so that a customer does not need to manage software or hardware. Even if a team has the technical skills, they may not want to spend the time required for Databricks, which is perhaps one reason why Snowflake is reporting decent growth in the Fortune 500 and other key accounts.
The architecture of a data lakehouse allows for business intelligence and machine learning through a more open paradigm. The idea is to combine the best of data warehouses and data lakes to span unstructured and semi-structured data while keeping costs low. By combining both, teams can move faster and without duplicating the data. This is a key benefit to Databricks DeltaLake, and this is especially important for data analytics and machine learning. With that said, Databricks is more advanced and expert-level.
I want to point out that Snowflake is very clear as to why it's done well – which is that it handles migrations to the public cloud from legacy on-premise systems better than the competitors. Snowflake's priority is to compete with other SQL databases right now, although the company will need to eventually compete with Databricks. Management has discussed rolling out support for unstructured data, for instance, but no timeline has been set.
Looking longer-term, what Snowflake needs to answer is how will it compete with Databricks on machine learning? Databricks is superior here for ML as it’s built on top of Apache Spark and supports Spark, Python, Scala and also SQL. This was discussed in the thread on the forum here.
The forum thread points out that Databricks is more complex to upload the data, monitor and manage, but there are benefits to going through this hassle. One of the primary benefits is support for Python and Scala, which are programming languages for machine learning. For now, you must use an outside vendor or tool as connectors or integrations in order to support these programming languages and libraries with Snowflake. It’s also worth mentioning that Databricks is cheaper for processing a lot of data at petabyte scale.
Growth is the great equalizer when comparing products and my preliminary understanding is that Snowflake is growing much faster than Databricks and expects to continue to outpace the competitor. I will need to look into Databrick’s financials and see an earnings report or two to determine more about the competitor’s sustained growth rate.
What I find to be very intriguing is what Snowflake will do to compete on ML. This gap in product capability is not lost on the Snowflake team and management. Being laser-focused on the public cloud/cloud-native lends itself well for Snowflake to compete here theoretically, yet its laser-focus on SQL is getting in the way strategically speaking. The company is aware of this and plans to roll out support for unstructured data.
We have two strong products here yet the valuation on Snowflake is stretched and I imagine Databricks will be, too. It’s rare to see a company sustain higher than a 40 or a 50 forward P/S for an extended period of time. Right now, Snowflake is at a 79 forward P/S.
MongoDB:
Big Data applications require a flexible data model, which NoSQL supports. MongoDB is a database that can handle unstructured and semi-structured data, whereas SQL competitors require data to be structured and stored in tables. The predefined schema of the relational database is correlated due to common characteristics. SQL is well-supported as the original database management type yet NoSQL is also reaching critical mass.
The reason NoSQL has risen in popularity is because as data grows, there are more data types to work with outside of Excel spreadsheets/CSV or tabular structures. MongoDB and its competitors are a good choice for Big Data because NoSQL databases can process unpredictable and unstructured data. The most popular types of NoSQL databases include graph, key-value pairs, columnar and document.
Moving forward, we think NoSQL is going to take more market share, simply because it saves steps when dealing with Big Data as the unstructured data does not need to be converted and this is preferred for some machine learning models. This is why NoSQL is used by companies that generate the most data, like Amazon, Facebook, LinkedIN and Google. The extra bonus is that the JSON documents in NoSQL databases can be prepared for machine learning. Because you do not have to define a schema, this allows data to be directly loaded from any new source without changing lines of code. SQL is used in training machine learning models with most of this data coming from on-premise servers. Therefore, the migration to the cloud and various types of data that are generated is also helpful for companies like MongoDB in growing market share. This is because the cloud produces various forms of data.
MongoDB has a query language and secondary indexes for specific values to filter, sort and aggregate data. The leading NoSQL database also allows for the storage and retrieval of trained models as JSON documents. In this case, you can query MongoDB to pull up a previous model.
In the multi-cloud trend, MongoDB is a leader here as the company was the first cloud database to run applications simultaneously on all major cloud providers. The multi-cloud clusters allow developers to deploy applications across multiple cloud providers without having to manage the complexity. In addition, the technical team at MongoDB maintains that you can forego Hadoop and Spark, which requires complex functions and logic, and instead rely on Tensorflow.js, MongoDB and a browser for the same level of machine learning but with less complexity. In an example, a MongoDB representative was able to write a ML program with 88 lines of code. With that said, NoSQL requires more expertise than the universal language of SQL.
The takeaway is that Big Data companies prefer NoSQL for many reasons, and we think in the era of ML and AI, that more companies will lean towards having similar requirements as Big Data companies. This isn’t to say that SQL isn’t alive and well due to the sheer amount of support for structured data across various database systems. Financial transactions for instance fit well into SQL. This is not a “SQL will die” discussion, instead it’s a “NoSQL may see a bigger market thanks to big data and the sheer amounts of unstructured and semi-structured data that will continue to grow” discussion.
Although the SQL and NoSQL debate has lingered for some time with SQL being the leading database today, requirements may change and we think MDB is positioned well for this shift.
Also, refer to the fact that MongoDB is fifth in terms of database market share yet is tied for first place for most wanted database skills among software developers. Notably, MySQL and Oracle are the top database systems globally yet MySQL is fifth in terms of most wanted database skills. The demand for talent is typically an important indicator of where we are now and where the puck is going.
You can read more about MongoDB here in our deep dive research report including more details on Atlas.
Confluent:
The founding team of Apache Kafka worked at LInkedIN before leaving to start Confluent. Apache Kafka is used by thousands of companies for message streaming, such as LinkedIN, where a publish/subscribe model allows applications to share and create data in a serverless and microservices architecture. What Kafka solved for is the ingestion of events data in real-time and with low latency.
At the time that Kafka was developed, LinkedIN was ingesting 1 billion events a day. The company is now ingesting 1 trillion per day. Kafka does this through a log that writes messages to a topic and is able to retain messages for a long time. Kafka is also used in stream processing by parallelizing the pipelines. Kafka Streams were built to increase simplicity while retaining the same amount of performance as a Spark streaming job.
As with Spark and other open-source projects, there is a marketplace for making the frameworks easier to use. Confluent Kafka opens up the amount of data that can be integrated, for example, to combine transactional data (orders, inventory) with sentiment-driven data (likes, page clicks). This helps with predictive analytics and also machine learning because the “data flow” allows for algorithms to work as they are intended to. This is what is meant by the title slide of the S-1 filing “Set Data in Motion.” In order for data to be in motion, Confluent’s platform connects data from many different sources.
The end result for Confluent is that the company allows large amounts of data to be moved very quickly. This is needed for machine learning algorithms that are very data hungry. Kafka can be paired with Apache Spark and Apache Samza to route data and then load it into ElasticSearch, for instance, so it’s a bridge (or a nervous system according to Confluent’s marketing department).
The goal of Confluent is to reduce operational complexity. In the case of Kafka Streams, this is done by not requiring a cluster to be spun up, offering a single framework for streams of events, and reducing the number of pieces in a stream architecture. Confluent Cloud is growing rapidly at 200% year-over-year, primarily driven by event streaming.
Please note, that Confluent is on a partial lockup schedule. The partial lockup dates are 15% on the day of the IPO (June 24th), 25% on the second day of trading (August 09th) after the Q2 earnings, with the remaining at the earlier of the second day of trading after Q3 earnings and 181 days of the IPO.My note: Already up to 40% of the shares have already been released by the eligible employees. The full lock-up expiry is between November and December..
Elastic:
Elastic is a best-of-breed search company that has other benefits, as well. Elasticsearch is the core product that allows for the searching, storing and analyzing of data. This allows developers to build search features that pair Uber passengers with drivers, recommend grocery items on Instacart based on your history, match online data profiles for Tinder, or log events for Fitbit at a rate of 250,000 logs per second. In addition to searching and storing data, Logstash and Beats are ingestion tools to ingest data from applications and to query external systems. Kibana is an open-source tool for visualizing the data. We’ve covered Elastic Stack in more detail here.
Since 2018, the Elastic License has been free and open source with paid proprietary features. As Bradley detailed in this write-up, Amazon began to profit from Elastic’s open-source software and did not contribute back. According to Elastic, over 90% of new downloads choose Elastic’s License. As of January 2021, the company dual-licensed Elasticsearch and Kibana under SSPL or “Server Side Public License,” which requires Amazon or any others to publish modifications and the entirety of their source code. We think the multi-cloud trend is one reason that Elastic has been able to overcome Amazon as the primary driver is to avoid vendor lock-in. Notably, Elastic is cloud neutral so it does not rely on any specific external services for machine learning like AWS’s OpenSearch. Basically, this goes back to the points we made about multi-cloud earlier in this analysis.
We also discussed Elastic’s move into XDR is important because security is a primary concern for those who are on multi-cloud deployments. The SIEM and XDR space is not without its competitors yet it could be Elastic’s combination of already having ingestion tools for thousands of applications and sensors that lends itself well to monitoring and detection. SIEM is security, information and event management while XDR stands for extended, detection and response (XDR). SIEM was first used as a compliance product and often works alongside endpoint and network security products in order to offer a narrower yet deeper set of activity. This last piece has become critical over time. For Elastic’s product, XDR builds on the SIEM and EDR (endpoints) combination for more accuracy and applies machine learning models to detect anomalies.
Where there is data, there will be new opportunities for growth as the AI/ML landscape goes from nascent to mature (i.e. not all uses cases have arrived for big data and analytics companies). Due to Elastic being essentially a pretrained model for extracting keywords and synonyms and “term co-occurrences”, it lends itself well to natural language processing (NLP). With Elastic, terms can be filtered by significance and offer out-of-the box shortcuts to Python with its REST API. Cognitive search is a new form of search that uses AI to improve search queries and to extract information from multiple data sets. Cognitive search can combine a traditional search engine with NLP to extract more useful information since keyword search is limited in the variety of data that can be searched. Cognitive search uses machine learning algorithms for its greatly improved search results and will be a $6 billion market by 2025. We think it's impressive that Elastic was named a Leader in the Gartner Magic Quadrant for cognitive search in the first year it was added as a new entrant, blowing past Microsoft, AWS and even Google.
Conclusion:
I wanted to cover Big Data and Analytics broadly and horizontally rather than vertically by company because it paints a better picture of what we are positioning for and why. It’s easy to get lost in the jargon when discussing companies individually especially with technical companies like these. But what really separates each of them? We think the side-by-side comparison can be more conducive at times when setting up a microtrend.
We had a few goals with this analysis that I hope we accomplished:
- Bring to your attention this trend (and the common thread) and pull-out names from the general “cloud” list to discuss why they may have a unique catalyst. There will be many winners in this space and we are limited in terms of number of positions we can enter. It’s easy to get caught up in “stock picks” yet we also want to offer you microtrends to help inform your individual portfolio decisions.
- We think big data and analytics from best-of-breed companies could become a solid post-covid cloud play due to the sheer number of companies that migrated to the cloud yet have multi-cloud and hybrid deployments
- Third, I want to make sure and elaborate on where the MongoDB, Confluent and Elastic positions are coming from that the I/O Fund recently entered. We offer deep dives on companies but we also want to anchor our readers with the underlying microtrends that we are investing in. For instance, Snowflake is a great choice, yet the valuation is high and that range above 50 has not treated us well in the past (i.e., personal choice). Perhaps for your investment profile, you prefer Snowflake right now, etc.
This is a big space and it’d be impossible for me to cover everything but we pulled out the critical pieces. We think it’s important to simplify the key drivers of a microtrend and illustrate the ways that specific companies are serving the trend. You can expect to see MongoDB and perhaps Confluent added to the LTBH portfolio as the thesis should take about 3-5 years to fully play out. The main thing to know is this means we will have to remove a name or two from the current LTBH portfolio. We will keep you in the loop as we weigh these decisions.






































