The AI Simple Stack
By Mike Marg and Justin Reidy, Craft Ventures
It’s been a little over a year since the AI gold rush was ignited by the launch of ChatGPT. We’ve seen countless, excellent market maps emerge, documenting in detail how the AI landscape is exploding.
These market maps typically outline a few key areas: compute, AI models, infrastructure and apps.
The problem with the expansive market map is that it visually represents the market as if all sections of the map have equal weight. In reality, market momentum and buyer appetite varies considerably both within and across sections. It’s important to consider the traction of key use cases, and especially important to consider the buying ideal customer profile (ICP), in order to identify the products that have the best chances of taking off and becoming long term winners.
Fortune 5,000 companies (with legitimate budget, headcount, and renewal potential) have begun adopting AI technologies for both their IT stack and their product offering with incredible speed. From an investment standpoint, we thought it would be helpful to understand how these companies are starting to build their AI stacks.
The New “AI Simple Stack”
Although products will emerge to serve every AI use case you can think of, we’re starting to see the emergence of the “AI Simple Stack,” the core areas that every company will soon be evaluating — if they aren’t already. The staples of the new, streamlined stack will include at least three core components:
- An AI model
- The role of the AI model is to be the best, fastest, and most accurate input/output machine:
- the end user enters a command or question
- the AI model must interpret the question and work on a solution
- the AI model produces the best output
2. A vector database
- Vector embeddings + vector search will make it easy to classify all product or enterprise data so that pulling the relevant information to a generative query will be faster, more accurate, and more complete.
3. A data retrieval mechanism
- The role of the retrieval mechanism is to access all potential data (including both internal data and application level data) that might be needed for problem solving, gain permission to that data, and grab that data so the best output can be delivered. Without a retrieval mechanism, an AI model will be largely limited to knowledge that it has been pre-trained on, which would severely inhibit the enterprise use cases. And without the vector database, the retrieval mechanism won’t know exactly which data to pull.
This stack will help business leaders and technology decision makers to supercharge both their employees, as well as their products. These are the two key areas of investment, and these executives are likely to pursue a minimum viable stack at first, as opposed to dozens of products that could be costly and time intensive to deploy.
The Emerging Stack
The “AI Simple Stack” as it stands today is based around retrieval augmented generation (RAG).
It allows an AI app to connect to an end user’s existing work data (think files, folders, app data, etc.) The AI enabled app will be able pull that data via a data retrieval mechanism into a context window. The AI model + data pulled into the context window + pre programmed prompts = generative output.
Over time, these outputs will more effectively travel to other applications via a data retrieval mechanism, and power workflows that can live outside of the AI assistant or AI enabled product.
1. The AI Model
The race is on to build the best AI model for employees. As CIOs rush to deploy the AI model as an enterprise friendly copilot (or “AI assistant”) for end users, product orgs rush to integrate the best AI model into their product to supercharge their features. Each of these represents a buying cycle, a wave of evaluations that is well underway.
Put yourself in the buyer’s shoes. Here is what you’re likely looking for:
- A model that reliably produces quality outputs from the given inputs
- A model that excels in enterprise friendly use cases, like reading comprehension, business writing, creative writing, instruction following, data analysis, and others
- A model with the ability to improve steadily (or better yet, rapidly) over time
- A model that is affordable, speedy, and produces high quality results
- A model that is easy to deploy, maintain, secure and monitor
- A model that is secure, and whose brand inspires confidence. Think of the maxim “nobody ever got fired for buying IBM.” The same will be true in AI.
Under this lens, it makes sense why OpenAI is making strides to win the AI model category. They serve the employee-centric AI assistant market (with ChatGPT and now ChatGPT for Teams), and they also serve the developer-centric product integration market with their API. They have launched rapid improvements from GPT3, to GPT 3.5 Turbo, to GPT 4, and GPT5 is soon to be released.
For others to compete, performance, output quality, brand perception, cost, and most importantly, ease of use will be critical. Anthropic, Grok, & Writer are names to watch here, along with Google (fka Bard, now Gemini) and Meta’s efforts (LLaMA).
Closed source models typically handle a great deal of complexity on behalf of their customers, and appeal to many because they come “batteries included.” Open source will be a viable path as well, though it will require much more expertise and maintenance, and a “bring your own” infrastructure approach.
2. The Vector Database
We have spoken with numerous technology leaders who have recently made a decision to adopt a vector database to supercharge their in-app search experiences: we believe vector search/vector database market will split into closed source and open source markets, and both will have respective winners.
In the old world, a search experience relied on finding the exact words of a given query. In the vector powered present & future, everything in a database can be represented as a multi dimensional concept.
Old world: a search for “Q4 2022 metrics” would be very difficult for a search engine to understand. It would be able to find that exact phrase, but not much more.
New world: search for “Q4 2022 metrics” within a vector database, and it will be able to identify phrases and passages that likely fit your query within a given set of data. This is a major breakthrough.
For generative queries, your AI stack would be able to find and pull the right data for a given job in a “just in time” manner.
While Pinecone seems to be leading the closed source vector DB market so far, Chroma, Weaviate, and Qdrant are making great strides on the open source side. Some customers just want a managed service to make their lives fast and easy. Some customers will never feel comfortable with a closed source product as core infrastructure, and they will turn to open source solutions as a result. We expect vector databases to not only infiltrate in-product data stacks, but also the internal company data stack.
3. The Data Retrieval Mechanism
Without data retrieval (and ever expanding “context windows”) an end user will be limited to outputs that their AI of choice has been trained on. This is a challenge in an enterprise context.
A lot of data that you need to get your work done is locked away in different apps you use, or different spreadsheets or files. Accessing this data is a challenge, and understanding which data a user has access to (ie, navigating access control) is an even bigger challenge.
Glean is a product to watch in the data retrieval space. They started out as an enterprise search product to handle these exact nuances for the search use case. They spent a lot of time and energy building the best search capability, secure integrations, and permissioning that maps to key data sources, and it shows. Glean is emerging as an exciting enterprise RAG (and enterprise AI assistant) solution.
With a RAG (again, “retrieval augmented generation”) approach, end users can perform generative AI actions on top of their existing work and data instead of having to feed their data and files proactively into an AI system. This will help users avoid a laborious, slow, frustrating, and potentially even insecure process. This ability to generate on top of your existing work, wherever it may reside, is critical to the promise of enterprise AI.
In contrast to Glean’s cross-silo approach, Elastic is working on workplace search solutions, and Microsoft & Google will probably have an interest in releasing this type of functionality within their suites of applications soon.
What’s next
The heat map of the most in-demand use cases will evolve over time, and new layers of the AI Simple Stack will continue to emerge.
We’re excited about better data pipelines, AI security/monitoring, collaboration between AI native platforms, memory/reinforcement learning, data retrieval from the internet, and improvements to the agent ecosystem. It’s just too early to predict where areas of opportunity will emerge, and which will warrant a full fledged buying decision vs. becoming a feature of an already well-entrenched platform.
Many top contenders for this emerging stack will be closed source, and will appeal to many buyers for their speed and ease of deployment.
Others will take an open source approach, the “do it yourself” path, which appeals to enterprises that are more security focused and less content to trust their core infrastructure with managed service providers.
As nearly every company on earth rushes to figure out their AI strategy, the staples of the Simple Stack will become extremely valuable. These core areas will be hotly contested and leaders will make their solutions easy to try, easy to deploy, easy to scale, and ultimately, easy to buy. It’s hard to overestimate the opportunity: a near endless B2B total addressable market is up for grabs.
