Skip to content

a16z-infra/llm-app-stack

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 

Repository files navigation

LLM App Stack

aka Emerging Architectures for LLM Applications

2657 Emerging LLM App Stack R2 Clean

This is a list of available tools, projects, and vendors at each layer of the LLM app stack.

Our original article included only the most popular options, based on user interviews. This repo is meant to be more comprehensive, covering all available options in each category. We probably still missed some important projects, so please open a PR if you see anything missing.

We also included Perplexity and Cursor.sh prompts to make searching and markdown table formatting easier.

Table of Contents

  1. Data Pipelines
  2. Embedding Models
  3. Vector Databases
  4. Playgrounds
  5. Orchestrators
  6. APIs / Plugins
  7. LLM Caches
  8. Logging / Monitoring / Eval
  9. Validators
  10. LLM APIs (proprietary)
  11. LLM APIs (open source)
  12. App Hosting Platforms
  13. Cloud Providers
  14. Opinionated Clouds

Project List

Data Pipelines

Name (site) Description Github Pip Installs
Databricks A unified data platform for building, deploying, and maintaining enterprise data solutions, including products (like MosaicML and MLflow) purpose-built for AI
Airflow A data pipeline framework to programmatically author, schedule, and monitor data pipelines and workflows, including for LLMs
Unstructured.io Open-source components for pre-processing documents such as PDFs, HTML, and Word documents for usage with LLM apps
Fivetran A platform that extracts, loads, and transforms data from various sources for analytics, AI, and operations N/A
Airbyte An open-source data integration engine that helps consolidate data in data warehouses, lakes, and databases
Anyscale An AI compute platform that allows developers to scale data ingest, preprocessing, embedding, and inference computations using Ray
Alluxio An open-source data platform at the intersection of compute and storage, bringing data closer to compute, to accelerate model training and serving, boost GPU utilization, and reduce costs for AI workloads GitHub Repo stars PyPI - Downloads

^ Back to Contents ^

Embedding Models

Name (site) Description Github Pip Installs
OpenAI Ada Embedding 2 OpenAI's most popular embedding model for capturing semantic relationships in text n/a
Cohere AI An independent commerical provider of LLMs, with particular focus on embeddings for semantic search, topic clustering, and vertical applications
Sentence Transformers An open-source Python framework for sentence, text, and image embeddings

^ Back to Contents ^

Vector Databases

Name (site) Description Github Pip Installs
Pinecone A managed, cloud-native vector database with a simple API for high-performance AI applications n/a
Weaviate An open-source vector database that stores both objects and vectors
ChromaDB An AI-native, open-source embedding database platform for developers
Pgvector An open-source vector similarity search for Postgres, allowing for exact and approximate nearest-neighbor search
Zilliz (Milvus) An open-source vector database, built for developing and maintaining AI applications
Qdrant A vector database and vector similarity search engine
Metal io A managed service for developers to build applications with ML embeddings N/A
LanceDB A serverless vector database for AI applications

^ Back to Contents ^

Playgrounds

Name (site) Description Github Pip Installs
OpenAI Playground A web-based platform for experimenting with various machine-learning models developed by OpenAI N/A N/A
nat.dev A platform that allows users to test prompts with multiple language models and compare their performance
Humanloop A platform that helps developers build applications on top of LLMs
Parea AI Platform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground.

^ Back to Contents ^

Orchestrators

Name (site) Description Github Pip Installs
Langchain An open-source library that gives developers the tools to build applications powered by LLMs
LlamaIndex A data framework for LLM applications to ingest, structure, and access private or domain-specific data
Autogen A framework for automating and streamlining LLM workflows using customizable, conversable agents for complex AI applications
Microsoft Semantic Kernel A lightweight open-source orchestration SDK
Haystack LLM orchestration framework to build customizable, production-ready LLM applications
Vercel AI SDK An open-source library for developers to build streaming UIs in JavaScript and TypeScript (node/npm)
Vectara AI A search and discovery platform for AI conversations utilizing your own data N/A
ChatGPT An AI chatbot that uses natural language processing to create humanlike conversational dialogue N/A N/A

^ Back to Contents ^

APIs / Plugins

Name (site) Description Github Pip Installs
Serp API A real-time API to access Google search results, as well as handling proxies, solving captchas, and parsing structured data
Wolfram Alpha API A web-based API providing computational and presentation capabilities for integration into various applications N/A
Zapier API AI Plugin A plugin that allows you to connect 5,000+ apps and interact with them directly inside ChatGPT N/A N/A

^ Back to Contents ^

LLM Caches

Name (site) Description Github Pip Installs
Redis An in-memory data structure store used as a database, cache, message broker, and streaming engine
SQLite A self-contained, serverless, zero-configuration, transactional SQL database engine
GPTCache An open-source tool for improving the efficiency and speed of GPT-based applications by implementing a cache to store the responses N/A

^ Back to Contents ^

Logging / Monitoring / Eval

Name (site) Description Github Pip Installs
Braintrust Data An AI product stack featuring evaluations, prompt playgrounds, continuous integration, dataset management, and access to various AI models through a single API
Arize AI An observability platform for both LLMs and supervised ML
Weights & Biases An MLOps platform for streamlining ML workflows
MLflow A platform to streamline ML development
PromptLayer A platform for tracking, managing, and sharing LLM prompt engineering
Helicone An open-source observability platform for LLMs
Quotient AI Quotient AI is a platform for evaluating AI products on real-world use-cases, during research, development, and in production N/A N/A
Portkey AI A platform to develop, launch, maintain, and iterate generative AI apps and features N/A N/A
Freeplay AI A platform to prototype, test, and optimize LLM features for customers N/A N/A
Gentrace An API and SDKs for evaluating and observing generative data, with features like AI, heuristic, and human grading evaluations, as well as production data observation N/A
Patronus AI An automated evaluation and benchmarking platform for LLMs, providing tools for testing, scoring, and evaluating LLMs in real-world scenarios N/A N/A
Autoblocks AI A collaborative cloud-based workspace designed for rapid iteration on GenAI products, offering features like prompt management, observability, continuous evaluations, fine-tuning, prototyping, debugging, and scalable data ingestion & search, all in a provider-agnostic environment N/A
Context AI Tools for pre-launch LLM evaluations and post-launch analytics, with features such as testing, performance monitoring, user conversation analysis, and support for various models and libraries N/A
E2b dev Services to deploy, test, and monitor AI agents, including a sandbox with a secure, long-running cloud environment for various LLMs with features like internet access
Agentops Toolkit for evaluating and developing AI agents, providing tools for agent development, monitoring capabilities, and replay analytics
Zenoml AI evaluation platform that enables data visualization, model performance analysis, and the creation of interactive reports for various data types
Baserun Tools for model configuration, prompt playground, monitoring, and prototype workflow, as well as features for full visibility into LLM workflows and end-to-end testing
WhyLabs AI Observability platform for ML and GenAI including LLM monitoring, guardrails and security
Log10 AI-powered LLMOps platform that automatically optimizes prompts and models with built-in logging, debugging, metrics, feedback, evaluations and fine-tuning

promptfoo Open-source LLM eval framework with support for model/prompt/RAG eval, dataset generation, local models, and self-hosting. (node/npm)
Parea AI Platform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground.
Galileo Galileo is a platform for evaluation, fine-tuning and real-time observability, powered by high-accuracy hallucination guardrails. N/A N/A

^ Back to Contents ^

Validators

Name (site) Description Github Pip Installs
Guardrails AI An open-source Python package for specifying structure and type, validating, and correcting the outputs of LLMs
Rebuff An open-source framework designed to detect and protect against prompt injection attacks in LLM apps
Microsoft Guidance A guidance language for controlling LLMs, providing a syntax for architecting LLM workflows
LMQL An open-source programming language and platform for language model interaction
Outlines A tool for helping developers guide text generation to build robust interfaces with external systems and guarantee that outputs match a regex or JSON schema
LLM Guard An open-source, comprehensive tool designed to fortify the security of Large Language Models (LLMs).

^ Back to Contents ^

LLM APIs (proprietary)

Name (site) Description Github Pip Installs
OpenAI A company providing many leading LLMs, including the GPT-3.5 and GPT-4 families N/A
Anthropic The developer of Claude, an AI assistant based on Anthropic’s research N/A
Cohere AI An LLM vendor with particular focus on embeddings for semantic search, topic clustering, and vertical applications
LLM A CLI utility and Python library for interacting with Large Language Models, both via remote APIs and models that can be installed and run on your own machine.

^ Back to Contents ^

LLM APIs (open source)

Name (site) Description Github Pip Installs
Hugging Face A hub for open-source AI models and inference endpoints, including leading base LLMs and LoRAs/fine-tunes
Replicate An AI hosting platform and model inference hub that allows software developers to integrate AI models into their apps
Anyscale An AI API and compute platform that allows developers to scale inference, training, and embedding computations with any model using Ray
Ollama Get up and running with large language models locally
GPT4ALL An ecosystem of open-source on-edge large language models.

^ Back to Contents ^

App Hosting Platforms

Name (site) Description
Vercel A cloud platform designed for front-end engineers, built with first-class support for LLM apps
Netlify An enterprise cloud computing company that offers a development platform for web applications and dynamic websites
Steamship An SDK and hosting platform for AI agents and tools, both a package manager and package hosting service for AI
Streamlit An open-source Python library designed for creating and sharing custom web apps for ML and data science
Modal A platform that enables running distributed applications using the modal Python package

^ Back to Contents ^

Cloud Providers

Name (site) Description
Amazon Web Services A cloud computing platform, offering services from data centers globally
Google Cloud Platform A cloud computing platform, offering services from data centers globally
Microsoft Azure A cloud computing platform, offering services from data centers globally
CoreWeave A specialized cloud provider that delivers GPUs on top of flexible deployment infrastructure

^ Back to Contents ^

Opinionated Clouds

Name (site) Description
Databricks (MosaicML) Databricks acquired Mosaic ML in 2023, along with its tooling and platform for efficient pre-trainining, fine-tuning and inferencing LLMs
Anyscale An AI compute platform that enables developers to scale inference, training, and embedding computations with any model using Ray
Modal A platform that eables running distributed applications using the Modal Python package
Runpod A cloud computing platform designed for AI and ML applications
OctoML A compute service that allows users to run, tune, and scale generative models
Baseten A inference service that allows users to deploy, serve, and scale custom and open-source models
E2B Secure sandboxed cloud environments made for AI agents and AI apps

^ Back to Contents ^




Formatting Prompt Templates

We were able to partialy automate this - particularly finding Github and PyPI links - using this Perplexity search prompt. It worked roughly ~75% of the time and could handle ~3 projects at a time, pulling data from 20-30 sources in each iteration.

image

Once you have the data you would like to add, if you don't want deal with the markdown formatting here, it is easy to correctly format using a tool like Cursor.

See the prompt below that works as an inline edit, just make sure you highlight 4-5 previous examples so Cursor can infer the format itself:

image

^ Back to Contents ^