Tag: RAG

Posted in Artificial Intelligence

RAG Frameworks Compared in 2026: LangChain vs LlamaIndex vs Haystack

Posted on by Filip Keser

Retrieval-augmented generation (RAG) is the pattern that lets an LLM answer questions using your documents instead of (or in addition to) whatever it memorized during training: you retrieve relevant chunks from an index, stuff them into the prompt, and let the model generate a grounded answer. Three open-source frameworks dominate how developers actually build this in 2026: LangChain, LlamaIndex, and Haystack. They overlap heavily, but they were built with different centers of gravity — and that shows up in how painless each one makes a RAG pipeline versus a general agent. This is a RAG-specific comparison. If you’re choosing a framework for autonomous multi-step agents rather than document retrieval, see our separate comparison of LangGraph, CrewAI, and AutoGen. And if you want the mechanics of how the vector search underneath any of these three actually works, we cover that in our explainer on vector databases and ANN search.

At a Glance

FrameworkLicenseOpen SourceBest For
LangChainMITYesGeneral-purpose chain and agent orchestration, with RAG as one capability among many; best if you’ll also need tool-calling agents, not just retrieval
LlamaIndexMITYesData-centric, retrieval-first applications where ingesting, indexing, and querying your own documents is the primary job
HaystackApache 2.0YesProduction NLP/RAG pipelines built by deepset, with an emphasis on modular, serializable pipelines for deployment at scale

Short Verdict

All three are solid, genuinely open-source choices, and the “right” one depends on what your RAG pipeline needs to do beyond retrieval. LlamaIndex is the most retrieval-native of the three — its APIs are built around getting documents in and answers out, with layered high-level and low-level interfaces for customizing the retrieval step. LangChain is the most general-purpose: it treats RAG as one tool an agent can reach for, built on top of LangGraph for orchestration, which makes sense if your end goal is an agent that sometimes retrieves and sometimes does other things. Haystack sits closest to a production-engineering mindset, with serializable, modular pipelines, Kubernetes support, and hybrid retrieval with self-correction loops baked in — a good fit if you’re deploying a RAG system as a long-lived service rather than prototyping an app.

What Each One Is Optimized For

LangChain: Agent and Chain Orchestration First, RAG Included

LangChain’s own documentation frames the framework around `create_agent`, described as “a minimal, highly configurable harness” for composing a model, tools, a prompt, and middleware. Agents built this way run on LangGraph underneath, which provides durable execution, human-in-the-loop support, and persistence. RAG in LangChain is implemented as retrievers and vector store integrations that an agent (or a simpler chain) can call as a tool, rather than being the framework’s sole purpose. That said, LangChain’s retriever and vector-store abstractions are extensive: its current integrations list includes Pinecone, Chroma, Qdrant, Weaviate, Milvus, PGVector, Redis, Elasticsearch, Astra DB, Azure Cosmos DB, Google AlloyDB, FAISS, Neo4j, and dozens more, which makes swapping the underlying store a low-friction decision.

LlamaIndex: Data Indexing and Retrieval First

LlamaIndex describes itself as a framework for building LLM-powered agents and applications “over your own data,” and its documentation is explicit that the framework is built around getting external data into a model at answer time — RAG is its best-known pattern, though the docs now present it as one tool an agent can call rather than the only thing the framework does. Its core primitives are data connectors, indexes, vector stores, and query/chat engines, offered through both a high-level API (documents to answers in a few lines) and a lower-level API for customizing or replacing individual components. For messy source documents, LlamaIndex’s companion product LlamaParse handles complex parsing — scanned PDFs, tables, embedded charts — and feeds parsed output straight into an index. Its documented vector store integrations include Pinecone, Qdrant, Chroma, Weaviate, Milvus, Elasticsearch, and multiple pgvector-based options.

Haystack: Production Pipelines by deepset

Haystack is maintained by deepset GmbH and is described on its own site as “Open Source AI framework for building production-ready, LLM-powered agents and applications,” built around modular components, serializable pipelines, and Kubernetes support for production deployment. Its RAG capabilities are explicitly called out as “Advanced RAG,” including hybrid retrieval and self-correction loops, alongside standardized tool calling for agents and support for multimodal tasks like image processing and audio transcription. Haystack’s document-store integration list is the broadest of the three by raw count: dedicated vector databases (Pinecone, Qdrant, Weaviate, Chroma, Milvus, LanceDB, FAISS, Vespa, Upstash Vector, Amazon S3 Vectors), relational stores with vector support (pgvector, Supabase, AlloyDB, MariaDB, Oracle AI Vector Search, SingleStore), search engines (Elasticsearch, OpenSearch, Solr, Azure AI Search), and several cloud/NoSQL databases (AstraDB, DynamoDB, Azure CosmosDB, MongoDB, Neo4j, and others).

Licensing and Paid Hosted Offerings

FrameworkCore LicenseGitHub Stars (approx.)Paid Hosted Offering
LangChainMIT~149k (langchain-ai/langchain)LangSmith (observability/evals, free dev tier then $39/seat/mo+) and LangGraph Platform (agent deployment, bundled into LangSmith’s Plus/Enterprise plans)
LlamaIndexMITWidely used; exact star count not independently verified hereLlamaCloud (LlamaParse document parsing/extraction plus a hosted indexing pipeline), free tier of 10,000 credits/month (roughly 1,000 pages), paid tiers beyond that
HaystackApache 2.0~27k (deepset-ai/haystack)Haystack Enterprise Platform (formerly marketed as deepset Cloud) — cloud or self-hosted deployment, governance/RBAC/audit logging, visual pipeline design, and Forward Deployed Engineer support; also a lighter “Haystack Enterprise Starter” tier

All three frameworks themselves are free and unrestricted open source — none of them gate core RAG functionality behind a license key the way some observability tools do. The commercial products are additive layers: managed deployment, document processing at scale, governance, and support, not a different version of the open-source library. Worth noting precisely: we could not independently verify LlamaIndex’s exact current GitHub star count from its own site in this research pass, so that cell is left general rather than guessed.

Pros and Cons

LangChain

  • Enormous ecosystem and integration surface — dozens of vector stores, model providers, and tool integrations
  • Natural fit if your RAG app will grow into a multi-step agent that does more than retrieve-and-answer
  • LangGraph underneath gives durable execution and human-in-the-loop for free
  • More moving parts than you need if all you want is straightforward document retrieval
  • Abstractions have changed significantly across versions, which can mean migration friction on older tutorials and codebases
  • RAG-specific documentation is spread across a broader agent-first framework rather than presented as a first-class use case

LlamaIndex

  • Purpose-built abstractions for indexing and querying data, with a genuinely simple high-level API for basic RAG
  • LlamaParse handles messy real-world documents (scans, tables, charts) better than generic text loaders
  • Layered API means you can start simple and drop down to custom retrieval logic without switching frameworks
  • Less natural than LangChain if your end goal is a complex multi-tool agent rather than a data-grounded Q&A system
  • LlamaCloud’s free tier (10,000 credits/month, ~1,000 pages) is modest for large document sets
  • Smaller model-provider integration surface than LangChain in some areas, though vector store coverage is comparable

Haystack

  • Apache 2.0 licensing is unambiguous and permissive, with no separate enterprise-edition carve-out inside the open-source repo
  • Broadest documented list of document-store/vector-database integrations among the three
  • Built with production deployment (Kubernetes, serializable pipelines, logging/monitoring) as a first-class concern, not an afterthought
  • Smaller community and ecosystem than LangChain, meaning fewer third-party tutorials and templates
  • Pipeline-based mental model has a steeper initial learning curve than LlamaIndex’s “few lines to a working RAG app” approach
  • The enterprise platform rebrand (deepset Cloud → Haystack Enterprise Platform) means older articles referencing “deepset Cloud” may describe outdated branding

Who It’s a Good Fit For

LangChain fits teams building an agent that needs retrieval as one capability among several (web search, code execution, other tool calls), or teams already invested in the LangChain/LangGraph/LangSmith ecosystem who want RAG without adopting a second framework.

LlamaIndex fits teams whose primary problem is “how do we get an LLM to answer questions accurately from our documents,” especially when those documents are messy (PDFs, tables, scanned files) and parsing quality matters as much as the retrieval algorithm.

Haystack fits teams, especially ones with existing NLP/MLOps practices, who are deploying a RAG pipeline as a durable production service and want modular, inspectable, Kubernetes-friendly pipelines plus an enterprise support path if they need it later.

Who Should Look Elsewhere

If you need a fully autonomous multi-agent system with complex role delegation rather than retrieval-grounded Q&A, none of these three is the primary tool for the job — see our LangGraph vs. CrewAI vs. AutoGen comparison instead, since RAG frameworks assume retrieval is central and agent orchestration is secondary (or absent). If your team has no NLP/ML engineering experience and just wants a no-code RAG chatbot over a handful of PDFs, a hosted no-code platform may get you there faster than adopting any of these three open-source libraries directly.

Code Example: Minimal RAG Query Engine in LlamaIndex

LlamaIndex’s high-level API gets you from a folder of documents to a queryable RAG pipeline in a handful of lines:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Load documents from a local folder
documents = SimpleDirectoryReader("./data").load_data()

# Build an in-memory vector index over the documents
index = VectorStoreIndex.from_documents(documents)

# Create a query engine and ask a question grounded in your data
query_engine = index.as_query_engine()
response = query_engine.query("What does the Q3 report say about churn?")
print(response)

The Bottom Line

LangChain, LlamaIndex, and Haystack are all permissively licensed, actively maintained, and genuinely capable of building a solid RAG pipeline — the differences are about where each one puts its center of gravity. LangChain is agent-first with RAG as a tool. LlamaIndex is data-and-retrieval-first with agents as an extension. Haystack is production-pipeline-first, built by an NLP company that cares about what happens after your RAG system ships. None of them require a commercial license to use fully; the paid products (LangSmith/LangGraph Platform, LlamaCloud, Haystack Enterprise Platform) all sit on top as optional managed services, not gated versions of the open-source core.

FAQ

Are LangChain, LlamaIndex, and Haystack all free to use commercially?

Yes. LangChain and LlamaIndex are both MIT licensed, and Haystack is Apache 2.0 licensed — all three are permissive open-source licenses that allow commercial use without royalties or a required license purchase. The paid products each vendor sells (LangSmith, LlamaCloud, Haystack Enterprise Platform) are optional hosted/managed add-ons, not different licensing terms on the core framework.

Can I use LangChain and LlamaIndex together?

Yes, and plenty of teams do — for example, using LlamaIndex for document parsing and indexing, then handing retrieved context into a LangChain/LangGraph agent for the rest of the orchestration. They are not mutually exclusive, since both expose standard retriever interfaces.

Which framework is easiest for a first RAG prototype?

Based on each framework’s own documentation, LlamaIndex’s high-level API is explicitly designed to get from documents to answers “in a few lines,” which makes it the fastest on-ramp for a first prototype. LangChain and Haystack both require slightly more setup (chains/agents in LangChain, pipeline definitions in Haystack) before you get a working retrieval loop.

Is deepset Cloud still a thing, or has it been replaced?

As of this research, deepset’s own site markets its commercial hosted/managed offering as the “Haystack Enterprise Platform” (with a lighter “Haystack Enterprise Starter” tier), rather than using the “deepset Cloud” name. If you see “deepset Cloud” referenced elsewhere, treat it as the predecessor branding for what is now sold as the Haystack Enterprise Platform.

Do any of these frameworks require a specific vector database?

No. All three are vector-database agnostic and document dozens of supported stores — Pinecone, Chroma, Qdrant, Weaviate, Milvus, pgvector-based options, Elasticsearch, and more appear across all three frameworks’ integration lists. For background on how the underlying search algorithms (like HNSW and IVF) actually work, see our vector databases and ANN search explainer.

Sources

Posted in Engineering

Vector Databases and Approximate Nearest Neighbor Search Explained: HNSW vs IVF for AI Retrieval

Posted on by Filip Keser

Every retrieval-augmented generation (RAG) pipeline, semantic search box, and recommendation engine built since 2023 depends on the same underlying problem: given a query vector, find the handful of vectors in a database of millions (or billions) that are closest to it. Doing this exactly, by comparing the query to every stored vector, is called k-nearest neighbor (kNN) search, and it does not scale — brute-force comparison is O(N) per query. So almost every real vector database instead does approximate nearest neighbor (ANN) search: trading a small, tunable amount of recall for orders-of-magnitude faster lookups. Two families of ANN algorithm dominate production systems in 2026: graph-based methods, led by HNSW, and partition-based methods, led by IVF. This piece explains how each actually works, what their real trade-offs are, and when to reach for which.

At a Glance

FactorHNSW (graph-based)IVF (partition-based)
Core ideaMulti-layer navigable graph; search by walking edges toward the queryCluster vectors into cells with k-means; search only the nearest cells
Key build parametersM (connections per node), efConstructionnlist (number of cells)
Key query parameterefSearchnprobe (cells searched)
Query speedVery fast, roughly logarithmic in dataset sizeFast, scales with nprobe/nlist fraction scanned
Memory overheadHigher — stores the full graph on top of vectors (~M × 2 × 4 extra bytes/vector)Lower — mainly list IDs plus chosen vector encoding
Build timeSlower to build, especially at high M/efConstructionFaster to build; k-means training step is comparatively cheap
Handles deletions well?No — removing nodes breaks the graph structureYes — easier to add/remove vectors from a cell
Typical usersQdrant, Weaviate, Milvus (default), pgvector (HNSW index)FAISS, Milvus (IVF variants), many large-scale offline pipelines

Why Exact Search Doesn’t Scale

An embedding model turns text, images, or audio into a dense vector, typically a few hundred to a few thousand floating-point numbers, positioned so that semantically similar inputs land close together under a distance metric like cosine similarity or Euclidean distance. Finding the true nearest neighbors to a query vector means computing that distance against every vector in the collection. At a few thousand vectors this is trivial. At tens of millions, it means billions of floating-point operations per query — workable for a batch job, unworkable for an interactive RAG endpoint that needs to return results in tens of milliseconds. ANN algorithms accept a small, measurable chance of missing a true nearest neighbor in exchange for search that scales sub-linearly.

How HNSW Works

HNSW (Hierarchical Navigable Small World) was introduced by Yu. A. Malkov and D. A. Yashunin in their 2016 paper “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs,” building on earlier skip-list and small-world-graph ideas. Conceptually, HNSW organizes vectors into several stacked graph layers, like a multi-level transit map:

  • The top layer is sparse, with a small number of nodes connected over long links — good for covering large distances quickly.
  • Each layer below is progressively denser, with more nodes and shorter, more local connections.
  • The bottom layer contains every vector in the collection, densely interconnected.

A search starts at an entry point in the top layer and greedily walks toward the query vector, hopping to whichever neighbor is closer at each step. When no neighbor in the current layer is closer, the search drops down a layer and continues, refining its position until it bottoms out with a set of nearby candidates. This is similar to zooming from a country-level map down to a street-level map rather than scanning every street in the country directly.

Three parameters control the trade-off, as documented by vector database vendors such as Qdrant:

  • M — the maximum number of connections per node (commonly 8–64; Qdrant defaults to 16). Higher M improves recall but increases memory and build time.
  • efConstruction — how many candidate neighbors are evaluated when inserting a new vector into the graph (commonly 100–500). Higher values build a better-connected graph at the cost of slower indexing.
  • efSearch (sometimes called hnsw_ef) — how many candidates are evaluated per query at search time (commonly 50–200+). This is the main dial for trading query latency against recall, and unlike the other two, it can be changed per query without rebuilding the index.

HNSW’s practical downside is memory and mutability. The graph’s edges sit on top of the raw (or quantized) vectors, adding roughly M × 2 × 4 bytes per vector according to FAISS’s own documentation, and removing a vector cleanly is not supported because deleting a node can disconnect the graph around it — most implementations instead mark vectors as deleted and periodically rebuild.

How IVF Works

IVF (Inverted File index), as implemented in Meta’s FAISS library, takes a completely different approach: partition first, then search only the relevant partitions. FAISS’s own wiki documentation describes it as a “cell-probe method”:

  • During index training, a coarse quantizer (usually k-means) splits the vector space into nlist cells, each represented by a centroid.
  • Every vector in the collection is assigned to exactly one cell — whichever centroid it’s closest to — and stored in that cell’s inverted list.
  • At query time, the query vector is compared only against the nlist centroids to find the nprobe closest cells, then compared against the actual vectors inside just those cells.

FAISS’s documentation gives a rule of thumb for sizing nlist at roughly C × sqrt(n) for a dataset of n vectors. The fraction of the database actually scanned per query is approximately nprobe / nlist (in practice usually a bit more, since cells are rarely perfectly balanced). Raising nprobe improves recall by checking more cells, at a roughly linear cost in query time. Recall suffers specifically when a query’s true nearest neighbor happens to sit in a cell that wasn’t among the nprobe probed — a structural risk near cell boundaries that has no direct HNSW equivalent.

IVF is commonly paired with vector compression. IVFFlat stores full-precision vectors (about 4d + 8 bytes per vector, where d is dimensionality); IVFPQ (product quantization) compresses vectors much further at some recall cost, making it attractive for billion-scale, memory-constrained deployments. Because cells are just lists, adding and removing vectors is straightforward compared to HNSW’s graph surgery problem.

A Minimal Illustration of the Search Logic

The pseudocode below is a simplified sketch of the two query paths — not production code — meant only to show where each algorithm spends its effort.

function ivf_search(query, nprobe):
    centroids = get_cell_centroids()
    nearest_cells = top_k(centroids, query, k=nprobe)
    candidates = []
    for cell in nearest_cells:
        candidates.extend(cell.vectors)
    return top_k(candidates, query, k=final_k)

function hnsw_search(query, ef_search):
    layer = top_layer
    current = entry_point
    while layer > 0:
        current = greedy_walk(layer, current, query)
        layer = layer - 1
    candidates = explore_neighbors(current, query, ef=ef_search)
    return top_k(candidates, query, k=final_k)

HNSW vs IVF: When to Use Which

Neither structure is strictly better; they sit at different points on the recall/speed/memory curve, and the right choice depends on dataset size, update frequency, and hardware budget.

  • Choose HNSW when query latency matters most, the dataset fits comfortably in memory, recall needs to stay high even at low latency, and updates are frequent but deletions are rare (most RAG and real-time semantic search use cases). It’s the default or primary index type in Qdrant, Weaviate, and pgvector’s HNSW mode.
  • Choose IVF (often IVF+PQ) when the collection is very large (hundreds of millions to billions of vectors), memory is tight, and some recall can be traded for dramatically smaller index size. It’s also easier to shard and rebuild incrementally, which is why it shows up heavily in FAISS-based offline and batch retrieval pipelines.
  • Hybrid approaches exist: FAISS supports using an HNSW graph as the coarse quantizer inside an IVF index (IVF_HNSW), combining fast coarse routing with inverted-list storage.

In practice, most teams don’t hand-pick an algorithm from first principles — they pick a vector database (Qdrant, Weaviate, Pinecone, Milvus, or Postgres with pgvector) and then tune its exposed parameters. Pinecone’s newer serverless indexes go a step further and abstract the index type away entirely, separating storage from query compute and exposing on-demand read capacity rather than a pod size the user has to choose. That’s a trend worth watching: the HNSW-vs-IVF decision is increasingly made by the database vendor, not the application developer, with the practitioner left tuning recall targets and cost rather than graph parameters directly.

Trade-offs Summary

  • Pros of HNSW: excellent recall-to-latency ratio, logarithmic-ish query scaling, tunable at query time via efSearch without rebuilding.
  • Cons of HNSW: larger memory footprint, slower builds at high quality settings, poor support for deletions.
  • Pros of IVF: lower memory overhead (especially with PQ), faster builds, straightforward updates and deletions, scales well to very large, sharded datasets.
  • Cons of IVF: recall is more sensitive to data distribution and cell boundaries; needs careful nlist/nprobe tuning and periodic retraining of centroids as data shifts.

The Bottom Line

HNSW and IVF solve the same problem — fast approximate retrieval over dense vectors — with opposite structural bets: a navigable graph versus a partitioned index. For most RAG and application-facing semantic search workloads where the data fits in memory and queries need to be fast and fresh, HNSW’s recall-per-millisecond is hard to beat, which is why it has become the default in most purpose-built vector databases. For very large-scale, storage-constrained, or heavily batch-oriented retrieval, IVF (particularly IVF+PQ) remains the more economical choice. The practical skill isn’t memorizing one algorithm — it’s knowing which knobs (efSearch, nprobe, M, nlist) to turn when recall, latency, or memory becomes the bottleneck in production.

FAQ

Is ANN search ever exact?
No — by definition, approximate nearest neighbor search can miss some true nearest neighbors. Both HNSW and IVF expose parameters (efSearch, nprobe) that push recall closer to exact search as you increase them, at the cost of query speed.

Can I combine HNSW and IVF?
Yes. FAISS supports using an HNSW graph as the coarse quantizer inside an IVF index, and several systems layer quantization (scalar or product quantization) on top of either structure to shrink memory further.

Which one do most vector databases use by default?
Most purpose-built vector databases aimed at application developers — Qdrant, Weaviate, Milvus’s default configuration, pgvector’s HNSW index type — default to HNSW because of its strong recall-to-latency ratio on in-memory datasets. IVF (and IVF+PQ specifically) is more common in large-scale, cost-sensitive, or research/offline retrieval pipelines built directly on FAISS.

Does deleting vectors work the same way in both?
No. FAISS’s own documentation notes that HNSW does not support true vector removal because deleting a node can disconnect the graph; most systems instead tombstone and periodically rebuild. IVF’s list-based structure makes removal and insertion more straightforward.

How does this relate to RAG?
A RAG pipeline embeds a user’s query, runs ANN search (HNSW or IVF, depending on the vector store) against a corpus of pre-embedded chunks, and feeds the top results into an LLM’s context window as retrieved evidence. The ANN index’s recall directly caps how good the retrieved context — and therefore the generated answer — can be.

Sources

Related reading on this site: B-Trees vs LSM-Trees: How Databases Choose Their Storage Engine and Oracle Database 23ai and the Oracle–Nvidia Partnership, which covers vector search built directly into a relational database engine.