How RAG Works, Visually

The mental model behind chunks, retrieval, and grounded answers

Pranjal Rawat

The problem, and the fix

Ask a plain language model a question about documents it has never seen, and it will still answer. Confidently, and often wrong.

Retrieval-augmented generation is the fix: look the answer up before writing it.

This deck shows the mental model, not a library’s API. By the end you should be able to picture what happens, step by step, when a RAG system and an agent built on top of it answer a question over a real corpus. The running example throughout is a fictional company, Aurora Cloud, with 48 real documents: HR policy, product docs, engineering runbooks, and some deliberately unrelated pages. Every figure in this deck is a real run against that real corpus - nothing is a mockup.

Ask it something it has never seen

A plain model has no way to check itself against real documents. It fills the gap with whatever is most probable, not what is true.

Same question, three different answers, and no way to tell which one is real.

This is the hallucination problem in its plainest form: no grounding means no way to adjudicate between guesses. Note the number changes each time - that’s a deliberate callback to the near-duplicate PTO docs (hr-01 at 15 days, hr-02 at 20) that show up later in the corpus.

Meet Aurora Cloud

Aurora Cloud is the fictional company behind every example in this deck: 48 documents across four topics.

This is the corpus every later slide in the deck runs against, for real: chunking, embedding, retrieval, reranking, evaluation. Nothing here is a mockup - it’s the same 48-document object every topic’s run.py imports from corpus.py.

What retrieval changes

Retrieval-augmented generation adds one step in front of writing: look before you answer.

The answer is no longer memory. It is a copy of what the corpus actually says.

Search, cite, write - three steps, each of which later topics in the deck open up in detail (retrieval, reranking, prompt assembly). For now this is just the shape of the fix.

The rule: cite or refuse

Grounding is a contract, not a suggestion. A covered question gets a citation. An uncovered one gets a decline, never a guess.

A refusal is a correct answer too, when the corpus genuinely doesn’t cover it.

This is the cite-or-refuse gate the rest of the deck builds toward: a similarity threshold decides which branch a question takes. Topic 10 (generation) is where this becomes a concrete number.

Cite-or-refuse, the contract behind every answer this deck builds toward: a claim traces to a retrieved chunk and names it with [n], or the system declines instead of guessing.

Retrieval turns a prompt into evidence; generation must cite it or refuse. Every concept from here on is one piece of making that contract real.

The Problem, and the Fix

The Problem, and the Fix

The Problem, and the Fix

The Problem, and the Fix

The Problem, and the Fix

Two clocks, one system

Building the index and answering a question are two different jobs, on two different clocks.

One runs once, ahead of time. The other runs every time someone types a question.

Two different jobs, two different clocks: the offline build (chunk, embed, index) happens once; the online loop (retrieve, augment, generate) reruns for every question.

This is the map for the whole deck. Everything that follows - chunking, embeddings, hybrid search, reranking, prompts, refusals - is a closeup of one box in this picture. Keep this shape in mind and the details never get confusing about when they happen.

The corpus becomes a frozen index

The raw documents get chunked and turned into vectors a single time. After that, the index just sits there, ready to be searched.

Notice there is no question anywhere in this picture. The corpus doesn’t know what anyone is going to ask. It just gets chunked and indexed, once, and then it waits.

Every question walks the same four steps

A question comes in, gets turned into a search, and the results get folded into a prompt before the model ever runs.

Retrieve, augment, generate: this is the whole online loop. No matter how the retrieval or the prompt gets fancier later in the deck, it still runs inside these four boxes.

Same shape, very different rhythm

Put both loops side by side and the contrast is the whole point: one column runs once, the other runs again for every question that comes in.

The left column happens once and then it’s done. The right column is the loop - it fires again for every question, always starting cold from the same frozen index. That’s why it’s called the naive RAG loop: nothing carries over between questions except the index itself.

This is the whole map

Everything from here on zooms into one box from this picture: how retrieval actually scores a match, how the prompt gets assembled, how the model is kept from making things up.

Worth restating before diving into details: the split between “build the index” and “answer a question” is the single most useful fact to hold onto in this whole deck. Get that right and every later concept has an obvious home.

The whole pipeline, five stages

ingest

raw documents: HR, product, engineering, off-topic

index

chunk, embed, and fit the vectors into one structure

retrieve

score every chunk against the query, keep the top-k

rerank

score just those candidates again, more carefully

generate

cite the evidence, or refuse

The Naive RAG Loop

The Naive RAG Loop

The Naive RAG Loop

The Naive RAG Loop

The Naive RAG Loop

One document holds many facts

A document mixes several separate facts into one block of text. A question only needs one of them back, not the whole page.

This is the motivating problem for chunking. Indexed whole, this document returns three unrelated facts for any query that touches it. Chunked, a query about accrual only pulls the accrual sentence. This exact document (hr-01, “Paid Time Off Policy v1”) is the one the walkthrough later splits for real.

Chunk, the unit retrieval actually indexes and hands back: one self-contained slice of a document, small enough that a question only pulls the fact it needs.

Equal, non-overlapping cuts can break an idea

A window of fixed size slides across the words. The simplest version just cuts, edge to edge, with nothing shared.

This is the naive version of chunking: slide the window by its full size, no sharing. It works until an idea happens to straddle the cut - then half of it ends up in chunk 1, half in chunk 2, and neither chunk contains the whole thing.

A fact that straddles a chunk boundary survives whole in neither chunk. Fixed-size, non-overlapping cuts guarantee this happens somewhere in a large corpus, overlap (the next slide) is the guard.

Overlap keeps the idea whole

Shift the window by less than its full size, and consecutive chunks share a few words at the seam.

Same window, shifted by less than its width. The two chunks now share a band of words. Whatever idea lands in that band is complete in chunk 1 and complete again in chunk 2 - overlap is just insurance against the cut in the previous slide. The walkthrough proves this against real words: literally chunk_i.text.split()[-5:] == chunk_{i+1}.text.split()[:5].

Size is a tradeoff

A bigger window swallows unrelated ideas into one chunk. A smaller window slices a single idea into fragments that lose their surrounding context.

Both failure modes are real. Too big, and a single chunk answers for three different questions, diluting what a retriever matches against. Too small, and a chunk stops being a coherent unit at all - just a shard of a sentence. Aurora Cloud’s pipeline uses size=40 with overlap=10 as its working default; the walkthrough shows what a much smaller window does instead.

Most of Aurora Cloud’s documents need no splitting at all

Under the window this pipeline actually uses, nearly every one of the 48 documents collapses to a single chunk. Shrink the window, and one of them finally splits for real.

Every square here is one of Aurora Cloud’s real documents. Under the pipeline’s actual default (size=40, overlap=10), all 48 stay whole - confirmed empirically, not assumed. The flipbook that follows shrinks the window to size=15, finds a document that really does split into multiple chunks, and checks the overlapping words against each other directly.

Chunk size is a tradeoff, not a default: too big dilutes a chunk across unrelated facts, too small shatters one fact into pieces with no context. Overlap fixes the second failure, not the first.

Chunking the Corpus

Chunking the Corpus

Chunking the Corpus

Chunking the Corpus

Embeddings are geometry

A chunk of text turns into a list of numbers. That list is a location - somewhere you could point to on a map.

Once meaning has a location, comparing meanings becomes measuring distance.

Embedding, the list of numbers a chunk (or a query) turns into: a location in a shared space where distance stands in for how related two pieces of text are.

This is the idea that makes every later concept (retrieval, hybrid search, reranking) possible. Before this, a “search” is a distance calculation.

A chunk becomes a point in space

Any vectorizer works here: TF-IDF weighted word counts, or a neural embedding model. The pipeline in this deck uses TF-IDF (real, computed, not staged) because it is transparent - every coordinate traces back to actual words in the chunk.

Hundreds of numbers, two you can draw

This is the honest caveat: the pictures in this deck are a 2D projection (TruncatedSVD on the real TF-IDF matrix), not the literal embedding. A projection can’t preserve every distance perfectly, but it preserves enough of the structure that near-in-2D reliably means near-in-reality here.

Related chunks land close together

Three HR questions huddle together because they share vocabulary and topic. A kitchen-knife question shares almost none of that vocabulary, so it lands somewhere else entirely. This is the whole trick, drawn as plainly as possible.

Search by meaning is search by distance

The query gets vectorized exactly like every chunk, then dropped onto the same map. Whichever dot it lands nearest is the answer.

This reframe is the payoff of the whole concept: “search by meaning” was always going to need a precise mechanical definition, and the definition turns out to be nearest-neighbor distance on this map. Retrieval (the next concept) is exactly this, run at real scale with a real distance metric (cosine similarity).

The next screens are the real thing

Not a mockup: the real 48-document Aurora Cloud corpus, chunked and turned into real TF-IDF vectors, projected to 2 axes. Each dot is a real chunk, colored by its document’s tag. The star is a real query, placed by the same vectorizer, at its real projected position.

Once meaning has a location, “search by meaning” has a precise mechanical definition: nearest-neighbor distance on that map. Retrieval is exactly this, run at real scale.

Embeddings Are Geometry

Embeddings Are Geometry

Embeddings Are Geometry

Embeddings Are Geometry

Building the Index

Every chunk now has a vector. Those vectors don’t sit around loose - they get packed together into one structure: the index. Building it happens once. Answering a question against it never does.

Vector index, every chunk’s embedding stored together in one fitted structure. Built once from the whole corpus; every later question is a read against it, never a rebuild.

This is the hinge between “embeddings as geometry” and “retrieval”: once each of the 48 chunks is a point in vector space, those points get stored together as a single fitted object - the TF-IDF matrix plus the vectorizer that produced it. Fitting that object is a one-time cost. Every question that follows, no matter how different, is just a lookup against the same object.

Vectors get stored together

48 separate vectors. One structure holds all of them.

Under the hood this is literally one VectorIndex object: a fitted TfidfVectorizer plus the matrix of every chunk’s vector, together. There is no per-chunk index - there is one index that happens to contain 48 rows.

Fit once, then query without limit

The fit is a single event. Every search after it is free to repeat.

This is the piece people miss: fitting (building the vocabulary, the matrix, the whole structure) is asymptotically irrelevant once it has happened, because it happens exactly once. Search cost is what recurs, and search is cheap.

Many questions, one index

Three unrelated questions. One already-built index answers all of them.

The three questions are deliberately unrelated: an HR question, an engineering question, and one the corpus barely covers. All three hit .search() on the exact same fitted object built at the top of this section.

The index never rebuilds

The wrong mental model refits per question. The right one fits once.

The failure mode to rule out: treating “build the index” as a step inside the query loop. It isn’t. It runs once, before the first question ever arrives, and every subsequent question is a pure read against it.

Fitting the index is a one-time cost; search is what recurs, and search is cheap. Treating “build the index” as a per-question step is the mistake to rule out.

Building the Index

Building the Index

Building the Index

Building the Index

Retrieval: The Nearest Points Win

The index holds 48 points. Retrieval is the one operation that matters: drop the query onto the same map, and read off whichever points sit closest.

Nothing about retrieval is mysterious once the geometry is in place. It’s a distance calculation, run once per question.

Retrieval, scoring every chunk in the index against the query’s vector and keeping the top few. One distance calculation, run once per question, before generation ever sees the corpus.

This is the payoff of “embeddings are geometry” and “building the index.” Both of those concepts existed to make this one moment possible: a question arrives, and answering it becomes “which points are nearest,” not “which document sounds right.”

From a question to a ranked list

The query is embedded through the exact same vectorizer that embedded every chunk - that’s what makes the comparison meaningful. Cosine similarity scores all 48 chunks at once; sorting and slicing to the top-k is the only step left.

Nearest wins, and “nearest” is a number

There’s no threshold logic inside retrieval itself - it always returns its top-k, confidently, whether the best score is 0.48 or 0.02. Deciding what to do with a weak top score is generation’s job (cite or refuse), a few concepts from now.

The next screens are the real thing

Not a mockup: the real 48-document Aurora Cloud corpus, the real fitted index from the last concept, one real query run through .search(). The ranked list, the scores, and the map are all computed live.

Retrieval always returns its top-k, confidently, whether the best score is 0.48 or 0.02. Deciding what a weak score means is generation’s job, not retrieval’s.

Watch the same query land in the embedding space right where the top-ranked hr documents sit, and watch a second, uncovered question come back with uniformly low scores - retrieval still answers, it just isn’t confident about any of it.

Retrieval: The Nearest Points Win

Retrieval: The Nearest Points Win

Retrieval: The Nearest Points Win

Retrieval: The Nearest Points Win

Retrieval: The Nearest Points Win

One ranking isn’t enough

Dense search ranks by meaning. Sparse search ranks by exact words. Run the same query through both and they don’t agree - fusing the two beats trusting either alone.

Three notions of “relevant”

lexical

BM25 / TF-IDF, rewards exact term overlap, blind to paraphrase

semantic

dense cosine similarity, rewards topical closeness, blind to rare exact terms

hybrid

reciprocal rank fusion, combines both rankings by position, not raw score

Dense vs. sparse: two different notions of “relevant”

Dense turns the query into a vector and measures how close it lands to every doc’s vector - a graded, continuous notion of “about the same thing.” Sparse never leaves the words: it rewards a document for containing the query’s rare terms, “escalation” and “on-call” here, whether or not the rest of the document is topically similar.

Same query, two different rankings

Nobody changed the corpus or the query between these two lists. The only thing that changed is which notion of “relevant” is doing the ranking, and that alone reorders the results. Neither list is wrong; they’re answering slightly different questions.

Fusion: combine by rank, not raw score

Reciprocal rank fusion never looks at the raw cosine similarity or the raw BM25 score, only at where each chunk landed in each list. That sidesteps a real problem: the two scores live on incomparable scales, so averaging them directly would let whichever scale happens to be bigger quietly dominate. A chunk that’s decent in both rankings can out-rank a chunk that’s great in one and absent from the other.

Why fusion is more robust than either alone

Neither method is strictly better; each has a blind spot the other one covers. Dense misses a rare product name it’s never seen phrased that way; sparse misses a paraphrase that shares no words with the document. Fusing them means the system doesn’t have to bet on which failure mode this particular query will hit.

Dense and sparse have different blind spots, paraphrase versus rare exact terms. Fusing them means the system never has to bet on which failure mode a given query will hit.

Hybrid Search

Hybrid Search

Hybrid Search

Hybrid Search

Hybrid Search

Reranking: A Second, More Careful Pass

Retrieval scores every chunk against the query with one cheap, global signal, across the whole corpus at once. That is what makes it fast, and it is also why the order it returns is only approximately right.

Reranking takes just the candidates retrieval already found and looks at each one again, one at a time, with a narrower, more literal question: how much of the query’s actual wording is in this chunk.

Set up the handoff cleanly before showing anything moving. Retrieval already ran and already narrowed 48 documents down to 5 candidates - reranking never sees the other 43. It is a second pass over a short list, not a second search.

Reranking, a second, narrower pass over just the candidates retrieval already found, scoring each one on its own instead of against the whole corpus at once. It reorders; it never fetches a new document.

The First Pass Is Approximate

A single global signal, spread across every word in the query, does not always put the right document on top.

This slide is the abstract version of the mechanism, no real numbers yet. A doc that scores well by scattering partial matches across many query words can still lose to a doc with one precise match, once each is judged on its own instead of against the whole corpus at once.

A single global signal, scored across the whole corpus at once, is what makes retrieval fast, and exactly why its order is only approximately right. Reranking trades that speed for a second, narrower look.

What Actually Happened: “on-call handoff day”

Tried the near-duplicate pairs first (PTO days, expense threshold, outage lag) - none of them swap, because the one word that tells those two docs apart never appears in the query. This query, tried next, produces a real swap at #1.

This is the real run, not a mockup: index.search("on-call handoff day", k=5) then rerank(...) on those same five results, in build/topics/reranking/run.py. TF-IDF’s top hit is genuinely the wrong document for this query. The reranker’s plain exact-word-count fixes it, and it is honest to say why: “day” is common and diluted by idf, “handoff” is rare and, once scored on its own per chunk, decisive.

Reranking Reorders. It Does Not Fetch.

Ties directly to the assert in run.py: set(ids_before) == set(ids_after). Worth saying plainly - if retrieval missed the right document entirely, no amount of reranking gets it back. That is still retrieval’s job, not reranking’s.

What Reranking Can’t Fix

The corpus has three near-duplicate pairs, same wording, one changed number. None of them swap under this reranker.

That is an honest limit, not a bug: a reranker that only counts word overlap can’t tell two documents apart on the one word neither of them is being asked about.

Confirmed empirically, not assumed: run the PTO query and the pair’s relative order is identical before and after reranking, down to the score (both land on the same overlap count and the same tie-break). Whatever actually resolves this pair - picking one canonical version, deduplication upstream - is a different concept’s job, not reranking’s.

A reranker that only counts word overlap can’t break a tie on the one word neither candidate mentions. Aurora Cloud’s near-duplicate PTO pair (15 days vs. 20) never swaps under this reranker.

Reranking

Reranking

Reranking

Reranking

The chunks aren’t the input

Retrieval hands back three objects. The model reads one string. Something has to build that string.

A chat model takes plain text in. Three Chunk objects are not plain text yet.

Every retrieval call in this deck so far has ended at a ranked list of chunk objects. That list is still structured data: a doc title, a score, a body of text. Before generation can run, something has to flatten that into the one string a language model actually consumes.

assemble_prompt() turns them into one string

The retrieved chunks flow into a single card, and each one gains a number as it lands.

This is the “A” in RAG. assemble_prompt() is a plain string builder: it walks the chunk list in order and writes each one out behind a bracketed number. Nothing clever happens here, and that is the point - the model’s grounding depends on this step being completely mechanical and inspectable.

Every excerpt is numbered, not named

A chunk’s doc title never reaches the model directly. Its [n] does.

The numbering is positional, not semantic: chunk one becomes [1] because it is first in the list assemble_prompt() was handed, full stop. That is also why generation can later write [2] in its answer and the app can resolve it back to a real source: the mapping was fixed the moment the prompt was built.

An instruction wraps the excerpts

Three zones, in this exact order in the real string: an instruction that scopes the model to the excerpts, the numbered excerpts themselves, and a closing rule that demands a citation for every claim. The closing rule is what makes cite-or-refuse enforceable downstream - it is written directly into the prompt, not bolted on after the model answers.

What crosses the boundary to the model

The model never sees the corpus. It never sees the index. It sees one string, built once, per question.

This is why grounding is checkable: the model’s entire universe for this answer is the literal text assemble_prompt() returned. Nothing else it might “know” from training can slip in through this channel, because nothing else is in the channel.

The model’s entire universe for an answer is the literal prompt string assemble_prompt() built. Corpus, index, and every other chunk never cross that boundary, which is exactly what makes grounding checkable.

Augmentation: Building the Prompt

Augmentation: Building the Prompt

Augmentation: Building the Prompt

Augmentation: Building the Prompt

Cite or Refuse

Retrieval and augmentation hand the model a stack of excerpts. Generation is the last step: turn that stack into a sentence a human can trust, or say, honestly, that the corpus doesn’t have the answer.

Cite or refuse is the discipline that makes a RAG system trustworthy.

Generation is not “the model writes whatever it wants.” Every claim either traces to a retrieved chunk, or the system declines. There is no third option. This is the same demo query that ran through retrieval, hybrid search, and augmentation in the earlier slides, now reaching its two possible endings.

Two ways a question can end

Same model, same prompt template, same threshold check. The only difference between these two cards is what retrieval found. A well-covered question gets a cited sentence. A genuinely uncovered one gets an honest no.

A citation is a pointer, not decoration

The bracket number is not a footnote for style. It resolves back to one specific chunk in the index. Anyone reading the answer can walk that pointer and check the claim against the source themselves.

Where the line gets drawn

The gate is one number: the best similarity score retrieval returned. Set it too low and the model starts guessing on thin evidence. Set it too high and it refuses questions the corpus actually answers. The threshold is a real, tunable knob, not a metaphor.

No relevant chunk retrieved, without a refusal gate, the model falls back on parametric memory and answers fluently anyway, with no warning it is guessing. The similarity threshold is the guard that turns “no evidence” into an honest decline instead of a confident hallucination.

No partial credit

A plain language model never has this branch - it always produces text. Cite or refuse gives the system a way to say “I don’t know” instead of a fluent guess. That one branch is the difference between a tool people can trust and one that merely sounds confident.

A plain model always produces text. Cite-or-refuse gives the system a branch a plain model doesn’t have: the ability to say “I don’t know” instead of a fluent guess.

Generation: Cite or Refuse

Generation: Cite or Refuse

Generation: Cite or Refuse

Generation: Cite or Refuse

Generation: Cite or Refuse

Agentic RAG: The Tool-Use Loop

Every concept so far runs retrieval exactly once per question. An agent gets to check its own result first, and search again if it wasn’t good enough.

This is the last mechanical concept before evaluation. Nothing about the retrieval machinery changes here - the same .search() from every earlier topic is still doing the work. What’s new is a decision wrapped around it: look at the score, decide whether to trust it, and only pay for a second call when the first one earns it.

One search, or two?

The retrieval loop so far always asks once. An agent gets to check its own work first.

Topics so far (retrieval through generation) are one fixed pass: embed the query, pull the top chunks, answer. An agent wraps that same search_corpus function as a tool it can call, inspect, and call again - the retrieval machinery underneath doesn’t change, only who decides when to stop.

The decision loop

One tool call, one confidence check, and a fork: answer now, or search again first.

This is run_agent(): a scripted loop, not a free-ranging agent. It calls search_corpus once, and only takes the loop-back edge if the top score comes back under 0.2. The retry is capped at exactly one - there’s no path back through the diamond a second time, so this can never turn into an unbounded search spiral.

A real trace

The question: “what’s up with the database when it just kind of goes down.” Vague on purpose.

DEMO_QUERY_WEAK (“what happens during a database outage”) was the first thing tried here, and despite the name it scores 0.212 - just over the 0.2 bar, so it never triggers a second call. This awkward, filler-heavy phrasing is what actually pushes the first score under threshold. Both calls land on the same document in this case; the reformulation didn’t need to find something new, it just had to clear the confidence bar.

How the rewrite happens

The reformulation is a word filter, not a rewrite from scratch: drop anything four letters or shorter.

The rule is literally " ".join(w for w in question.split() if len(w) > 3). It has nothing to do with meaning - it just strips short filler words, the same words a person trims when repeating themselves more plainly. It’s a stand-in for what a real agent’s reformulation step does conceptually, kept simple so the trace stays deterministic and checkable.

Not blind repetition

The agent only pays for a second search when the first one earns it.

Same corpus, same k, same code path - the only thing that changes the shape of the trace is the confidence of the first hit. That’s the actual point of the design: an agent that searched twice on every question would double its latency and cost for no benefit on the questions it already nailed. Gating the retry on a real threshold means the expensive path only runs when it’s earning its keep.

The only difference between one-shot and agentic retrieval is the ability to check a confidence score and try again. The retry is capped at one call, gated on a real threshold, never an unbounded search spiral.

Agentic RAG: The Tool-Use Loop

Agentic RAG: The Tool-Use Loop

Agentic RAG: The Tool-Use Loop

Evaluation: How You Know It’s Working

A demo can look convincing and still be wrong. The only way to trust a RAG system is to run it against questions with known answers, and grade it honestly.

Every concept so far - chunking, retrieval, hybrid search, reranking, generation - is a design choice. Evaluation is how you find out whether those choices actually work.

This is the closing concept of the deck. Nothing built so far should be trusted just because it looks reasonable in a one-off demo. A real evaluation is a fixed, repeatable question set with a known correct answer for each question, run automatically every time the pipeline changes.

A frozen set of questions, each with a known answer

Write down real questions a user would ask, then name the one document that actually answers each. Freeze the list. Score every future change against it, unchanged.

Three of the four questions have a real answer in the corpus. The fourth is deliberately unanswerable - it belongs in the set for exactly the same reason as the other three, to check what the system does, not just whether it sounds confident.

Recall@k is a looser bar than faithful

Recall@k asks whether the right document is anywhere in the top few results. Faithful asks a stricter question - is it the very top hit, with a real similarity score worth trusting.

A pipeline can pass recall@k and still be untrustworthy at the top. The right document is in the list, but something else outranks it - and that is what actually gets cited. Faithful is the harder, more honest bar.

Recall@k, whether the right document lands anywhere in the top k results. A looser bar than faithful, which asks whether it’s the #1 hit with a score worth trusting.

Running it for real

This is a real run of the real toy pipeline against these four frozen questions. Three pass recall and faithful together. The fourth fails both, on purpose - there is no document in this corpus about the meaning of life, and the honest answer is to say so, not to guess from the nearest chunk.

Back to where this deck started

A plain language model would have answered the meaning-of-life question anyway, fluently and with no warning that it was guessing. This system checks itself against real evidence first. It only speaks when it has some.

Every design choice in this deck, chunk size, hybrid fusion, reranking, the refusal threshold, is just a hypothesis until it’s run against frozen questions with known answers. Evaluation is how you find out whether those choices actually work.

Evaluation: How You Know It Works

Evaluation: How You Know It Works

Evaluation: How You Know It Works

Making It Real, With Chroma

Everything so far ran a hand-built TF-IDF index. Swap in chromadb, a real vector store, and chonkie, a real chunker, and the retrieval story does not change, only the plumbing underneath it does.

Chroma, an embedded vector store, pip install chromadb, no server to run. A collection stores documents, embeds them, and answers .query() calls with the nearest matches, all inside the same Python process.

The Toy Index, Now Real Chroma

The Toy Index, Now Real Chroma

The Toy Index, Now Real Chroma

The Toy Index, Now Real Chroma

The hand-built TF-IDF index becomes one real library call, coll.query(). Same corpus, same query, a relevant document retrieved, now backed by a real vector store instead of a few hundred lines of hand-rolled code.

chroma

embedded and local, runs in-process, no server, the vector store this coda actually calls

milvus, pinecone

managed vector stores built for scale, a server (or a cloud service) to run and operate

chonkie, docling

chonkie chunks text, used above for real, docling ingests PDFs and other documents into text, named here, not run

References

Guides Lewis et al., Retrieval-Augmented Generation (2020) · Robertson & Zaragoza, BM25 and Beyond (2009) · LangChain, retrieval concepts · Anthropic, Contextual Retrieval

Siblings How Git Works, Visually · How Inference & Serving Work, Visually · How Evals & Observability Work, Visually · How Agentic Harnesses Work, Visually · How LangChain & LangGraph Work, Visually

Every figure in this deck: a real TF-IDF pipeline over the 48-document Aurora Cloud corpus, regenerable with build/regen_all.py