How Evals & Observability Work, Visually

One coffee shop, one bot: fixed test sets, judges, drift, spans, tokens and cost, offline scoring versus online tracing

Pranjal Rawat

Two ways to ask “is it working”

Grading a bot against a fixed test set answers one question: did it get these known cases right. Watching a live run answers a different one: what actually happened, span by span, token by token, dollar by dollar. Most teams build only the first and are surprised when it doesn’t explain the second.

Offline, scored. A fixed test set, a metric, a number that regresses or doesn’t. Online, traced. A live run, instrumented into spans, read after the fact.

Offline-scored vs. online-traced

Offline-scored vs. online-traced. Offline-scored means graded against a test set with known-correct answers. Online-traced means watched as it runs, with no answer key, via spans, tokens, and cost.

Meet Beanline Coffee, graded

Every figure ahead is a photograph of real code grading (or tracing) one small coffee shop’s order-taking bot: a fixed set of orders, a scripted model that replies the same way every run, and real langchain/langgraph underneath.

Part 1 · Offline, scored

A frozen test set with known-correct answers, and a metric that turns a prediction into a number. Deterministic by construction: same script, same score, every run.

Test set. A frozen list of (input, expected) pairs, checked into the repo like code, the eval’s ground truth. Byte-identical every run.

Grading needs an answer key

Grading needs an answer key. A reference-based metric is a pure function of (prediction, expected). Without a known-correct answer to compare against, it cannot be honest.

What Is a Test Set?

Two ways to check an answer

Exact match. Does the prediction equal the expected string, verbatim. The simplest metric, and the hardest one to argue with.

Recall@k. Of the items that were actually relevant, how many did the top-k surface. Ragas’s non-LLM version of this is NonLLMContextRecall.

Offline Metrics: Exact Match, Recall, and the BLEU Trap

The trap in surface metrics

BLEU / ROUGE score a correct paraphrase low. Two answers can share the same meaning and almost no surface tokens, token overlap alone cannot tell “reworded” from “wrong.” Target what the answer actually claims (a number, a fact), not its word choice.

LLM as Judge, and How It Fails

A model grading a model

LLM-as-judge. A second model reads a candidate answer against a rubric and returns a verdict. Useful when there’s no exact string to match against, and only as reliable as what the rubric actually asks it to check.

A judge asked to check agreement will rubber-stamp a false premise. Write rubrics against ground truth (the menu, the till, the stock room), never against a claim the prompt itself supplied.

Pairwise Comparison: A vs B

Three families of offline metric

Reference-based

Exact match, recall@k. Compares the output against a known-correct answer. Deterministic, cheap, and blind to anything the reference didn’t anticipate.

Surface

Token-overlap scores like BLEU/ROUGE. Cheap to compute, but grades word choice, not correctness, the trap from the last section.

LLM-judge

A model scores another model, against a rubric or in pairwise preference. Handles open-ended output, at the cost of a rubric that must target truth.

Is the drop real?

Statistical significance (for a pass rate). A pass rate is a proportion; a proportion measured on a small sample has a wide confidence interval. The same nominal drop can be noise at n=20 and signal at n=200, report the interval, not just the rate.

Is the Drop Real? Statistical Significance

Part 2 · The pivot to observability

A test set answers “did it get the known cases right.” It cannot answer “what happened on the live run nobody scripted.” That takes watching, not scoring: nested spans, token counts, cost, latency, read after the fact.

Watching, not scoring

Trace / span. A trace is the record of one real run; a span is one step inside it (a model call, a tool call), with a start, a duration, and its own attributes, nested under whichever span called it.

Observability watches a live run with no answer key. The question changes from “was this right” to “what happened, and what did it cost.”

The two instruments, side by side

Offline · scored

Fixed inputs, known-correct answers, a metric that is a pure function of (prediction, expected). One number, comparable run to run.

Online · traced

Live inputs, no known answer, a nested span tree with tokens, cost, and latency read after the fact. Not a score, a record of what happened.

The Pivot: Scored Offline, Traced Online

A Real Trace: Spans, Tokens, Cost

Drift Over Time: Same Eval, Two Model Versions

Part 3 · Real tools, no stubs

Three widely used libraries, run for real and deterministically: DeepEval’s own metric objects, Ragas’s own non-LLM metrics, and the Langfuse SDK’s own span emission, captured offline with an in-memory OpenTelemetry exporter.

A real library’s own objects and formulas can run deterministically without ever calling a hosted LLM, the point of Part 3 is the code path, not the difficulty of the metric.

DeepEval docs · Ragas docs · Langfuse v4 (OpenTelemetry-based) docs

DeepEval, For Real

Ragas: Real RAG Metrics

Langfuse, Captured Offline

References

Guides Anthropic, Demystifying evals for AI agents · OpenAI, A practical guide to building agents · Anthropic, Building effective agents

Libraries DeepEval docs · Ragas docs · Langfuse docs

Siblings How Git Works, Visually · How RAG Works, Visually · How Inference & Serving Work, Visually · How Agentic Harnesses Work, Visually · How LangChain & LangGraph Work, Visually

Every figure in this deck: a real run of langchain 1.3 / langgraph 1.2 / deepeval / ragas / langfuse at Beanline Coffee, regenerable with build/regen_all.py · versions pinned in build/requirements.txt