One tiny model, one sentence: real tokens, real logits, a real cache, real sampling, then two real calls to a real local server
Pranjal Rawat
One call, taken apart
“Inference” is one function call: tokens in, one more token out, repeated. Everything a serving stack adds on top, caching, batching, sampling, quantization, a server in front of it, is there to make that one call cheaper or faster, never to change what it computes.
Most figures ahead are photographs of gpt2 (124M params, CPU) really completing one sentence, plus two real calls to a real local server, a handful (7 of 39) are clearly-labeled concept cards, drawn to carry ideas this tiny deck has no way to benchmark.
The running example
One sentence, one small model, completed token by token, the mechanics shown by real runs and a few labeled concept cards.
Part 1 · The mechanics of one token
Everything in Part 1 happens inside a single process, on this laptop’s CPU: a string becomes ids, ids become logits, logits become a distribution, and a growing cache and a batch dimension change nothing about the answer, only its cost.
Every topic ahead in Part 1 leaves the logits untouched: sampling only changes how a token gets picked from them, and the cache and the batch dimension only change what it costs to get there, never the answer.
A token is one piece of gpt2’s byte-pair vocabulary, the atomic unit the model actually reads and writes. The fixed sentence becomes a short list of integer ids before anything else happens.
Tokenization
Tokenization
Tokenization
Tokenization
Tokenization
A logit is the model’s raw, unnormalized score for one entry in gpt2’s ~50,257-word vocabulary, one real number per candidate token, before any softmax turns them into probabilities.
The Forward Pass
The Forward Pass
The Forward Pass
The Forward Pass
The Forward Pass
The Logit Distribution
The Logit Distribution
The Logit Distribution
The Logit Distribution
Sampling
Sampling
Sampling
Three ways to decode the same logits
greedy
Always takes the argmax of the real logits. Deterministic: the same token every run.
temperature
Divides the logits before softmax. This deck’s temperature 1.6 spreads probability mass and raises entropy versus temperature 0.7.
top-p
Keeps the smallest sorted prefix of the distribution whose cumulative mass reaches p, then renormalizes. A smaller p keeps a smaller, more confident set.
Greedy is the temperature-to-zero limit of the same softmax: deterministic, always the same token. This deck’s higher temperatures (0.7, 1.6) trade that determinism for diversity, never for a different forward pass.
The KV cache stores each layer’s past key and value tensors so a new token’s forward pass can attend to the whole prefix without recomputing it from scratch.
The KV Cache
The KV Cache
The KV Cache
The KV Cache
The KV Cache
The KV Cache
The KV Cache
The KV Cache
It is tempting to assume a growing cache changes the model’s answer as context grows. It does not: this topic’s oracle proves the cached next-token logits equal a full from-scratch recompute at every one of the 4 generation steps, the cache only removes redundant computation.
Batching
Batching
Batching
Batching
Two ideas, before the code
The next two topics are concept-only: no serving engine lives in this deck to honestly benchmark, so no figure claims a speed number.
Why vLLM Is Fast
Why vLLM Is Fast
Why vLLM Is Fast
Why vLLM Is Fast
Quantization stores each parameter in fewer bits, fp16 instead of fp32, which changes how a number is stored and how much memory it costs to move, never how many parameters exist or what they compute.
Quantization
Quantization
Quantization
Quantization
Local vs hosted is where the weights live. Hosted serving calls someone else’s GPUs and API over the network; local serving runs the weights yourself, in this deck a real qwen2.5:0.5b behind Ollama on localhost. Same call shape, different tradeoffs in control, privacy, and who runs the ops.
Hosted vs Local
Hosted vs Local
Hosted vs Local
Part 2 · Serving it for real
Part 1 stayed inside one python process. Part 2 leaves it: a real HTTP call to a real local model server, twice, then the one call shape that also reaches a hosted provider, then where the throughput path takes over once Ollama is outgrown.
Ollama and litellm calls in this section hit a real qwen2.5:0.5b server on localhost:11434, not a mock.
Ollama, litellm, and vLLM are three ways to make the identical call cheaper or faster to run at scale, never a way to change what it computes.
Ollama, For Real
Ollama, For Real
Ollama, For Real
Ollama, For Real
litellm, One Shape
litellm, One Shape
litellm, One Shape
litellm, One Shape
vLLM, The Throughput Path
vLLM, The Throughput Path
vLLM, The Throughput Path
Three serving options, one call shape
local runtime
Ollama: a real HTTP call to a real local qwen2.5:0.5b server on localhost:11434, right for one process serving one or a few users.
gateway
litellm: wraps that same Ollama server behind the OpenAI-shaped request and response every hosted provider also speaks, one call shape, whichever provider sits behind it.
throughput engine
vLLM: the path teams reach for once request volume outgrows what naive batching on one GPU can absorb, via paged attention and continuous batching (drawn, not run, in this deck).
The Whole Path
One sentence, five ways to run it
Every figure in this deck traced one idea: the math stays fixed, only the cost changes.
Most figures: real runs of gpt2 (transformers) plus real Ollama/litellm calls to a local qwen2.5:0.5b server; a handful are clearly-labeled concept cards. Regenerable with build/regen_all.py · versions pinned in build/requirements.txt
