Recursive Language Models: Context as a Variable
A recursive language model keeps its input as a variable and pokes at it with code — no context rot, inputs 100x past the window. Here's how RLMs work.
This article has an interactive companion
Don't just read it — step through it interactively, one state change at a time.
The default way to use a language model is to make the prompt be the input: paste the whole document, ask the question, take the answer. It works until the document is 200,000 tokens of legal filings or a 2-million-line codebase — and then it quietly stops working, because the model was never good at reading the middle of a giant context in the first place.
A Recursive Language Model (RLM) flips the setup. Instead of loading the input into the prompt, it binds the input to a variable that a root model never fully reads. The root writes Python, runs it in a REPL, and interacts with the input the way you would with a huge file you can't hold in your head: check its size, search for the part you need, slice out a window, hand that window to a sub-model, and compose the answer.
The payoff is threefold: no context rot (no single model call ever sees a bloated prompt), inputs that scale 100x beyond any context window, and — not coincidentally — the exact architecture behind the real-world fight between Claude Code's agentic search and Cursor's indexed retrieval. This is the map of the interactive deep dive on YeetCode, where every number below is recomputed live from a real document you pick and query.
Why does long context rot?
Attention is O(n²): every token attends to every other token. Jam a 484-token report into one forward pass and the model pays that quadratic cost whether the question is hard or trivial. Worse, accuracy is not uniform across the window.
The published finding here is "Lost in the Middle" (Liu et al., 2023): long-context models recall facts near the start and end of the prompt reliably, but accuracy sags in a U-shaped curve for facts buried in the middle. So the failure isn't only cost — it's that a real fact sitting at position 0.5 of the document is the fact most likely to be missed.
That is the deep flaw of prompt-is-input: you pay full attention over everything, and the payment doesn't even buy you uniform recall.
Context as a variable, not a prompt
The RLM move is to stop treating the document as text to attend over and start treating it as data to query with code. Consider a counting question — "how many times does revenue appear in this report?" Fuzzy attention is genuinely bad at counting; an exact string op is perfect at it.
# --- traditional LLM: prompt IS the input ---
chars = len(context) # full document
tokens = chars // 4 # ~4 chars/token
answer = llm(f'{context}\n\ncount revenue') # attends over ALL tokens
# --- RLM: context is a variable ---
hits = re.findall("revenue", context) # exact op, 0 model tokens
count = len(hits) # exact, groundedSame document, same answer — but the RLM spent zero model tokens to get the count, because re.findall runs in the REPL, not the GPU. The model only spends tokens on the small slices it decides to read. On the deep dive's sample report those add up to ~135 model tokens — a ~100-token peek plus a ~35-token delegate slice — against a ~484-token document read in full: roughly a 3.6x attention reduction for the same exact answer, and no dependence on where in the document the fact happened to live.
The agentic REPL loop
The engine of an RLM is a five-step loop the root model drives with code. It never reads the full context; it reads about the context, then reads one focused slice.
context = open('report.txt').read() # bound to a variable, not the prompt
# 1. PEEK — how big is it, what's at the top?
print(len(context), repr(context[:400]))
# 2. LOCATE — find every offset of the query term
hits = [m.start() for m in re.finditer(query, context)]
# 3. SLICE — pull a focused window around the first hit
start, end = hits[0] - 50, hits[0] + 90
window = context[start:end]
# 4. DELEGATE — a sub-LM reads ONLY that window
answer = rlm_agent(query, window) # tiny, focused context
# 5. FINAL — compose the computed value into a reply
FINAL(f'{query!r} → {answer}')Each step is either a free string op (len, re.finditer, slice) or a delegation to a sub-model with a clean, tiny window. The FINAL() marker is the loop's exit condition. If the query matches nothing, the loop halts honestly — an absent term returns zero hits, and a real RLM broadens the search rather than inventing an answer.
Walkthrough: answering "revenue" over a 1,935-char report
Take the deep dive's annual report: 1,935 characters, ~484 tokens, six sections. The query is "revenue". Every value in this trace is a real string op, not a narrated guess.
| Step | Operation | Real result | Tokens the model reads |
|---|---|---|---|
| 1. Peek | len(context), context[:400] | 1935 chars; reads first 400 chars | ~100 |
| 2. Locate | re.finditer("revenue") | 9 hits at offsets [82, 331, 384, 419, 458, 490, 1393, 1562, 1657] | 0 (string op) |
| 3. Slice | context[32:172] | a 140-char window around hit #1 | 0 (string op) |
| 4. Delegate | rlm_agent("revenue", window) | regex over the slice extracts "$4.2B" | ~35 |
| 5. Final | FINAL("revenue → $4.2B") | answer composed | 0 |
The whole answer touched 400 + 140 = 540 of 1,935 characters — about 28% of the document, and none of the read text came from the risky middle. The locate step is the quiet hero: it costs zero model tokens because re.finditer is deterministic. Edit the query to "APAC" and the offsets change to a different 6-hit spread; the trace recomputes because it was never canned.
The recursion math: past the context window
Peek-locate-slice handles one focused answer. Recursion is what lets a single input run 100x beyond any window. The root splits the context into f equal chunks, and each chunk recurses into its own fresh sub-model window.
def rlm(context, query, depth=0):
if len(context) <= LEAF or depth >= MAX_DEPTH:
return sub_lm(query, context) # leaf: read the slice
chunks = split(context, fanout) # split into f equal chunks
parts = [rlm(c, query, depth + 1) for c in chunks]
return synthesize(parts) # results bubble back upWith a uniform fan-out f and depth d, the arithmetic is a conservation law:
calls at depth d = f^d
tokens per call = total / f^d
tokens per level = f^d × (total / f^d) = total ← conservedThe total work is conserved at every level — splitting never reduces how many tokens get read overall. What it shrinks is the context any single call must hold. That is the whole trick, and it's why an RLM has no context rot: every model invocation sees a small, fresh slice regardless of how large the original was.
| Total tokens | Fan-out | Depth | Leaf calls (f^d) | Tokens per leaf |
|---|---|---|---|---|
| 1,000,000 | 2 | 4 | 16 | 62,500 |
| 100,000,000 | 8 | 5 | 32,768 | ~3,052 |
| 10,000,000 | 2 | 5 | 32 | 312,500 |
A hundred million tokens — far past any real window — is processed by 32,768 leaf calls, each reading only about 3,000 tokens. Feed the input an order of magnitude larger and you add depth, not context-per-call. The leaf window stays bounded; the total simply distributes across more calls.
Agentic search vs. indexed RAG: Claude Code vs. Cursor
The "locate" step has two production answers, and they are billion-dollar bets. Claude Code searches your repo with agentic shell tools — grep, glob, ls, cat — running exact searches on demand. Cursor builds a vector index: embeddings in a vector DB, Tree-sitter chunking, Merkle-tree incremental sync, retrieval by cosine similarity.
Cosine similarity is the normalized dot product, which measures the angle between a query vector and a chunk vector so length doesn't skew the comparison:
cosine(a, b) = dot(a, b) / (‖a‖ · ‖b‖)Run both over the same chunks and their strengths separate cleanly:
| Agentic grep (Claude Code) | Indexed RAG (Cursor) | |
|---|---|---|
| Matching | Literal / regex, exact | Cosine over embeddings |
| Best at | Named symbols: function names, error codes, identifiers | Concepts: "where do we handle errors" |
| Model-token cost to locate | Zero (shell op) | Zero at query, but index must be built + kept in sync |
| Failure mode | Clean miss — a conceptual query finds nothing verbatim | Similar-but-wrong — ranks a plausible near-neighbor with a non-zero score |
Search cosineSimilarity in the codebase and grep returns both real occurrences — the definition in util.ts and its call site inside rankResults in search.ts — with zero false positives, because the symbol is literal. Ask "where do we handle errors" and grep returns nothing — no chunk contains those literal words — while embeddings surface the try/catch block that actually does the work. The gap between the top and second cosine score is the signal-to-noise of the retrieval, and it's exactly where the "similar but wrong" risk hides.
What to remember
- Context is a variable, not a prompt. The root model interacts with the input through code, so it never pays full attention over a bloated window.
- Exact ops beat fuzzy attention for exact questions. Counting, locating, and slicing run in the REPL for zero model tokens and never rot.
- Recursion conserves work but bounds context.
f^dcalls each readingtotal/f^dtokens sum back to the total — every individual call still sees only a leaf-sized slice. - Locate has two production shapes. Agentic grep wins on namable symbols; indexed RAG wins on concepts. Each carries its own failure mode.
Keep going
- How LLMs Actually Work: The Next-Token Loop — the forward pass an RLM is trying to keep small.
- How to Read AI Research Papers Without Drowning — for chasing down "Lost in the Middle" and its successors.
- From Hand-Written Rules to LLMs: How AI Learned to Learn — how the field got from grep-style rules to models that write their own grep.
- Neural Networks from Scratch: How Machines Actually Learn — the gradient-descent machinery under every sub-model.
Then open the interactive deep dive to edit the query, dial the fan-out, and watch every offset, slice, and cosine score recompute live.
FAQ
What is a recursive language model?
A recursive language model (RLM) is an architecture where the input is stored as a variable rather than pasted into the prompt. A root LLM drives a Python REPL to peek at the input's size, search it, slice out relevant windows, and delegate those slices to sub-models — recursing on large chunks until it calls FINAL(). Because no single model call ever holds the whole input, there is no context rot, and inputs can scale far past any context window.
How does an RLM avoid context rot?
It never lets a full document into one forward pass, which is exactly where the rot lives. The root model works through code instead: it runs zero-token string operations to pinpoint the span it needs, then hands a sub-model a short, fresh window rather than the whole thing. Because every model call only ever sees a leaf-sized slice, where a fact sits stops mattering — the familiar degradation that plagues facts parked deep inside a long prompt has nothing to attach to when no prompt is ever long in the first place.
How can an RLM handle inputs 100x larger than the context window?
Through recursion with a conserved-work invariant. With fan-out f and depth d, there are f^d leaf calls, each reading total / f^d tokens, which multiply back to total. Splitting never reduces the total tokens read — it shrinks the context any single call must hold. So a 100-million-token input becomes 32,768 leaf calls of ~3,000 tokens each (fan-out 8, depth 5). A ten-fold larger input just grows the tree by another level or wider fan-out; each leaf still reads the same bounded slice, so no single window ever overflows however large the input gets.
Is agentic grep better than RAG for code search?
Neither dominates; they fail differently. Agentic grep (Claude Code style) is exact and deterministic, so it's ideal for namable symbols — function names, error codes, identifiers — with zero false positives, but it returns nothing for a conceptual query phrased in words that don't appear verbatim. Indexed RAG (Cursor style) uses cosine similarity over embeddings, so it finds meaning that grep misses, but it can rank a plausible-but-wrong near-neighbor with a non-zero score. Match the tool to the query: symbols favor exact search, concepts favor embeddings.
Does an RLM cost more than a single large prompt?
For a focused question, usually far less. A traditional call pays the full O(n²) attention bill over every token even when the answer is a single number buried in one line. An RLM only spends model tokens on the window it actually decides to read; the peeking, counting, and locating happen as free string operations off the model entirely, in the REPL. The savings scale with how much of the input is irrelevant to the question — the larger the document relative to the answer, the more lopsided the win. Recursion does eventually read the total token count spread across all its leaves, but it swaps one impossible giant call for a fan of cheap, bounded ones that each stay well inside the window.
Make it stick: run this one yourself
Don't just read it — step through it interactively, one state change at a time.