Every team that has fed a large language model a huge input has hit the same wall. You paste in a 600,000-line log file, a whole codebase, a stack of contracts, or a thousand research documents. Then even a model advertised with a million-token window starts missing things. It forgets details that are plainly present and contradicts itself, and the more you give it the worse it reasons. Researchers call this “context rot”: the measured drop in a model’s recall and reasoning accuracy as its window fills up, even on trivial tasks, even when the answer sits right there in the input.
Why does it happen? Two forces compound. Attention is a fixed budget spread across every token, so the more you add, the thinner each one gets attended to. And models are trained overwhelmingly on far shorter sequences, which makes a million-token prompt out-of-distribution territory they rarely practiced on. The window is technically available. Reliability falls off long before you reach the limit, and a bigger window does not fix that. It can make it worse.
A Recursive Language Model (RLM) is a recent answer to exactly this. Researchers at MIT CSAIL introduced it in late 2025. There is no new model and no new architecture in it. What you get is a way of calling the models you already use. Instead of stuffing the entire context into one forward pass, the model gets the long input as something it can examine with code, break into pieces, and farm out to cheaper sub-calls. No single call ever has to read the whole thing.
Key takeaways
- In MIT CSAIL’s tests, the approach built on a smaller model beat a flagship by over 34 points on the OOLONG long-context benchmark at around 132,000 tokens, at comparable cost per query.
- It needs no retraining and no new architecture. It runs as a drop-in wrapper around the frontier models you already call.
- The verdict for now: this is a research-grade pilot capability, not a production default, and nothing to put in front of customers yet.
- What holds it back is latency. The reference implementation is unoptimized, and a single query can run from a few seconds to several minutes.
- It earns its keep on one specific job: precise, traceable reasoning over a single enormous, messy input where lossy chunking would lose information.
What are Recursive Language Models?
It’s an inference-time technique, introduced by MIT CSAIL researchers in late 2025, that wraps an existing language model and lets it explore a long input as code instead of reading it all in one context window. Put another way: the long context stops being tokens inside the attention window and becomes an external environment the model queries on demand.
The mechanics are straightforward. The long prompt gets loaded into a small Python session as a variable in memory. A “root” model receives only your question, not the full context. From there it writes code to peek at that variable, search it, slice it into chunks, and when needed spawn fresh, smaller calls to a language model over individual pieces. The results flow back, the root model synthesizes them, and out comes a normal answer.
From the outside this looks identical to an ordinary model call. You swap one completion call for another. Under the hood, the model is orchestrating its own reading of the input, and anyone already building generative AI systems can adopt the pattern without touching a single weight.
The work surfaced first in an October 2025 blog post by Alex L. Zhang at MIT, then got formalized in a December 2025 paper, “Recursive Language Models,” by Zhang, Tim Kraska, and Omar Khattab of MIT CSAIL (later revised). Khattab is also known for earlier work on retrieval and prompting frameworks, which gives the idea a credible pedigree.
What a Recursive Language Model is not
The name collides with a few unrelated ideas. Worth being precise about each.
Not a recurrent neural network (RNN)
Recurrent networks are a sequential model architecture with memory loops, the kind people used before transformers took over. This has nothing to do with that. There is no architecture here at all, only a way of orchestrating calls to today’s transformer-based models.
Not recursive self-improvement
That phrase refers to an AI improving its own weights or capabilities over time, an alignment-flavored topic about systems that bootstrap their own intelligence. The technique here changes no weights and learns nothing permanently. It just decomposes one query at run time.
Not just recursive prompting
Calling a model in a loop, or chaining prompts together, is a different animal. Here the model writes and runs actual code to inspect and partition data that never fully enters its context, and it decides for itself what to read and what to delegate. The structure is emergent, not a fixed script.
How do Recursive Language Models work?
The method runs as a control loop. Each turn, the root model reasons, writes a small block of code, the code runs, and a truncated version of the result comes back into view. State persists between turns, so intermediate findings accumulate.
Step 1: Turn the context into an environment, not a prompt
The long input goes into the programming session as a variable, typically a plain string or a list of strings. The root model never sees the full text. It sees metadata about it (the type, the length, a short preview) plus the running history of the code it has executed.
Step 2: Inspect and chunk the input
With the context sitting in a variable, the root model decides how to attack it. No hard-coded chunking rules. Common emergent moves: peek at the first couple thousand characters, search the text with regular expressions, filter lines that match a pattern, or partition the whole thing into chunks to process one at a time.
Step 3: Make recursive sub-calls
When a chunk needs real language understanding, the model spawns a sub-call. Effectively a function that runs a fresh, clean model invocation over just that snippet. These can fire one at a time or batch many in parallel. Because each sub-call sees only a small, focused slice, its own context stays short and clean, which is the whole reason accuracy holds up even when the total input is enormous. This recursion stays a single level deep in practice; the sub-calls themselves do not recurse further (more on why in the limitations).
Step 4: Aggregate results and terminate
Sub-call results return as variables into the root model’s view, truncated to avoid bloat. The model combines them, reasons over the synthesis, and signals it is done by emitting a final answer. That can be a value directly, or a pointer to a long string it has built up in the environment, so the final output itself can exceed what fits in one message.
A 10MB file costs the same in prompt tokens as a 10KB one here, because it gets searched programmatically rather than poured into the window. That is how the technique can address inputs running up to roughly two orders of magnitude beyond a model’s native context window.
Recursive Language Models vs RAG, long context, and AI agents
It is tempting to file this under “rebrand of something I already know.” The distinctions are real and practical. The table sums up how each approach handles a large input.
| Approach | How it handles context | Best for | Cost / latency | Key limitation |
|---|---|---|---|---|
| Recursive Language Model (RLM) | Keeps the whole input as a variable and explores it adaptively with code, spawning short sub-calls | Precise reasoning over one large, messy input | Median cost comparable to a single big call, but latency from seconds to minutes | Unoptimized, unpredictable runtime; backfires on small inputs |
| RAG / CAG | Fetches a fixed set of chunks up front by similarity (RAG) or preloads a stable corpus once (CAG) | Fast, specific lookups against a defined knowledge base | Cheap, fast, predictable | Lossy chunking can drop information; no exact scans |
| Long-context window | Loads everything into one attention pass | Inputs that comfortably fit and stay well under the limit | Single call, but quality degrades as tokens grow | Context rot, accuracy falls as the window fills |
| AI agents | Coordinate specialized roles that talk to each other | Multi-step workflows split by responsibility | Varies with orchestration overhead | Splits work by role, not by context size |
How to choose. Default to RAG or CAG for any scoped lookup against a defined knowledge base; they are cheaper and faster. Stay with a plain long-context call when your input genuinely fits well under the limit. Reach for the recursive approach only when you need precise, auditable reasoning over one large, messy input and lossy chunking would throw away something you cannot afford to lose. If you are weighing the retrieval side, our deep dive on RAG and CAG enterprise knowledge assistants goes deeper. And these are not mutually exclusive: the technique is itself a form of agentic decomposition that splits work by context rather than by role, so it can sit as one node inside a broader multi-agent system.
Where Recursive Language Models help most
The pattern earns its keep on high-stakes tasks where accuracy and traceability beat millisecond latency, and where the input is genuinely huge. At Alltegrio we build retrieval and long-context LLM systems for legal and finance teams as well as engineering work, so these trade-offs are familiar terrain.
Whole-codebase comprehension
Reasoning across an entire repository, not just the handful of files a retriever happened to surface, is a natural fit.
Legal and contract review
Reviewing a whole policy, a long contract, or a discovery set end to end. The kind of work where missing a single clause is unacceptable and lossy summarization is a liability.
Financial audit and due diligence
Reconciling giant spreadsheets, multi-year document archives, and technical due-diligence materials, where deterministic, auditable computation is a genuine selling point for compliance.
Log forensics and root-cause analysis
Scanning massive log files deterministically, counting and filtering and tracing, instead of hoping the relevant lines got retrieved.
Multi-document and deep research synthesis
Synthesizing across hundreds or thousands of documents. This is the case where standard retrieval-based systems showed the sharpest drop-off and the recursive approach held up best.
These share a profile. The input is large and sprawling, the stakes are high, and nobody minds if the answer takes seconds or even a couple of minutes to come back.
Who needs Recursive Language Models?
Not everyone. The teams most likely to benefit are the ones whose pain is specifically about reasoning, not lookup, over very large inputs that standard approaches already choke on.
That covers engineering teams doing whole-codebase analysis; legal, compliance, and finance teams working through massive document sets where traceability is mandatory; security and operations teams doing log forensics; and research-heavy functions synthesizing across large corpora. If your workload is FAQ-style lookups against a stable, modest knowledge base, you almost certainly do not need this. Well-tuned retrieval will serve you better and cheaper. Organizations weighing where it fits often start with generative AI consulting to map it against their actual data.
Limitations and when Recursive Language Models backfire
Both sides matter. Here are the constraints in plain terms.
Latency. The reference implementation is explicitly unoptimized. Sub-calls are blocking, with no caching and no parallelism, so a single query can take anywhere from a few seconds to several minutes. In the December 2025 MIT CSAIL paper, the authors note the sequential steps add meaningful latency over a single pass and frame the implementation as a research prototype, ill-suited to real-time, conversational use.
Unpredictable cost. Most tokens go to cheap sub-calls, so the median query stays affordable, often comparable to or below a single big-model call. But the distribution has heavy tails. A long recursive trajectory can spike a single query’s bill, which is exactly why the authors offer no strong guarantee on total API cost or runtime per call.
It backfires on the wrong tasks. On short or simple inputs the orchestration overhead is not worth it, and the approach can underperform a plain call, including on some math and tool-use tasks. It is justified only above a meaningful context-size threshold.
Code fragility. Because the model writes its own code, a slicing bug can crash a run. Depth matters too: the MIT CSAIL authors capped every reported experiment at depth-one recursion and found deeper nesting unnecessary. Their design treats a single level as doing most of the work, and hitting the reported numbers still took careful engineering of the inference loop.
What to consider before adopting a Recursive Language Model
When we scope a long-context pilot for a client, we work it in this order:
-
Start with the problem, not the technique. Find a real long-context pain point where RAG or a long-context model has already failed you. No million-token reasoning pain, no reason to move now.
-
Define what success means. Decide up front what accuracy, traceability, and turnaround you actually need, and whether seconds-to-minutes latency is acceptable for that use case.
-
Plan for cost and runtime guardrails. With no hard guarantees on spend, cap iterations or sub-calls per query, and budget for the occasional expensive outlier rather than the average.
-
Mind the sandbox. The model executes code it wrote itself, so run it in an isolated environment rather than in-process. This matters most for sensitive data and overlaps with private LLM and data-residency concerns.
-
Keep humans and fallbacks in the loop. Hold on to RAG or CAG for everyday lookups, keep an off-ramp for when a query runs long, and keep people reviewing high-stakes outputs.
-
Pilot, measure, iterate. Try it on internal data first, keep it off latency-sensitive customer flows, and only scale once the numbers hold on your workload. Operationalizing any of this sensibly leans on solid LLM operations practice.
If you are already comfortable training your own GPT model, note that purpose-built recursive models are an active research direction. The gains so far, though, come from retrofitting the pattern onto models never trained for it, and the open tooling is pip-installable today.
Bringing it together
The technique reframes a stubborn problem. Rather than fighting context rot by buying ever-larger windows, it keeps each model call short and lets the model read its input with code. The early benchmarks on genuinely huge inputs are hard to wave away.
For now, though, this is promising research, not a production default. The latency is real, the cost is unpredictable, and the tooling is young. For most organizations the right move is a narrow, well-instrumented pilot on a real long-context pain point, retrieval kept for everyday work, and a plan to revisit as purpose-built, trained models arrive. If you want a second opinion on whether your data and use case fit, book a working session.
