Context engineering is the practice of deciding what information a language model sees at each step of a task: the instructions, the retrieved documents, the tool definitions, the results of earlier tool calls and the conversation so far. The goal is to give the model everything it needs to act correctly and nothing that distracts it. It matters most in AI agents, which run many steps and accumulate information as they go.
Context Engineering vs Prompt Engineering
Prompting is about writing good instructions. Context engineering covers the whole input, and it is repeated on every model call, not written once. Anthropic’s engineering team describes it as curating “the smallest possible set of high-signal tokens” for each step, and treats context as a finite resource with an “attention budget” that every extra token draws down (Anthropic).
A simple chatbot with one system prompt barely needs the distinction. An agent that reads files, calls ten tools and works for an hour depends on it.
Why It Matters
Models do not use long inputs evenly. Research from 2023 showed that models answer best when the relevant information is at the start or end of the input and noticeably worse when it sits in the middle (Lost in the Middle, arXiv 2307.03172). Chroma’s 2025 study of 18 models found performance grows less reliable as input length increases, even on simple tasks, and that irrelevant but similar “distractor” text hurts further (Chroma). This effect is often called context rot. A bigger context window raises the ceiling; it does not remove the need to choose what goes in.
What Goes Into the Context
- System instructions: role, rules, output format, what to do when unsure.
- Tool definitions: names, descriptions and parameters of every tool the model may call. Each one costs tokens and adds a choice the model can get wrong.
- Retrieved knowledge: passages from retrieval-augmented generation (RAG), database rows, files.
- Tool results: output of searches, API calls and code execution, often the largest and noisiest part.
- History: earlier turns and steps.
- Memory: facts saved from earlier sessions and loaded when relevant.
Main Techniques
- Retrieve just in time. Instead of loading every document up front, give the agent lightweight references (file paths, query tools) and let it fetch what it needs.
- Trim tool output. Return the 20 relevant rows, not 5,000. Summarise or truncate large results before they enter the context.
- Compact long sessions. When the context fills, summarise the work so far and continue in a fresh context with that summary.
- Take structured notes. Let the agent keep a task list or notes file outside the context and reload it when needed.
- Split work across sub-agents. A sub-agent explores in its own clean context and returns a short result to the coordinating agent.
- Design fewer, clearer tools. Overlapping tools with vague descriptions lead to wrong calls.
- Order deliberately. Keep stable content (instructions, tool definitions) at the start, which also makes prompt caching effective, and put the most important material where the model attends best.
Common Failure Modes
- Context stuffing: loading everything “just in case”, which raises cost and lowers accuracy.
- Stale context: a summary written early in the session that no longer reflects the current state.
- Poisoned context: untrusted text (a web page, an email, a retrieved file) carrying instructions the model then follows. This is indirect prompt injection.
- Tool overload: dozens of overlapping tool definitions in every call, so the model spends tokens on choices it should never face and picks the wrong tool more often.
- Lossy compaction: a summary that drops a constraint or a decision made earlier, so the agent continues as if it had never been agreed.
Failures caused by the size limit itself, such as silent truncation or an answer squeezed out by too much input, are covered under context window.
Related Terms
Context engineering includes RAG, memory and tool design, and it is the main lever for reducing AI hallucination in agent systems. Its effect is measured with evaluations on real tasks, not by inspecting prompts.
Context Engineering in Practice
When an agent gives a bad answer, the first thing we open is the trace in LangSmith that shows exactly what the model saw at that step; more often than not the cause is there, in a tool result that buried the relevant row or a stale summary. Each tool therefore has a written output contract: which fields it returns, how many rows at most, and what it says when it truncates. Long-running agents built with LangGraph checkpoint their state and compact at points chosen in the design, with the constraints of the task copied into every summary. A change to what enters the context is treated like a code change and rerun against the evaluation set before release.
See also: AI agent development.