Agent orchestration is the control layer around AI agents. It decides which step runs next, keeps the shared state that steps read and write, saves progress so a run can resume after a failure, and pauses the run when a person needs to approve something. The model reasons; the orchestration layer runs the process.
What the Orchestration Layer Does
- Sequencing. It routes from one step to the next: a fixed order, a branch based on a classifier, or a choice the model makes.
- State. It holds the data the steps share, such as the user request, intermediate results and tool outputs, in a defined structure rather than a growing chat log.
- Persistence. It writes a checkpoint after each step. If a server restarts or an API times out, the run continues from the last checkpoint instead of starting over.
- Interrupts. It can stop at a chosen point, wait hours or days for a decision, and resume with that decision. See human-in-the-loop.
- Limits. It enforces step caps, timeouts, retries and budgets that a model cannot be trusted to enforce on itself.
Common Patterns
Anthropic’s engineering guide names the patterns most systems are built from: prompt chaining, routing, parallelisation, orchestrator-workers, and evaluator-optimizer (Building effective agents). The guide also separates workflows, where code fixes the path, from agents, where the model chooses it. Most production systems mix both: deterministic steps for what must always happen, model-driven steps where the path cannot be known in advance.
| Pattern | Who decides the next step | Good for |
|---|---|---|
| Chain | Code, fixed order | Draft, check, format |
| Router | A classifier or model, once | Sending a request to the right specialist |
| Orchestrator-workers | A lead model, at runtime | Research and other open-ended tasks |
| Evaluator-optimizer | A checker, in a loop | Output that must meet a clear standard |
Frameworks
LangGraph describes itself as “a low-level orchestration framework and runtime for long-running, stateful agents”. Its documentation lists durable execution through checkpointed state, human-in-the-loop pauses, short-term and long-term memory, streaming, and graphs that mix deterministic code with model-driven steps (LangGraph overview). Other options include vendor agent SDKs and general workflow engines. A short hand-written loop is often enough for a single agent with three tools; a framework pays off when runs are long, need to survive restarts, or wait for people.
Orchestration vs Neighbouring Terms
A single AI agent has its own loop. A multi-agent system is a design with several agents; orchestration is what runs that design. Interoperability protocols such as A2A connect agents across systems, while orchestration controls the flow inside one system.
Common Failure Modes
- Control flow hidden in prompts. “If the customer is in the EU, ask legal first” written in a prompt will be skipped sometimes. Written as a graph edge, it will not.
- Non-repeatable side effects. A step that sends an email and then fails will send it again on retry unless it records what it already did.
- Unbounded state. Everything is appended to the state and later steps drown in it.
- No timeouts on human steps. A pending approval with no owner blocks a run forever.
In Practice
The first design decision in orchestration is which steps the graph fixes in code and which it leaves to the model. In our AI agent development work, LangGraph graphs in Python route between those steps, and PostgreSQL stores checkpoints, so a run that pauses for an approval or hits a timeout resumes where it stopped instead of starting over. Every branch the graph can take has a test, and production runs are traced in LangSmith, which shows a reviewer which path a run took and why.