What Is a Voice Agent?

A voice agent is an AI system that holds a spoken conversation, over the phone or inside an app, and can act on what it hears: check an order, book an appointment, update a ticket or pass the caller to the right person. It listens, works out what the caller wants, calls tools to get or change data, and answers in a natural voice, all fast enough that the conversation does not feel like waiting for a machine.

How a Voice Agent Works

There are two main designs.

Pipeline (cascaded). Three models in a chain:

  1. Speech-to-text turns the caller’s audio into text.
  2. A large language model decides what to say or which tool to call.
  3. Text-to-speech turns the reply into audio.

The pipeline is easy to inspect, because every turn exists as text, and each part can be swapped independently.

Speech-to-speech. One model takes audio in and produces audio out, keeping tone and timing that text loses. OpenAI’s Realtime API documentation, for example, describes a model that works directly with audio, keeps conversation state, calls tools, and supports interruptions and natural turn-taking, connected over WebRTC in the browser or WebSocket on the server.

Around either design sit the parts that make a phone call work:

  • Turn detection (voice activity detection) decides when the caller has finished speaking.
  • Barge-in lets the caller interrupt, and the agent stops talking.
  • Telephony connects the agent to phone numbers and existing call centre systems.
  • Tools through function calling or the Model Context Protocol connect it to the CRM, booking system or knowledge base.
  • Handoff passes the call, with a summary, to a human agent.

What Makes Voice Harder Than Chat

  • Latency. In a text chatbot two seconds is fine. On a call it feels broken. Every stage, including tool calls and retrieval, must fit a tight budget, which usually rules out slow reasoning models on the live path.
  • Recognition errors. Names, addresses, order numbers and accents are easy to mishear. Critical values should be read back and confirmed.
  • No screen. Long lists and links do not work. Answers must be short and spoken.
  • Interruptions and silence. People talk over the agent, pause, or speak to someone else in the room.

Rules That Apply

Disclosure is a legal matter, not only good manners. Under Article 50 of the EU AI Act, applicable since 2 August 2026, a system that interacts with people must make clear they are dealing with AI unless it is obvious. Call recording, consent for outbound calls and the handling of personal data fall under telecoms, consumer-protection and data protection law in each market, and US rules on artificial voices in telemarketing calls are strict. Check the rules for every country you call. See EU AI Act.

Common Failure Modes

  • Acting on a misheard value without confirmation.
  • Answering beyond its knowledge instead of handing over to a person.
  • Too many permissions. A voice agent that can issue refunds or change accounts without approval is an easy target for social engineering and prompt injection.
  • No way out. Callers who cannot reach a person quickly lose trust in the whole channel.

In Practice

A voice agent starts with low-risk call types, such as opening hours, order status or appointment reminders, and takes on more as recorded calls show it handles them. Latency is budgeted end to end, since a pause of more than a second or so makes callers talk over the agent. Every call offers a way to reach a person (human in the loop), and a change to a booking or a payment is read back to the caller and confirmed before it is made.

We build voice agents as part of AI agent development and chatbot development services, connected to existing systems through AI workflow automation.

Back to Glossary

Let's talk about your project

We'll help you harness AI and ML technologies to create innovative software solutions

File requirements: pdf, doc, docx, rtf, ppt, pptx — max 15 MB

No file selected