Skip to main content
All posts
AI & Data14 min read

Voice AI Agent Architecture: Latency, Barge-In, Compliance

How to build an enterprise voice AI agent: the realtime loop, a per-hop latency budget, barge-in, speech-to-speech trade-offs and EU compliance.

Published Updated: 15 November 2026

In May 2026 OpenAI shipped gpt-realtime-2, which it says brings GPT-5-class reasoning to voice agents. In September Google followed with Gemini 3.8 Live. The money is moving too: CRM Magazine reported a Gartner survey in which customer-service AI spending rose 38 percent while service budgets grew 2 percent.

Every voice pilot we have seen hits the same wall first. It is not accuracy or cost. It is the silence after the caller stops talking. A chatbot that answers in two seconds feels quick; a phone agent that answers in two seconds feels broken. This article is for the people who sign off the architecture: CTOs, contact-centre owners, architects and CISOs. We take the realtime loop apart, budget each hop in milliseconds, and set out which European rules apply.

TL;DR / Key takeaways

  • Latency is a budget, not a feature. The reference cascaded pipeline in the Pipecat primer takes 1,293 ms voice-to-voice, half of it waiting for the LLM's first token. Budget per hop and aim for a median near 800 ms before telephone transit.
  • Turn detection decides how the agent feels. Silence timeouts either cut callers off or keep them waiting. Semantic end-of-turn models are now the baseline.
  • Barge-in separates demos from production. Stop playout, cancel in-flight work, and trim the context to what the caller actually heard.
  • Speech-to-speech is a trade. Fewer hops and better prosody, in exchange for transcript fidelity, free choice of voice and some tool-calling reliability. Choose per call flow.
  • EU compliance is already live. AI Act Article 50 disclosure has applied since 2 August 2026; recordings bring GDPR, § 201 StGB and PCI DSS into scope.
The enterprise voice AI agent. A cascaded realtime loop (1–4) with the barge-in path (5), the enterprise controls zone (6) and the speech-to-speech alternative (7). The ms values are our planning targets.
The enterprise voice AI agent. A cascaded realtime loop (1–4) with the barge-in path (5), the enterprise controls zone (6) and the speech-to-speech alternative (7). The ms values are our planning targets.

Download: Editable draw.io file · Open in diagrams.net · Chart data (CSV)

Why this matters now

Gartner's spending gap means service AI projects must prove value or be cut. And in a customer survey Gartner published in August 2026, 87 percent of respondents said companies using generative AI for service must offer a route to a human. A voice agent that traps people fails on both counts.

The building blocks have matured. OpenAI's Realtime API accepts SIP calls directly, with an EU endpoint (sip-eu.api.openai.com). Azure's Voice Live API, generally available since the end of September 2025, now lists gpt-realtime-2.1 with global, data-zone and regional processing, next to cascaded options on Azure speech. Amazon Nova 2 Sonic has been generally available since December 2025, and AWS lists Europe (Stockholm) among its regions.

The physics have not changed. ITU-T G.114 says that below 150 ms mouth-to-ear, most applications "experience essentially transparent interactivity", and it caps one-way network delay at 400 ms for planning. A voice agent must then listen, think and speak inside the pause a human would leave, which the Pipecat primer puts at about 500 ms. A 2009 PNAS study of ten languages found speakers everywhere minimise silence between turns.

The realtime loop, hop by hop

The numbers match the diagram callouts. Each hop carries our planning target, not a vendor promise.

① Telephony and the media edge

Calls arrive over the telephone network, mostly as 8 kHz narrowband audio, cross a SIP trunk and session border controller, and land in the media gateway. In the primer's reference pipeline this transport layer costs 203 ms across both directions, and the primer notes a telephony provider adds "a few hundred milliseconds" more. The only lever is placement: media gateway, speech services and model endpoint in one region. Target: 200 ms.

② Streaming STT and turn detection

Transcription is fast; deciding that the caller has finished is not. A short silence timeout cuts off people reading out an IBAN; a long one slows every answer. Semantic end-of-turn models listen to how someone speaks, not just whether they stopped. The primer gives about 250 ms for Pipecat's open-source Smart Turn; Deepgram publishes a p90 of about one second for its Flux end-of-turn detection (p95: 1.5 seconds), a reminder to watch the tail and not just the median. OpenAI's Realtime API offers semantic_vad with an eagerness setting. Target: 250 ms, tested on your callers, languages and 8 kHz audio.

③ The LLM and its tools

The biggest hop: 650 of 1,293 ms in the reference. What counts is time to first token, so use a fast model, a short cacheable prompt, a regional endpoint and streaming end to end. The quiet killer is a tool call before the first word. Let the agent say "let me pull up your order" while the CRM lookup runs in parallel. Our note on small language models versus API models covers the cost side. Target: 250 ms, which keeps large reasoning models off the critical path.

④ Streaming TTS and playout

The orchestrator streams the first speakable clause to TTS (20 ms of chunking in the primer). ElevenLabs gives about 100 ms median for Flash v2.5, excluding network and application latency; the primer's table has 120 ms to first audio. Target: 100 ms. Test that the voice pronounces your product names, numbers and addresses correctly over a narrowband line.

⑤ Barge-in: the path nobody demos

When the caller talks over the agent, the barge-in controller must flush the playout buffer, cancel LLM generation and TTS synthesis, and trim the history to the words actually played. Teams forget the third step. OpenAI's Realtime API has a conversation.item.truncate event for it; in a cascaded pipeline you build it from TTS word timestamps. The primer warns of the opposite failure too: coughs, keyboard clicks, background voices and echo leaks trigger false interruptions. Tune both error rates together.

⑥ Handoff, recordings and the controls zone

Give the agent narrowly scoped, read-first tools and treat every spoken input as untrusted; voice is just another channel for indirect prompt injection. Make human handoff first-class: a warm transfer (for example SIP REFER) with a summary. Our post on human-in-the-loop approval gates covers when to hand over. Store recordings and transcripts with consent, redaction and retention enforced, and trace every turn with per-hop timings as in our AI agent observability guide. Without per-hop timings there is no budget to manage.

⑦ The speech-to-speech alternative

One realtime model takes audio in and returns audio out: OpenAI's gpt-realtime family, Gemini Live, Amazon Nova Sonic or the realtime models in Azure Voice Live. Hops ② to ④ disappear as separate services, and the tone of voice a transcript discards survives. The Pipecat primer lists the costs: these models "do not follow instructions or call tools as reliably as text-mode LLMs", their transcripts may fall short for compliance and evals, and context handling is less flexible. It also finds them better with mixed-language speech.

What the numbers say

Latency waterfall: where 1,293 ms go in a cascaded reference voice pipeline, compared with a typical human response and our planning target
Latency waterfall: where 1,293 ms go in a cascaded reference voice pipeline, compared with a typical human response and our planning target

The waterfall uses the Pipecat primer's published latency table: 16 rows for a WebRTC client with no telephone leg, which we grouped into the diagram's five hops. By the end of transcription the human 500 ms is already gone, and the LLM's first token is half the total.

Per hop: published reference latency compared with our planning target, in milliseconds
Per hop: published reference latency compared with our planning target, in milliseconds

Our targets are planning values, not measurements; the CSV gives the basis for each. Transport barely moves and turn detection gains a little. Of the 473 ms we need to save, 400 must come from the LLM. That makes model choice for voice a latency decision first and a quality decision second.

In a recent engagement…

We recently reviewed a voice pilot at a European service organisation. The demo had impressed the board; on the phone, callers hung up. The traces showed why: media gateway and model endpoint in different regions, a long policy document in the prompt on every turn, and a blocking CRM lookup before the first word.

None of the fixes needed a new model. We moved everything into one EU region, shortened and cached the prompt, and had the agent acknowledge first and look up second. Semantic end-of-turn detection replaced the silence timeout, and context trimming on barge-in fixed a bug logged as "the bot repeats itself". The hardest question was organisational: who owns a budget spread across three vendors?

Cascaded or speech-to-speech? The decision table

RequirementCascaded STT → LLM → TTSSpeech-to-speech
Exact transcript for compliance and QANative: the transcript is the inputBy-product; verify accuracy
Reliable multi-step tool callingStrongest todayImproving; test on your flows
Free choice of voice and custom voiceAny TTSCoupled to the model (Azure Voice Live adds Azure voices)
Natural prosody and emotionLost at the transcriptPreserved
Mixed-language callersHarderBetter, per the Pipecat primer
Per-hop tuning and vendor swapYesNo: one opaque hop
EU data residencyAssemble per componentCheck the model's processing options
Engineering effortHigherLower

Our rule of thumb: open-ended service conversations can start on speech-to-speech. Authentication, payments, claims and anything you must replay word for word stay cascaded until your own tests prove otherwise.

Governance for European enterprises

EU AI Act. Article 50(1) has applied since 2 August 2026: people must be told they are talking to an AI, at the latest at first interaction, unless it is obvious. On the phone it never is, so put it in the greeting. Providers of systems generating synthetic audio must mark it machine-readably under Article 50(2); the AI Omnibus, Regulation (EU) 2026/1744, gives systems on the market before 2 August 2026 until 2 December 2026. Watch emotion analytics. Article 5(1)(f) has prohibited inferring employees' emotions since February 2025, and the Commission's guidelines cite call-centre voice monitoring of staff as an example. Emotion recognition on customers is not prohibited but is an Annex III high-risk use, deferred to 2 December 2027, while the Article 50(3) duty to inform the people analysed already applies. Our agent platform reference architecture shows where these controls sit.

GDPR and German criminal law. Recordings, transcripts, traces and eval sets hold personal data: document a legal basis and retention per store. Voiceprint authentication is biometric data for unique identification under Article 9 GDPR. In Germany, § 201 StGB makes recording the non-public spoken word without authorisation a criminal offence: the consent announcement is not a formality.

PCI DSS. Spoken card numbers end up in audio, transcripts and LLM context. Keep them out with DTMF masking or a dedicated payment flow, as the PCI SSC's telephone-payments guidance recommends.

A pragmatic adoption path

  1. Pick one flow and instrument it. A high-volume, low-risk call type such as order status. Trace every hop and record p50 and p95 voice-to-voice on real phone calls, not WebRTC.
  2. Set the budget before choosing vendors. Write per-hop targets, then pick components that meet them in one EU region.
  3. Build the unglamorous paths first. Barge-in with context trimming, warm handoff, the AI disclosure and recording consent.
  4. Test both architectures on the same calls. Score task success, latency, false interruptions and transcript accuracy.
  5. Scale by flow, not by channel. Add call types only while containment and complaint rates hold; keep identity checks and payments on deterministic paths.

The verdict

Enterprise voice AI is production-ready in 2026. The models are no longer the bottleneck; the architecture around them is. Teams that give every hop a budget and an owner, build barge-in and handoff before polishing the persona, and choose cascaded or speech-to-speech per flow will ship agents callers stay on the line for.

If a voice pilot feels slow, our AI architecture team runs this per-hop latency and compliance review, and you leave with the budget filled in for your stack.

FAQ

What is a good latency target for an enterprise voice AI agent?

A median voice-to-voice latency well under one second before telephone network transit. Our planning budget is about 820 ms: 200 ms transport, 250 ms transcription and end-of-turn, 250 ms to the LLM's first token, 20 ms chunking and 100 ms to first TTS audio. The reference pipeline in the Pipecat primer comes to 1,293 ms. Track the 95th percentile too.

Should we use a speech-to-speech model or a cascaded STT, LLM and TTS pipeline?

Use a cascaded pipeline when you need reliable tool calling, an exact transcript, your own voice and model, and per-hop tuning. Use speech-to-speech, such as OpenAI's gpt-realtime family, Gemini Live, Amazon Nova Sonic or Azure Voice Live, when prosody, mixed-language callers and fewer moving parts matter more. You can run both, chosen per call flow.

What is barge-in and why is it hard?

Barge-in lets the caller interrupt the agent mid-sentence. The system must detect real speech rather than a cough or its own echo, stop playout, cancel the LLM and TTS work in flight, and trim the conversation history to what the caller actually heard. Skip the trimming and the agent believes it said things nobody heard.

Does the EU AI Act apply to voice bots in the contact centre?

Yes. Since 2 August 2026, Article 50(1) requires telling callers they are talking to an AI unless it is obvious. Providers of synthetic audio must mark it machine-readably under Article 50(2), with a grace period to 2 December 2026 for older systems. Inferring your staff's emotions is prohibited; emotion recognition on customers is Annex III high-risk from 2 December 2027.

Can we keep voice agent data inside the EU?

Largely, if you choose deliberately. OpenAI documents an EU SIP endpoint for its Realtime API, Azure Voice Live offers data-zone and regional variants of its realtime models, and AWS lists Nova 2 Sonic in Stockholm. Check every component separately: speech, model, recordings, transcripts and traces can each live somewhere else.

Can a voice agent take card payments over the phone?

Not by listening to card numbers. Spoken card data in recordings, transcripts or LLM context pulls all of them into PCI DSS scope. Hand the payment step to DTMF masking or a payment IVR and return to the agent with a token, as the PCI Security Standards Council's telephone-payments guidance recommends.

Sources

All sources checked on 15 November 2026. Vendor latency figures are the vendors' own and are measured under their own conditions. The ms targets in the diagram and in chart 2 are our planning values.

Tier A: primary, standards and regulatory

Tier B: research and analysis

Tier C: vendor and news, directional

Topics

voice AI agent architectureenterprise voice agentvoice agent latency budgetspeech-to-speech vs cascaded pipelinebarge-in voice botcontact centre AIrealtime API SIPEU AI Act Article 50 voice

Frequently Asked Questions

A median voice-to-voice latency well under one second before telephone network transit. Our planning budget is about 820 ms: 200 ms transport, 250 ms transcription and end-of-turn, 250 ms to the LLM's first token, 20 ms chunking and 100 ms to first TTS audio. The reference pipeline in the Pipecat primer comes to 1,293 ms. Track the 95th percentile too.

Expert engagement

Need expert guidance?

Our team specializes in cloud architecture, security, AI platforms, and DevSecOps. Let's discuss how we can help your organization.

Get in touchNo commitment · No sales pressure

Related articles

All posts