Building a Production Voice AI Stack: Architecture, Latency & Real-Time Orchestration

Voice AI demos are easy to build. Production voice systems are not. The difficult part is keeping audio, models, tools, state and interruptions coordinated while everything is happening in real time.

The real problem with Voice AI is not the voice

From the outside, a voice agent seems straightforward: listen to the user, understand what they said, work out what to do, and say something back.

Underneath that simple interaction is a distributed realtime system. Audio is moving continuously. Transcripts can change while the user is still speaking. Models may be generating responses while external tools are running. The user can interrupt at any point. Meanwhile, the client still has to play audio smoothly without jitter or buffering clicks.

This is why connecting an ASR provider, an LLM and a TTS provider together is only the beginning. The engineering challenge is coordinating those pieces without making the conversation feel like an awkward collection of discrete API calls.

Two ways to architect a modern Voice AI system

There is no single architecture that makes sense for every voice application. In practice, production systems converge on two broad patterns:

1. The Chained Architecture: The traditional approach separates the system into specialized, swappable components:

Audio In
  ↓
Speech Recognition (Streaming ASR)
  ↓
LLM / Reasoning Layer
  ↓
Tools & Application Logic (APIs, CRM, DB)
  ↓
Speech Synthesis (Streaming TTS)
  ↓
Audio Out

This approach remains useful because each layer can be controlled, evaluated, and replaced independently. Teams can choose their own specialized transcription provider, reasoning model, voice engine, and business logic. It is also straightforward to inspect: you can analyze transcripts, inspect model tokens, examine API payloads, and listen to generated audio as separate, traceable stages.

2. Full-Duplex Realtime Voice: Newer voice systems increasingly treat conversation as a continuous audio interaction rather than a sequence of completed turns. A full-duplex model listens while speaking, handles interruptions natively, and maintains conversational timing without requiring every utterance to cross discrete text-to-speech and speech-to-text boundaries.

This changes the architecture considerably. The conversational voice path stays focused on keeping interaction fluid and responsive, while deeper reasoning, tool execution, and database lookups happen asynchronously when needed.

1. Audio transport and voice activity detection

Everything starts before an AI model processes a single token. The system first has to capture raw microphone audio, transport it reliably across unpredictable networks, and accurately determine when human speech is occurring.

Depending on the client environment, transport typically involves WebRTC data/media tracks for browser and mobile apps, or SIP/WebSockets for telephony integrations.

VAD is more important than it looks.

Voice activity detection (VAD) determines whether incoming audio contains speech. In theory this sounds simple, but real-world audio is messy. Callers breathe, pause mid-thought, say "um", speak softly, or cough. Street noise, sirens, and background voices can easily trick an untrained detector.

A VAD or endpointing algorithm that is too aggressive will cut users off mid-sentence. One that waits too long introduces 600–900ms of dead air after every turn, making the assistant feel sluggish and unresponsive. Endpointing alone can materially change the perceived speed of an AI assistant.

2. Streaming speech recognition

In a real-time conversational agent, waiting for an entire utterance to finish before transcribing defeats the purpose of streaming. Instead, the speech recognition engine must emit partial results while the user is still vocalizing, followed by a committed final transcript once silence is confirmed.

Domain vocabulary matters.

General-purpose speech recognition works remarkably well until customers speak company-specific jargon, product SKUs, foreign names, or internal system identifiers:

"I want to cancel my VectoRise Core subscription."

If the recognizer transcribes "VectoRise Core" as "vector eyes core", the downstream LLM may hallucinate a response based on an inaccurate entity. Keyterm prompting, specialized vocabulary dictionaries, and phonetic fuzz-matching against catalog databases are essential safeguards before executing business operations.

3. The reasoning layer is not just an LLM

Calling the middle tier of a voice agent "the LLM" obscures most of the engineering required for production stability. The reasoning layer must continuously orchestrate:

• Intent classification and conversational goal tracking
• Multi-turn context and session state preservation
• Decision logic on whether more information is needed before taking action
• Executing external API integrations and validating return payloads
• Enforcing safety and verification boundaries on sensitive actions
• Handling users who change their mind or contradict earlier statements mid-conversation
• Formatting responses specifically tailored for spoken audio rather than screen display

Voice changes how an LLM should respond.

A response written for a monitor can comfortably display bullet points, hyperlinks, and complex tables. Spoken aloud, that same output is exhausting and unnatural. Production voice prompts enforce succinct phrasing, conversational cadence, single-idea clauses, and verbal confirmations.

4. Tool calls are where voice agents become useful

A voice agent that can only recite FAQ answers is a glorified IVR. A voice agent that can act in backend systems delivers genuine operational leverage. Typical enterprise workflows include:

• Checking order tracking and delivery status in real-time
• Scheduling, modifying, or cancelling calendar bookings
• Updating customer records in Salesforce or HubSpot
• Authenticating account credentials via multi-factor SMS
• Creating and prioritizing support tickets in Zendesk or Jira
• Verifying warehouse stock and processing inventory reservations
• Triggering internal dispatch and operational workflows

The fundamental challenge is that enterprise APIs are rarely realtime. A CRM lookup may take 250ms, an inventory database query may take 1.8 seconds, and an external payment gateway may occasionally timeout altogether. The voice layer must never stall the conversation while waiting for slow backend services.

5. Streaming text-to-speech

Once the reasoning layer begins producing tokens, the synthesizer must convert text into audio bytes without waiting for the complete sentence. Waiting for full model completion adds hundreds of unnecessary milliseconds of latency.

Modern neural TTS engines stream audio via WebSockets. However, the orchestrator faces a delicate balance: chunking text too early (1–2 words) produces fragmented, robotic prosody because the synthesizer lacks context. Chunking too late (full sentence) delays first-audio playback.

6. Barge-in: when the user talks over the assistant

Barge-in is the single clearest test separating amateur prototypes from enterprise-grade voice systems. Consider a typical interaction:

Assistant: "Your appointment is scheduled for Thursday at two in the afternoon..."
User:      "No, Thursday doesn't work. Make it Friday."

Naive implementations continue playing the original audio stream while simultaneously attempting to parse the interruption. The result feels deaf, clumsy, and unnatural. True barge-in requires coordinated cancellation across the entire software stack:

1. Detect incoming user vocalization via VAD within 20–50ms
2. Immediately instruct client audio drivers to stop and flush playback buffers
3. Send cancellation frames to the active TTS synthesis WebSocket
4. Cancel in-flight LLM token generation to avoid wasted compute and credit spend
5. Resolve whether in-flight tool calls should be aborted, rolled back, or allowed to complete in the background
6. Accurately reconcile conversation history based on the exact words the user heard before interrupting

That last point is critical: the model may have generated 20 words, the synthesizer may have buffered 12 words, but the user only heard 5 words before speaking. If your conversation memory includes the unplayed words as "spoken context", state desynchronization is guaranteed.

7. Latency is a budget, not a single number

Claiming "our voice agent has 300ms latency" is meaningless without specifying what was measured. In conversational audio, response speed is the sum of multiple distributed pipeline stages:

Crucially, these stages do not execute sequentially. While the user is speaking, ASR is streaming frames. While turn endpointing resolves, context is pre-fetched. While the LLM generates tokens, early words stream into TTS. The architecture operates as a concurrent pipelined system.

8. What happens when something goes wrong?

Production systems are defined not by their happy paths, but by how gracefully they recover when dependencies fail:

• ASR mishears an essential name or address: The agent should seek gentle clarification rather than committing a faulty transaction.
• CRM API takes 3+ seconds to respond: The voice layer uses filler phrases or proceeds asynchronously rather than freezing in dead silence.
• User interrupts in the middle of a database commit: The orchestrator evaluates idempotency keys and state safety before aborting.
• TTS provider WebSocket disconnects: The client immediately falls back to a secondary synthesis provider or cached prompts.
• Client network packet drops: Audio transport reconnects seamlessly without resetting the conversation session.

9. Observability: measure the conversation, not just the APIs

Standard application APM metrics (like HTTP 200 counts) are useless for voice agents. A server may register 100% API success while users endure a frustrating, disjointed conversation. True voice observability tracks interaction telemetry at the session level:

• Endpointing duration and false-positive cut-off counts
• P50, P90, and P99 Time-to-First-Audio latency distribution
• Interruption and barge-in frequency per conversation
• Token generation cancellation rates and wasted compute costs
• ASR confidence scores and entity correction frequency
• Tool execution failure rates and retry volumes
• Task completion rate versus human agent transfer/escalation rate

Monitoring latency distribution is vital: an average response time of 400ms can easily conceal a P99 tail where 5% of users wait 3.5 seconds in awkward silence.

10. Security and data boundaries belong in the architecture

Voice callers frequently speak sensitive data they would hesitate to enter into a web form: national identifiers, card numbers, account details, and private personal circumstances. Security must be baked into the architecture from day one:

• Mutual TLS encryption for all SIP, WebRTC, and WebSocket transports
• Zero-retention policies for raw audio recordings containing regulated data
• Granular tool-level authorization ensuring agents cannot exceed least privilege
• Human-in-the-loop confirmation gates for financial or consequential actions
• Prompt injection safeguards against malicious voice jailbreaks and tool exploits
• Audit-ready tamper-evident event logs for all agent actions and decisions

11. When voice is the wrong interface

Voice is not universally superior simply because it is conversational. Good software engineering is about choosing the right interface for the job. A screen is vastly better when users need to:

• Review long, detailed legal documents or contracts
• Compare dozens of multi-attribute products side-by-side
• Enter complex alphanumeric strings (like serial keys or tracking numbers)
• Inspect rich graphical dashboards, charts, and visualizations

Voice excels when users are mobile, hands or eyes are occupied, workflows require conversational clarification, or when routine Tier-1 inquiries can be resolved immediately without hold times.

What actually makes a Voice AI system production-ready?

The difference between a flashy demo and a resilient enterprise voice system lies entirely in the engineering around the model:

Conclusion: Build the system around the conversation

Voice AI has progressed far beyond connecting an off-the-shelf speech-to-text API to an LLM and reading the result through text-to-speech. While that chained approach offers valuable modularity and control, full-duplex architectures are transforming conversational fluidity.

The true engineering challenge sits in the middle: the orchestration layer that preserves state, the transport layer that streams audio reliably, the cancellation logic that enables seamless barge-in, and the validation boundaries that prevent bad transcripts from triggering catastrophic transactions.

At VectoRise, we treat Voice AI as an infrastructure and systems engineering problem. The goal is always the same: build an AI system that holds up when real users do real things with it.

Frequently asked questions