Voice AI has matured from novelty to production-ready. Combining Whisper for speech-to-text with GPT-4 for understanding creates powerful voice interfaces. Here's the architecture, latency optimization, and UX patterns that work.
Rahul Sharma
Head of AI Engineering
A production voice AI system has four components: speech-to-text (STT), natural language understanding (NLU), dialogue management, and text-to-speech (TTS). Each component adds latency. The engineering challenge is minimizing total round-trip time while maintaining accuracy — users perceive responses over 1.5 seconds as "slow" and over 3 seconds as "broken."
The modern stack: Whisper (Open Source) or Deepgram for STT, GPT-4 or Claude for NLU and dialogue, and ElevenLabs or Azure Neural TTS for speech output. Each component can be cloud-hosted or self-hosted depending on your latency, cost, and privacy requirements. This is the stack our conversational AI development team assembles for most voice-first client projects.
Whisper comes in five sizes: tiny, base, small, medium, and large. For production, we use large-v3 for accuracy-critical applications (medical transcription, legal) and medium for general-purpose voice interfaces where latency matters more than perfect accuracy. The medium model achieves 92% of large-v3's accuracy at 4× the speed.
Self-hosting Whisper requires GPU infrastructure. We run Whisper on NVIDIA A10G instances with faster-whisper (a CTranslate2-based implementation that's 4× faster than the official implementation with identical accuracy). For cost optimization, we use spot instances with a fallback to on-demand instances when spot capacity is unavailable.
Users don't wait until they finish speaking before expecting a response. Streaming STT provides partial transcriptions as the user speaks, enabling the system to start processing before the utterance is complete. Whisper doesn't natively support streaming, so we use a chunked approach: buffer 3 seconds of audio, transcribe, then transcribe the next overlapping chunk. The final transcription stitches overlapping segments together.
Deepgram's streaming API is a better choice when streaming is a first-class requirement. It provides word-level timestamps and partial results with sub-300ms latency, which enables more responsive voice UX patterns.
GPT-4's role in a voice AI system is twofold: understanding user intent from the transcription and generating natural responses. The key is structuring the prompt to handle ASR errors — Whisper's transcriptions contain homophone errors, missing punctuation, and occasional hallucinations on silence.
We include the transcription confidence score in the prompt and instruct the model to handle low-confidence segments gracefully. For voice-first applications, we also include conversation history as a compressed summary (not raw transcript) to stay within context limits while maintaining conversational coherence.
TTS quality has improved dramatically. ElevenLabs produces near-human speech with proper prosody and emotion. Azure Neural TTS offers the best latency for real-time applications (sub-200ms time-to-first-audio-byte). For production voice AI, we use Azure for interactive applications where latency matters and ElevenLabs for content generation where quality matters more than speed.
Streaming TTS is essential — start playing audio while the rest of the response is still being generated. Both Azure and ElevenLabs support streaming audio output. The implementation: pipe the LLM's streaming text output to the TTS service in sentence-sized chunks, and play audio chunks sequentially.
Total voice AI latency has five components: audio capture buffer (200–500ms), STT processing (300–2000ms depending on model), LLM inference (500–3000ms for first token), TTS processing (200–500ms for first audio byte), and network round-trips. Optimizing each component:
Use VAD (Voice Activity Detection) to detect speech end before sending audio to STT — this eliminates the "thinking pause" latency. Use streaming STT with partial results to start LLM processing before transcription is complete. Use LLM streaming with sentence-level TTS chunking. The result: perceived latency drops from 4–6 seconds to 1.5–2.5 seconds.
Voice UX is fundamentally different from text UX. Three patterns that work: visual feedback showing the system is listening and processing (animated waveform), barge-in support (user can interrupt the system's speech), and confirmation for irreversible actions ("Did you say transfer $500 to John?"). The biggest mistake in voice AI UX is not handling silence and ambiguity gracefully — always have a fallback prompt that guides the user without sounding like an error message. Our NLP development team designs these fallback flows alongside the underlying language understanding pipeline, and clients building this in-house often hire AI developers with voice UX experience specifically.
Tags
Continue Reading
A practical guide to building AI agents that actually work outside of demos — covering tool selection, memory, error recovery, and deployment patterns we've validated in production.
Two approaches, very different trade-offs. RAG is faster, cheaper, and keeps your data current. Fine-tuning gives you style and behaviour control. Here's when each wins — and when to use both.
Most prompt engineering guides are written for demos. Production prompts face hallucinations, adversarial inputs, context length limits, and latency budgets. Here's what actually holds up.
From AI agents to mobile apps — iMOBDEV builds what your business needs.
