Voice & real-time AI

Talking to AI instead of typing changes the user experience completely. The technology underneath has its own moving parts: transcription, real-time streaming, voice synthesis, and tight latency budgets.

What voice AI actually does

Voice AI takes the same language models you've used in chat and wraps them in audio: you speak, the system transcribes, the model thinks, the system speaks back. Done well, it feels like a phone call with a competent assistant. Done poorly, it feels like an IVR robot.

The whole game is latency — the gap between you finishing your sentence and the model starting to speak. Under ~300ms feels human; over a second feels broken. Everything else — accent handling, voice cloning, interruption — comes second to that.

That's the whole concept. Below: the two architectures used today, where the latency budget goes, and the consent/scam implications of cheap voice cloning.

When you'd reach for it

When the interface should not be a keyboard:

• Phone-based customer service — replacing IVR menus with conversational agents.
• Hands-busy contexts — driving, cooking, accessibility.
• Field workers — warehouse, healthcare, on-site technicians using voice instead of forms.
• Anything that benefits from tone — companions, tutors, therapy adjuncts.

You wouldn't reach for voice when the user wants precision (writing code, editing a contract) or when a transcript is needed (voice has noise; text doesn't).

How it's actually built

A voice AI conversation, simplified:

That's the classic pipeline — three model calls glued together. It works, but each stage adds latency. Naive implementations hit 3–6 seconds end-to-end, which feels broken.

Modern real-time multimodal models (OpenAI Realtime API, Anthropic voice mode, Gemini Live) collapse the pipeline into a single end-to-end model that takes audio in and produces audio out, with sub-second latency. They handle interruptions, "uhh"s, and tone — all the things that make a conversation feel natural.

Where it bites in real life

The hard parts of voice AI

Latency budget

For natural conversation, you have ~300–500ms total between the user finishing speaking and the model starting to reply. Distributed across:

• Voice activity detection (VAD): is the user done talking? ~50ms
• Speech-to-text: ~100–200ms
• LLM inference (TTFT): ~150–400ms
• Text-to-speech start: ~100–200ms

Modern unified models cut several of these by working directly on audio, but the budget is unforgiving.

Turn-taking and interruptions

A real conversation has interruptions, overlaps, "mhm"s, false starts. Voice AI has to detect when the user is actually done vs. just pausing, and stop talking when interrupted (rather than ploughing through). Cheap implementations break here; expensive ones don't.

Domain accuracy

Whisper-class transcription is excellent for general speech but degrades on: heavy accents, specialised vocabulary (medical, legal, technical jargon), low-quality phone audio, and overlapping speakers. Production deployments often combine general transcription with domain-specific lexicons.

Voice cloning and authenticity

You can clone a voice from minutes of audio. ElevenLabs and similar tools make this trivial. Major implications: scams (deepfake calls from "your boss"), accessibility wins (people who lost their voice keeping it), creative misuse (impersonating public figures). Vendors are adding watermarking and consent requirements, but the genie is out of the bottle.

Architecture choices

Pipeline (Whisper → LLM → TTS)

• ✅ Mix and match best-in-class components.
• ✅ Easier to debug each stage.
• ❌ Higher latency (sum of three models).
• ❌ Loses tone, emotion, "uhh" handling between stages.

Unified real-time model (OpenAI Realtime, Gemini Live)

• ✅ Sub-second latency, natural conversation.
• ✅ Handles tone, interruptions, non-verbal sounds.
• ❌ Less customisable.
• ❌ Newer, fewer providers, more expensive per-minute.

For most production deployments today, a hybrid: real-time for the conversation feel, with pipeline components for specialised tasks (e.g. medical-grade transcription).

Under the hood (optional)

A skeleton call to OpenAI's Realtime API: open a streaming connection, set instructions and voice, then loop on incoming audio events. About 15 lines. Skip if you don't code.

import asyncio
from openai import AsyncOpenAI
 
async def voice_chat():
    client = AsyncOpenAI()
    async with client.beta.realtime.connect(model="gpt-4o-realtime-preview") as conn:
        await conn.session.update(session={
            "modalities": ["audio", "text"],
            "instructions": "You are a friendly receptionist. Be brief.",
            "voice": "alloy",
        })
        # Stream audio in (from microphone) and audio out (to speakers).
        # The library handles VAD, turn-taking, interruption.
        async for event in conn:
            if event.type == "response.audio.delta":
                play_audio_chunk(event.delta)

Try it yourself (~5 minutes)

Try voice AI yourself — there are free options:

1. ChatGPT mobile app: tap the headphone icon. Have a conversation. Try interrupting mid-sentence.
2. Gemini Live: in the Gemini app, similar feature.
3. Claude voice (where available): Anthropic has been rolling this out — check your region.

Notice: how natural does interruption feel? Can you hear emotion? How long does the model wait before answering? These are the design choices that distinguish products.

Further reading

• OpenAI — Realtime API guide — official guide, with audio examples.
• Whisper paper (Radford et al. 2022) — the most cited transcription model.
• ElevenLabs — voice cloning & TTS — the leading voice synthesis platform.
• LiveKit Agents — open-source framework for building real-time voice agents.
• Pipecat — open-source voice AI orchestration framework.