Voice AI is learning when to speak

How does a voice assistant know when you're finished speaking?

A pause could mean you're done. It could mean you're searching for a word. Answer too early and it talks over you; wait too long and the conversation drags.

A basic voice assistant transcribes what I say, sends the words to a language model, and reads the answer aloud. Each step can work well on its own while the conversation still feels awkward. OpenAI calls this a chained approach. Newer systems can handle incoming and outgoing audio together, which makes interruptions possible. They still have to decide when to take their turn.

I start with Block because it is asking what a model can learn from speech itself. Then I move to the app I use and the companies building different parts of a voice conversation.

Block and Dessa

Jack Dorsey's Block introduced Dessa, a research model trained on spoken conversations without written transcripts or a pretrained language model. It can listen while it speaks. In Block's demos, it laughs and makes brief sounds of agreement during another person's turn. That matters because a transcript misses timing and tone. Dessa is still a research preview with limited access, so a demo cannot show how reliably it handles everyday calls.

OpenAI and ChatGPT Voice

ChatGPT Voice is my daily reference point. I can ask something, change my mind midway, and interrupt an answer. OpenAI also gives developers a choice: use a realtime voice model for the conversation, or combine transcription, a language model, and speech generation. The chained approach gives developers a transcript to inspect when something goes wrong.

ElevenLabs

ElevenLabs offers the pieces for building a voice agent: speech recognition, a language model, and a spoken reply. Its conversation controls let developers adjust how long an agent waits through silence and whether a person can interrupt it. Those settings answer a familiar question from my ChatGPT calls: was I finished, or just thinking?

Alibaba and Qwen

Qwen3.8-Omni-Flash-Realtime can take live video as well as audio. Point a camera at an object and ask a question, and the assistant has to connect what it hears with what it sees. Qwen also provides a desktop harness for trying that interaction. The model has to know which object I mean, even if I never name it.

ByteDance

Seeduplex focuses on when to answer. ByteDance says the model listens while speaking and uses more than a moment of silence to decide whether a person has finished. It also aims to avoid reacting to background voices. ByteDance reports fewer early replies in its own tests; I would want to hear how it handles ordinary pauses and interruptions before judging that claim.

MiniMax

MiniMax Speech 2.8 tackles the sound of the reply. It can generate expressive speech, including breaths and hesitations, and its API can stream audio as it is made. That could make an answer pleasant to hear. It does not, by itself, tell the assistant when to speak or whether the answer is right.

When I try a voice assistant now, I pay attention to the pause before it answers, what happens when I interrupt, and whether it still understands after I correct myself. I can forgive an ordinary voice. I stop using the app when it keeps talking over me.