Voice agents demo well and deploy badly. The distance between "it answered my question in a quiet room" and "it handled a real customer on a real phone line" is where most of the engineering lives, and until recently there was no shared way to measure it.
τ-Voice measures a whole call, which is the part almost nothing else tests. Its tasks run up to twenty minutes of conversation and the hard ones take dozens of exchanges to get through, and nothing counts until the job is actually done at the end of it. A benchmark built on single exchanges cannot tell you whether an agent falls apart in minute eleven.
A benchmark that behaves like a phone call
It measures a whole call, not a moment
τ-Voice extends τ²-bench into voice. It keeps 278 customer-service tasks across retail (114), airline (50) and telecom (114), and keeps the tasks, tools and policy documents almost byte-for-byte identical to the text leaderboard. That identity is the point: it makes the voice–text gap directly measurable rather than a matter of opinion.
The gap is large. The τ-Voice paper reports GPT-5 (reasoning) at 85% on the text version of these tasks, while the voice agents it measured reached 31–51% under clean audio and 26–38% under realistic audio — "retaining only 30–45% of text capability." Most of those failures, the authors find, come from the agent rather than from the simulated caller.
Buying a better opponent
The benchmark runs on its own clock rather than a real one. Both sides stream audio continuously and can talk over each other, but a step in the simulation doesn't have to finish in the time it would take on an actual phone line.
That sounds like a technicality and isn't. It means the simulated customer can use a frontier model to decide when to cut in, when to hold the floor, when to drop in an "mm-hm" — none of which has to happen in real time. Build the same thing on a real clock and you are forced into a fast, weak simulator, which leaves you measuring your agent against a bad conversation partner instead of a hard one. Here the customer is as sharp as the best model available, and the clock waits.
Difficulty comes from everywhere
The difficulty doesn't come from one place. Three that we run into constantly:
The audio. Diverse accents, continuous background noise, burst events about once a minute, packet loss, dynamic muffling, and the band-limiting every phone call gets. Coughs, sneezes and sniffles land in the middle of utterances. Everyone who has run phone traffic recognizes this list. It is not adversarial noise designed to break models; it is Tuesday.
The person. Callers vary in accent, age and temperament — elderly, impatient, prone to escalation. In telecom they also vary in competence: the "Hard" persona is a 64-year-old who finds technology overwhelming, for whom "APN" may as well be a foreign language, and who — the line that actually hurts — only shares information when specifically asked. Some checks only they can run, so the agent has to talk them through it — and they can still get it wrong. The simulator also produces speech that is not addressed to the agent at all ("Hold on a second.", "I'm on the phone."), low-pass filtered as if the speaker turned away from the handset.
The task. Real policy documents, real database state. Some tasks are a single lookup. Others require reading a policy correctly and refusing something the customer wants. And some stack a half-dozen independent faults that must each be found and cleared before the customer's problem goes away.
Did the job get done, and did the conversation work
Two things, kept separate.
Task success is pass^1, and it is mostly not a matter of opinion: it is checked against the state the world was left in — the record in the database, the setting on the simulated handset. Either the refund was issued against the right order, or it was not. One voice-specific trap: if the call ends for any reason other than a clean stop, the score is zero before any evaluator runs. Correct but slow fails exactly like wrong.
Interaction quality is scored separately, from the recording rather than from a judge's opinion: how often the agent answers and how fast, how quickly it gets out of the way, how often it talks over the caller, and — the number we'd highlight — selectivity, meaning what it correctly ignores. Optimizing for responsiveness is easy and mostly fake, since you can just answer everything. What an agent declines to answer is what separates a conversation from a hair trigger.
Reproduced, not reported
Submissions are not taken on trust. You provide full trajectories and audio that pass validation; the maintainers recompute the interaction metrics from your trajectories rather than accepting your numbers; and then they re-run the system themselves, with their own voices, so that published results are comparable across entries. Only then does anything appear on the board.
What a hard task looks like
Nine things wrong with one phone
A 64-year-old customer, travelling in France, cannot send picture messages. He wants that fixed. The benchmark has broken nine things on the way there.
Airplane mode is on, the SIM is unseated, the network is pinned to 2G, mobile data is off, Wi-Fi calling is misconfigured, the messaging app is missing permissions, the APN is broken — and two more that are not on the phone at all: roaming is disabled on the account, and the line is over its data cap.
The score is a single bit: at the end, can the phone send an MMS? Fixing eight of the nine scores exactly the same as fixing none.
Where agents typically lose it: order, because some fixes do not take effect until the device reboots or re-searches for the network, so a correct configuration applied in the wrong sequence still fails; the account side, because an agent focused on the handset never thinks to check whether the carrier has roaming switched off; and the policy, because the data top-up has a cap that exists only in the policy document and is not enforced by the API, so an agent that tops up a sensible-looking amount passes every check the system performs and still fails the task. All of it has to be pulled out of a customer who volunteers nothing and is anxious about losing his photos.
Five reservations, one forbidden shortcut
A customer has five reservations. She wants every trip containing a flight longer than four hours cancelled, and every trip of three hours or less upgraded to business — and she wants the total upgrade cost before anything is charged. She is busy, so the agent has to work out which is which.
Three reservations qualify for the upgrade; the total comes to $1387, and the agent has to say that number out loud.
Where agents typically lose it: one reservation does contain a long flight, so the customer wants it gone — but it turns out not to be cancellable, and the agent has to work that out and say no. One reservation is already in the past and simply must not be touched; the database check is exact, so one stray write fails the whole task. And there is a shortcut: upgrade the uncancellable trip to business first, and business fares are cancellable. The customer pre-emptively forbids it. An agent clever enough to find that path fails for finding it.
Where we landed
Sierra re-ran our system independently and published their own numbers. On the standard leaderboard — every entry driven by the benchmark's own user simulator — this is where things stand:
Source: τ³-Voice leaderboard.
We submitted a second run as well. The benchmark's standard user simulator is driven by GPT-4.1; in this one we swapped that single model for GPT-5.5 at its highest reasoning effort and changed nothing else — same voices, same audio pipeline, same turn-taking policy, same tasks.
Source: τ³-Voice leaderboard.
Because it changes the simulator, that row is classified as custom and is not comparable to the standard entries — including our own. It is a stress test, not a ranking. The first row is the one on the standard board, classified as standard.
Infrastructure errors are counted as failures rather than dropped from the denominator — one of them in the standard run, four in the GPT-5.5 run, all in telecom.
Worth stating plainly: nothing here was trained on the benchmark. Every model in the path — recognition, the conversational model, the reasoning model, synthesis — is the same one serving production traffic at 19pine.ai. We did not fine-tune for these domains, and no file in the benchmark repository was modified.
What we built
Our production system is a full-duplex, streaming voice agent, built for the long, difficult conversations that happen in a real-world setting. Three things have to be true at once, and they pull against each other. The conversation runs in real time and can't pause while the system thinks. The task still has to get done, accurately, while that conversation is going on. And real audio carries far more than words, so understanding and generation both have to reach past the linguistic channel.
So the agent listens and speaks in real time while it reasons and acts in parallel, and it recognizes and generates prosody, background noise and the other paralinguistic cues that are essential to natural conversation.
Full-duplex, streaming architecture
Turn-based systems — where one side waits for the other to finish before it starts composing a reply — are a poor match for how people actually talk. Speakers overlap. They project the end of a turn before it arrives. They drop in short continuers that encourage the other speaker to keep the floor rather than yield it. An agent that can't do the same either jumps in too early or leaves audible silence where a competent participant wouldn't.
The system streams audio continuously in both directions. Our model takes the caller's speech and its own past speech as input and produces speech in real time. It doesn't wait for a turn to finish before responding, and it can interrupt itself, backchannel, or yield the floor mid-utterance — which is what holds the conversation together through overlap, interruption and background noise.
Hearing its own voice is what makes floor management coherent. Whether to yield mid-sentence depends enormously on whether you're three words into a clarifying question or two-thirds of the way through reading back an account number. Same audio from the caller; opposite right answer.
This architectural class isn't ours alone. It's shared with frontier real-time voice systems such as OpenAI's GPT-Live and xAI's Grok Voice. We treat it as necessary rather than sufficient.
Splitting the talking from the thinking
The most useful voice agents need two things no single model currently does well: the sub-second responsiveness that keeps a conversation natural, and the long reasoning that gets work done. A fast model answers on time but can't think. A reasoning model thinks but can't answer on time. Neither is enough on its own, and no amount of tuning collapses the tradeoff.
So we run two models concurrently over shared context.
The fast-thinking model conducts the live conversation. It listens, thinks and speaks at low latency, and because it hears both the caller's audio and its own, it maintains a unified representation of the exchange rather than a one- sided one. Its work is the surface of the interaction: tracking pauses, hesitations, interruptions, turn-taking cues, and the short particles like "uh- huh" and "oh" that give a conversation its moment-to-moment texture.
The slow-thinking model runs alongside it and carries none of that timing burden. Freed from the sub-second deadline, it can follow the discussion at depth, work out what the customer is actually after, invoke whatever tools the task requires, plan multi-step actions, and manage the slow asynchronous operations that real work involves. It writes back through the shared context, so what it concludes can extend, redirect or correct whatever the fast model is in the middle of saying. It can also start acting on a request before the caller has finished making it, so that when they stop, the answer is often already waiting.
Fast thinking keeps the conversation moving. Slow thinking gets the job done.
Hearing more than words
Recognition in production audio looks nothing like recognition on a curated corpus. Callers speak from moving cars, open-plan offices, kitchens and streets. They bring a wide range of accents, down microphones, headsets and phone lines of wildly different quality. Non-speech sounds turn up constantly — coughs, sneezes, keyboard clicks, background music, other voices in the room — and they often land right on top of the utterance the agent is trying to understand.
Our model is trained on that distribution rather than an idealized one. It handles partial occlusion of speech by concurrent sounds, and it distinguishes speech intended for the agent from other audio in the environment. Past the words, it picks up non-verbal events — laughter, sighs, coughs, background music — and the paralinguistic cues that carry emotional state, like frustration or amusement. Encoding all of that is what lets the agent respond to the situation the way a person would, acknowledging a noisy room or softening its tone for a frustrated caller, instead of either discarding the context or reacting to it in ways that break the conversation.
The hard part at this stage isn't accuracy on clean speech. It's the two failure modes at the audio edges: missing speech that was actually spoken, and hallucinating speech that wasn't. The second one is especially damaging in a full-duplex system, because fabricated text is indistinguishable from the caller genuinely talking — so the agent stops mid-sentence, or answers something nobody said.
It has to sound real
Speech that carries only the words is intelligible but not conversational. Human listeners lean on prosody — pitch, timing, loudness, voice quality — to work out intent, emotion, emphasis, and whether the speaker is about to hand over the floor or keep it. Flatten those dimensions and you get output that's technically correct and interpersonally wrong, and callers hear the difference immediately.
So synthesis treats prosody as a first-class signal rather than a byproduct of text. Tone, pitch and pacing are modulated according to conversational context, so a clarifying question, a confirmation, a piece of bad news and a routine acknowledgment don't all arrive with the same contour. Intonation does real work for turn-taking, emphasis and engagement: the same words can end on a rising or falling boundary depending on whether the agent is inviting a response or carrying on with its own turn.
And the small imperfections characteristic of spontaneous speech — brief hesitations, mid-sentence self-corrections, the occasional filled pause — stay in rather than being smoothed away. Their absence is itself a strong cue to a listener that the interlocutor is not a person.
What we're still working on
Thinking costs time we don't have
The fast-thinking model has to answer inside the timescale of a natural conversation, and it has to understand the exchange well enough for the answer to be worth hearing. Those two demands do not fully commute. Give a response more thinking and it gets better, but the lag becomes audible; cut the thinking budget and the timing holds, but the answer sometimes misses what the caller actually wanted. Both failure modes show up in our current system.
The problem isn't to make the model faster. It's to get more capability per thinking token, in the one place that can't afford to spend them freely. We're exploring ways to let the fast model's thinking budget flex with what the moment demands.
Crowded lines
Everything above assumes a two-party call. Plenty of real settings involve more than two speakers — meetings, conference calls, multi-party customer conversations — and the problems shift accordingly. Speaker diarization becomes central. Addressee detection becomes ambiguous. Turn-taking stops being a single handover and becomes a floor negotiated among several people. And the reasoning models have to maintain a per-speaker representation of intent and recent history.
Our current system only handles a small number of participants. Extending it to larger groups while preserving conversational quality and coherence is active work.
The obstacles that τ-Voice constructs are the ones we encounter in production: accents that push recognizers past their training distribution, callers whose replies address a different question than the one asked, background audio that is easily mistaken for speech, and tasks whose difficulty lies not in a single hard step but in a sequence of ordinary ones that must be executed in the correct order, over a degraded connection, with a caller who is not enjoying the conversation. These are not synthetic adversarial conditions engineered to defeat a model; they are the ordinary substance of production traffic. A benchmark that reproduces them makes progress on deployed voice agents measurable rather than a matter of opinion, and gives the field a shared target for the work that remains.
