Who Said That? Multiperson Detection and ID Tracking for Real-Time Voice Conversations

Picture a real call. Robert, a real estate client, phones in with his wife Arlene and their mortgage advisor on the line. Three people, one conversation, one voice AI trying to keep up. Arlene says, "We need the closing pushed to the 15th." The advisor says, "I can make that work if the paperwork lands by Friday." Robert says, "Fine, but I want the inspection report first."

Now the AI has to follow up. Who gets the inspection report? Who was promised what? The transcript says "you" fourteen times and every single one of them points somewhere different.

This is the failure mode nobody talks about. Every voice AI on the market is built for a two-party fiction: one human, one agent. The moment a second human joins the call, the model's mental model of the conversation degrades. With three or more people, it collapses. Speaker labels go missing or get invented. Commitments attach to the wrong person. The agent asks Robert a question meant for Arlene, and the humans on the line exchange a look that says: this thing is not ready.

The cost is not embarrassment. It is misattributed action. A follow-up email goes to the wrong stakeholder. A verbal agreement gets logged under the wrong name. In regulated or high-stakes conversations (real estate closings, financial advice, medical intake), a wrong "you" is a liability event, not a comedy beat.

The industry's current answer is diarization: chop the audio into segments and label them "Speaker 1," "Speaker 2," "Speaker 3." That is a transcript feature, not an identity system. It tells you that somebody spoke. It does not tell you who they are, what they want, or what they were promised ten minutes ago. We need to go further: not just who spoke, but who they are, tracked continuously, with a confidence score the agent can act on.

Human beings solve this problem without thinking about it. You hear a new voice on a call and within two seconds you have a working hypothesis: male or female, roughly what age, a familiar cadence or a stranger. As the conversation continues, that hypothesis hardens. The voice you could not place at minute one is unmistakable by minute ten. You are running an identity model the whole time, updating it on every turn.

A voice agent should do exactly this. The mechanism is an attribute ladder: a sequence of increasingly specific signals, each one confirming and reinforcing the last.

Rung one: the fast first cut. Gender is the cheapest high-value signal in the stack. Male versus female voices separate cleanly on fundamental frequency and formant structure, and a classifier can make that call in under a second of speech with very high accuracy. This is not the whole answer. It is the opening move. On a call with one man and one woman, it resolves the majority of turns by itself. On a call with three men, it resolves nothing, which is why it is rung one and not the ladder.

Rung two: the voiceprint. Pitch range, timbre, speaking rate, pause patterns, accent markers. These are the features a speaker embedding captures: a compact numerical signature of how a person sounds, computed per turn and compared against the signatures already seen in the call. Two turns with highly similar embeddings are probably the same person. Two turns with very different embeddings are probably not. This is where identity starts to persist across a conversation rather than being re-guessed from scratch every time someone opens their mouth.

Rung three: the language fingerprint. People do not just sound different; they talk different. Vocabulary choice, sentence length, filler words, how they hedge ("I think maybe" versus "here is the number"), domain language (the advisor says "amortization," Robert says "the monthly thing"). A per-speaker language model, even a lightweight one, adds a signal that is independent of acoustics. When two brothers with similar voices are on the same call, and acoustics cannot separate them, the fact that one talks like an accountant and the other talks like a contractor can.

The confidence score. No rung is trusted blindly. Every speaker turn gets an identity assignment with a confidence score: high when the acoustic and language signals agree and match a known participant, low when they conflict or when the voice is new. The agent's behavior keys off that score. High confidence: act normally, attribute commitments, address the person by name. Medium confidence: proceed but avoid name-specific claims. Low confidence: ask. "Sorry, was that Robert or Arlene?" is a perfectly good sentence, and it is infinitely better than confidently attributing Arlene's request to Robert. The protocol is not "never be wrong." It is "know how sure you are, and behave accordingly."

Two speakers is a solved-ish problem. Three is where production systems go to die, and the reasons are specific.

Overlapping speech. Humans interrupt each other. On a three-person call, two people routinely talk at once: Arlene starts answering while the advisor is still finishing. Naive diarization assigns the overlapped segment to one speaker and silently drops the other. The dropped half-sentence is often the important one ("but not before the inspection" vanishing into crosstalk). A real system must detect overlap, flag it, and either separate the streams or mark the turn as contested rather than guessing.

Similar voices. Family members share vocal tract genetics. Two brothers, a mother and daughter, business partners of the same age and background: their embeddings cluster close together. A threshold tuned for strangers will merge them into one speaker; a threshold tuned for family will split one stranger into two. The fix is adaptive: the system should tighten or loosen its similarity threshold based on how many distinct speakers the conversation seems to contain and how the confidence scores are trending, not use one fixed number for every call.

Speakers entering and leaving. "Sorry, I have to jump to another call, Arlene will take it from here." Now the participant set changed mid-conversation and the agent's speaker inventory is stale. Or the reverse: someone joins twenty minutes in ("Sorry I am late, what did I miss?") and the system has no voiceprint for them. Identity tracking has to be dynamic: enroll new voices on the fly, retire departed ones, and never confuse the newcomer with someone who left. The language fingerprint helps here too. The advisor who just joined talks like an advisor, not like anyone already on the call.

Short turns. "Yes." "Right." "Mm-hm." Backchannel utterances carry almost no acoustic or linguistic signal, and on a multi-person call they are constant. Assigning every "mm-hm" to a speaker with high confidence is a fool's errand. The right move is to assign them cheaply (probably the person who has been driving this topic) and mark them low-confidence, so they never anchor an important attribution.

The paper becomes an offering the moment speaker identity is something the agent can act on rather than something the transcript merely records. Three capabilities, in order:

1. Who said what. Per-speaker attribution on every turn of every call, with confidence scores. The call summary stops saying "they agreed" and starts saying "Arlene agreed to Friday; Robert asked for the inspection report first." This alone is worth money to any business that takes commitments on calls.

2. Per-speaker memory. Once you know who is who, each participant gets their own memory thread: what they asked for, what they were promised, what they prefer. Arlene calls back next week and the agent knows she is the one who cares about the closing date, not the inspection. Identity is the key that unlocks continuity.

3. Identity confidence as a control signal. The agent modulates its own behavior on the score: name people it is sure about, hedge when it is not, ask when it must. This is the difference between a system that occasionally embarrasses itself and one that degrades gracefully. People forgive "who am I speaking with?" They do not forgive being called by the wrong name after a twenty-minute conversation.

Who buys it. Service businesses that live on multi-stakeholder calls. Real estate teams, mortgage brokers, financial advisors, home-service companies where the spouse joins the estimate call, clinics doing intake with a patient and a family member. Anywhere the current answer is "Speaker 2 said..." and somebody has to reconstruct who Speaker 2 was.

Price shape. This ships as an attribution layer on top of the voice line, not a standalone offering. Base voice line carries two-speaker handling as standard. Multiperson tracking is a per-line add-on: a base monthly per-line figure (Michael sets it) for up to five concurrent identified speakers, with per-speaker memory as the upsell tier. The wedge is the call summary. Nobody buys "diarization." Everybody buys "never lose track of who promised what on a call again."