**Identify Each Speaker in Real Time: Create Multi-Speaker AI Applications Using NVIDIA Nemotron 3 Diarization**

If you build voice apps, meeting tools, or call-analytics pipelines, you already know that transcribing words is only half the job. Figuring out who said what—especially when people talk over each other—is what makes transcripts actually useful. This article walks through NVIDIA Nemotron 3 Diarization, a new open-weight, 100M-parameter model that ranked #1 in Voice Arena's initial Diarization-Bench results, and explains how it works, how accurate it is, and how to think about latency versus quality.

Why speaker diarization matters

Every conversation carries two layers of information: the words themselves, and who spoke them. Speech recognition (ASR) handles the words. Speaker diarization answers the second question—identifying which time intervals belong to which speaker, including moments when voices overlap.

Imagine a transcript of a meeting, customer call, or podcast where every sentence is accurate but nothing is attributed to anyone. You can read it, but you can't reliably tell who made a promise, who pushed back, or who cut whom off. Search, summaries, action items, conversation analytics, and voice-agent memory all lose much of their value without attribution.

NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that tops Voice Arena's Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). It supports up to eight speakers in both live and recorded audio, handles overlapping speech, processes audio in chunks so recordings of any length work, and lets you tune streaming latency.

Earlier models like NVIDIA Streaming Sortformer laid the groundwork for four-speaker diarization, including the diar_streaming_sortformer_4spk-v2.1 checkpoint used as the baseline in the evaluations below. Nemotron 3 Diarization doubles the speaker limit and improves both accuracy and throughput.

How Nemotron 3 Diarization works

One model for offline and streaming conversations

Diarization systems face two linked problems. First, detect speech and assign it to the right speaker. Second, keep that assignment consistent across the whole conversation—even through silences, interruptions, and long gaps between a speaker's turns.

Streaming makes the second problem much harder. An offline model sees the entire recording at once, but a streaming system only receives a small chunk of new audio plus limited context. Without a solid memory mechanism, the speaker assigned to one output channel in this chunk might get mapped to a different channel in the next.

Nemotron 3 Diarization follows the Sortformer approach: speakers are ordered by when they first appear. The first new voice becomes channel one, the next becomes channel two, and so on. This arrival-time ordering keeps the model's generic speaker labels stable and eliminates the need to re-solve a speaker permutation for every chunk.

The model was trained on public and licensed speech data, including multispeaker-annotated real-world conversations licensed from David AI. Additional licensed David AI audio provided source material for large-scale simulated English and multilingual mixtures covering 21 languages. Adding David AI data reduced compound diarization error rate by 0.77 absolute points, from 11.19% to 10.42%, at both the offline-style and ultra-low-latency operating points.

The model supports up to eight speaker channels. These are anonymous labels, not real identities: it can report that speaker_2 talked from one timestamp to another, but it cannot tell you who speaker_2 actually is. Downstream apps can map these channel IDs to real identities using meeting metadata, user profiles, or active-speaker verification models.

From audio to speaker activity

The model takes 16 kHz, single-channel audio and converts it into Mel-spectrogram features with a 10 ms frame step. Those features are stacked by a factor of eight into 80 ms frames, which feed a 31-layer Transformer encoder using rotary positional embeddings (RoPE).

Above the Transformer, a Conv1D layer upsamples predictions back to input-feature resolution. The default output is a [T, 8] floating-point tensor: T time steps by eight speaker channels, where each value is the probability a given speaker is active at that moment. The default stride is 10 ms and can be set to any multiple of 10 ms.

This design handles overlap naturally: when two people speak simultaneously, two channels can be active in the same frame. Postprocessing turns those probabilities into generic speaker labels with start and end timestamps.

During streaming inference, two forms of memory supply context:

• The Arrival-Order Speaker Cache (AOSC) keeps useful information about speakers seen in earlier chunks, organized by their arrival-ordered channels.
• A first-in, first-out (FIFO) queue provides recent frame context just before the current chunk.

The input buffer also includes right context—audio immediately following the current chunk. More right context helps the model interpret speaker transitions; less right context shortens the wait before results appear. Together, the current chunk, right context, FIFO queue, and speaker cache let one model operate at multiple latency points.

Chunked inference removes any fixed model-imposed maximum on audio duration. Performance can still drop on unusually long recordings, or audio with heavy noise, reverberation, far-field capture, or domain shift.

Diarization and speaker-attributed ASR are different tasks

Standalone diarization produces speaker activity and timestamps—not the words. ASR produces text but doesn't necessarily preserve attribution. A speaker-attributed transcription pipeline combines both outputs: diarization timestamps are merged with ASR text so each spoken word maps to a specific speaker.

This separation matters when designing your system. Diarization errors include missed speech, false detections, wrong speaker assignments, and boundary errors, while ASR errors affect the words themselves. Evaluate both components and the combined pipeline on your actual audio.

Balance latency and accuracy

The same model supports recommended input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds. Shorter buffers mean faster responses; more context generally means better accuracy and throughput. These values measure only the audio buffered before inference—computation, networking, ASR, and application processing all add to end-to-end latency. Technically the model can run with an 80 ms buffer, but 0.32 seconds is the lowest recommended configuration.

Benchmark results: ranked #1 in Voice Arena's initial Diarization-Bench

In Voice Arena's initial Diarization-Bench results, NVIDIA Nemotron 3 Diarization placed first among 12 systems (17 total configurations) evaluated on 139 English conversations totaling roughly 22 hours. With overlapping speech scored, system-generated speech activity detection, and no boundary collar, it achieved a 14.72% DER versus 19.3% for the next-ranked system—a roughly 24% relative reduction. It also ranked first with 100 ms and 250 ms collars, and across both in-person and online recordings. Note that these initial results may shift as Voice Arena completes its Version 1 evaluation and paired statistical analysis.

Measuring diarization accuracy

The primary metric is the diarization error rate (DER), which combines three error types:

• Missed speech: a reference speaker was active but the system detected nothing.
• False alarm: the system marked a speaker active when the reference contained no speech.
• Speaker confusion: speech was detected at the right time but assigned to the wrong speaker.

DER sums these errors and divides by total reference speaker time, with overlapping reference speakers each contributing to the denominator.

Benchmark settings can materially change DER, so the protocol is part of the result. The Nemotron 3 evaluation uses 901 condition-specific recordings spanning multilingual telephone speech, meetings, near- and far-field microphones, multi-microphone capture, and difficult acoustics. Overlapping speech is scored in every evaluation. DIHARD III, AliMeeting, AMI, and NOTSOFAR1 use a zero-second collar (no boundary tolerance excluded); CALLHOME-Part2 uses a 0.25-second collar. Results came from the NeMo e2e_diarize_speech.py evaluation script.

An average 40% relative DER reduction at 1.04-second latency

At 1.04-second input-buffer latency, Nemotron 3 Diarization lowers DER on all eight listed evaluation conditions, with relative reductions ranging from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The unweighted mean of the eight per-dataset relative reductions is 41.0%—that's an average of relative improvements across conditions, not a pooled DER computed over every recording combined.

The gains hold across operating points too: at every shared latency (30.4, 1.04, and 0.32 seconds), the final model posts lower full-set DER on every evaluated dataset. (The previous baseline has no 0.64-second configuration.)

Improvements grow with more speakers

Eight-speaker support enables meetings and group conversations with more than four participants, and the benchmark advantage widens in higher-speaker-count subsets of DIHARD III, CALLHOME-Part2, and NOTSOFAR1.

One nuance remains: on the two-speaker CALLHOME subset, the new model scores 5.98% DER versus 5.68% for the previous baseline. Across the full CALLHOME-Part2 evaluation, though, DER improves from 10.32% to 9.10%, with larger gains in the higher-speaker-count subsets. Also note that DIHARD III groups recordings with five through nine speakers into one result—nine speakers exceed Nemotron 3 Diarization's supported maximum of eight, so that aggregate includes audio beyond the specified speaker-count limit.