Duplex-MPE: Evaluating Multi-Party Conversation in Full-Duplex Dialogue
Duplex-MPE benchmarks voice assistants in group talk using 2,000 audio-only scenarios, testing speaking, answering, and silence timing.
Key Takeaways
- Duplex-MPE is a new benchmark for testing voice assistants in multi-party conversations, focusing on when models should speak, answer, or stay silent.
- It uses 2,000 paired scenarios with three or four human speakers, contrasting explicit addressing and implicit addressing.
- The benchmark evaluates response initiation, answer correctness, staying silent when appropriate, and stopping once a human resolves a request.
- It uses audio-only inputs without transcripts or turn-boundary cues, making evaluation more realistic for messy group conversations.
- Duplex-MPE is especially useful for developers and researchers building assistants for shared devices, meetings, and home environments.
Meta description: Duplex-MPE, a new benchmark for voice assistants in multi-party talk, tests when models should speak, answer, or stay silent across 2,000 paired scenarios.
Tags: voice assistants, multi-party conversations, benchmark, Duplex-MPE, speech AI
Featured image: Abstract friendly illustration of overlapping rounded speech bubbles, soft blue and warm yellow gradients, calm geometric shapes, no real people, no logos, no text.
The Duplex-MPE benchmark has been introduced for developers and researchers who build voice assistants for group conversations, and its data and code are planned for release before December. It evaluates whether an assistant should join a continuous multi-party discussion, answer correctly, or stay quiet when it is not addressed. The benchmark contains 2,000 paired scenarios with three or four human speakers and an assistant, contrasting explicit and implicit addressing.
Why Group Talk Is Harder Than One-on-One Speech
A voice assistant that works well in a single-user prompt can still behave poorly in a group conversation. Several people may talk, pause, interrupt, or shift topics. The assistant must decide not only how to respond, but also whether a response is appropriate.
This is a practical problem for shared devices, meeting spaces, home assistants, and other environments where users are not always alone. A model that speaks too eagerly may interrupt a person. A model that stays too silent may miss a request. A model that answers a question it should not answer may also fail to stop when a human has already resolved the issue.
Duplex-MPE is designed for that gap. It gives developers a way to test social timing in conversation, not only the correctness of a final answer.
What Duplex-MPE Measures
The benchmark uses 2,000 paired scenarios to compare different ways people may address an assistant. One key distinction is explicit addressing, where a speaker clearly directs attention to the assistant. Another is implicit addressing, where the assistant could respond from context even if no one directly calls it.
The evaluation separates several behaviors:
- response initiation: whether the assistant should begin speaking;
- answer correctness: whether the assistant gives an appropriate response when it does speak;
- remaining silent: whether the assistant can avoid speaking when it should not;
- stopping when a human resolves a request: whether the assistant can disengage once the issue is no longer open to it.
This structure matters because a model can pass a simple question-answering test while still failing in a group setting. It may know the right answer but deliver it at the wrong moment, or keep listening when the human has already solved the problem.
Why Audio-only Inputs Matter
Models receive audio without transcripts or supplied turn boundaries. A turn boundary is a marker that tells a system when one speaker has finished and another has begun. In many controlled tests, systems receive clean text or clear speaker changes. In real conversations, those cues are often messy.
By using audio only, Duplex-MPE places the model in a harder and more realistic position. It must infer who is speaking, whether the assistant is being addressed, and when the conversation has moved on without being handed a tidy transcript.
This makes the benchmark useful for testing the full chain of behavior: speech understanding, conversational awareness, and response timing.
Where to Look and What Comes Next
The project page at https://step-out.github.io/Duplex-MPE-Page/ provides audio examples and details about the benchmark design. The team behind the project says data and code are being prepared for release before December.
Once available, the benchmark can be used to compare models on the same group-conversation tasks. For developers, it offers a clearer way to test whether an assistant is not just correct, but also appropriately quiet. For researchers, it adds a focused measure for one of the most difficult parts of conversational AI: deciding when to speak.

Comments (0)